Haiku r1beta6 can silently store stale pages when a write() gets its data
from a memory mapped file, see https://dev.haiku-os.org/ticket/20392 .
That is what broke rust builds there (corrupt .rlib archives, rustc ICEs).
scripts/haiku_mmap_pread_shim.c replaces private mappings of regular files
by anonymous memory filled via pread(). The haiku job builds it and preloads
it into everything it runs.
With that, the previous workarounds are removed: the pip install retry loop,
the serialized rust build and the relocated TMPDIR / cargo target dir.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The moto S3 test server install already pulls in the cryptography
manylinux wheel, so a plain "pip install --no-binary cryptography"
kept that wheel (statically linked OpenSSL) and the dynamic libcrypto
check failed.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
ssh:// now addresses a current repository via REST over stdio over ssh
(served by a remote "borg serve --rest"); with --from-borg1, it still
addresses a legacy borg 1.x repository via the legacy RPC protocol.
The testsuite uses ssh://__testsuite__/path (like the legacy client) to
run "borg serve --rest" locally instead of via ssh.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
illumos does not overcommit memory: fork() must reserve swap for the
child's entire address space. The big pytest-xdist workers therefore
sporadically failed to spawn subprocesses (borg serve, for the
remote_archiver tests) with OSError: [Errno 12] "Not enough space"
(illumos' strerror for ENOMEM) when the image's small swap was
exhausted - the swap-backed tmpfs /tmp and ZFS-ARC-squeezed availrmem
eat into the same reservation pool. Seen e.g. in the omniOS job of
runs 35858263044 (PR #10411) and 35913850908 (master).
Add a 4 GiB sparse swap zvol before running the tests so fork()
reservations have room; being sparse, it takes no pool space unless
actually paged to. Also log swap -lh / swap -sh for diagnosability.
Co-Authored-By: Claude Fable 5 <[email protected]>
So the borg mount tests also run on Windows and the Windows binary
can mount (if WinFsp is installed).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
MinIO withdrew its community binaries, so CI was pinned to its last GitHub
releases that still carry assets (#10360) - a frozen snapshot that gets no
fixes and could vanish as well.
Use moto in server mode (moto_server) instead: it comes from PyPI, so there
is no server binary to download, and the pins in the new
requirements.d/s3-test-server.txt are kept up to date by dependabot. moto's
"server" extra pulls in the dependencies of all AWS services it can mock,
for S3 the server only needs flask and flask-cors on top of the "s3" extra.
The mc client is not needed either, boto3 creates the bucket.
This replaces the S3 server of the native_tests job and the one the
oldglibc_binary job smoke-tests the frozen binary against.
Also add test_s3_repo_compact: compact rewrites a mixed pack via
store.defrag, which for the S3 backend is ranged GETs plus one PUT. A server
that ignores the requested range goes unnoticed by test_s3_repo_basics, but
makes compact fail.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The S3 tests download a MinIO server (and the mc client) from dl.min.io. Those URLs
now answer HTTP 410 Gone, so "curl -fsSL" exits 22 and the step fails before borg is
even installed. That takes down three native_tests jobs, both oldglibc_binary jobs
(their S3 smoke test needs the same server), the codecov upload (no tests ran, so no
coverage) and, via fail-fast, the rest of the native_tests matrix.
MinIO stopped publishing binaries on GitHub as well: the releases from
RELEASE.2025-10-15T17-29-55Z on carry no assets. So pin the last releases that do
publish them - minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z -
and check each download against the sha256 published next to it.
Release and sums live in the workflow-level env block, shared by the two steps that
download MinIO, so a bump is a one-place change.
Verified: the amd64 and arm64 binaries match the pinned sums, and running the patched
step in a linux/arm64 container downloads both, starts the server and creates the
bucket.
Co-Authored-By: Claude Opus 5 <[email protected]>
#10346 did this only in the oldglibc_binary job (glibc239 binaries). The
Linux binaries from native_tests (ubuntu-26.04, glibc243) still bundled
two OpenSSLs: the system's libcrypto, which borg's crypto extension uses
and PyInstaller bundles, and the copy statically linked into the
manylinux wheel of cryptography (a paramiko dependency, sftp extra).
Build cryptography from source against the system OpenSSL there, too,
for the Linux "binary" matrix entries - the runner image has the Rust
toolchain. The step checks that the extension links libcrypto
dynamically and loads the same OpenSSL version as the system's openssl
program; the smoke test of the built binary checks that the bundled
extension links the bundled libcrypto, and prints the bundle sizes.
The stripping of the debug symbols and the trimming of the botocore
models from #10346 live in the shared PyInstaller spec, so they already
apply to these binaries. See #10345.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The Linux binaries were about 70 MiB as a single file and unpacked to
152 MiB at every start, see #10345. Three changes:
Strip the debug symbols on Linux: pythons built with the default CFLAGS
carry -g, and the spec had strip=False everywhere, so about half of the
Linux binaries was debug info. Enable PyInstaller's strip for the
collected shared libraries on Linux (only there: on macOS it would
interfere with code signing, and it is not recommended on Windows).
Python tracebacks do not need the symbols, only gdb would. Measured on
the glibc239 x86_64 bundle: 152 -> 78 MiB unpacked, 69.6 -> 43.4 MiB
compressed.
Bundle only the botocore service models borg needs: botocore (boto3, s3
extra) ships the JSON service models of all ~430 AWS services, 23 MiB,
about 14 MiB in the compressed binary. borg only ever creates an s3
client - also for S3-compatible services like MinIO or Backblaze B2,
which just get a different endpoint URL - so the spec keeps only the s3
model, the shared top-level files and the models the credential
providers may need (sts, sso, sso-oidc).
In the oldglibc_binary job, build cryptography (a paramiko dependency
from the sftp extra) from source against the OpenSSL built there,
instead of using the manylinux wheel that statically links its own copy
of OpenSSL - the binary contained two OpenSSLs. The runner image has the
Rust toolchain this needs.
The job now checks that the bundled OpenSSL and Python libraries are the
ones built there by their GNU build IDs (the files are stripped now),
that cryptography links and loads that OpenSSL, that no bundled shared
library still carries debug info, that the other botocore models are
gone, prints the bundle sizes, and smoke-tests the frozen binary against
a MinIO server - the test suite's S3 test runs borg from the venv, not
the binary.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The Linux binaries from native_tests are built on ubuntu-26.04 and thus
need glibc 2.43, and the x86_64 one does not run on CPUs below x86-64-v3
("Illegal instruction"). The new oldglibc_binary job builds on
ubuntu-24.04 and ubuntu-24.04-arm (glibc 2.39, the oldest GitHub-hosted
Ubuntu image, with its baseline x86-64 toolchain and packages) and
produces borg-linux-glibc239-{x86_64,arm64}-gh.
Fixes#10342.
Ubuntu 24.04 only has OpenSSL 3.0 and Python 3.12 (and setup-python's
3.14 for it links the system OpenSSL 3.0), so the job builds OpenSSL
3.5.8 (no-tests, no-docs - the OpenSSL test suite is very slow and is
never run) and Python 3.14.7 from source, pinned by version and sha256,
links Python and borg's crypto extension against that OpenSSL, and
PyInstaller bundles those libraries. OpenSSL is built with an rpath to
its prefix, so that its libssl and the openssl program do not pick up
the system's OpenSSL 3.0 libraries, which have the same SONAMEs. Python
is a PGO build like the setup-python ones; its PGO training run (a
subset of python's test suite) runs in parallel worker processes. The
job checks that the ssl module, borg's crypto extension and the frozen
binary really use the OpenSSL and Python built there and that no
bundled ELF file needs a newer glibc than the binary's name promises.
The binary is built on every run of the job and the test suite runs
against it. On tags, the binaries get a provenance attestation and the
release job (which now depends on this job) attaches them and their
sigstore bundles to the GitHub release.
Also: scripts/glibc_check.py no longer crashes on an ELF file without
versioned glibc symbols.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The main CI workflow never runs for docs-only changes: its pull_request
paths filter lists code patterns only and additionally excludes docs/**.
So a docs-only PR triggers no CI at all, and the docs toxenv - which runs
sphinx-build with -W - never gets to see the change.
#10314 is an example: it touched docs/ only, no CI ran, and broken rst
markup reached master. The push-to-master matrix has no docs entry
either, so the merge commit did not build the docs.
Give the docs build its own workflow, triggered on docs/**,
pyproject.toml and the workflow file itself, for pushes to master and for
PRs. A docs-only PR now gets its markup checked without paying for the
whole test matrix.
Drop the docs entry from ci.yml's pull_request matrix, and the "docs"
toxenv conditions that go dead with it - the new workflow owns the docs
build now.
Co-Authored-By: Claude Opus 5 <[email protected]>
a6970c7e7 did this for the sdist: its provenance attestation is only
stored in the GitHub attestations API, and fetching it from there needs
a recent gh and a GitHub token, so the sigstore bundle is attached to
the release as <asset>.sigstore.jsonl for offline verification.
Do the same for the standalone binaries: the jobs that build them attest
them, so they are also the ones that have the bundle. Each of them stores
it under the name of every asset it attested (one attestation covers both
the single-file binary and the .tgz) and uploads it in the artifact that
carries the binaries, so the release job attaches it to the release along
with them - it uploads whatever is in the artifacts.
vm_tests attested after uploading its artifact, so the attestation is now
made before the upload, like in the other two jobs.
Also document the offline verification in the binaries README (which is a
release asset itself) and in the release checklist.
Co-Authored-By: Claude Opus 5 <[email protected]>
This VM hands back data that does not match what was written to it: rustc
crashed on crate metadata it had written itself ("assertion failed:
bytes[len] == STR_SENTINEL"), and pip found sha256 mismatches reading
wheels back from its own cache - twice on cython (1.3MB), once on
virtualenv (5.5MB), with a different wrong hash every time and none of
them matching any published artifact. Only the big files were hit.
Retry the installs with a purged pip cache, so a damaged file does not
fail the job - and does not get stored in the actions cache, from where
it would fail every following run until someone deletes the cache.
Co-Authored-By: Claude Opus 5 <[email protected]>
The sdist provenance attestation is only stored in the GitHub
attestations API, and fetching it from there requires a recent gh
version and a GitHub token - both inconvenient for CI/CD jobs that
verify the sdist before building borg from source (see the discussion
in #10187).
Attach the sigstore bundle produced by actions/attest-build-provenance
to the release as borgbackup-<version>.tar.gz.sigstore.jsonl, so the
sdist can be verified offline and without a GitHub token using generic
sigstore tooling (cosign verify-blob / verify-blob-attestation, or
gh attestation verify --bundle), and document that in the generated
release notes.
Co-Authored-By: Claude Fable 5 <[email protected]>
default-days, semver-minor-days, semver-major-days now read 14, 30, 90
in both ecosystems, so the escalation is obvious at a glance.
Co-Authored-By: Claude Opus 5 <[email protected]>
The pip ecosystem set semver-major-days and semver-minor-days, but no
default-days, so patch releases had no cooldown at all and could land
within the weekly update cycle. Patch bumps are the usual shape of a
compromised PyPI release, so this was the least protected case.
Use the same 14 days as the github-actions ecosystem above.
Co-Authored-By: Claude Opus 5 <[email protected]>
backport.yml declared contents: write and pull-requests: write for the
whole workflow. It has only one job, so nothing actually ran with more
than it does now, but this is the workflow triggered by
pull_request_target, which is where the scope should be tightest.
The comments there were also the wrong way round: contents: write is what
lets backport-action push the backport branch, not what lets it comment.
codeql-analysis.yml had no workflow-level permissions at all. Its one job
is scoped correctly, but a second job added later would silently fall
through to the repository default - which is "write" for this repository.
Co-Authored-By: Claude Opus 5 <[email protected]>
The inline "# zizmor: ignore[...]" comments had to go after the version
comment on the `uses:` line, which left two comments there:
uses: actions/cache@55cc8345... # v6.1.0 # zizmor: ignore[cache-poisoning]
Dependabot maintains those version comments when it bumps a pin (see
.github/dependabot.yml, the github-actions ecosystem is updated weekly),
and zizmor's own ref-version-mismatch audit stopped recognising the
version, so the line is better left in the plain "@sha # version" form.
The explanations stay in the workflows next to what they explain; the new
config file only records which findings were accepted.
Co-Authored-By: Claude Opus 5 <[email protected]>
pull_request_target is dangerous when the workflow checks out or runs code
from the pull request, because it runs with a writable token. This one
does not: the checkout takes the base branch (no `ref:`), and the only
thing that looks at pull request content is backport-action cherry-picking
commits. Say so, so the next reader does not have to re-derive it, and
mark the finding as reviewed.
Co-Authored-By: Claude Opus 5 <[email protected]>
${{ ... }} is expanded into the script before the shell sees it, so the
value becomes code rather than data. matrix.binary is a literal from the
matrix in this very workflow, so nothing is exploitable here, but going
through env is the form that stays correct if the value ever stops being
a literal - and it silences zizmor's template-injection audit.
Co-Authored-By: Claude Opus 5 <[email protected]>
native_tests, vm_tests and windows_tests all restore a pip/tox cache and,
on tags, also build the release binaries, attest their provenance and
upload them. A cache entry is writable by any job that can write to the
cache scope, so this is the cache-poisoning path zizmor points at: a
poisoned dependency could end up inside an attested release artifact.
Skip the caches on tag pushes. Branch and pull request runs - where the
speed actually matters - keep caching, and release builds start cold,
which for a handful of tag pushes per year is a good trade.
zizmor's own autofix for this sets lookup-only: true, which would disable
cache restore on *every* run; that is why these are marked ignored instead.
While here, drop the second pip restore-key. "<os>-<arch>-" is a prefix
of the tox cache keys ("<os>-<arch>-tox-..."), so a pip cache miss could
restore a tox cache archive.
Co-Authored-By: Claude Opus 5 <[email protected]>
1.5.0 adds support for Haiku R1/beta6, so bring haiku back into the
vm_tests matrix - it was removed in 30ed2bcbf because r1beta5 (2024) was
too painful to work with, see #9463.
Also: build blake3 outside /tmp, one rustc at a time
rustc 1.94.1 ICEd while compiling the blake3 crate ("assertion failed:
bytes[len] == STR_SENTINEL"), reading back metadata that does not match
what it wrote. Try building outside /boot/system/cache/tmp with a single
rustc at a time, in case that corruption comes from disk or memory
pressure there.
Co-Authored-By: Claude Opus 5 <[email protected]>
actions/checkout leaves the job token behind in .git/config, so anything
that later archives the workspace also archives the credential (zizmor's
artipacked audit). None of these workflows push with it - release.yml
authenticates gh via GH_TOKEN, and peter-evans/create-pull-request uses
its own token input - so they can all opt out.
backport.yml is the exception and keeps the credentials: korthout/backport-action
pushes the backport branch with a plain `git push`, which only works with
what actions/checkout left in .git/config. Marked accordingly.
Co-Authored-By: Claude Opus 5 <[email protected]>
Both workflows were added after 4dfcf29ad ("CI: pin all GitHub Actions to
commit SHAs") and used floating tags, so they were the only two workflows
not following that policy. zizmor flagged all eight uses as unpinned-uses.
The SHAs for checkout, setup-python and cache are the ones already used by
the other workflows. docker/setup-qemu-action was not pinned anywhere yet;
v4 currently resolves to 96fe6ef7f3 (v4.2.0), so this is a no-op today.
Co-Authored-By: Claude Opus 5 <[email protected]>
The auto-generated release notes claimed the macOS binaries are built
with FUSE support; they are built with the "none" FUSE variant
(toxenv py314-none) and no workflow installs macFUSE, matching what
00_README.txt says. Also drop the corresponding stale macFUSE step
from the release checklist.
Co-Authored-By: Claude Fable 5 <[email protected]>
All our other CI machines are 64-bit, so nothing exercises borg where a
pointer, size_t, Py_ssize_t and off_t are 32 bits wide. armv7 is the 32-bit
platform borg users still run (Raspberry Pi OS 32-bit, NAS / router /
embedded boxes); it additionally does not tolerate unaligned accesses
everywhere and has an unsigned plain "char", so it exercises C portability,
too.
Modelled on the big-endian s390x job: debian trixie (python 3.13, and its
OpenSSL 3.5 satisfies our OpenSSL >= 3.2 requirement - bookworm's 3.0 would
not) under qemu user-mode emulation, the same trigger policy (native/format
relevant paths, weekly, manual), the word size sensitive tests, and a
cross-architecture interoperability test - a repository written on the
emulated 32-bit machine is read (and written to) by the 64-bit runner and the
other way round, requiring identical chunk IDs on both sides.
Co-Authored-By: Claude Opus 5 <[email protected]>
The 2.0.0b23 Windows binaries call themselves
"2.0.0b24.dev0+g116435fa4.d20260823": actions/checkout runs Git for
Windows, whose system config sets core.autocrlf=true, so the worktree
gets CRLF line endings. The msys2 git the build steps use does not set
it, thus sees every text file as modified, and setuptools-scm builds a
guessed-next .devN version instead of the version of the tag.
- .gitattributes: store and check out text files with LF everywhere.
logo.pdf has no NUL byte within the first 8000 bytes, so git would
auto-detect it as text - it needs an explicit -text.
- ci.yml: also switch core.autocrlf off before checkout, so that
building an older tag (which has no such .gitattributes) works, too.
- release.yml: refuse to publish a sdist that is not the tag version.
Co-Authored-By: Claude Opus 5 <[email protected]>
Borg on Linux now needs OpenSSL >= 3.2 for argon2, and ubuntu-24.04 ships
OpenSSL 3.0.x, so the crypto extension no longer builds there. Move all
Ubuntu-based jobs to ubuntu-26.04 (and ubuntu-24.04-arm to
ubuntu-26.04-arm): ci.yml, canary.yml, bigendian.yml, codeql-analysis.yml,
black.yaml, backport.yml and release.yml. The jobs that never build borg
move along with them, so we do not end up with a confusing mix of runners.
Ubuntu 26.04 ships glibc 2.43, so rename the built binaries from glibc239
to glibc243, in the native_tests matrix and in the EXPECTED_ASSETS list of
the release workflow, which has to stay in sync with it.
The Python versions stay as they are: actions/python-versions publishes
3.10 through 3.15 builds for the ubuntu-26.04 image, so setup-python can
still provide the whole 3.11 .. 3.15-dev range we test. (When the image was
new it only worked with its default 3.14, which is what the canary removed
in the previous commit was watching for.)
Co-Authored-By: Claude Opus 5 <[email protected]>
- src/borg/crypto/low_level.pyx: implement `argon2_hash` using OpenSSL's
`EVP_KDF` API for ARGON2 (requires OpenSSL >= 3.2.0).
- src/borg/crypto/key.py: switch to the native `argon2_hash` implementation,
removing `argon2-cffi` dependency.
- setup.py: require OpenSSL >= 3.2.0 for the crypto extension to ensure
ARGON2 KDF support is available.
- pyproject.toml: drop `argon2-cffi` dependency.
- docs: update installation requirements and security documentation to
reflect the transition to OpenSSL for Argon2.
The lanes are computed in parallel via OpenSSL's thread pool: OpenSSL's
Argon2 only uses threads if OSSL_set_max_threads() enabled the thread pool
and a "threads" parameter > 1 is given. Passing threads=1 would compute the
4 lanes of our default parallelism=4 sequentially, and unlocking would get
slower than it was with argon2-cffi, which used real threads for the lanes.
The thread count only affects speed, never the derived key (only the lanes
parameter does), so this is fully compatible with existing keys.
Apple M-series (OpenSSL 3.6.3), borg defaults (t=3, m=64 MiB, p=4):
threads=1: 83 ms, threads=4: 29 ms (2.8x faster).
Co-Authored-By: Claude Opus 5 <[email protected]>
The inverted canary added in #10097 watched for actions/setup-python
gaining Python 3.12 support on the ubuntu-26.04 image. It went red on
2026-08-15 and 2026-08-16: the "Try to set up Python 3.12" step now
succeeds, which was the whole point of the probe. Its job is done, so
remove it.
Older Pythons are therefore no longer categorically unavailable on
ubuntu-26.04, so the matrix can gain multi-version coverage there once
we move to those runners.
Co-Authored-By: Claude Opus 5 <[email protected]>
The cross-platform-actions FreeBSD image uses a plain UFS root without
the acls mount option, so are_acls_working() failed and the FreeBSD ACL
tests were always skipped. Remount / with acls enabled (this also covers
/tmp, where the tests create their temp files) and hard-fail the job if
the option did not take effect.
Co-Authored-By: Claude Fable 5 <[email protected]>
The interpreter is a pypy nightly build, but the job itself should run like the
other test jobs: on pushes to master and on pull requests, with the same paths
filter and concurrency group as ci.yml. No daily schedule.
Still continue-on-error, so it reports without blocking.
Co-Authored-By: Claude Opus 5 <[email protected]>
Scheduled (and manually triggerable) job that installs a pypy-3.11-nightly via
actions/setup-python and runs the test suite with it. It is not part of ci.yml
and it is continue-on-error, so it never blocks anything - it only reports.
pypy nightlies change daily and borg's C extension code keeps running into
cpyext differences, so both borg regressions and pypy regressions show up here
early. No fuse extras: the fuse bindings do not support pypy, those tests skip.
The unlocked development requirements are used, because the lock file pins
versions/wheels for CPython.
Co-Authored-By: Claude Opus 5 <[email protected]>
The macOS test jobs took ~3-4 hours (vs 10-20 minutes on comparable linux
runners), nearly all of it in the remote (rest://) tests. Root cause, found
via #9470: the runner hostname is a long, unresolvable .local name and macOS
needs ~35s to give up resolving it. borg resolves the fqdn at import time
(platform/base.py), so EVERY spawned borg process paid those 35s - a remote
test spawns ~10 serve/client processes, i.e. ~400s per test.
Diagnosed and verified on a macos-15 runner:
- getaddrinfo(gethostname(), AI_CANONNAME) alone: 35.0s
- .local is mDNS territory: an /etc/hosts entry alone does NOT help
- after scutil --set HostName + /etc/hosts entry: getaddrinfo 0.04s,
borg --version 35.9s -> 0.28s, one remote test 394s -> 5.2s (76x)
Co-Authored-By: Claude Fable 5 <[email protected]>
