Since the PyPy nightly of 2026-10-11 (PyPy 8.0.2-alpha0, rev 544bd2f768f9), 12 tests
in archive_test.py (test_missing_chunk, test_corrupt_chunk, test_resync_*) fail with:
BufferError: Existing exports of data: object cannot be re-sized
at `del self._buffered_data[:drop_len]` in RobustUnpacker._search().
_search() scanned the buffer inside `with memoryview(self._buffered_data) as data:`
and called valid_msgpacked_dict(data[offset:]) for every offset, which also sliced the
key out of that view. CPython frees these unreferenced child views immediately, so the
bytearray is exportless again as soon as the with block releases the parent view.
PyPy has no refcounting: the child views stay alive until the GC runs. PyPy 8.0.0 did
not count child views as exports of the bytearray, the new nightly does (like CPython
always did), so the resize after the with block fails.
Fix: valid_msgpacked_dict(d, offset=0) only indexes d, it never slices it, and _search()
passes (data, offset) instead of a slice. Together with the explicitly released slice in
_try_item_start(), no unreleased view of _buffered_data exists anymore when it is
resized. The scan loop also gets a bit faster on CPython (no slice object per offset).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Update the data structures and security docs: the lock objects are in the
store object envelope, locking needs the key (it is loaded before waiting for
the lock), unreadable lock objects are treated as exclusive locks, and what
the store still sees about the locks. Add a change log entry.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
borg 1.x protected the key of the authenticated and authenticated-blake2
modes with the passphrase (pbkdf2, "sha256" algorithm), like the key of
the encrypted modes. The legacy authenticated key classes lacked
Pbkdf2FileMixin, so borg 2 could not decrypt that key and --from-borg1
(e.g. borg transfer) failed with "passphrase is incorrect".
The test uses keys exported from borg 1.4.5 authenticated and
authenticated-blake2 repositories.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
--from-borg1 via ssh:// (e.g. borg transfer from a borg 1.x repository)
failed with a borg 1.x "borg serve" on the remote host:
- the client called info() after open, which borg 1.x servers do not
have ("Borg server is too old for info"),
- then close() sent the close RPC, which borg 1.x servers do not have
either ("RPC method close is not valid"), masking the first error,
- get_manifest is also missing on borg 1.x servers,
- borg 1.x servers pack bytes into the msgpack "raw" type, which the
client unpacks to str, so e.g. the repository id and object data
arrived as str.
For borg 1.x servers, the client now uses repository version 1 (the only
version borg 1.x serves), does not send the close RPC (borg 1.x closes
the repository when stdin gets closed), gets the manifest object via get
and converts the bytes results back to bytes.
Tested with a real borg 1.4.5 "borg serve": repo-list, version,
repo-create --other-repo and transfer with --from-borg1, repokey and
keyfile mode. The testsuite emulates a borg 1.x server.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Describe ssh:// repository URLs for current repositories (served by a
remote "borg serve --rest") and for legacy borg 1.x repositories (with
--from-borg1, served by a remote "borg serve") and add a change log
entry.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
EVP_EncryptUpdate() / EVP_DecryptUpdate() take the input length as a
C int, and the AEAD ciphers passed the whole message in one call, with
all lengths held in C ints: a message of 2 GiB or more failed with
SystemError (negative size for the result allocation), and above 4 GiB
the payload was silently truncated to its length modulo 2^32.
Feed the input (data as well as AAD) to OpenSSL in chunks of at most
1 GiB and keep every byte count in a Py_ssize_t. Both AEAD modes (OCB,
chacha20-poly1305) stream, so chunked calls give the same result as a
single one; the chunk size is a module variable so a test can shrink it
and cover chunk boundaries that fall inside a cipher block. A second
test round-trips a 2 GiB message, skipped unless BORG_TESTS_BIG_MEMORY
is set.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- the envelope AAD also holds the repository id and a tag that tells a full
object name and the namespace of a hashed-name object (index/) apart, so
objects of another repository using the same key material, or envelopes
made for the other kind of name, fail the authentication.
- index fragments are not hashed on load to verify their names: the
authentication already proves their content (borg check still verifies
the names). The error message says "authentication failed".
- a corrupt chunk index (a fragment that fails the authentication or does
not deserialize) aborts every command that needs it, with the new
CorruptChunkIndexFragment error (rc 94): run "borg check --repair".
Only the commands that rewrite the whole index anyway, under an exclusive
lock (borg check --repair, borg compact, borg repo-compress), rebuild it
from the packs; borg compact --dry-run aborts.
- borg check: such a fragment is an index error, --repair rebuilds the
index from the packs. A check without --repair never rebuilds the index,
and the index cross-check is skipped if the store hash check already found
the index corrupt. Help text updated.
- remember the plaintext hash of the fragments read in a session and do not
store a fragment with known content again (unless forced), so e.g. a
crashed repack converges again.
- debug get-obj/put-obj/delete-obj: load the key via with_repository(),
including the security checks.
- tests: open_repository() always loads the real key, more tests; docs.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Describe the store object envelope and update what the store sees of the
index/ and cache/ objects, see #9819, #10235.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
- repo-info: append "empty passphrase" to the Encrypted: line if the key was unlocked with an
empty passphrase (for repokey and keyfile storage). The key now remembers this in the new
empty_passphrase attribute, which (unlike logically_encrypted) is independent of the storage.
- security: the "Attempting to access a previously unknown unencrypted repository" warning now
says why the repository counts as unencrypted - a mode that does not encrypt the data
(none-* / authenticated-*) or a repokey with an empty passphrase. Previously, users with an
encrypted repository could not tell why borg called it unencrypted, as repo-info said
"Encrypted: Yes".
- docs (repo-create, quickstart, repo-info): an empty passphrase still encrypts with a random
key, when it may be acceptable (keyfile storage), and that a passphrase can be added later
with "borg key change-passphrase" (unlike with the none-* modes).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
When the chunk ids can not be compared (the archives were created with different
--chunker-params), ItemDiff._content_diff() emitted a "modified" content change
unconditionally, and equal() returned False on the ctime/mtime difference before
ever comparing the content, so a file that was only touched showed up as
"modified: (can't get size)" in the default output mode (--content-only was
correct). Hardlink siblings whose ctime changed were affected the same way.
Now _content_diff() compares the sizes and then the content itself (once - the
chunk iterators can be consumed only once) and emits the count-less "modified"
change only if the content really differs; _content_equal() reuses that result.
Also: cover the default output mode with different chunker params in the tests,
add a changelog entry.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
#10346 did this only in the oldglibc_binary job (glibc239 binaries). The
Linux binaries from native_tests (ubuntu-26.04, glibc243) still bundled
two OpenSSLs: the system's libcrypto, which borg's crypto extension uses
and PyInstaller bundles, and the copy statically linked into the
manylinux wheel of cryptography (a paramiko dependency, sftp extra).
Build cryptography from source against the system OpenSSL there, too,
for the Linux "binary" matrix entries - the runner image has the Rust
toolchain. The step checks that the extension links libcrypto
dynamically and loads the same OpenSSL version as the system's openssl
program; the smoke test of the built binary checks that the bundled
extension links the bundled libcrypto, and prints the bundle sizes.
The stripping of the debug symbols and the trimming of the botocore
models from #10346 live in the shared PyInstaller spec, so they already
apply to these binaries. See #10345.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The Linux binaries were about 70 MiB as a single file and unpacked to
152 MiB at every start, see #10345. Three changes:
Strip the debug symbols on Linux: pythons built with the default CFLAGS
carry -g, and the spec had strip=False everywhere, so about half of the
Linux binaries was debug info. Enable PyInstaller's strip for the
collected shared libraries on Linux (only there: on macOS it would
interfere with code signing, and it is not recommended on Windows).
Python tracebacks do not need the symbols, only gdb would. Measured on
the glibc239 x86_64 bundle: 152 -> 78 MiB unpacked, 69.6 -> 43.4 MiB
compressed.
Bundle only the botocore service models borg needs: botocore (boto3, s3
extra) ships the JSON service models of all ~430 AWS services, 23 MiB,
about 14 MiB in the compressed binary. borg only ever creates an s3
client - also for S3-compatible services like MinIO or Backblaze B2,
which just get a different endpoint URL - so the spec keeps only the s3
model, the shared top-level files and the models the credential
providers may need (sts, sso, sso-oidc).
In the oldglibc_binary job, build cryptography (a paramiko dependency
from the sftp extra) from source against the OpenSSL built there,
instead of using the manylinux wheel that statically links its own copy
of OpenSSL - the binary contained two OpenSSLs. The runner image has the
Rust toolchain this needs.
The job now checks that the bundled OpenSSL and Python libraries are the
ones built there by their GNU build IDs (the files are stripped now),
that cryptography links and loads that OpenSSL, that no bundled shared
library still carries debug info, that the other botocore models are
gone, prints the bundle sizes, and smoke-tests the frozen binary against
a MinIO server - the test suite's S3 test runs borg from the venv, not
the binary.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The Linux binaries from native_tests are built on ubuntu-26.04 and thus
need glibc 2.43, and the x86_64 one does not run on CPUs below x86-64-v3
("Illegal instruction"). The new oldglibc_binary job builds on
ubuntu-24.04 and ubuntu-24.04-arm (glibc 2.39, the oldest GitHub-hosted
Ubuntu image, with its baseline x86-64 toolchain and packages) and
produces borg-linux-glibc239-{x86_64,arm64}-gh.
Fixes#10342.
Ubuntu 24.04 only has OpenSSL 3.0 and Python 3.12 (and setup-python's
3.14 for it links the system OpenSSL 3.0), so the job builds OpenSSL
3.5.8 (no-tests, no-docs - the OpenSSL test suite is very slow and is
never run) and Python 3.14.7 from source, pinned by version and sha256,
links Python and borg's crypto extension against that OpenSSL, and
PyInstaller bundles those libraries. OpenSSL is built with an rpath to
its prefix, so that its libssl and the openssl program do not pick up
the system's OpenSSL 3.0 libraries, which have the same SONAMEs. Python
is a PGO build like the setup-python ones; its PGO training run (a
subset of python's test suite) runs in parallel worker processes. The
job checks that the ssl module, borg's crypto extension and the frozen
binary really use the OpenSSL and Python built there and that no
bundled ELF file needs a newer glibc than the binary's name promises.
The binary is built on every run of the job and the test suite runs
against it. On tags, the binaries get a provenance attestation and the
release job (which now depends on this job) attaches them and their
sigstore bundles to the GitHub release.
Also: scripts/glibc_check.py no longer crashes on an ELF file without
versioned glibc symbols.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Only BORG_PASSPHRASE was removed from the environment given to subprocesses
(the command given in BORG_PASSCOMMAND, --paths-from-command / --content-from-command commands, tar
filter commands, borg with-lock commands, fusermount/umount, fakeroot
detection, ssh for legacy borg 1.x repos). Now also remove:
- BORG_NEW_PASSPHRASE, BORG_OTHER_PASSPHRASE: secrets, like BORG_PASSPHRASE.
- BORG_PASSCOMMAND, BORG_OTHER_PASSCOMMAND: often embed secrets or paths to them.
- BORG_PASSPHRASE_FD, BORG_OTHER_PASSPHRASE_FD: the file descriptors are not
inherited by the subprocess (Popen closes them), so the numbers would be
dangling and misleading there.
- BORGSTORE_REST_PASSWORD: borgstore's REST authentication password.
The other borg env vars (BORG_REPO, BORG_KEY_FILE, ...) are not secrets and
with-lock scripts may rely on them, so they are kept.
Also document this in the BORG_PASSPHRASE help text and add tests.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Statistics.usize (the size of the chunks that were new to the repository) is
accumulated while an archive is created and printed as "Deduplicated size" in
the text --stats block, but Statistics.as_dict() did not include it, so the
--json output (which replaces the text block) had no way to get it.
as_dict() now reports it as "deduplicated_size", as borg 1.x did (there it was
the deduplicated compressed size, in borg 2 it is the deduplicated original
size, see the 2.0.0a2 changelog). This covers borg create --json, borg
import-tar --json and the archive_progress JSON lines.
Archive.calc_stats() (used by borg info) can not compute the deduplicated size
of an existing archive (that is expensive, see borg analyze), so it now marks
usize as unknown (None) and as_dict() leaves the key out there instead of
reporting a misleading 0.
Also fix the create help text: borg 2 prints a single stats block, there is no
"This Archive" / "All archives" table anymore.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
A leading "*" opens inline emphasis, so the two "*-aes" occurrences made
sphinx report "Inline emphasis start-string without end-string". The docs
tox env runs sphinx-build with -W, so this failed the docs build.
Use literal markup instead.
Co-Authored-By: Claude Opus 5 <[email protected]>
gcc at -O2 - Debian's python flags - keeps the VAES/AVX-512 path's two
loops rolled: the 15 stride-2 roll steps run with an index chain and the
9 AES rounds over 8 zmm vectors come with 72 zmm register copies per
group. Spelled out by macro (like the aes-ni path now), the group loop is
straight-line code.
Zen 4, gcc 14 -O2, borg benchmark cpu with BORG_AES_CHUNKER_KERNEL=vaes:
toeplitz-aes 871 -> 1000 MB/s, rabin-aes 879 -> ~1010, goldilocks-aes
489 -> ~555. At the kernel level toeplitz-aes goes from 7.06 to 6.40
cycles per byte. Cut points are unchanged (the hw/vaes/evp equivalence
tests pass).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Same change as for buzhash64: the loop accumulating `cand |= ...` over the
8 lanes is vectorised by gcc >= 12 at -O2, which reloads the eight 8-byte
stores of s[] as 16-byte vectors and fails store-to-load forwarding on
every block - the "blockwise 8x slower than scalar" seen on Zen CPUs in
#10160. As a short-circuit chain it compiles to eight test-and-branch
pairs that leave the loop at the first candidate. Measured at the kernel
level (bit-identical cut points and hash state):
- Zen 4, gcc 14 -O2 (Debian flags): 6.04 -> 1.92 cycles per byte,
834 -> 2607 MB/s of scanned data (the x86-64 default stays the
sequential loop at 1.07)
- Apple M3 Pro, Apple clang 17 -O3: 1.91 -> 1.09 cycles per byte,
2081 -> 3640 MB/s (the aarch64 default stays NEON at 0.93)
So the portable kernel - the default on every platform that is neither
x86-64 nor aarch64, and what BORG_FASTCDC_KERNEL=blockwise selects -
is now within 15-20% of the fastest kernel instead of 2-3x behind.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The blockwise kernel tested its 8 lanes with a loop accumulating
`hit |= ((c ^ s[k]) & M[k]) == 0`. clang lowers that into a chain of
conditional increments; gcc >= 12 at -O2 vectorises it and reloads the
eight 8-byte stores of s[] as 16-byte vectors, which fails store-to-load
forwarding on every block - the "blockwise 8x slower than scalar" seen on
Zen CPUs in #10160. Written as a short-circuit chain it compiles to eight
test-and-branch pairs that leave the loop at the first hit.
Measured at the kernel level (bit-identical cut points and hash state):
- Apple M3 Pro, Apple clang 17 -O3: blockwise 2450 -> 2950 MB/s of scanned
data, i.e. +24% over the shipped aarch64 default, the NEON kernel (2370)
- Zen 4, gcc 14 -O2 (Debian flags): blockwise 766 -> 1707 MB/s (2.2x);
the x86-64 default stays the sequential loop (2390)
So the aarch64 default becomes blockwise: the NEON kernel already lost to
the shipped blockwise kernel there (the file's own note said so while
bz64_kernel_default() still picked neon), and it loses more now. It stays
selectable via BORG_BUZHASH64_KERNEL=neon. The default tables in the tests
are now per chunker family, and the env var docs say which kernel each
family defaults to on aarch64.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The aes-arm64 scan path of the toeplitz-aes, rabin-aes and goldilocks-aes
chunkers was only compiled when __ARM_FEATURE_AES was defined, i.e. when
the whole build targeted the crypto extension. Apple's default target
does, a Linux or BSD python extension build (-march=armv8-a baseline, no
-march flags from setup.py) does not: there the hardware path silently
did not exist, every such build ran the portable OpenSSL EVP path (1.5 to
1.9x slower per scanned byte on an Apple M3, 976 vs 529 MB/s for
toeplitz-aes on the same core under Linux) and BORG_AES_CHUNKER_KERNEL=
aes-arm64 was rejected with "not a kernel of this build" while the valid
values still listed it.
Enable the crypto extension per function with a target attribute (gcc >=
6, clang >= 14), exactly like the x86-64 path does with target("aes,sse2"),
and decide at run time whether the CPU has the instructions: getauxval on
Linux, elf_aux_info on FreeBSD, always on Apple Silicon, and only for
builds that target +crypto as a whole elsewhere. A compiler too old for
the attribute now reports aes-arm64 as "not compiled into this build"
instead of "not a kernel of this build".
Add a regression test: on aarch64 the hardware path must be the default
whenever the CPU has the AES instructions.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
a6970c7e7 did this for the sdist: its provenance attestation is only
stored in the GitHub attestations API, and fetching it from there needs
a recent gh and a GitHub token, so the sigstore bundle is attached to
the release as <asset>.sigstore.jsonl for offline verification.
Do the same for the standalone binaries: the jobs that build them attest
them, so they are also the ones that have the bundle. Each of them stores
it under the name of every asset it attested (one attestation covers both
the single-file binary and the .tgz) and uploads it in the artifact that
carries the binaries, so the release job attaches it to the release along
with them - it uploads whatever is in the artifacts.
vm_tests attested after uploading its artifact, so the attestation is now
made before the upload, like in the other two jobs.
Also document the offline verification in the binaries README (which is a
release asset itself) and in the release checklist.
Co-Authored-By: Claude Opus 5 <[email protected]>
borg called platformdirs with the appname only, so on Windows the
appauthor component defaulted to the appname and all default
directories ended up under a doubled AppData\...\borg\borg path.
Pass appauthor=False everywhere (no vendor directory level; on
Linux/macOS the appauthor is unused, so nothing changes there) and
roaming=True for the config dir, restoring the layout the CHANGES
upgrade notes had documented as intended all along:
- config (incl. keys): C:\Users\<user>\AppData\Roaming\borg
(follows the user's roaming profile in domain environments)
- data (incl. security state): C:\Users\<user>\AppData\Local\borg
- cache: C:\Users\<user>\AppData\Local\borg\Cache
- runtime: C:\Users\<user>\AppData\Local\Temp\borg
This matches what comparable tools do on Windows (pip, rclone and
kopia keep the configuration in the roaming AppData and machine
state/caches in the local AppData, all without a vendor directory).
The cache keeps platformdirs' opinionated "Cache" subdir so the
CACHEDIR.TAG written into the cache root stays out of the data dir.
Only Windows is affected. The first Windows binary was only released
with 2.0.0b23, so beta users needing to migrate should be rare: move
the keys from the old %LOCALAPPDATA%\borg\borg\keys to the new
location (or set the BORG_* dir env vars to the old paths); cache and
security state are recreated as needed.
Docs: update the environment help topic (and the generated
environment.rst.inc), FAQ, key/transfer usage docs and CHANGES.
Co-Authored-By: Claude Fable 5 <[email protected]>
borg calls platformdirs without appauthor/roaming, so on Windows the
config/data/cache/runtime directories all resolve under
AppData\Local\borg\borg (appauthor defaults to the appname, Roaming is
never used), matching what the fs tests assert. Fix the FAQ, the
CHANGES upgrade notes and the environment help topic accordingly and
point to #10237 where the directory scheme review is tracked.
Co-Authored-By: Claude Fable 5 <[email protected]>
Copying is cheap and fast: no file content is read or written, only a new
archive metadata object is created. Like any deduplicated archives, the two
archives share their data, so a copy needs almost no additional repository
space.
The copy is an independent archive: deleting either of the two archives
keeps the other one intact, because "borg compact" only frees chunks that no
remaining archive references.
Archive.copy() is Archive.rename() without removing the original archive
entry. Copying an archive to its own name is refused: the new metadata would
be identical, thus have the same archive ID, and no second archive would be
created. NEWNAME may well be the name of some other existing archive, the
copy then just joins that archive series.
OLDNAME must match precisely one archive, so it accepts an archive name (if
it is unique) or an archive ID like "aid:d34db33f".
Co-Authored-By: Claude Fable 5 <[email protected]>
msgpack 1.2.2 only brings bug fixes relevant to borg's usage:
correct Timestamp.from_datetime for datetimes far from the epoch,
reject out-of-range timestamp nanoseconds, copy ExtraData.extra out of
a temporary contiguous buffer, and a new RuntimeError guard against
re-entrant Unpacker.feed() (borg never feeds from inside a hook).
Co-Authored-By: Claude Opus 5 <[email protected]>
A "rest://host/" URL (no repository path) did not match rest_re, fell through
to local_re and became the local directory "./rest:/host", which then failed
with a confusing "Repository ... does not exist" message. Only "ssh://" and
"file://" were excluded from local_re, "rest://" was not.
Exclude all schemes borg parses itself from local_re, in the opaque form too,
so no "rest:...", "ssh:..." or "file:..." input can silently be taken for a
local path, and tell the accepted forms when such a URL does not parse.
A local path that really starts with such a prefix can still be given as
"./rest:..." or as an absolute path.
"Failed to create/acquire the lock <Store(backend='PosixFS')> (timeout)."
told neither which repository was locked nor what was holding it, although
the lock objects have the owner's host and pid.
Now both the message while waiting and the timeout error name the lock(s)
in the way:
Waiting for the lock. Repository is locked by: exclusive lock on host
myhost@1234567, pid 51939, age 2.000 seconds.
Failed to create/acquire the lock /path/to/repo (timeout). Repository
is locked by: exclusive lock on host myhost@1234567, pid 51939,
age 6.361 seconds.
The repository is named by its canonical path (credentials stripped), not
by the store's repr. break-lock logs each lock it breaks, and a refresh
that has to give up because our own lock was killed now says so instead of
just claiming a timeout.
Co-Authored-By: Claude Opus 5 <[email protected]>
borg --repo repo repo-create
error: the following arguments are required: repo-create.encryption
"repo-create.encryption" is the key that argument has in a config file,
not something you can type on the command line - and the usage shown
above the message was the one of borg itself, not of repo-create.
Translate the message to the one naming the option strings, and point
the usage and the help hint at the subcommand the argument belongs to:
borg --repo repo repo-create
usage: borg [options] repo-create [-h] ...
tip: For details of accepted options run: borg repo-create --help
error: the following arguments are required: -e/--encryption
jsonargparse worded that message differently before 4.51 (and reported
only the first missing argument), so match both wordings - borg supports
jsonargparse >= 4.47.
Co-Authored-By: Claude Opus 5 <[email protected]>
Four issues reported for the borg2 completions in #3086:
- -r/--repo and --other-repo completed nothing. They now complete
directories - a local repository is one, and a remote location like
ssh://... simply does not match any.
- The generated script hardcoded "borg": it registered its completions
for that command and, worse, ran "borg repo-list" to look up archive
names and tags. A borg 2 installed as "borg2" next to a borg 1.x
therefore asked the borg 1.x for the archives of a borg 2 repository,
which of course completed nothing. It now uses the command borg was
invoked as, as long as that is a borg* name - borg is also invoked in
ways that give no usable command name (python -m borg, the test
suite), and then "borg" stays the right guess.
- Completing "borg repo-create --encryption=" showed the same help text,
listing all the modes, next to each of the modes. zsh and fish now get
a short description of what that one mode does; bash and tcsh have no
completion descriptions, so they just complete the mode names.
- The help texts are written in RST and the shells show them verbatim as
the completion description, markup included ("**(required)**",
"``--format``"). Strip the inline markup for the completions.
The encryption modes need shtab >= 1.11.0: only from that release does a
.complete function take precedence over an argument's choices. Older
shtab offers both, so bash would complete every mode twice. That release
also has the fish path-completion fix borg was working around, so drop
the local FILE override again.
Also fix two stale encryption mode names: --encryption=repokey-aes-ocb
in the "Encryption NOT enabled" hint and the "none" mode in the
--key-location help are both from before #9104. The latter has to say
``none-*``: a bare "none-*" starts RST emphasis, and the docs are built
with -W.
Co-Authored-By: Claude Opus 5 <[email protected]>
Until now every chunker ran its simplest implementation unless a
BORG_*_KERNEL env var named another one, because which kernel is fastest
is not predictable from the instruction set. It is predictable per
platform though, now that #10160 has benchmarks from a range of CPUs, so
choose accordingly instead of leaving speed on the table by default:
- fastcdc / buzhash64: neon on aarch64, scalar (the plain sequential
loop, which the compiler folds into a single instruction there) on
x86-64, blockwise elsewhere.
- toeplitz-aes / rabin-aes / goldilocks-aes: wider is simply faster
here, so vaes, else aes-ni on x86-64, aes-arm64 on aarch64, and evp
where there is no AES hardware path.
Both fall back where a build or a CPU does not have the preferred
kernel. An env var still means exactly what it says: a kernel that
cannot run here is an error, never a silent fallback.
On an Apple M3 this makes the default chunkers 1.6x (goldilocks-aes) to
2.1x (buzhash64) faster.
With "--digests ALGOS", borg computes hash digests over the full content of each
regular file and stores them in the item's new "digests" dict (hash algorithm name
-> digest), e.g. --digests=blake3 gives {"blake3": ...} (32 bytes). ALGOS is a
comma-separated list of hash algorithm names - the same algorithms that are available
as "borg list" format keys - or "none". Default: none, digests are opt-in.
Unlike the chunk ids, such a digest depends neither on the chunker nor on borg's key,
so it is the same digest that e.g. "b3sum" computes for that file. "borg list --format
{blake3}" uses a stored digest if the item has one, instead of reading the file content
from the repository.
The content is hashed while it is read and processed anyway: the first chunk inline,
from the second (big enough) chunk on by a background thread. Our hash implementations
release the GIL, so the hashing can overlap with chunking, id hashing, compression and
encryption instead of adding to their runtime. ContentDigester.update() takes ownership
of the chunk data and releases it after hashing, so nothing is copied for this and the
memory is still freed timely.
Measured (12 cores, medians of 5 runs, relative to --digests=none, which is as fast as
borg without this):
1GiB new 1GiB known 20000 small files
--digests=blake3 free free +6..12%
--digests=blake3,sha256 free +10% +16%
--digests=blake3,sha256,sha512 +7% +60% +24%
One background thread hashes all requested algorithms one after the other, so their
combined throughput is the harmonic sum, while many small files are hashed inline by
the main thread. Hashing is free exactly while it is faster than the rest of the
pipeline - a single algorithm is, several may not be. That is why digests are opt-in.
Digests are not computed if borg does not read the full content: an unchanged file gets
them from the files cache and additional hard links from the first hard link, but
"borg create --reuse-from" items do not have digests. Files that already are in the
files cache without digests get them when they change (and thus are read again), so
changing --digests only affects the files that borg reads.
When showing a whole repository, the archive directories were named like
the archives, with -{id:.8} appended to names that are not unique (the
archives of a series). Now the BORG_MOUNT_ARCHIVE_DIR_FORMAT environment
variable gives the format of these names, using the placeholders of
borg repo-list --format (default: {name}, the same naming as before).
Names that are still not unique (or empty, "." or "..") get -{id:.8}
appended, "/" and NUL are replaced, an invalid format string is an error.
As this is implemented in the shared ArchiveVFS, it applies to borg mount
and borg webdav alike.
