Haiku r1beta6 can silently store stale pages when a write() gets its data
from a memory mapped file, see https://dev.haiku-os.org/ticket/20392 .
That is what broke rust builds there (corrupt .rlib archives, rustc ICEs).
scripts/haiku_mmap_pread_shim.c replaces private mappings of regular files
by anonymous memory filled via pread(). The haiku job builds it and preloads
it into everything it runs.
With that, the previous workarounds are removed: the pip install retry loop,
the serialized rust build and the relocated TMPDIR / cargo target dir.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Describe ssh:// repository URLs for current repositories (served by a
remote "borg serve --rest") and for legacy borg 1.x repositories (with
--from-borg1, served by a remote "borg serve") and add a change log
entry.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Every borg 2 mode has a borg key now: the AEAD modes (aes256-ocb,
chacha20-poly1305) and the authenticated-sha256 / authenticated-blake3
modes. There is no compatibility with repositories older borg 2 betas
created in a none-* mode.
- crypto.key: remove ChecksumKeyBase, ChecksumKey, Blake3ChecksumKey and
the SHA256NONE / BLAKE3NONE key types.
- remove KeyBase.id_check_is_authentication: all borg 2 envelopes
authenticate the chunk id, so RepoObj.parse verifies the id only at
the BORG_ASSERT_ID places and the mandatory ones.
- repo-create: always show the key / passphrase hint.
- a borg 1.x "none" repository has unkeyed chunk ids, which no borg 2
mode has, so transferring it needs --chunker-params (docs updated).
- tests: use authenticated-sha256 where none-sha256 was used as the
cheap default mode, add make_test_key() for tests needing some key.
The repository swap detection tests now re-label the repokey of the
attacker's repository for the faked repository id.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The Linux binaries were about 70 MiB as a single file and unpacked to
152 MiB at every start, see #10345. Three changes:
Strip the debug symbols on Linux: pythons built with the default CFLAGS
carry -g, and the spec had strip=False everywhere, so about half of the
Linux binaries was debug info. Enable PyInstaller's strip for the
collected shared libraries on Linux (only there: on macOS it would
interfere with code signing, and it is not recommended on Windows).
Python tracebacks do not need the symbols, only gdb would. Measured on
the glibc239 x86_64 bundle: 152 -> 78 MiB unpacked, 69.6 -> 43.4 MiB
compressed.
Bundle only the botocore service models borg needs: botocore (boto3, s3
extra) ships the JSON service models of all ~430 AWS services, 23 MiB,
about 14 MiB in the compressed binary. borg only ever creates an s3
client - also for S3-compatible services like MinIO or Backblaze B2,
which just get a different endpoint URL - so the spec keeps only the s3
model, the shared top-level files and the models the credential
providers may need (sts, sso, sso-oidc).
In the oldglibc_binary job, build cryptography (a paramiko dependency
from the sftp extra) from source against the OpenSSL built there,
instead of using the manylinux wheel that statically links its own copy
of OpenSSL - the binary contained two OpenSSLs. The runner image has the
Rust toolchain this needs.
The job now checks that the bundled OpenSSL and Python libraries are the
ones built there by their GNU build IDs (the files are stripped now),
that cryptography links and loads that OpenSSL, that no bundled shared
library still carries debug info, that the other botocore models are
gone, prints the bundle sizes, and smoke-tests the frozen binary against
a MinIO server - the test suite's S3 test runs borg from the venv, not
the binary.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The Linux binaries from native_tests are built on ubuntu-26.04 and thus
need glibc 2.43, and the x86_64 one does not run on CPUs below x86-64-v3
("Illegal instruction"). The new oldglibc_binary job builds on
ubuntu-24.04 and ubuntu-24.04-arm (glibc 2.39, the oldest GitHub-hosted
Ubuntu image, with its baseline x86-64 toolchain and packages) and
produces borg-linux-glibc239-{x86_64,arm64}-gh.
Fixes#10342.
Ubuntu 24.04 only has OpenSSL 3.0 and Python 3.12 (and setup-python's
3.14 for it links the system OpenSSL 3.0), so the job builds OpenSSL
3.5.8 (no-tests, no-docs - the OpenSSL test suite is very slow and is
never run) and Python 3.14.7 from source, pinned by version and sha256,
links Python and borg's crypto extension against that OpenSSL, and
PyInstaller bundles those libraries. OpenSSL is built with an rpath to
its prefix, so that its libssl and the openssl program do not pick up
the system's OpenSSL 3.0 libraries, which have the same SONAMEs. Python
is a PGO build like the setup-python ones; its PGO training run (a
subset of python's test suite) runs in parallel worker processes. The
job checks that the ssl module, borg's crypto extension and the frozen
binary really use the OpenSSL and Python built there and that no
bundled ELF file needs a newer glibc than the binary's name promises.
The binary is built on every run of the job and the test suite runs
against it. On tags, the binaries get a provenance attestation and the
release job (which now depends on this job) attaches them and their
sigstore bundles to the GitHub release.
Also: scripts/glibc_check.py no longer crashes on an ELF file without
versioned glibc symbols.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Drives `borg benchmark cpu --chunking --json` once per scan kernel
combination and prints one throughput table over all chunkers and all
kernels this build and CPU can run, which is how the per-platform kernel
defaults were and are decided (#10160).
`borg benchmark cpu` reports a chunker it cannot construct - e.g. because
an unusable kernel was requested - as an {"error": ...} row and carries on
with exit code 0, so the script treats such a row as the run's failure;
that is also how it discovers which kernels exist (an invalid name makes
borg list the valid ones).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The generated borgfs.1 was corrupted: jsonargparse injects --config and
--print_config into the borgfs parser (a top-level parser), and
--print_config's "[=flags]" metavar is invalid rST option-list syntax,
so the man page embedded a docutils WARNING and an unformatted blob of
options. Hide jsonargparse-injected actions (detected structurally by
their module, so future injected options stay filtered) and borg's
top-level-only --cockpit from the borgfs docs via help=SUPPRESS, which
all make.py writers already honor. -V/--version stays documented, as
in borg 1.x. Regenerate borgfs.1 accordingly (now also showing the
REPOSITORY positional).
Remove docs/usage/borgfs.rst{,.inc}: build_usage intentionally stopped
generating a borgfs HTML page long ago (#3404), so the leftover
.rst.inc still documented the borg 1.x CLI (REPOSITORY_OR_ARCHIVE,
--prefix, ...) and was published as an orphan page. Nothing references
these files; borgfs is documented on the "borg mount" page.
Co-Authored-By: Claude Fable 5 <[email protected]>
"--encryption=none" and "--encryption=authenticated" have not existed since
the id hash became part of the mode name for the modes that do not encrypt:
they are now "none-sha256"/"none-blake3" and "authenticated-sha256"/
"authenticated-blake3". The examples in docs/usage/repo-create.rst still used
the old names, so copying them gave an argparse error.
The "authenticated --id-hash=blake3" example was doubly wrong: those modes
name their id hash themselves and reject a separate --id-hash. Replaced it
with plain "authenticated-blake3"/"none-blake3" examples and dropped the
now-wrong claim that the non-encrypting modes only support sha256.
Also:
- "--encryption=aes-ocb" (docs/usage/key.rst, docs/quickstart_example.rst.inc)
is not a valid mode either, it is "aes256-ocb".
- scripts/rest_perf_test.sh used "--encryption none".
- regenerated the affected man pages; borg-repo-create(1) additionally picked
up the already-fixed --key-location help text.
All examples were verified by actually running them.
All our other CI machines are 64-bit, so nothing exercises borg where a
pointer, size_t, Py_ssize_t and off_t are 32 bits wide. armv7 is the 32-bit
platform borg users still run (Raspberry Pi OS 32-bit, NAS / router /
embedded boxes); it additionally does not tolerate unaligned accesses
everywhere and has an unsigned plain "char", so it exercises C portability,
too.
Modelled on the big-endian s390x job: debian trixie (python 3.13, and its
OpenSSL 3.5 satisfies our OpenSSL >= 3.2 requirement - bookworm's 3.0 would
not) under qemu user-mode emulation, the same trigger policy (native/format
relevant paths, weekly, manual), the word size sensitive tests, and a
cross-architecture interoperability test - a repository written on the
emulated 32-bit machine is read (and written to) by the 64-bit runner and the
other way round, requiring identical chunk IDs on both sides.
Co-Authored-By: Claude Opus 5 <[email protected]>
- src/borg/crypto/low_level.pyx: implement `argon2_hash` using OpenSSL's
`EVP_KDF` API for ARGON2 (requires OpenSSL >= 3.2.0).
- src/borg/crypto/key.py: switch to the native `argon2_hash` implementation,
removing `argon2-cffi` dependency.
- setup.py: require OpenSSL >= 3.2.0 for the crypto extension to ensure
ARGON2 KDF support is available.
- pyproject.toml: drop `argon2-cffi` dependency.
- docs: update installation requirements and security documentation to
reflect the transition to OpenSSL for Argon2.
The lanes are computed in parallel via OpenSSL's thread pool: OpenSSL's
Argon2 only uses threads if OSSL_set_max_threads() enabled the thread pool
and a "threads" parameter > 1 is given. Passing threads=1 would compute the
4 lanes of our default parallelism=4 sequentially, and unlocking would get
slower than it was with argon2-cffi, which used real threads for the lanes.
The thread count only affects speed, never the derived key (only the lanes
parameter does), so this is fully compatible with existing keys.
Apple M-series (OpenSSL 3.6.3), borg defaults (t=3, m=64 MiB, p=4):
threads=1: 83 ms, threads=4: 29 ms (2.8x faster).
Co-Authored-By: Claude Opus 5 <[email protected]>
Converts thin_dump XML (full mode: allocation map of one thin LV) and
thin_delta XML (delta mode: differences between two thin snapshots)
into the borg create --map format. The docstring documents the full
workflow including the reserve/release_metadata_snap steps and the
snapshot discipline needed for --reuse-from.
Also add a docs section pointing from the --read-special LVM example
to the input map based approach.
Co-Authored-By: Claude Fable 5 <[email protected]>
The hardcoded cython_sources list had drifted from reality and was missing 8 of
the 21 .pyx modules: the chunkers base/fastcdc/goldilocks_aes/phte_chunker/
rabin_aes/toeplitz_aes, legacy/crypto/low_level and platform/netbsd.
So "make.py clean" left their generated .c files and compiled extension modules
behind. Stale extensions built for another Python version then linger in the
working tree, which is confusing at best.
Discover the sources by globbing src/borg/**/*.pyx instead, so newly added
modules are covered automatically and the list cannot go stale again. The glob
reproduces setup.py's cython_sources list exactly.
Co-Authored-By: Claude Opus 5 <[email protected]>
All our CI machines are little-endian, so the big-endian code paths in the
native code (e.g. the __builtin_bswap64 calls in the chunker kernels) are
never executed and the architecture independence of the repository format
is not tested at all.
Build and test borg on s390x in an emulated container, running the tests of
the endianness sensitive parts (chunkers, crypto, compression, hashindex,
item, repository format).
Additionally run a cross-architecture interoperability test: a repository
written on the big-endian side is checked, extracted and compared on the
little-endian side and the other way round. Both sides also have to produce
identical chunk ids for the same data - if they did not, the archives would
still be correct, but deduplication between machines of different endianness
would silently not work.
Emulating everything is slow (~35min), so this does not run for every pull
request, but only for pull requests touching native or format relevant code,
plus weekly and on demand.
Add an "environment" help topic (alias: "env") showing the documentation
about the environment variables borg supports, like borg help already does
for patterns, placeholders, compression and match-archives.
The help topic is now the single source of that documentation:
scripts/make.py build_usage generates docs/usage/general/environment.rst.inc
from it (and skips it on the misc. help docs page, where it would duplicate
the "General" chapter's section), and build_man already generates a
borg-environment(1) man page from any help topic.
The text was adapted a bit so that it renders well both via sphinx and via
nanorst on the terminal: the list-table, note directive and hyperlink
targets were replaced by constructs nanorst understands.
nanorst: end a code block when a following line is less indented than the
code block's contents (instead of the hardcoded "less than 4 spaces" rule),
so code blocks nested inside definition lists do not swallow the text
following them any more. Also, borg help <topic> now resolves :ref:
references like the command epilogs already did.
Co-Authored-By: Claude Fable 5 <[email protected]>
The binary build broke with
File "src/borg/chunkers/buzhash64.pyx", line 26, in init borg.chunkers.buzhash64
from .kernel_env import kernel_error, requested_kernel
ModuleNotFoundError: No module named 'borg.chunkers.kernel_env'
kernel_env is imported only from .pyx files, and the chunkers are Cython
extensions, so PyInstaller's static analysis of the Python sources cannot
see the import at all and never collects the module. The spec already
lists borg.chunkers.base and .phte_chunker as hiddenimports for the same
reason (they are cimported at C level); this adds kernel_env and widens
the comment, since the rule is "anything a compiled chunker pulls in has
to be listed here", not just the cimported base classes.
Verified both ways with a local pyinstaller build: with the entry the
binary runs "borg benchmark cpu" through all chunkers, and rebuilding
with the entry removed reproduces the traceback above exactly.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01TwV6jAxVttxcnJ5AVhfPxm
fish completions are now generated by shtab, like the bash, zsh and tcsh
ones, extended with a fish preamble that provides the same dynamic
completions the bash/zsh scripts have (archive names, aid: archive IDs,
tags, sort keys, files cache modes, compression specs, chunker params,
relative time markers, timestamps, file sizes, help topics).
This needs shtab >= 1.9.3, which brings the fixed fish backend (see
tqdm/shtab#227, #228, #229, #230 and #231), so the pin was bumped.
Arguments taking a filesystem path are now typed, so that all shells can
complete them: such a path was either untyped or just a str, so the
shells relied on their default completion (bash/zsh/tcsh) or offered
nothing at all (fish). FilesystemPathSpec is used for arguments taking a
file (--exclude-from, --patterns-from, TARFILE, the borg debug file
arguments, borg key export/import PATH), the new FilesystemDirSpec for
arguments taking a directory (MOUNTPOINT, borg benchmark crud PATH).
Besides the completions, they now also reject empty paths and slashify
them on Windows, like the other filesystem path arguments do.
Note that borg key export/import PATH completed directories before
(it takes a file) and borg benchmark crud PATH completed files (it takes
a directory) - both are fixed by this.
shtab's fish FILE completion needs a little help to also complete
absolute paths, see tqdm/shtab#245 (fix: tqdm/shtab#246).
The completion script is now always generated for the "borg" command,
no matter how borg was invoked while generating it - completions
generated by "python -m borg completion ..." were useless before, as
the shells would complete a "__main__.py" command.
The hand-written, unmaintained fish completions in
scripts/shell_completions/ are removed, as are their tests.
Co-Authored-By: Claude Fable 5 <[email protected]>
The CDC chunkers each carried a private copy of the same buffering
machinery: fill(), chunkify(), __iter__/__next__, the buffer/stream
state, the normalized-chunking parameter setup and the buffer part of
__dealloc__ - about 340 near-identical lines across buzhash, buzhash64,
fastcdc and ChunkerPHTE. Every buffering change (e.g. the recent
zero-copy fill) had to be made in four places.
Promote the ChunkerPHTE shape into a general ChunkerBase (base.pyx +
base.pxd for cross-module cdef inheritance):
- state + _setup_common() (sizes, buffer, counters, NC masks - with a
high_masks flag choosing high-bit (Gear) or low-bit (uniform hash)
mask placement), fill(), chunkify(), __iter__(), __next__(), and the
buffer __dealloc__ live in ChunkerBase, once.
- the window-less process() also moves into the base, parameterized by
the _scan() / _restart() hooks (one virtual call per scan span and
per chunk start, never per byte).
- ChunkerFastCDC becomes a ChunkerBase subclass and drops its own
process() entirely: it only keeps the keyed Gear table, the kernel
property and a _scan() hook (295 -> 105 lines).
- ChunkerPHTE shrinks to the 64-byte digest warm-up (_restart ->
_digest64) and kernel-name plumbing (245 -> 54 lines); rabin-aes,
goldilocks-aes and toeplitz-aes are unchanged.
- ChunkerBuzHash64 inherits everything but keeps its window-based
process() (400 -> 277 lines).
- buzhash (32-bit) is left untouched: it is frozen borg 1.x
compatible legacy.
The only C-level difference: the memmove in what was ChunkerPHTE's
fill() now runs with the GIL released, like the other copies already
did.
Net -210 lines. No behavior change: chunk cut points verified
bit-identical per-chunker over 1 GiB of synthetic data for all six CDC
chunkers, chunker test suite passes. Throughput unchanged (the hook
indirection is per-span, not per-byte).
Co-Authored-By: Claude Opus 5 <[email protected]>
rabin-aes, goldilocks-aes and toeplitz-aes differed only in their rolling
universal hash, but each carried its own copy of the AES layer, the
two-lane scan structure, the chunker class and the test suite - about 79%
of the goldilocks/toeplitz code was line-identical to rabin-aes.
Three shared pieces replace the copies:
- phte_core.h: AES-128 key expansion, endian-explicit LE helpers, hardware
detection, single-block AES helpers and the context base (round keys, EVP
context, path selection).
- phte_scan.h: the scan itself as a compile-time template. Each kernel
defines its rolling functions and PH_PREFIX/PH_CTX/PH_ROLL/PH_ROLL2/
PH_DIGEST64, then includes it, so the rolling code is still expanded
inline into fully specialized scan loops - no function pointers, no
cross-kernel calls, identical code generation.
- phte_chunker.pyx/.pxd: ChunkerPHTE, the shared chunker driver (buffering,
min/max clamping, normalized chunking, scan loop, sparse handling). The
subclasses implement _scan()/_digest64() as Cython cdef methods, i.e. one
vtable call per scan span (up to a whole chunk) and per chunk start, never
per byte.
Likewise for the tests: the behavior every one of these chunkers must have
(kernel equivalence, size distribution, shift resilience, param parsing,
fuzzing) now runs parametrized over all three in phte_chunkers_test.py; the
per-chunker modules keep only their frozen golden chunk points and the
tests of their own hash arithmetic and key derivation.
Net -2335 lines. Each kernel .c is now 84-125 lines (was 523-564) and each
chunker .pyx no longer contains the ~250-line class body.
Verified unchanged behavior and performance:
- all three frozen golden chunk-point tests unchanged, 508 tests pass,
mypy/black/ruff clean, e2e create+extract with all three chunkers OK;
- C-level A/B of the production scan (same harness, HEAD kernels vs these,
64 MiB, best of 8): rabin 683.6 -> 682.5, goldilocks 331.8 -> 331.8,
toeplitz 687.4 -> 694.7 MB/s hw; EVP paths 376.6 -> 376.6, 251 -> 250,
394.2 -> 394.0 - all within +-1%;
- disassembly of ra_scan keeps the identical AES and table-lookup structure
(853 vs 850 instructions; the delta is prologue register shuffling caused
by the changed struct field offsets).
Also excludes *.pxd from the wheel (build-time only, borg's first .pxd).
Co-Authored-By: Claude Fable 5 <[email protected]>
Third UHF-then-PRF chunker, and on the evidence the best construction of
the three: the universal hash is a tabulated LFSR-based Toeplitz hash
(Krawczyk, CRYPTO '94) - digest = sum_j x^(63-j) * T[b_j] over
GF(2)[x] mod P - with a secret random table T of 256 uint64 (2 KiB of
key material) and a FIXED PUBLIC irreducible P of degree 64
(x^64 + x^4 + x^3 + x + 1, the CLHASH polynomial, top bit implicit).
Collision bound: two distinct 64-byte windows collide with probability
EXACTLY 2^-64 over T - optimal for a 64-bit digest and unconditional
(rabin-aes: ~2^-55 probabilistic over the sampled P; goldilocks-aes:
~2^-58). Sketch: the difference is sum_v c_v * T[v] where some c_v is a
nonzero polynomial of degree < 64, hence an invertible element mod the
irreducible degree-64 P, making the sum uniform. This needs the fixed
64-byte window (distinct powers x^0..x^63); a plain-rotation variant
(keyed buzhash) fails - rotations satisfy R^64 = I, so e.g. two
all-same-byte windows collide with probability 1/2.
The rolling step d' = d*x ^ x^64*T[out] ^ T[in] reduces via a branchless
shift+masked-XOR and indexes tables only by plaintext bytes: no loads and
no secret-dependent memory access on the serial chain. Setup needs no
irreducibility sampling (P is public; the secrecy is the table).
Same C kernel architecture as rabin-aes/goldilocks-aes (two even/odd
lanes, grouped AES on arm64/AES-NI, batched OpenSSL EVP, bit-identical
paths), fully wired (constants, ChunkerParams, get_chunker, completion,
borg benchmark cpu, docs, tests with frozen golden chunk points),
verified end-to-end with repo-create/create/extract.
Measured (1 GiB random, 19,23,21, 5 runs x 2 reps): hw 666/713 (nc2)
MB/s - on par with rabin-aes 661/698; EVP 424/450 vs 405/434 (~+5%).
The tie on the hw path is itself a finding: at ~700 MB/s the grouped-AES
path is AES/transfer-bound, not roll-chain-bound, so the cheaper roller
only shows on the EVP path. Equal speed, strictly better bound.
Co-Authored-By: Claude Fable 5 <[email protected]>
Add a second UHF-then-PRF chunker implementing the reference
construction of "Breaking and Fixing Content-Defined Chunking"
(Truong et al., eprint 2025/558) with its original universal hash: a
rolling polynomial hash over the Goldilocks prime field GF(p),
p = 2^64 - 2^32 + 1, at a secret random evaluation point K, over the
same 64-byte window - followed by the same AES-128 PRF layer as
rabin-aes. The C kernel mirrors rabin_aes_impl.c exactly (two even/odd
rolling lanes, groups of 8 pipelined AES blocks on arm64/AES-NI,
batched OpenSSL EVP fallback, bit-identical paths); only the rolling
arithmetic differs. Negated removal tables keep the field multiply as
the only critical-path operation; all values stay canonical since the
state feeds AES verbatim.
Verified equivalent to the paper's Rust artifact on shared keys and
random data: rolling states, AES input blocks and ciphertexts are
bit-identical per position; only the cut rule differs (they mask the
last ciphertext bytes via a big-endian u128, we mask the first 8
little-endian), so cut positions do not coincide - by choice, both
rules see uniform PRF bits.
Purpose is an evidence baseline, not a new default: on an Apple
M-series (256 MiB random, 19,23,21), 354 MB/s hw / 267 MB/s EVP versus
rabin-aes 670 / 401 - the GF(p) multiply chain is ~2x the GF(2)
shift+xor chain, table loads hide off-path in both. Non-speed
trade-offs versus rabin-aes: slightly better collision bound (63/p ~=
2^-58 vs ~2^-55), trivial key sampling (no irreducibility search), no
secret-dependent table indices in the hot loop, and a cheaper wide-
digest upgrade path (two independent K vs GF(2^127) clmul); rabin-aes
keeps the 2x throughput. Wired into get_chunker ("goldilocks-aes",
own key domain) and chunker_bench.py only.
Co-Authored-By: Claude Fable 5 <[email protected]>
Add a "UHF-then-PRF" content-defined chunker following the Chk-PHTE
construction of "Breaking and Fixing Content-Defined Chunking"
(Truong et al., eprint 2025/558): a rolling Rabin fingerprint over
GF(2)[x]/P(x) - with P a secret, random, irreducible polynomial of
degree 63 - hashes the last 64 bytes (a universal hash family), and
AES-128 with a secret key is applied to the digest; the cut decision
only looks at the AES output. Observed chunk boundaries are thus
pseudo-random and yield no usable equations about the chunking
secrets, unlike chunkers cutting directly on (keyed) rolling-hash
bits. Both secrets are derived from the id key (domain "rabin-aes").
The C scan kernel has three bit-identical paths: batched OpenSSL EVP
AES-128-ECB (portable), arm64 crypto-extension intrinsics and x86-64
AES-NI (runtime detected), interleaving the AES work with the serial
rolling-hash chain. Batching ahead of the cut decision is possible
because the digest stream does not depend on where cuts happen.
Throughput on an Apple M-series (1 GiB random, 19,23,21,2): 489 MB/s
(arm64 hw path), 334 MB/s (EVP path); chunk-size distribution, shift
resilience and dedup behavior are identical to fastcdc/buzhash64.
Normalized chunking, param shape rabin-aes,min,max,mask,nc_level.
Co-Authored-By: Claude Fable 5 <[email protected]>
The workflow no longer runs on pushes to FAME.md. The "Lines" column is
blame of the current checkout and master sees dozens of commits a week, so
merging one of these pull requests immediately produced the next one:
#10008 was merged at 10:26:09 and #10009 opened 37 seconds later. Both
files are written by the same run and cannot drift apart, so the weekly run
plus manual dispatch is enough.
git-fame also leaves people with the same commit count in its own order,
which is stable for one checkout but not across checkouts, so tied ranks
kept shuffling and added diff noise. Sort by commits, then surviving
lines, then name. The next run will apply that order to FAME.md.
The logo green (#00dd00) is made for a dark background; on white it only
reaches a contrast of 1.9:1, so the light scheme uses a darker green of the
same hue.
git-fame's own --format=svg renders the whole table as monospaced text,
which works for a handful of contributors but not for a few hundred, so
draw a bar chart of the top 20 (--svg-top) instead. FAME.md embeds it.
The chart follows the reader's colour scheme, sizes its viewport in user
units plus a viewBox (em on the root <svg> would resolve against the wrong
font size) and escapes author names, which are commit-controlled input.
Bots keep their row in the table, but do not take a chart slot away from a
person.
The workflow now also runs when FAME.md changes on master, so that the two
files cannot drift apart.
The workflow runs scripts/fame.py against master every Monday and opens a
pull request if the contributor statistics changed.
For that to be quiet in weeks where nothing changed, fame.py now leaves
FAME.md alone when only the generation date would differ.
reimplemented for the pack-based repository format: the repository is
processed pack after pack - each pack is read as a whole and, if it has
objects not stored with the desired compression, those get recompressed
and the pack is rewritten (objects already fine are copied into the new
pack unchanged), the chunk index is updated and the old pack is deleted.
a pack whose objects all already match is not touched at all.
- new Repository.transform_pack: rewrite one pack, passing each indexed
object's bytes through a transform callback. gap bytes are handled
like in compact_pack: superseded duplicates are dropped, other
unindexed bytes are carried forward for "borg check --repair".
- factored the check_pack_objects / superseded_gap_ranges helpers out
of compact_pack, now shared with merge_packs and transform_pack.
- crash safety like compact (#9748): stored chunk indexes are
invalidated before the first store change, a full updated index is
written back at the end. SIGINT stops cleanly at a pack boundary,
saving a valid index; a later run recompresses the remaining packs.
- --stats reports rewritten packs, per-object outcomes (recompressed /
already had the desired compression / kept as-is because
recompression brings no gain) and the repository size change.
- re-adding the fish completions also restores the compression_methods
definition that the create/transfer/recreate/import-tar completions
kept referencing after it was removed together with the old
repo-compress.
Co-Authored-By: Claude Fable 5 <[email protected]>
git blame / git shortlog listed a number of contributors twice, either because
the same address was used with different spellings of the name, or because the
same person committed from several addresses. Merge them in .mailmap: this
brings the number of distinct authors down from 380 to 370.
Note that two of the merges are judgement calls, see the diff for the entries
in question: one address was used with two different real names and a handle
(the most recent non-handle name is used), and another was used mostly with a
handle and a few times with a real name (the real name is used).
Add scripts/fame.py, which writes FAME.md from git-fame output, sorted by
commit count. It bakes in an exclusion pattern for generated files - without
it, the three Excalidraw sources of the pack format figures contribute 39230
lines (a third of the repository) to a single contributor, because they are
pretty-printed JSON.
Co-Authored-By: Claude Opus 5 <[email protected]>
blake3 can hash a single input in parallel, but borg always used the
single-threaded path. Let id_hash use blake3's internal thread pool once the
input is big enough.
Multi-threading is not a win at every size: blake3 parallelises over its binary
hash tree, so the speedup peaks at powers of two and decays in between, and for
small inputs the thread dispatch costs much more than it saves. Measured on a
12-core machine (Apple M3 Pro): 0.16x at 8 KiB, still below 1x at 240 KiB, then
2.0x at 256 KiB, 3.2x at 512 KiB, 5.6x at 2 MiB and 7.3x at 8 MiB.
256 KiB was the smallest size that never lost there, so that is the default and
smaller chunks keep hashing single-threaded. Chunk sizes are content-defined, so
we can not rely on hitting the sizes that happen to parallelise well and have to
be conservative.
The optimal value depends on the core count and therefore differs per machine,
so BORG_BLAKE3_MT_THRESHOLD (in KiB) can override the default. It is evaluated
on first use and then cached: get_blake3_mt_threshold() runs for every chunk and
os.environ.get() costs ~425ns (~30% of id_hash for a 512B chunk), while the
cached call costs ~45ns. Evaluating it lazily rather than at import time also
means an invalid value is reported by borg's normal error handling instead of a
traceback while importing.
scripts/blake3-optimize-mt-threshold.py measures the best value for a machine:
it sweeps input sizes, verifies a candidate by probing the sizes that are
usually worst, re-measures the point that determines the threshold to keep
scheduling noise out, and optionally writes an HTML or SVG chart.
This only affects repos using --id-hash blake3. The resulting chunk ids are
unchanged - that single- and multi-threaded blake3 agree is the blake3
project's invariant, so there is no test for it here.
Co-Authored-By: Claude Opus 5 <[email protected]>
Set BORG_DEBUG_PROFILE to a filename to get an execution profile written
there; a ".pyprof" suffix still selects the Python-compatible format.
Note that this now applies to EVERY borg invocation while the variable is
set, not just to a single command like the option did - the docs say so and
the test switches it off around the "borg debug convert-profile" call, which
would otherwise profile itself over the profile it is reading.
Unlike the other options removed recently, --iec did work for borg2 repos -
but the same setting is better placed in the environment: it is a display
preference one usually wants for all borg invocations, not per command.
BORG_IEC=yes now selects IEC units (1KiB = 1024B); "true" and "1" are
accepted, too, because jsonargparse's automatic BORG_<OPTION> environment
variables accepted these for --iec. Note that removing the option also
removes that automatic environment variable, so the new use_iec_units()
helper reads BORG_IEC explicitly.
The iec=... plumbing through Archive, Statistics, ArchiveFormatter and Cache
is unchanged, only the places reading args.iec now call use_iec_units().
Both were only ever read by the legacy (borg 1.x, ssh://) remote repository
code, so they had no effect on borg2 repositories - borgstore does not do
rate limiting or upload buffering at all.
For the legacy code path they are not worth keeping either: we only support
transferring FROM borg1 repos, so there is not much traffic in the upload
direction of a borg1 repo anyway.
Removes the SleepingBandwidthLimiter class with them; the only thing left
from it is write_to_fd(), which turns a BrokenPipeError into the nicer
ConnectionBrokenWithHint. Without an upload buffer size limit, the "queue
more calls" condition simplifies to "the send buffer is empty".
Like --remote-path, --rsh was only honoured on the legacy (borg 1.x, ssh://)
code path. For borg2 repositories the remote shell command comes from
borgstore, which reads its own BORGSTORE_RSH environment variable, so --rsh
silently had no effect there.
Remove the option; BORG_RSH is now the only way to set the remote shell
command. For borgstore-based repositories, borg gives the BORG_RSH value to
borgstore as BORGSTORE_RSH, except if that was set explicitly (then the user
wants a different command for borgstore).
borg benchmark crud does not need to inherit --rsh / --remote-path into its
sub-invocations any more (environment variables are inherited anyway), so the
parse_args wrapper is gone.
The --remote-path option was only honoured on the legacy (borg 1.x, ssh://)
code path in legacy/remote.py. For borg2 rest:// repositories, the remote
borg is determined by rest_serve_command(), which only ever looked at the
BORG_REMOTE_PATH environment variable - so giving --remote-path silently had
no effect and the far end failed with "command not found: borg".
Rather than plumbing args into rest_serve_command(), drop the option
completely and use BORG_REMOTE_PATH consistently for both repo types. Also
apply replace_placeholders() to it for rest:// repos, as the legacy code
path already did.
Add a new "fastcdc" content-defined chunker selectable via --chunker-params.
It uses the FastCDC Gear rolling hash (fp = (fp << 1) + Gear[byte]), which is
window-less and cheaper per byte than buzhash's cyclic-polynomial update, so it
chunks noticeably faster (see "borg benchmark cpu" output), while producing
the same chunk-size distribution and deduplication.
The Gear table is keyed: it is derived from the repo id key via CSPRNG (own
"fastcdc" domain), exactly like the buzhash64 table, so chunk cut points stay
unpredictable without the key (anti-fingerprinting). It implements the same
FastCDC techniques as buzhash64 (sub-minimum skipping, normalized chunking with
a required nc_level, min/max clamping); the mask uses the high bits of the hash
(Gear accumulates entropy there).
chunker-params: "fastcdc,chunk_min,chunk_max,chunk_mask,nc_level" - there is no
window field, because Gear is window-less. e.g. fastcdc,19,23,21,2
Also: borg benchmark cpu now measures the fastcdc chunker; tests in
borg.testsuite.chunkers (golden vector, size distribution, keyed gear table,
param parsing, slow fuzz); docs and changelog.
Benchmarks (scripts/chunker_bench.py, buzhash64 vs fastcdc, both nc_level=2,
incompressible data unless noted):
5 GiB, 2 MiB target (default params):
buzhash64: CV 0.294, 1011 MB/s
fastcdc: CV 0.295, 1313 MB/s (+30%)
64 MiB, 64 KiB target:
buzhash64: CV 0.374, shift-resilience 0.9928, 963 MB/s
fastcdc: CV 0.359, shift-resilience 0.9929, 1331 MB/s (+38%)
Re-backup of a 2.5 GiB file after scattered single-byte edits (dedup ratio,
0.5 = v2 fully deduplicated, lower is better):
64 edits: buzhash64 0.5237, fastcdc 0.5236
320 edits: buzhash64 0.6133, fastcdc 0.6161
borg benchmark cpu, 1 GB: fastcdc 3.80s, buzhash 4.36s, buzhash64 8.13s,
fixed 0.56s.
Chunk-size distribution, deduplication and shift-resilience match buzhash64
within noise; fastcdc is consistently faster.
Also: fix bug when computing the mask, one needs to use 1ULL instead of
1, so the shifting computation is done in a uint64, not in a 32bit int.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Normalized chunking switches between a stricter and a looser cut mask
around the target chunk size. This greatly tightens the chunk-size
distribution (coefficient of variation ~0.9 -> ~0.3 in tests) and removes
the dedup-hostile max-size-clamped chunks, with unchanged deduplication.
chunker-params for buzhash64 gains a required 6th field, nc_level:
buzhash64,chunk_min,chunk_max,chunk_mask,window_size,nc_level
Use nc_level=2 for the new default, nc_level=0 to disable (then behavior
is byte-identical to the previous single-mask chunker).
buzhash (32bit) is untouched and stays bit-compatible with borg 1.x.
The mask transition point (normal_size) defaults to a principled formula
(target minus the expected loose-phase tail) so the mean stays near the
target; it can be tuned via the normal_size constructor arg.
scripts/chunker_bench.py: evidence harness used to measure chunk-size
distribution, dedup ratio, throughput and shift-resilience.
Measurements (before = nc_level 0, after = nc_level 2; both at the default
params buzhash64,19,23,21,4095; measured with scripts/chunker_bench.py):
5 GiB of incompressible data (~2000-2700 chunks, statistically stable):
before: CV 0.739, 49 max-size-clamped (8 MiB) chunks, 953 MB/s
after: CV 0.311, 0 max-size-clamped chunks, 1024 MB/s
Re-backup of a 2.5 GiB file after a few scattered single-byte edits
(deduplication ratio; 0.5 = v2 fully deduplicated against v1, lower is
better):
64 edits: before 0.5424 -> after 0.5235
320 edits: before 0.6791 -> after 0.6142
Normalized chunking deduplicates better after edits: removing the
max-size-clamped chunks means a single-byte change invalidates much less
data (about 36% less dedup overhead at 320 edits). Throughput was also
consistently higher with nc_level=2 at this scale.
Also: fix bug when computing the mask, one needs to use 1ULL instead of
1, so the shifting computation is done in a uint64, not in a 32bit int.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The multi-key feature (#9743) added the "borg key add", "borg key list"
and "borg key remove" subcommands but never wired up their docs:
- scripts/make.py: map key_add/key_list/key_remove to the "key" usage
group, so build_man can locate their examples (it previously aborted
with FileNotFoundError: docs/usage/key_add.rst).
- docs/usage/key.rst: include the three new generated snippets so they
show up on the HTML key page.
- regenerate the affected docs: new key_{add,list,remove}.rst.inc usage
snippets, new borg-key-{add,list,remove}.1 man pages, and borg-key.1
(SYNOPSIS + SEE ALSO now reference the new subcommands).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Ports #9755 (1.4-maint) to master.
macOS flags files whose content lives in cloud storage (e.g. iCloud Drive)
and is not present locally with SF_DATALESS. Reading such a file triggers
downloading its content.
This adds --exclude-dataless to borg create: the flags are checked right
after stat() and before any open(), so excluded files/dirs never get
materialized. Skipped paths are reported with the usual 'x' status,
analogous to --exclude-nodump.
Notes:
- stat.SF_DATALESS only exists from Python 3.13 on, so there is a fallback
to the value from macOS' sys/stat.h (0x40000000).
- The check covers directories too (they can also be dataless), and we skip
before opening/recursing into them.
- Test fakes get_flags via monkeypatch since SF_DATALESS cannot be set from
userspace.
- Fish shell completion updated; bash/zsh completions don't exist in master.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
for packs, this needs to get implemented differently to perform well.
processing needs to be pack-after-pack and the index needs to be
updated correctly and carefully, e.g. considering interruptions
of repo-compress.
