Add a new "fastcdc" content-defined chunker selectable via --chunker-params.
It uses the FastCDC Gear rolling hash (fp = (fp << 1) + Gear[byte]), which is
window-less and cheaper per byte than buzhash's cyclic-polynomial update, so it
chunks noticeably faster (see "borg benchmark cpu" output), while producing
the same chunk-size distribution and deduplication.
The Gear table is keyed: it is derived from the repo id key via CSPRNG (own
"fastcdc" domain), exactly like the buzhash64 table, so chunk cut points stay
unpredictable without the key (anti-fingerprinting). It implements the same
FastCDC techniques as buzhash64 (sub-minimum skipping, normalized chunking with
a required nc_level, min/max clamping); the mask uses the high bits of the hash
(Gear accumulates entropy there).
chunker-params: "fastcdc,chunk_min,chunk_max,chunk_mask,nc_level" - there is no
window field, because Gear is window-less. e.g. fastcdc,19,23,21,2
Also: borg benchmark cpu now measures the fastcdc chunker; tests in
borg.testsuite.chunkers (golden vector, size distribution, keyed gear table,
param parsing, slow fuzz); docs and changelog.
Benchmarks (scripts/chunker_bench.py, buzhash64 vs fastcdc, both nc_level=2,
incompressible data unless noted):
5 GiB, 2 MiB target (default params):
buzhash64: CV 0.294, 1011 MB/s
fastcdc: CV 0.295, 1313 MB/s (+30%)
64 MiB, 64 KiB target:
buzhash64: CV 0.374, shift-resilience 0.9928, 963 MB/s
fastcdc: CV 0.359, shift-resilience 0.9929, 1331 MB/s (+38%)
Re-backup of a 2.5 GiB file after scattered single-byte edits (dedup ratio,
0.5 = v2 fully deduplicated, lower is better):
64 edits: buzhash64 0.5237, fastcdc 0.5236
320 edits: buzhash64 0.6133, fastcdc 0.6161
borg benchmark cpu, 1 GB: fastcdc 3.80s, buzhash 4.36s, buzhash64 8.13s,
fixed 0.56s.
Chunk-size distribution, deduplication and shift-resilience match buzhash64
within noise; fastcdc is consistently faster.
Also: fix bug when computing the mask, one needs to use 1ULL instead of
1, so the shifting computation is done in a uint64, not in a 32bit int.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Normalized chunking switches between a stricter and a looser cut mask
around the target chunk size. This greatly tightens the chunk-size
distribution (coefficient of variation ~0.9 -> ~0.3 in tests) and removes
the dedup-hostile max-size-clamped chunks, with unchanged deduplication.
chunker-params for buzhash64 gains a required 6th field, nc_level:
buzhash64,chunk_min,chunk_max,chunk_mask,window_size,nc_level
Use nc_level=2 for the new default, nc_level=0 to disable (then behavior
is byte-identical to the previous single-mask chunker).
buzhash (32bit) is untouched and stays bit-compatible with borg 1.x.
The mask transition point (normal_size) defaults to a principled formula
(target minus the expected loose-phase tail) so the mean stays near the
target; it can be tuned via the normal_size constructor arg.
scripts/chunker_bench.py: evidence harness used to measure chunk-size
distribution, dedup ratio, throughput and shift-resilience.
Measurements (before = nc_level 0, after = nc_level 2; both at the default
params buzhash64,19,23,21,4095; measured with scripts/chunker_bench.py):
5 GiB of incompressible data (~2000-2700 chunks, statistically stable):
before: CV 0.739, 49 max-size-clamped (8 MiB) chunks, 953 MB/s
after: CV 0.311, 0 max-size-clamped chunks, 1024 MB/s
Re-backup of a 2.5 GiB file after a few scattered single-byte edits
(deduplication ratio; 0.5 = v2 fully deduplicated against v1, lower is
better):
64 edits: before 0.5424 -> after 0.5235
320 edits: before 0.6791 -> after 0.6142
Normalized chunking deduplicates better after edits: removing the
max-size-clamped chunks means a single-byte change invalidates much less
data (about 36% less dedup overhead at 320 edits). Throughput was also
consistently higher with nc_level=2 at this scale.
Also: fix bug when computing the mask, one needs to use 1ULL instead of
1, so the shifting computation is done in a uint64, not in a 32bit int.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The multi-key feature (#9743) added the "borg key add", "borg key list"
and "borg key remove" subcommands but never wired up their docs:
- scripts/make.py: map key_add/key_list/key_remove to the "key" usage
group, so build_man can locate their examples (it previously aborted
with FileNotFoundError: docs/usage/key_add.rst).
- docs/usage/key.rst: include the three new generated snippets so they
show up on the HTML key page.
- regenerate the affected docs: new key_{add,list,remove}.rst.inc usage
snippets, new borg-key-{add,list,remove}.1 man pages, and borg-key.1
(SYNOPSIS + SEE ALSO now reference the new subcommands).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Ports #9755 (1.4-maint) to master.
macOS flags files whose content lives in cloud storage (e.g. iCloud Drive)
and is not present locally with SF_DATALESS. Reading such a file triggers
downloading its content.
This adds --exclude-dataless to borg create: the flags are checked right
after stat() and before any open(), so excluded files/dirs never get
materialized. Skipped paths are reported with the usual 'x' status,
analogous to --exclude-nodump.
Notes:
- stat.SF_DATALESS only exists from Python 3.13 on, so there is a fallback
to the value from macOS' sys/stat.h (0x40000000).
- The check covers directories too (they can also be dataless), and we skip
before opening/recursing into them.
- Test fakes get_flags via monkeypatch since SF_DATALESS cannot be set from
userspace.
- Fish shell completion updated; bash/zsh completions don't exist in master.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
for packs, this needs to get implemented differently to perform well.
processing needs to be pack-after-pack and the index needs to be
updated correctly and carefully, e.g. considering interruptions
of repo-compress.
Remove the handwritten bash and zsh shell completion scripts now that
auto-generated completions via borg completion bash/zsh (powered by
shtab, #9172) are tested and working. Fish completions are kept as
shtab does not yet support fish.
Replace string-matching tests with focused behavior tests: script size
sanity, shell syntax validation (bash -n / zsh -n), and tests that
invoke the custom preamble functions in bash (sortby key dedup,
filescachemode mutual exclusivity, archive name and aid: prefix
completion against a real repository).
Moved the `ChunkerFixed` implementation from `chunker` to a new `fixed` module for better modularity. Updated imports and type hints.
Removed now empty chunkers.chunker module.
Moved `buzhash` implementation from `chunker` to a new `buzhash` module for better separation of concerns. Updated imports, adjusted `setup.py` and build configuration accordingly. Removed deprecated `Chunker` definitions from `chunker.pyi`.
Extracted the `reader` logic from `chunker` into a dedicated `reader` module to improve modularity and maintainability. Updated imports, references, and build configurations accordingly.
Updated bash completions to include new commands such as `analyze`, `debug`, `repo-space`, `tag`, and `undelete`, along with their respective options. Fixed a typo in the `--upgrader` completions and improved completion handling for various commands.
thanks a lot to @sothix for helping with this!
removed pytest-forked, is not found anymore:
error: target not found: mingw-w64-ucrt-x86_64-python-pytest-forked
use a virtual env to avoid mixup of user with system packages.
remove old workaround for setuptools (SETUPTOOLS_USE_DISTUTILS: stdlib).
fix pip install
use --system-site-packages as a workaround for broken pip install python-cffi.
do not upgrade pip setuptools build wheel
use python -m pytest to use the one from the venv
Some features like append-only repositories rely on a server-side component
that enforces them (because that shall only be controllable server-side,
not client-side).
So, that can only work, if such a server-side component exists, which is the
case for borg 1.x ssh: repositories (but not for borg 1.x non-ssh: repositories).
For borg2, we currently have:
- fs repos
- sftp: repos
- rclone: repos (enabling many different cloud providers)
- s3/b3: repos
- ssh: repos using client/server rpc code similar as in borg 1.x
So, only for the last method we have a borg server-side process that could enforce some features, but not for any of the other repo types.
For append-only the current idea is that this should not be done within borg,
but solved by a missing repo object delete permission enforced by the storage.
borg create could then use credentials that miss permission to delete,
while borg compact would use credentials that include permission to delete.
Some features like repository quotas rely on a server-side component
that enforces them (because that shall only be controllable server-side,
not client-side).
So, that can only work, if such a server-side component exists, which is the
case for borg 1.x ssh: repositories (but not for borg 1.x non-ssh: repositories).
For borg2, we currently have:
- fs repos
- sftp: repos
- rclone: repos (enabling many different cloud providers)
- s3/b3: repos
- ssh: repos using client/server rpc code similar as in borg 1.x
So, only for the last method we have a borg server-side process that could enforce some features, but not for any of the other repo types.
For quotas the current idea is that this should not be done within borg,
but enforced by a storage specific quota implementation (like fs quota,
or quota of the cloud storage provider). borg could offer information
about overall repo space used, but would not enforce quotas within borg.
Thomas Waldmann