The workflow no longer runs on pushes to FAME.md. The "Lines" column is
blame of the current checkout and master sees dozens of commits a week, so
merging one of these pull requests immediately produced the next one:
#10008 was merged at 10:26:09 and #10009 opened 37 seconds later. Both
files are written by the same run and cannot drift apart, so the weekly run
plus manual dispatch is enough.
git-fame also leaves people with the same commit count in its own order,
which is stable for one checkout but not across checkouts, so tied ranks
kept shuffling and added diff noise. Sort by commits, then surviving
lines, then name. The next run will apply that order to FAME.md.
The logo green (#00dd00) is made for a dark background; on white it only
reaches a contrast of 1.9:1, so the light scheme uses a darker green of the
same hue.
git-fame's own --format=svg renders the whole table as monospaced text,
which works for a handful of contributors but not for a few hundred, so
draw a bar chart of the top 20 (--svg-top) instead. FAME.md embeds it.
The chart follows the reader's colour scheme, sizes its viewport in user
units plus a viewBox (em on the root <svg> would resolve against the wrong
font size) and escapes author names, which are commit-controlled input.
Bots keep their row in the table, but do not take a chart slot away from a
person.
The workflow now also runs when FAME.md changes on master, so that the two
files cannot drift apart.
The workflow runs scripts/fame.py against master every Monday and opens a
pull request if the contributor statistics changed.
For that to be quiet in weeks where nothing changed, fame.py now leaves
FAME.md alone when only the generation date would differ.
reimplemented for the pack-based repository format: the repository is
processed pack after pack - each pack is read as a whole and, if it has
objects not stored with the desired compression, those get recompressed
and the pack is rewritten (objects already fine are copied into the new
pack unchanged), the chunk index is updated and the old pack is deleted.
a pack whose objects all already match is not touched at all.
- new Repository.transform_pack: rewrite one pack, passing each indexed
object's bytes through a transform callback. gap bytes are handled
like in compact_pack: superseded duplicates are dropped, other
unindexed bytes are carried forward for "borg check --repair".
- factored the check_pack_objects / superseded_gap_ranges helpers out
of compact_pack, now shared with merge_packs and transform_pack.
- crash safety like compact (#9748): stored chunk indexes are
invalidated before the first store change, a full updated index is
written back at the end. SIGINT stops cleanly at a pack boundary,
saving a valid index; a later run recompresses the remaining packs.
- --stats reports rewritten packs, per-object outcomes (recompressed /
already had the desired compression / kept as-is because
recompression brings no gain) and the repository size change.
- re-adding the fish completions also restores the compression_methods
definition that the create/transfer/recreate/import-tar completions
kept referencing after it was removed together with the old
repo-compress.
Co-Authored-By: Claude Fable 5 <[email protected]>
git blame / git shortlog listed a number of contributors twice, either because
the same address was used with different spellings of the name, or because the
same person committed from several addresses. Merge them in .mailmap: this
brings the number of distinct authors down from 380 to 370.
Note that two of the merges are judgement calls, see the diff for the entries
in question: one address was used with two different real names and a handle
(the most recent non-handle name is used), and another was used mostly with a
handle and a few times with a real name (the real name is used).
Add scripts/fame.py, which writes FAME.md from git-fame output, sorted by
commit count. It bakes in an exclusion pattern for generated files - without
it, the three Excalidraw sources of the pack format figures contribute 39230
lines (a third of the repository) to a single contributor, because they are
pretty-printed JSON.
Co-Authored-By: Claude Opus 5 <[email protected]>
blake3 can hash a single input in parallel, but borg always used the
single-threaded path. Let id_hash use blake3's internal thread pool once the
input is big enough.
Multi-threading is not a win at every size: blake3 parallelises over its binary
hash tree, so the speedup peaks at powers of two and decays in between, and for
small inputs the thread dispatch costs much more than it saves. Measured on a
12-core machine (Apple M3 Pro): 0.16x at 8 KiB, still below 1x at 240 KiB, then
2.0x at 256 KiB, 3.2x at 512 KiB, 5.6x at 2 MiB and 7.3x at 8 MiB.
256 KiB was the smallest size that never lost there, so that is the default and
smaller chunks keep hashing single-threaded. Chunk sizes are content-defined, so
we can not rely on hitting the sizes that happen to parallelise well and have to
be conservative.
The optimal value depends on the core count and therefore differs per machine,
so BORG_BLAKE3_MT_THRESHOLD (in KiB) can override the default. It is evaluated
on first use and then cached: get_blake3_mt_threshold() runs for every chunk and
os.environ.get() costs ~425ns (~30% of id_hash for a 512B chunk), while the
cached call costs ~45ns. Evaluating it lazily rather than at import time also
means an invalid value is reported by borg's normal error handling instead of a
traceback while importing.
scripts/blake3-optimize-mt-threshold.py measures the best value for a machine:
it sweeps input sizes, verifies a candidate by probing the sizes that are
usually worst, re-measures the point that determines the threshold to keep
scheduling noise out, and optionally writes an HTML or SVG chart.
This only affects repos using --id-hash blake3. The resulting chunk ids are
unchanged - that single- and multi-threaded blake3 agree is the blake3
project's invariant, so there is no test for it here.
Co-Authored-By: Claude Opus 5 <[email protected]>
Set BORG_DEBUG_PROFILE to a filename to get an execution profile written
there; a ".pyprof" suffix still selects the Python-compatible format.
Note that this now applies to EVERY borg invocation while the variable is
set, not just to a single command like the option did - the docs say so and
the test switches it off around the "borg debug convert-profile" call, which
would otherwise profile itself over the profile it is reading.
Unlike the other options removed recently, --iec did work for borg2 repos -
but the same setting is better placed in the environment: it is a display
preference one usually wants for all borg invocations, not per command.
BORG_IEC=yes now selects IEC units (1KiB = 1024B); "true" and "1" are
accepted, too, because jsonargparse's automatic BORG_<OPTION> environment
variables accepted these for --iec. Note that removing the option also
removes that automatic environment variable, so the new use_iec_units()
helper reads BORG_IEC explicitly.
The iec=... plumbing through Archive, Statistics, ArchiveFormatter and Cache
is unchanged, only the places reading args.iec now call use_iec_units().
Both were only ever read by the legacy (borg 1.x, ssh://) remote repository
code, so they had no effect on borg2 repositories - borgstore does not do
rate limiting or upload buffering at all.
For the legacy code path they are not worth keeping either: we only support
transferring FROM borg1 repos, so there is not much traffic in the upload
direction of a borg1 repo anyway.
Removes the SleepingBandwidthLimiter class with them; the only thing left
from it is write_to_fd(), which turns a BrokenPipeError into the nicer
ConnectionBrokenWithHint. Without an upload buffer size limit, the "queue
more calls" condition simplifies to "the send buffer is empty".
Like --remote-path, --rsh was only honoured on the legacy (borg 1.x, ssh://)
code path. For borg2 repositories the remote shell command comes from
borgstore, which reads its own BORGSTORE_RSH environment variable, so --rsh
silently had no effect there.
Remove the option; BORG_RSH is now the only way to set the remote shell
command. For borgstore-based repositories, borg gives the BORG_RSH value to
borgstore as BORGSTORE_RSH, except if that was set explicitly (then the user
wants a different command for borgstore).
borg benchmark crud does not need to inherit --rsh / --remote-path into its
sub-invocations any more (environment variables are inherited anyway), so the
parse_args wrapper is gone.
The --remote-path option was only honoured on the legacy (borg 1.x, ssh://)
code path in legacy/remote.py. For borg2 rest:// repositories, the remote
borg is determined by rest_serve_command(), which only ever looked at the
BORG_REMOTE_PATH environment variable - so giving --remote-path silently had
no effect and the far end failed with "command not found: borg".
Rather than plumbing args into rest_serve_command(), drop the option
completely and use BORG_REMOTE_PATH consistently for both repo types. Also
apply replace_placeholders() to it for rest:// repos, as the legacy code
path already did.
Add a new "fastcdc" content-defined chunker selectable via --chunker-params.
It uses the FastCDC Gear rolling hash (fp = (fp << 1) + Gear[byte]), which is
window-less and cheaper per byte than buzhash's cyclic-polynomial update, so it
chunks noticeably faster (see "borg benchmark cpu" output), while producing
the same chunk-size distribution and deduplication.
The Gear table is keyed: it is derived from the repo id key via CSPRNG (own
"fastcdc" domain), exactly like the buzhash64 table, so chunk cut points stay
unpredictable without the key (anti-fingerprinting). It implements the same
FastCDC techniques as buzhash64 (sub-minimum skipping, normalized chunking with
a required nc_level, min/max clamping); the mask uses the high bits of the hash
(Gear accumulates entropy there).
chunker-params: "fastcdc,chunk_min,chunk_max,chunk_mask,nc_level" - there is no
window field, because Gear is window-less. e.g. fastcdc,19,23,21,2
Also: borg benchmark cpu now measures the fastcdc chunker; tests in
borg.testsuite.chunkers (golden vector, size distribution, keyed gear table,
param parsing, slow fuzz); docs and changelog.
Benchmarks (scripts/chunker_bench.py, buzhash64 vs fastcdc, both nc_level=2,
incompressible data unless noted):
5 GiB, 2 MiB target (default params):
buzhash64: CV 0.294, 1011 MB/s
fastcdc: CV 0.295, 1313 MB/s (+30%)
64 MiB, 64 KiB target:
buzhash64: CV 0.374, shift-resilience 0.9928, 963 MB/s
fastcdc: CV 0.359, shift-resilience 0.9929, 1331 MB/s (+38%)
Re-backup of a 2.5 GiB file after scattered single-byte edits (dedup ratio,
0.5 = v2 fully deduplicated, lower is better):
64 edits: buzhash64 0.5237, fastcdc 0.5236
320 edits: buzhash64 0.6133, fastcdc 0.6161
borg benchmark cpu, 1 GB: fastcdc 3.80s, buzhash 4.36s, buzhash64 8.13s,
fixed 0.56s.
Chunk-size distribution, deduplication and shift-resilience match buzhash64
within noise; fastcdc is consistently faster.
Also: fix bug when computing the mask, one needs to use 1ULL instead of
1, so the shifting computation is done in a uint64, not in a 32bit int.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Normalized chunking switches between a stricter and a looser cut mask
around the target chunk size. This greatly tightens the chunk-size
distribution (coefficient of variation ~0.9 -> ~0.3 in tests) and removes
the dedup-hostile max-size-clamped chunks, with unchanged deduplication.
chunker-params for buzhash64 gains a required 6th field, nc_level:
buzhash64,chunk_min,chunk_max,chunk_mask,window_size,nc_level
Use nc_level=2 for the new default, nc_level=0 to disable (then behavior
is byte-identical to the previous single-mask chunker).
buzhash (32bit) is untouched and stays bit-compatible with borg 1.x.
The mask transition point (normal_size) defaults to a principled formula
(target minus the expected loose-phase tail) so the mean stays near the
target; it can be tuned via the normal_size constructor arg.
scripts/chunker_bench.py: evidence harness used to measure chunk-size
distribution, dedup ratio, throughput and shift-resilience.
Measurements (before = nc_level 0, after = nc_level 2; both at the default
params buzhash64,19,23,21,4095; measured with scripts/chunker_bench.py):
5 GiB of incompressible data (~2000-2700 chunks, statistically stable):
before: CV 0.739, 49 max-size-clamped (8 MiB) chunks, 953 MB/s
after: CV 0.311, 0 max-size-clamped chunks, 1024 MB/s
Re-backup of a 2.5 GiB file after a few scattered single-byte edits
(deduplication ratio; 0.5 = v2 fully deduplicated against v1, lower is
better):
64 edits: before 0.5424 -> after 0.5235
320 edits: before 0.6791 -> after 0.6142
Normalized chunking deduplicates better after edits: removing the
max-size-clamped chunks means a single-byte change invalidates much less
data (about 36% less dedup overhead at 320 edits). Throughput was also
consistently higher with nc_level=2 at this scale.
Also: fix bug when computing the mask, one needs to use 1ULL instead of
1, so the shifting computation is done in a uint64, not in a 32bit int.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The multi-key feature (#9743) added the "borg key add", "borg key list"
and "borg key remove" subcommands but never wired up their docs:
- scripts/make.py: map key_add/key_list/key_remove to the "key" usage
group, so build_man can locate their examples (it previously aborted
with FileNotFoundError: docs/usage/key_add.rst).
- docs/usage/key.rst: include the three new generated snippets so they
show up on the HTML key page.
- regenerate the affected docs: new key_{add,list,remove}.rst.inc usage
snippets, new borg-key-{add,list,remove}.1 man pages, and borg-key.1
(SYNOPSIS + SEE ALSO now reference the new subcommands).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Ports #9755 (1.4-maint) to master.
macOS flags files whose content lives in cloud storage (e.g. iCloud Drive)
and is not present locally with SF_DATALESS. Reading such a file triggers
downloading its content.
This adds --exclude-dataless to borg create: the flags are checked right
after stat() and before any open(), so excluded files/dirs never get
materialized. Skipped paths are reported with the usual 'x' status,
analogous to --exclude-nodump.
Notes:
- stat.SF_DATALESS only exists from Python 3.13 on, so there is a fallback
to the value from macOS' sys/stat.h (0x40000000).
- The check covers directories too (they can also be dataless), and we skip
before opening/recursing into them.
- Test fakes get_flags via monkeypatch since SF_DATALESS cannot be set from
userspace.
- Fish shell completion updated; bash/zsh completions don't exist in master.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
for packs, this needs to get implemented differently to perform well.
processing needs to be pack-after-pack and the index needs to be
updated correctly and carefully, e.g. considering interruptions
of repo-compress.
Remove the handwritten bash and zsh shell completion scripts now that
auto-generated completions via borg completion bash/zsh (powered by
shtab, #9172) are tested and working. Fish completions are kept as
shtab does not yet support fish.
Replace string-matching tests with focused behavior tests: script size
sanity, shell syntax validation (bash -n / zsh -n), and tests that
invoke the custom preamble functions in bash (sortby key dedup,
filescachemode mutual exclusivity, archive name and aid: prefix
completion against a real repository).
Moved the `ChunkerFixed` implementation from `chunker` to a new `fixed` module for better modularity. Updated imports and type hints.
Removed now empty chunkers.chunker module.
Moved `buzhash` implementation from `chunker` to a new `buzhash` module for better separation of concerns. Updated imports, adjusted `setup.py` and build configuration accordingly. Removed deprecated `Chunker` definitions from `chunker.pyi`.
Thomas Waldmann