The logo green (#00dd00) is made for a dark background; on white it only
reaches a contrast of 1.9:1, so the light scheme uses a darker green of the
same hue.
git-fame's own --format=svg renders the whole table as monospaced text,
which works for a handful of contributors but not for a few hundred, so
draw a bar chart of the top 20 (--svg-top) instead. FAME.md embeds it.
The chart follows the reader's colour scheme, sizes its viewport in user
units plus a viewBox (em on the root <svg> would resolve against the wrong
font size) and escapes author names, which are commit-controlled input.
Bots keep their row in the table, but do not take a chart slot away from a
person.
The workflow now also runs when FAME.md changes on master, so that the two
files cannot drift apart.
The workflow runs scripts/fame.py against master every Monday and opens a
pull request if the contributor statistics changed.
For that to be quiet in weeks where nothing changed, fame.py now leaves
FAME.md alone when only the generation date would differ.
Keeps the jokes from borgbackup/borg#10000 in the repo, next to the SVG
that illustrates them: a Borg cube tractor-beams in 14 files and emits a
chunk store holding 4 unique chunks with their reference counts.
"Redundancy is futile, your files will be deduplicated!"
The "WE ARE BORG" headline is set in the borg logo typeface (Black Ops One,
SIL OFL 1.1) and stored as outlines, so the SVG stays self-contained.
Co-Authored-By: Claude Opus 5 <[email protected]>
reimplemented for the pack-based repository format: the repository is
processed pack after pack - each pack is read as a whole and, if it has
objects not stored with the desired compression, those get recompressed
and the pack is rewritten (objects already fine are copied into the new
pack unchanged), the chunk index is updated and the old pack is deleted.
a pack whose objects all already match is not touched at all.
- new Repository.transform_pack: rewrite one pack, passing each indexed
object's bytes through a transform callback. gap bytes are handled
like in compact_pack: superseded duplicates are dropped, other
unindexed bytes are carried forward for "borg check --repair".
- factored the check_pack_objects / superseded_gap_ranges helpers out
of compact_pack, now shared with merge_packs and transform_pack.
- crash safety like compact (#9748): stored chunk indexes are
invalidated before the first store change, a full updated index is
written back at the end. SIGINT stops cleanly at a pack boundary,
saving a valid index; a later run recompresses the remaining packs.
- --stats reports rewritten packs, per-object outcomes (recompressed /
already had the desired compression / kept as-is because
recompression brings no gain) and the repository size change.
- re-adding the fish completions also restores the compression_methods
definition that the create/transfer/recreate/import-tar completions
kept referencing after it was removed together with the old
repo-compress.
Co-Authored-By: Claude Fable 5 <[email protected]>
fastcdc is ~1.3x faster than buzhash/buzhash64 at the same deduplication
and (with normalized chunking) a tighter chunk size distribution.
It is also the better choice security-wise: its Gear table is derived
from secret key material, while the "buzhash" chunker only XORs a 32bit
seed into its table - so chunk cut points are much harder to predict
without the key (resistance against chunk-size fingerprinting attacks).
Both places using a chunker are switched:
- CHUNKER_PARAMS (file content data): fastcdc,19,23,21,2
- ITEMS_CHUNKER_PARAMS (item metadata stream): fastcdc,15,19,17,2
The metadata stream chunker was still using the 32bit seeded buzhash, so
this also closes that gap.
Also rename CHUNKER64_PARAMS -> BUZHASH64_PARAMS and add BUZHASH_PARAMS,
so the per-algorithm defaults are named consistently and CHUNKER_PARAMS
unambiguously means "the default chunker".
This is intentionally done during the beta phase so it gets plenty of
practical testing before borg2 is released for production.
All non-final callers used the same BORG_PROGRESS_FPS based dt, and the
final calls must always be shown - so apply the rate limit inside
show_progress (bypassed by final=True) instead of passing dt around.
Also initialize last_progress to -inf, so the first update is always
shown regardless of the platform's monotonic clock epoch.
Co-Authored-By: Claude Fable 5 <[email protected]>
git blame / git shortlog listed a number of contributors twice, either because
the same address was used with different spellings of the name, or because the
same person committed from several addresses. Merge them in .mailmap: this
brings the number of distinct authors down from 380 to 370.
Note that two of the merges are judgement calls, see the diff for the entries
in question: one address was used with two different real names and a handle
(the most recent non-handle name is used), and another was used mostly with a
handle and a few times with a real name (the real name is used).
Add scripts/fame.py, which writes FAME.md from git-fame output, sorted by
commit count. It bakes in an exclusion pattern for generated files - without
it, the three Excalidraw sources of the pack format figures contribute 39230
lines (a third of the repository) to a single contributor, because they are
pretty-printed JSON.
Co-Authored-By: Claude Opus 5 <[email protected]>
Like RESTIC_PROGRESS_FPS: maximum progress updates per second, default 5
(the traditional 0.2s interval). Fractional values below 1 are allowed,
e.g. BORG_PROGRESS_FPS=0.1 for one update every 10 seconds - useful when
the output goes into a logfile rather than to an interactive terminal.
Co-Authored-By: Claude Fable 5 <[email protected]>
For the AEAD encryption modes, the chunk id is part of the AAD, so a successful
decryption already proves that a repo key holder deliberately stored this exact
ciphertext for this exact chunk id: a malicious repository can not swap, splice
or substitute objects, whether we recompute the id hash over the plaintext or
not. What assert_id adds there is only the detection of chunks whose content
does not match their id, which only a malicious/compromised borg client that
had the borg key could have written (see #7362, which made the check mandatory).
Whether that extra full-plaintext hash pass is worth it depends on where borg
reads, so the new BORG_ASSERT_ID env var takes a comma-separated list of the
places that shall verify the chunk id:
read the general read path: extract, mount, export-tar, diff, ...
repair borg check --repair
transfer borg transfer (everything read from the source repo)
rechunk borg recreate --chunker-params (re-chunking reads)
Default (env var not set): "repair,transfer,rechunk", i.e. every place that
re-anchors content and would thus make a violation unnoticeable afterwards, but
not the hot read path. Unknown names are an error.
BORG_ASSERT_ID=read,repair,transfer,rechunk gives the old behaviour of always
verifying.
The id is verified no matter what the env var says:
- in borg check --verify-data - the audit that re-certifies the id/content
invariant for all chunks of the repository is what makes not verifying
elsewhere defensible, so it is not configurable (there is no "verify_data"
place name, giving one is an error).
- for "authenticated" and "none" mode repos and for borg 1.x repo objects:
there is no AEAD there, so assert_id IS the read path authentication
(KeyBase.id_check_is_authentication) and skipping it would demote
"authenticated" to "none".
Measured on an Apple M3 Pro (18GB RAM, hw accelerated sha256), reading one 20GiB
incompressible file back via "borg export-tar arch - > /dev/null", aes256-ocb,
--compression none, 4 runs alternating between the settings:
id hash BORG_ASSERT_ID run times [s] median throughput
sha256 default 20.43 21.13 20.79 20.93 20.86 982 MiB/s
sha256 ...,read 29.42 28.91 28.82 28.05* 28.87 709 MiB/s
blake3 default 19.99 19.97 20.00 19.93 19.98 1025 MiB/s
blake3 ...,read 24.70 24.95* 24.72 24.60 24.71 829 MiB/s
*) repeated run: the original one (31.64s resp. 28.37s) was disturbed by
unrelated background load on the test machine.
i.e. 28% faster for sha256 id hashes and 19% faster for blake3 ones (the file is
much bigger than the RAM, so this includes reading it from the SSD every time).
With the default, running borg check --verify-data periodically is the
recommended audit that re-certifies the id/content invariant for all chunks.
Co-Authored-By: Claude Opus 5 <[email protected]>
An unknown format key (or a malformed format string) made borg crash
with a KeyError/ValueError traceback, e.g.:
borg list -a aid:1234 --format '{nosuchkey}'
KeyError: 'nosuchkey'
BaseFormatter.validate_format existed, but was only used by borg check
and it did not know about keys that are only in KEY_GROUPS (like the
hash keys of ItemFormatter), so it could not be used as-is elsewhere.
- add BaseFormatter.known_keys() (KEY_DESCRIPTIONS + KEY_GROUPS + FIXED_KEYS)
and validate against that
- validate in BaseFormatter.__init__, so all formatters / commands
(list, repo-list, diff, prune, check) give a CommandError now
- borg list: validate early, format_needs_cache parses the format before
the ItemFormatter gets built
Co-Authored-By: Claude Opus 5 <[email protected]>
borg2 uses blake3 (as keyed hash) for chunk ids, but the --format
hash keys only offered the python stdlib hashes, so {blake3} raised
a KeyError.
The blake3 package is a hard dependency of borg2 anyway, so just add
blake3 to the set of supported hash algorithms. Output is the default
256bit hexdigest, like for the other hash keys.
Co-Authored-By: Claude Opus 5 <[email protected]>
borg.crypto.low_level has no AES class - the legacy AES wrapper (only used for
legacy key file encryption) lives in borg.legacy.crypto.low_level, which had no
type stub at all. So move the stub entry where the class actually is.
A truncated envelope (shorter than header + auth tag, for the legacy AES-CTR
modes: header + mac + iv) previously ended up calling EVP_DecryptUpdate with a
negative length, which OpenSSL luckily rejects, resulting in
CryptoError('EVP_DecryptUpdate failed').
But CryptoError means "malfunction in the crypto module" and no caller catches
it, so a truncated chunk (e.g. 32..47 bytes for the AEAD envelope layout) gave
an ugly crash instead of being handled like what it is: corrupted or tampered
data. Now we check the envelope length first and raise IntegrityError, which
the upper layers already handle properly.
Also: requirements_check() raised the NotImplemented constant (a TypeError at
runtime) instead of NotImplementedError, and the _AEAD_BASE key param docstring
called the AEAD key an "encrypt-then-mac key" - AEAD modes are exactly what
replaced borg 1.x's encrypt-then-MAC construction, they take a single AEAD key.
CryptoTestCase is part of the startup self test, so SELFTEST_COUNT needs to be
updated for the 2 test methods added here (otherwise every borg process aborts
with "Self-test count mismatch").
None of the ciphersuite classes in this module has next_iv(len(data)) - both the
AEAD and the legacy AES-CTR classes compute the next iv without arguments.
Also, header_len and aad_offset are constructor arguments, not encrypt/decrypt
arguments, and encrypt takes iv and aad.
"AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit IV we
provide" is only true for chacha20-poly1305: the ChaCha20 block function has a
32bit block counter besides the 96bit nonce, limiting a message to 2^32 blocks
(256GiB) per (key, nonce). AES-OCB has no such counter, it derives the per-block
offsets from the nonce (RFC 7253).
The conclusions drawn from that claim were fine for both ciphers, so this is a
comment/docs fix only:
- the 2^32 blocks per message check stays as it is - it is required for chacha
and just conservative for OCB (it can not trigger anyway, our messages are
limited to MAX_DATA_SIZE == 20MiB).
- incrementing the IV by 1 per message stays correct for both ciphers, because
the cipher blocks of a message do not consume IVs.
Also reworded the ValueError message, which claimed a counter overflow.
The forgery attempts (v) limit is the only AEAD limit borg does not enforce
(chacha20-poly1305: about 2^33 for our biggest messages at p 2^-50, AES-OCB with
its 128bit auth tag is far less restrictive). Document that this is intentional:
- a failed decryption means tampered or corrupted data, borg refuses it and
usually aborts the whole command (borg check and archive listing keep going,
but only to report the damage).
- reaching the limit would need way more tampered data than a real repository
will ever hold.
Also note it at the place where the failed decryptions actually happen.
The "48 bit IV is way more than needed" paragraph justified the IV size with the
amount of data encrypted in one session, but the IV only limits the number of
**messages** (2^48, borg refuses to encrypt more). How much **data** we may
encrypt with one session key is determined by the security bounds of the ciphers.
So now we state all 3 relevant quantities and the target probability we aim for:
- number of messages (q): 2^48 per session key, never the binding limit.
- data volume: AES-OCB only, 2^37 blocks (2TiB) per session key == p 2^-51
(RFC 7253's 4PiB rule of thumb corresponds to p 2^-32). chacha20-poly1305
does not have such a limit.
- forgery attempts (v): chacha20-poly1305, about 2^33 at p 2^-50 for our biggest
messages, counted over all session keys (so more sessions do not help there).
Also mention the session key change in the in-depth crypto docs (security.rst).
AES-OCB has a birthday-type security bound in the amount of data encrypted
using one key: the attacker's advantage is about 6 * sigma^2 / 2^128, sigma
being the number of 128bit cipher blocks (Krovetz/Rogaway OCB3, Theorem 1).
RFC 7253 derives its "at most 2^48 blocks (4PiB) per key" rule of thumb from
that bound, which corresponds to an advantage of 2^-32.
We now aim higher and start a new session after 2^37 blocks (2TiB), giving an
advantage of about 2^-51, in line with the target probability used in the
examples of draft-irtf-cfrg-aead-limits. Starting a new session is cheap (one
sha256 for the new session key) and fully transparent, because the session id
is part of every chunk header - so old chunks stay decryptable and nothing
changes for reading existing repositories.
Rekeying that often also is what helps in the multi-key setting: the advantages
of the individual session keys just add up, so the quantity that matters over
the lifetime of a borg key is sum(sigma_i^2) and not (sum sigma_i)^2.
chacha20-poly1305 does not need such a limit: its confidentiality bound does not
depend on the amount of data encrypted at all and its integrity bound only limits
the number of forgery attempts, counted over all keys.
Both generate broken zsh completions: shtab replaced escape_zsh() with
shlex.quote() for the zsh help text, but still interpolates the result inside
a double-quoted _arguments spec. shlex.quote returns a self-contained shell
word (single-quoted, with ' written as '"'"'), which is only valid at top
level, so the quote parity flips and the rest of the spec goes unquoted.
Every description with a space gained stray literal quotes, and an apostrophe
anywhere left an unterminated quote that cascaded to shtab's own
default='*::: :->{name}', whose -> then parsed as a redirection:
$ borg completion zsh | zsh -n
(stdin):715: parse error near `>'
Our --sort-by help for `borg diff` contains '>size_added,path', which is what
tripped test_zsh_completion_syntax. Reported as tqdm/shtab#224, fix proposed
in tqdm/shtab#225; drop the exclusions once a release carries it.
Only CI jobs with a cold .tox cache saw this (macOS 15 first), since shtab was
unpinned above 1.8.0 -- cached envs kept the working 1.8.1.
Fixes#9982
release chunk data memoryviews after use, see #1755
On pypy, a memoryview over a C-API-created bytes object pins the
buffer via cpyext: if the consumer drops the memoryview without
calling .release(), the memory is never reclaimed - not even by
gc.collect(). Since the chunkers wrap every yielded chunk in a
memoryview, borg create leaked roughly 2-3x the processed data
volume on pypy (a 20 GB create grew to a 43 GB memory footprint
and got killed by the OS).
Add release_chunk_data() to chunkers and call it at all places
that consume chunker output: ChunkBuffer.flush,
ChunksProcessor.process_file_chunks, ArchiveRecreater.chunk_processor,
transfer --rechunk and the chunker benchmark. Explicitly releasing
also frees the buffer timely on CPython instead of relying on
refcounting of the last reference. Document the consumer
obligation in the Chunk docstring.
Memory now stays flat for large creates on pypy (~160 MB for a
multi-GB create instead of linear growth).
Also fix a pre-existing AttributeError in transfer --rechunk
--dry-run: Chunk namedtuples have no .size attribute, all-zero
chunks need chunk.meta["size"].
Co-Authored-By: Claude Fable 5 <[email protected]>
BORG_UNITS determines how borg formats sizes in human-readable output:
- si (default): decimal units, e.g. 1.23 MB (1kB = 1000B)
- iec: binary units, e.g. 1.18 MiB (1KiB = 1024B)
- raw: exact byte counts, e.g. 1234567 B
raw is meant for scripts (e.g. monitoring) that want to parse sizes without
having to deal with scaled values and units. An invalid BORG_UNITS value is
warned about (once) and ignored.
BORG_IEC was removed, use BORG_UNITS=iec.
format_file_size now determines the units itself (via BORG_UNITS), so the iec
argument plumbing through Archive / Statistics / ArchiveFormatter / Cache /
ArchiveChecker / ArchiveGarbageCollector and use_iec_units() are all gone.
Beside being less code, this also makes the setting effective at the call
sites that never passed iec=use_iec_units(), e.g. borg repo-space.
The holes of a sparse file produce long runs of references to the same
all-zero chunk. Instead of fetching that chunk from the repository over
and over, detect it and serve it directly from the zeros constant.
Detection compares the chunk id against the id of an all-zero chunk of
the same size, memoized in the already existing zero_chunk_ids mapping
(now shared with the create side via the new zero_chunk_id() function).
To bound the memoized id computations to a few chunk sizes, they are
only done for ids occurring repeatedly within the requested stream -
a repeated id means repeating plaintext, which usually is a run of
zeros. Unique ids are only compared against already memoized sizes.
Extracting a file with a 256 MiB hole now does not access the
repository at all for the hole (before: 63 object fetches).
Co-Authored-By: Claude Fable 5 <[email protected]>
A content data stream may reference the same chunk many times (e.g. the
all-zero chunks of a sparse file converge on the same ids). Previously,
every repetition was decrypted, authenticated and decompressed again.
Add a small LRU cache of recently parsed chunks to DownloadPipeline,
keyed on (id, ro_type), so repeated chunks are served from the cache.
The repository is still asked for every occurrence (a cheap cached-pack
slice for local repos, and it keeps the legacy remote repos' pipelining
in borg transfer intact), only the parsing work is skipped.
Extracting a file with a 256 MiB hole now parses the all-zero chunk
once instead of 63 times.
Co-Authored-By: Claude Fable 5 <[email protected]>
borg info no longer computes per-archive deduplicated sizes (Archive.calc_stats
sets usize=0 as it is expensive), and the output only shows the original size.
The epilog still explained "this archive vs all archives deduplicated size",
which no longer matches what the command prints.
Replace it with a note that deduplicated sizes are not shown here and point to
`borg analyze` (deduplicated size of a set of archives) and `borg compact --stats`
(repository-wide deduplicated size).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
`borg analyze` now reports how much space a set of archives occupies and how much
deleting it would actually free. All figures come as source (uncompressed source
data) and stored (compressed, as stored in the repository) size, plus the
compression factor relating the two.
For the considered (matching) set of archives:
- deduplicated size of the set: the summed size of the union of chunks the set
references (chunks shared within the set counted once),
- exclusive size of the set: the summed size of chunks referenced only by the set
and by no other archive, i.e. what deleting the whole set would free,
- unreferenced chunks: chunks no non-deleted archive references, i.e. what
`borg compact` could free in the current state of the repository.
Following ThomasWaldmann's proposal in #5741, chunks are flagged as referenced by
the considered set (C) and/or by the rest of the archives (R); the exclusive size
is sum(size(x) for x in C - R). Without an archive filter the considered set is all
archives, so everything referenced is trivially exclusive to it - the report is then
labelled as covering the whole repository and that line is left out.
`--by-name` decomposes the whole repository by archive name instead: one row per
name with what is exclusive to it, one row for the chunks shared by 2+ names and one
for the unreferenced ones. Every chunk is counted in exactly one row, so the rows
add up to the repository's deduplicated size. Archives sharing a name form a series,
so a name usually groups all backups of one source; old-style archives that do not
form a series have one name each and group just as well. This needs a single pass:
flag bits 5..22 hold the name that first referenced a chunk (as index + 1, so 0 means
unreferenced) and bit 23 marks chunks referenced by more than one name. As the shared
and unreferenced rows require looking at every archive, --by-name always covers the
whole repository and rejects archive filters.
Rather than building a second index, this reuses the repository's own chunk index:
it already holds every chunk's stored size (obj_size). The flag bits are OR-ed in and
the source size (0 in the repo index, it is only recorded in the archives referencing
a chunk) is filled in from the per-archive references cache, keeping the pack location
intact via _replace. The mutations never persist: write_chunkindex_to_repo zeroes flags
and size, and close() only serializes F_NEW entries, of which there are none here.
Chunk membership and source sizes come from the per-archive references cache that
`borg compact` maintains, so unchanged archives usually need not be opened at all.
Its helpers (get_archive_references and friends, ArchiveReferences) move from
archiver/compact_cmd.py to cache.py as free functions, so analyze and compact share
one implementation; compact keeps its previous behaviour.
The pre-existing directory hot-spot report is unchanged and still needs 2+ archives.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
ThomasWaldmann