With "--digests ALGOS", borg computes hash digests over the full content of each
regular file and stores them in the item's new "digests" dict (hash algorithm name
-> digest), e.g. --digests=blake3 gives {"blake3": ...} (32 bytes). ALGOS is a
comma-separated list of hash algorithm names - the same algorithms that are available
as "borg list" format keys - or "none". Default: none, digests are opt-in.
Unlike the chunk ids, such a digest depends neither on the chunker nor on borg's key,
so it is the same digest that e.g. "b3sum" computes for that file. "borg list --format
{blake3}" uses a stored digest if the item has one, instead of reading the file content
from the repository.
The content is hashed while it is read and processed anyway: the first chunk inline,
from the second (big enough) chunk on by a background thread. Our hash implementations
release the GIL, so the hashing can overlap with chunking, id hashing, compression and
encryption instead of adding to their runtime. ContentDigester.update() takes ownership
of the chunk data and releases it after hashing, so nothing is copied for this and the
memory is still freed timely.
Measured (12 cores, medians of 5 runs, relative to --digests=none, which is as fast as
borg without this):
1GiB new 1GiB known 20000 small files
--digests=blake3 free free +6..12%
--digests=blake3,sha256 free +10% +16%
--digests=blake3,sha256,sha512 +7% +60% +24%
One background thread hashes all requested algorithms one after the other, so their
combined throughput is the harmonic sum, while many small files are hashed inline by
the main thread. Hashing is free exactly while it is faster than the rest of the
pipeline - a single algorithm is, several may not be. That is why digests are opt-in.
Digests are not computed if borg does not read the full content: an unchanged file gets
them from the files cache and additional hard links from the first hard link, but
"borg create --reuse-from" items do not have digests. Files that already are in the
files cache without digests get them when they change (and thus are read again), so
changing --digests only affects the files that borg reads.
Add a --sparse option to export-tar (BORG and PAX tar formats only):
files whose content contains runs of all-zero chunks are written as
sparse tar members in GNU sparse format 1.0 (what GNU tar creates in
POSIX mode), storing only a hole map and the non-hole data.
The hole map is computed from the item's chunk list metadata alone,
before any data is fetched: an all-zero chunk is detected by comparing
its id against the (memoized) id of an all-zero chunk of the same size,
reusing the zeros shortcut heuristic of DownloadPipeline.fetch_many
(now extracted into a shared zero_chunk_flags function). Additionally,
the memo is warmed up for power-of-two chunk sizes and each file's last
chunk size, so single (non-repeated) zero chunks - like the only chunk
of an all-zero file or a trailing hole's tail - get detected, too.
The zero chunks of a sparse member are never fetched from the
repository at all.
Notes:
- python's tarfile module reads GNU sparse members (import-tar has
always consumed such tarballs correctly), but cannot write them, so
the writing side crafts the pax records and the map itself and
streams map + data segments via tarfile.addfile().
- holes are shrunk to whole 512-byte tar blocks within a zero run
(GNU tar's sparse reader processes data segments block-wise and
desyncs on unaligned segments); the zero bytes shaved off the hole
edges are emitted as literal zeros, still without fetching the
all-zero chunks. Zero runs shorter than a block stay dense, as do
members where the map would cost more than the holes save.
- pax record order matters for in-order readers: path (mangled
GNUSparseFile.0/<name>, like GNU tar) comes first, GNU.sparse.name /
GNU.sparse.realsize come last, so the real name/size win.
- no pax size record must be written for a sparse member (readers
would desync recalculating the next-header offset from it), so the
stored size always lives in the ustar size field: standard octal
while it fits (< 8 GiB), else base-256 encoded (the GNU/star
big-number encoding, which GNU tar, libarchive and python's tarfile
all read in any tar format) - so stored data and logical file size
(GNU.sparse.realsize) are effectively unlimited.
- without --sparse, the output is unchanged.
Co-Authored-By: Claude Fable 5 <[email protected]>
With --tar-filter=auto, .tar.zst/.tar.zstd/.tzst tarballs (.tzst newly
recognized) are now (de)compressed in-process using the zstd module
(Python >= 3.14 stdlib / backports.zstd) instead of piping through an
external zstd process. borg requires the zstd module anyway, libzstd
multithreading (BORG_ZSTD_MT_WORKERS) runs outside the GIL, and no
external tool needs to be installed.
Compression uses level 3 (the zstd tool's default). An explicit
--tar-filter program is still run as an external filter, also for zstd.
Co-Authored-By: Claude Fable 5 <[email protected]>
fastcdc is ~1.3x faster than buzhash/buzhash64 at the same deduplication
and (with normalized chunking) a tighter chunk size distribution.
It is also the better choice security-wise: its Gear table is derived
from secret key material, while the "buzhash" chunker only XORs a 32bit
seed into its table - so chunk cut points are much harder to predict
without the key (resistance against chunk-size fingerprinting attacks).
Both places using a chunker are switched:
- CHUNKER_PARAMS (file content data): fastcdc,19,23,21,2
- ITEMS_CHUNKER_PARAMS (item metadata stream): fastcdc,15,19,17,2
The metadata stream chunker was still using the 32bit seeded buzhash, so
this also closes that gap.
Also rename CHUNKER64_PARAMS -> BUZHASH64_PARAMS and add BUZHASH_PARAMS,
so the per-algorithm defaults are named consistently and CHUNKER_PARAMS
unambiguously means "the default chunker".
This is intentionally done during the beta phase so it gets plenty of
practical testing before borg2 is released for production.
Thomas Waldmann