mirror of
https://github.com/borgbackup/borg.git
synced 2026-10-12 02:11:36 +00:00
Thomas Waldmann
b69023ce45
With "--digests ALGOS", borg computes hash digests over the full content of each
regular file and stores them in the item's new "digests" dict (hash algorithm name
-> digest), e.g. --digests=blake3 gives {"blake3": ...} (32 bytes). ALGOS is a
comma-separated list of hash algorithm names - the same algorithms that are available
as "borg list" format keys - or "none". Default: none, digests are opt-in.
Unlike the chunk ids, such a digest depends neither on the chunker nor on borg's key,
so it is the same digest that e.g. "b3sum" computes for that file. "borg list --format
{blake3}" uses a stored digest if the item has one, instead of reading the file content
from the repository.
The content is hashed while it is read and processed anyway: the first chunk inline,
from the second (big enough) chunk on by a background thread. Our hash implementations
release the GIL, so the hashing can overlap with chunking, id hashing, compression and
encryption instead of adding to their runtime. ContentDigester.update() takes ownership
of the chunk data and releases it after hashing, so nothing is copied for this and the
memory is still freed timely.
Measured (12 cores, medians of 5 runs, relative to --digests=none, which is as fast as
borg without this):
1GiB new 1GiB known 20000 small files
--digests=blake3 free free +6..12%
--digests=blake3,sha256 free +10% +16%
--digests=blake3,sha256,sha512 +7% +60% +24%
One background thread hashes all requested algorithms one after the other, so their
combined throughput is the harmonic sum, while many small files are hashed inline by
the main thread. Hashing is free exactly while it is faster than the rest of the
pipeline - a single algorithm is, several may not be. That is why digests are opt-in.
Digests are not computed if borg does not read the full content: an unchanged file gets
them from the files cache and additional hard links from the first hard link, but
"borg create --reuse-from" items do not have digests. Files that already are in the
files cache without digests get them when they change (and thus are read again), so
changing --digests only affects the files that borg reads.