blake3 can hash a single input in parallel, but borg always used the
single-threaded path. Let id_hash use blake3's internal thread pool once the
input is big enough.
Multi-threading is not a win at every size: blake3 parallelises over its binary
hash tree, so the speedup peaks at powers of two and decays in between, and for
small inputs the thread dispatch costs much more than it saves. Measured on a
12-core machine (Apple M3 Pro): 0.16x at 8 KiB, still below 1x at 240 KiB, then
2.0x at 256 KiB, 3.2x at 512 KiB, 5.6x at 2 MiB and 7.3x at 8 MiB.
256 KiB was the smallest size that never lost there, so that is the default and
smaller chunks keep hashing single-threaded. Chunk sizes are content-defined, so
we can not rely on hitting the sizes that happen to parallelise well and have to
be conservative.
The optimal value depends on the core count and therefore differs per machine,
so BORG_BLAKE3_MT_THRESHOLD (in KiB) can override the default. It is evaluated
on first use and then cached: get_blake3_mt_threshold() runs for every chunk and
os.environ.get() costs ~425ns (~30% of id_hash for a 512B chunk), while the
cached call costs ~45ns. Evaluating it lazily rather than at import time also
means an invalid value is reported by borg's normal error handling instead of a
traceback while importing.
scripts/blake3-optimize-mt-threshold.py measures the best value for a machine:
it sweeps input sizes, verifies a candidate by probing the sizes that are
usually worst, re-measures the point that determines the threshold to keep
scheduling noise out, and optionally writes an HTML or SVG chart.
This only affects repos using --id-hash blake3. The resulting chunk ids are
unchanged - that single- and multi-threaded blake3 agree is the blake3
project's invariant, so there is no test for it here.
Co-Authored-By: Claude Opus 5 <[email protected]>
