borgbackup/borg
1
0
Fork 0
mirror of https://github.com/borgbackup/borg.git synced 2026-10-12 02:11:36 +00:00
Code Issues Packages Projects Releases Wiki Activity

make fastcdc the default chunker, fixes #9957

Browse Source 
fastcdc is ~1.3x faster than buzhash/buzhash64 at the same deduplication
and (with normalized chunking) a tighter chunk size distribution.

It is also the better choice security-wise: its Gear table is derived
from secret key material, while the "buzhash" chunker only XORs a 32bit
seed into its table - so chunk cut points are much harder to predict
without the key (resistance against chunk-size fingerprinting attacks).

Both places using a chunker are switched:

- CHUNKER_PARAMS (file content data): fastcdc,19,23,21,2
- ITEMS_CHUNKER_PARAMS (item metadata stream): fastcdc,15,19,17,2

The metadata stream chunker was still using the 32bit seeded buzhash, so
this also closes that gap.

Also rename CHUNKER64_PARAMS -> BUZHASH64_PARAMS and add BUZHASH_PARAMS,
so the per-algorithm defaults are named consistently and CHUNKER_PARAMS
unambiguously means "the default chunker".

This is intentionally done during the beta phase so it gets plenty of
practical testing before borg2 is released for production.
This commit is contained in:
Thomas Waldmann
2026-08-02 00:40:58 +02:00
parent 7873273481
commit 61b1b83059
22 changed files with 72 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/changes.rst
+5
View File
@@ -244,6 +244,11 @@ New features:
- Chunkers:
- fastcdc: new chunker (keyed Gear hash, normalized chunking, ~1.3x faster), #9824
- fastcdc is the new default chunker, for file content data as well as for the
item metadata stream, #9957. Compared to the previous default "buzhash", it is
faster and its Gear table is derived from secret key material (instead of only
XORing a 32bit seed into the table), so chunk cut points are much harder to
predict without the key (better resistance against fingerprinting attacks).
- buzhash64: add normalized chunking (better chunk size distribution, less clamping)
- borg keys:
docs/faq.rst
+1 -1
View File
@@ -834,7 +834,7 @@ Compared to simply copying files (e.g. with ``rsync``), Borg has more work to do
This can make creation of the first archive slower, but saves time
and disk space on subsequent runs. Here what Borg does when you run ``borg create``:
- Borg chunks the file (using the relatively expensive buzhash algorithm)
- Borg chunks the file (using content-defined chunking, which needs to look at every byte)
- It then computes the "id" of the chunk (hmac-sha256 (slow, except
if your CPU has sha256 acceleration) or blake2b (fast, in software))
- Then it checks whether this chunk is already in the repo (local hashtable lookup,
docs/internals.rst
+4 -3
View File
@@ -18,9 +18,10 @@ specified when the backup was performed.
Deduplication is performed globally across all data in the repository
(multiple backups and even multiple hosts), both on data and file
metadata, using :ref:`chunks` created by the chunker using the
Buzhash_ algorithm ("buzhash" and "buzhash64" chunker) or a simpler
fixed block size algorithm ("fixed" chunker).
metadata, using :ref:`chunks` created by the chunker using a
content-defined chunking algorithm - the Gear rolling hash of FastCDC_
("fastcdc" chunker, the default) or Buzhash_ ("buzhash" and "buzhash64"
chunker) - or a simpler fixed block size algorithm ("fixed" chunker).
To perform the repository-wide deduplication, a hash of each
chunk is checked against the :ref:`chunks index <index>`, which is a
docs/internals/data-structures.rst
+14 -4
View File
@@ -399,7 +399,7 @@ A chunk is stored as an object as well, of course.
Chunks
~~~~~~
Borg has these chunkers:
Borg has these chunkers (the default is "fastcdc"):
- "fixed": a simple, low cpu overhead, fixed blocksize chunker, optionally
supporting a header block of different size.
@@ -466,7 +466,7 @@ which narrows down where cuts may be made, greatly reducing the amount of data
that is actually hashed for content-defined chunking.
``borg create --chunker-params buzhash,CHUNK_MIN_EXP,CHUNK_MAX_EXP,HASH_MASK_BITS,HASH_WINDOW_SIZE``
can be used to tune the chunker parameters, the default is:
can be used to tune the chunker parameters, the usual values are:
- CHUNK_MIN_EXP = 19 (minimum chunk size = 2^19 B = 512 kiB)
- CHUNK_MAX_EXP = 23 (maximum chunk size = 2^23 B = 8 MiB)
@@ -500,9 +500,19 @@ Like "buzhash64", the Gear table is cryptographically derived from secret key
material, so chunk cut points are unpredictable without the key.
``borg create --chunker-params fastcdc,CHUNK_MIN_EXP,CHUNK_MAX_EXP,HASH_MASK_BITS,NC_LEVEL``
can be used to tune the chunker parameters, the default is:
There is no window size (Gear is window-less). NC_LEVEL is the normalized
chunking level (0 disables it); 2 is a good default. E.g.: ``fastcdc,19,23,21,2``.
- CHUNK_MIN_EXP = 19 (minimum chunk size = 2^19 B = 512 kiB)
- CHUNK_MAX_EXP = 23 (maximum chunk size = 2^23 B = 8 MiB)
- HASH_MASK_BITS = 21 (target chunk size ~= 2^21 B = 2 MiB)
- NC_LEVEL = 2 (normalized chunking level, 0 disables it)
There is no window size (Gear is window-less). Normalized chunking varies the
cut-point mask around the target size, which tightens the chunk-size
distribution and reduces clamping at the min./max. chunk size.
This is the default chunker (``fastcdc,19,23,21,2``), also used for the item
metadata stream (with a finer granularity, ``fastcdc,15,19,17,2``).
.. _cache:
docs/usage/create.rst
+1 -1
View File
@@ -53,7 +53,7 @@ Examples
# Make a big effort in fine-grained deduplication (big chunk management
# overhead, needs a lot of RAM and disk space; see the formula in the internals docs):
$ borg create --chunker-params buzhash,10,23,16,4095 small /smallstuff
$ borg create --chunker-params fastcdc,10,23,16,2 small /smallstuff
# Backup a raw device (must not be active/in use/mounted at that time)
$ borg create --read-special --chunker-params fixed,4194304 my-sdx /dev/sdX
docs/usage/create.rst.inc
+2 -2
View File
@@ -103,7 +103,7 @@ borg create
+-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| | ``--timestamp TIMESTAMP`` | manually specify the archive creation date/time (yyyy-mm-ddThh:mm:ss[(+|-)HH:MM] format, (+|-)HH:MM is the UTC offset, default: local time zone). Alternatively, give a reference file/directory. |
+-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| | ``--chunker-params PARAMS`` | specify the chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE). default: buzhash,19,23,21,4095 |
| | ``--chunker-params PARAMS`` | specify the chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, NC_LEVEL). default: fastcdc,19,23,21,2 |
+-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| | ``-C COMPRESSION``, ``--compression COMPRESSION`` | select compression algorithm, see the output of the "borg help compression" command for details. |
+-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
@@ -175,7 +175,7 @@ borg create
Archive options
--comment COMMENT add a comment text to the archive
--timestamp TIMESTAMP manually specify the archive creation date/time (yyyy-mm-ddThh:mm:ss[(+|-)HH:MM] format, (+|-)HH:MM is the UTC offset, default: local time zone). Alternatively, give a reference file/directory.
--chunker-params PARAMS specify the chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE). default: buzhash,19,23,21,4095
--chunker-params PARAMS specify the chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, NC_LEVEL). default: fastcdc,19,23,21,2
-C COMPRESSION, --compression COMPRESSION select compression algorithm, see the output of the "borg help compression" command for details.
--tags TAG add tags to archive (comma-separated or multiple arguments)
docs/usage/import-tar.rst.inc
+2 -2
View File
@@ -43,7 +43,7 @@ borg import-tar
+-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| | ``--timestamp TIMESTAMP`` | manually specify the archive creation date/time (yyyy-mm-ddThh:mm:ss[(+|-)HH:MM] format, (+|-)HH:MM is the UTC offset, default: local time zone). Alternatively, give a reference file/directory. |
+-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| | ``--chunker-params PARAMS`` | specify the chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE). default: buzhash,19,23,21,4095 |
| | ``--chunker-params PARAMS`` | specify the chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, NC_LEVEL). default: fastcdc,19,23,21,2 |
+-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| | ``-C COMPRESSION``, ``--compression COMPRESSION`` | select compression algorithm, see the output of the "borg help compression" command for details. |
+-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
@@ -79,7 +79,7 @@ borg import-tar
Archive options
--comment COMMENT add a comment text to the archive
--timestamp TIMESTAMP manually specify the archive creation date/time (yyyy-mm-ddThh:mm:ss[(+|-)HH:MM] format, (+|-)HH:MM is the UTC offset, default: local time zone). Alternatively, give a reference file/directory.
--chunker-params PARAMS specify the chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE). default: buzhash,19,23,21,4095
--chunker-params PARAMS specify the chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, NC_LEVEL). default: fastcdc,19,23,21,2
-C COMPRESSION, --compression COMPRESSION select compression algorithm, see the output of the "borg help compression" command for details.
docs/usage/notes.rst
+2 -2
View File
@@ -14,12 +14,12 @@ resource usage (RAM and disk space) as the amount of resources needed is
(also) determined by the total number of chunks in the repository (see
:ref:`cache-memory-usage` for details).
``--chunker-params=buzhash,10,23,16,4095`` results in a fine-grained deduplication|
``--chunker-params=fastcdc,10,23,16,2`` results in a fine-grained deduplication|
and creates a large number of chunks and thus uses a lot of resources to manage
them. This is good for relatively small data volumes and if the machine has a
good amount of free RAM and disk space.
``--chunker-params=buzhash,19,23,21,4095`` (default) results in a coarse-grained
``--chunker-params=fastcdc,19,23,21,2`` (default) results in a coarse-grained
deduplication and creates a much smaller number of chunks and thus uses less
resources. This is good for relatively big data volumes and if the machine has
a relatively low amount of free RAM and disk space.
docs/usage/recreate.rst.inc
+2 -2
View File
@@ -73,7 +73,7 @@ borg recreate
+-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| | ``-C COMPRESSION``, ``--compression COMPRESSION`` | select compression algorithm, see the output of the "borg help compression" command for details. |
+-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| | ``--chunker-params PARAMS`` | rechunk using given chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE) or `default` to use the chunker defaults. default: do not rechunk |
| | ``--chunker-params PARAMS`` | rechunk using given chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, NC_LEVEL) or `default` to use the chunker defaults. default: do not rechunk |
+-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
.. raw:: html
@@ -123,7 +123,7 @@ borg recreate
--comment COMMENT add a comment text to the archive
--timestamp TIMESTAMP manually specify the archive creation date/time (yyyy-mm-ddThh:mm:ss[(+|-)HH:MM] format, (+|-)HH:MM is the UTC offset, default: local time zone). Alternatively, give a reference file/directory.
-C COMPRESSION, --compression COMPRESSION select compression algorithm, see the output of the "borg help compression" command for details.
--chunker-params PARAMS rechunk using given chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE) or `default` to use the chunker defaults. default: do not rechunk
--chunker-params PARAMS rechunk using given chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, NC_LEVEL) or `default` to use the chunker defaults. default: do not rechunk
Description
docs/usage/transfer.rst
+1 -1
View File
@@ -55,7 +55,7 @@ locations and passphrases first:
# The AEAD cipher does not matter (everything must be re-encrypted and
# re-authenticated anyway); you could also choose -e chacha20-poly1305 -i blake3.
$ borg repo-create -e aes256-ocb -i blake3
$ export CHUNKER_PARAMS="buzhash64,19,23,21,4095,2"
$ export CHUNKER_PARAMS="fastcdc,19,23,21,2"
# 2. Check what and how much it would transfer:
$ borg transfer --from-borg1 --chunker-params=$CHUNKER_PARAMS --dry-run
docs/usage/transfer.rst.inc
+2 -2
View File
@@ -27,7 +27,7 @@ borg transfer
+-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| | ``--recompress MODE`` | recompress data chunks according to `MODE` and ``--compression``. Possible modes are `always`: recompress unconditionally; and `never`: do not recompress (faster: re-uses compressed data chunks w/o change).If no MODE is given, `always` will be used. Not passing --recompress is equivalent to "--recompress never". |
+-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| | ``--chunker-params PARAMS`` | rechunk using given chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE) or `default` to use the chunker defaults. default: do not rechunk |
| | ``--chunker-params PARAMS`` | rechunk using given chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, NC_LEVEL) or `default` to use the chunker defaults. default: do not rechunk |
+-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| .. class:: borg-common-opt-ref |
| |
@@ -71,7 +71,7 @@ borg transfer
--upgrader UPGRADER use the upgrader to convert transferred data (default: no conversion)
-C COMPRESSION, --compression COMPRESSION select compression algorithm, see the output of the "borg help compression" command for details.
--recompress MODE recompress data chunks according to `MODE` and ``--compression``. Possible modes are `always`: recompress unconditionally; and `never`: do not recompress (faster: re-uses compressed data chunks w/o change).If no MODE is given, `always` will be used. Not passing --recompress is equivalent to "--recompress never".
--chunker-params PARAMS rechunk using given chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE) or `default` to use the chunker defaults. default: do not rechunk
--chunker-params PARAMS rechunk using given chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, NC_LEVEL) or `default` to use the chunker defaults. default: do not rechunk
:ref:`common_options`
src/borg/archiver/create_cmd.py
+1 -1
View File
@@ -1006,7 +1006,7 @@ class CreateMixIn:
default=CHUNKER_PARAMS,
action=Highlander,
help="specify the chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, "
"HASH_MASK_BITS, HASH_WINDOW_SIZE). default: %s,%d,%d,%d,%d" % CHUNKER_PARAMS,
"HASH_MASK_BITS, NC_LEVEL). default: %s,%d,%d,%d,%d" % CHUNKER_PARAMS,
)
archive_group.add_argument(
"-C",
src/borg/archiver/recreate_cmd.py
+1 -1
View File
@@ -166,7 +166,7 @@ class RecreateMixIn:
default=None,
action=Highlander,
help="rechunk using given chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, "
"HASH_MASK_BITS, HASH_WINDOW_SIZE) or `default` to use the chunker defaults. "
"HASH_MASK_BITS, NC_LEVEL) or `default` to use the chunker defaults. "
"default: do not rechunk",
)
src/borg/archiver/tar_cmds.py
+1 -1
View File
@@ -522,7 +522,7 @@ class TarMixIn:
action=Highlander,
metavar="PARAMS",
help="specify the chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, "
"HASH_MASK_BITS, HASH_WINDOW_SIZE). default: %s,%d,%d,%d,%d" % CHUNKER_PARAMS,
"HASH_MASK_BITS, NC_LEVEL). default: %s,%d,%d,%d,%d" % CHUNKER_PARAMS,
)
archive_group.add_argument(
"-C",
src/borg/archiver/transfer_cmd.py
+1 -1
View File
@@ -394,7 +394,7 @@ class TransferMixIn:
default=None,
action=Highlander,
help="rechunk using given chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, "
"HASH_MASK_BITS, HASH_WINDOW_SIZE) or `default` to use the chunker defaults. "
"HASH_MASK_BITS, NC_LEVEL) or `default` to use the chunker defaults. "
"default: do not rechunk",
)
src/borg/constants.py
+7 -7
View File
@@ -136,24 +136,24 @@ CH_FASTCDC = "fastcdc"
CH_FIXED = "fixed"
CH_FAIL = "fail"
# buzhash chunker params
# chunker params
CHUNK_MIN_EXP = 19 # 2**19 == 512 KiB
CHUNK_MAX_EXP = 23 # 2**23 == 8 MiB
HASH_WINDOW_SIZE = 0xFFF # 4095 B
HASH_WINDOW_SIZE = 0xFFF # 4095 B (buzhash / buzhash64 only, fastcdc is window-less)
HASH_MASK_BITS = 21 # results in ~2 MiB chunks statistically
# buzhash64-only: normalized chunking level (0 disables it). buzhash (32bit) does not support this
# and must stay bit-compatible to borg 1.x, so it has no nc_level param.
# not supported by buzhash (32bit): it must stay bit-compatible to borg 1.x, so it has no nc_level param.
NC_LEVEL = 2 # FastCDC-style normalized chunking: tightens chunk-size distribution (much lower variance)
# defaults, use --chunker-params to override
CHUNKER_PARAMS = (CH_BUZHASH, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE)
CHUNKER64_PARAMS = (CH_BUZHASH64, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE, NC_LEVEL)
# fastcdc uses a window-less Gear hash, so it has no window_size parameter.
FASTCDC_PARAMS = (CH_FASTCDC, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, NC_LEVEL)
BUZHASH_PARAMS = (CH_BUZHASH, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE)
BUZHASH64_PARAMS = (CH_BUZHASH64, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE, NC_LEVEL)
CHUNKER_PARAMS = FASTCDC_PARAMS # the default chunker for file content data
# chunker params for the items metadata stream, finer granularity
ITEMS_CHUNKER_PARAMS = (CH_BUZHASH, 15, 19, 17, HASH_WINDOW_SIZE)
ITEMS_CHUNKER_PARAMS = (CH_FASTCDC, 15, 19, 17, NC_LEVEL)
# normal on-disk data, allocated (but not written, all zeros), not allocated hole (all zeros)
CH_DATA, CH_ALLOC, CH_HOLE = 0, 1, 2
src/borg/testsuite/chunkers/buzhash64_self_test.py
+1 -1
View File
@@ -78,6 +78,6 @@ class ChunkerBuzHash64TestCase(BaseTestCase):
self.input = self.input[:-1]
return self.input[:1]
chunker = get_chunker(*CHUNKER64_PARAMS, sparse=False)
chunker = get_chunker(*BUZHASH64_PARAMS, sparse=False)
reconstructed = b"".join(cf(chunker.chunkify(SmallReadFile())))
assert reconstructed == b"a" * 20
src/borg/testsuite/chunkers/buzhash64_test.py
+2 -2
View File
@@ -110,8 +110,8 @@ def test_fuzz_bh64(worker):
def rnd_key():
return os.urandom(32)
# decompose CHUNKER64_PARAMS = (algo, min_exp, max_exp, mask_bits, window_size, nc_level)
algo, min_exp, max_exp, mask_bits, win_size, nc_level = CHUNKER64_PARAMS
# decompose BUZHASH64_PARAMS = (algo, min_exp, max_exp, mask_bits, window_size, nc_level)
algo, min_exp, max_exp, mask_bits, win_size, nc_level = BUZHASH64_PARAMS
assert algo == CH_BUZHASH64 # default chunker must be buzhash64 here
keys = [b"\0" * 32] + [rnd_key() for _ in range(10)]
src/borg/testsuite/chunkers/buzhash_self_test.py
+1 -1
View File
@@ -69,6 +69,6 @@ class ChunkerTestCase(BaseTestCase):
self.input = self.input[:-1]
return self.input[:1]
chunker = get_chunker(*CHUNKER_PARAMS, sparse=False)
chunker = get_chunker(*BUZHASH_PARAMS, sparse=False)
reconstructed = b"".join(cf(chunker.chunkify(SmallReadFile())))
assert reconstructed == b"a" * 20
src/borg/testsuite/chunkers/buzhash_test.py
+4 -4
View File
@@ -75,14 +75,14 @@ def test_buzhash_chunksize_distribution():
@pytest.mark.skipif("BORG_TESTS_SLOW" not in os.environ, reason="slow tests not enabled, use BORG_TESTS_SLOW=1")
@pytest.mark.parametrize("worker", range(os.cpu_count() or 1))
def test_fuzz_buzhash(worker):
# Fuzz the default chunker (buzhash) with random and uniform data of misc. sizes and seeds 0 or random int32 values.
# Fuzz the buzhash chunker with random and uniform data of misc. sizes and seeds 0 or random int32 values.
def rnd_int32():
uint = random.getrandbits(32)
return uint if uint < 2**31 else uint - 2**32
# decompose CHUNKER_PARAMS = (algo, min_exp, max_exp, mask_bits, window_size)
algo, min_exp, max_exp, mask_bits, win_size = CHUNKER_PARAMS
assert algo == CH_BUZHASH # default chunker must be buzhash here
# decompose BUZHASH_PARAMS = (algo, min_exp, max_exp, mask_bits, window_size)
algo, min_exp, max_exp, mask_bits, win_size = BUZHASH_PARAMS
assert algo == CH_BUZHASH
seeds = [0] + [rnd_int32() for _ in range(50)]
sizes = [random.randint(1, 4 * 1024 * 1024) for _ in range(50)]
src/borg/testsuite/chunkers/fastcdc_test.py
+14 -3
View File
@@ -9,7 +9,7 @@ from . import cf, cf_expand
from ...chunkers import ChunkerFastCDC, get_chunker
from ...chunkers.fastcdc import fastcdc_get_gear_table
from ...constants import * # NOQA
from ...helpers import hex_to_bin
from ...helpers import ChunkerParams, hex_to_bin
# from os.urandom(32)
@@ -106,11 +106,22 @@ def test_fastcdc_get_chunker():
assert b"".join(a) == data
def test_fastcdc_is_the_default_chunker():
# both the file content data chunker and the item metadata stream chunker default to fastcdc
assert CHUNKER_PARAMS == FASTCDC_PARAMS
assert CHUNKER_PARAMS[0] == CH_FASTCDC
assert ITEMS_CHUNKER_PARAMS[0] == CH_FASTCDC
# the defaults must be valid chunker params and must give working chunkers
for params in (CHUNKER_PARAMS, ITEMS_CHUNKER_PARAMS):
assert ChunkerParams(",".join(str(p) for p in params)) == params
data = os.urandom(1024 * 1024)
chunks = cf_expand(get_chunker(*params, key=None).chunkify(BytesIO(data)))
assert b"".join(chunks) == data
def test_fastcdc_params_parsing():
from argparse import ArgumentTypeError
from ...helpers import ChunkerParams
# fastcdc, chunk_min, chunk_max, chunk_mask, nc_level (no window field)
assert ChunkerParams("fastcdc,19,23,21,2") == (CH_FASTCDC, 19, 23, 21, 2)
assert ChunkerParams("fastcdc,10,23,16,0") == (CH_FASTCDC, 10, 23, 16, 0)
src/borg/testsuite/helpers/parseformat_test.py
+3 -1
View File
@@ -814,10 +814,12 @@ def test_eval_escapes():
@pytest.mark.parametrize(
"chunker_params, expected_return",
[
("default", ("buzhash", 19, 23, 21, 4095)),
("default", ("fastcdc", 19, 23, 21, 2)),
("19,23,21,4095", ("buzhash", 19, 23, 21, 4095)),
("buzhash,19,23,21,4095", ("buzhash", 19, 23, 21, 4095)),
("10,23,16,4095", ("buzhash", 10, 23, 16, 4095)),
("buzhash64,19,23,21,4095,2", ("buzhash64", 19, 23, 21, 4095, 2)),
("fastcdc,19,23,21,2", ("fastcdc", 19, 23, 21, 2)),
("fixed,4096", ("fixed", 4096, 0)),
("fixed,4096,200", ("fixed", 4096, 200)),
],