diff --git a/README.rst b/README.rst index 00c611bb7..1f23ae0d8 100644 --- a/README.rst +++ b/README.rst @@ -81,8 +81,7 @@ Main features authenticity. If you do not need confidentiality, there are also modes that only - authenticate (``authenticated-sha256`` / ``authenticated-blake3``) and - modes that do neither (``none-sha256`` / ``none-blake3``). + authenticate (``authenticated-sha256`` / ``authenticated-blake3``). **Hashing** You can choose between HMAC-SHA256 and Blake3. diff --git a/docs/internals/data-structures.rst b/docs/internals/data-structures.rst index a1bd9eff3..4d2ade5cc 100644 --- a/docs/internals/data-structures.rst +++ b/docs/internals/data-structures.rst @@ -122,15 +122,11 @@ function is selected via ``borg repo-create --id-hash``, independently of ``--encryption``. It is a keyed MAC over the plaintext (keyed by ``id_key``): ``sha256`` selects HMAC-SHA256, ``blake3`` selects a keyed BLAKE3. -For the modes **without encryption**, the id hash is what protects the data, so -it is part of the mode name and not separately selectable (giving ``--id-hash`` -in addition is only accepted if it agrees with the mode name): - -- ``authenticated-sha256`` / ``authenticated-blake3`` have key material and thus - use the same keyed MACs as the encrypted modes. -- ``none-sha256`` / ``none-blake3`` have no key at all, so the id is a plain - SHA-256 resp. BLAKE3 hash of the plaintext. All repositories of such a mode - therefore deduplicate identically, see :ref:`tagged_envelope`. +For the modes **without encryption** (``authenticated-sha256`` / +``authenticated-blake3``), the id hash is what protects the data, so it is part +of the mode name and not separately selectable (giving ``--id-hash`` in addition +is only accepted if it agrees with the mode name). These modes have key material +and thus use the same keyed MACs as the encrypted modes, see :ref:`tagged_envelope`. As the id / key is used for deduplication, id_hash must be a cryptographically strong hash or MAC. @@ -934,7 +930,7 @@ session key changes also keep the total advantage low over the lifetime of a bor Modes without encryption ~~~~~~~~~~~~~~~~~~~~~~~~ -The ``authenticated-*`` and ``none-*`` modes do not encrypt: the payload of a repository +The ``authenticated-*`` modes do not encrypt: the payload of a repository object slot (the compressed chunk data resp. the packed metadata, see `Repository objects`_) is stored as-is. Every slot still carries a 32 byte tag:: @@ -957,25 +953,17 @@ There is no nonce, no session and no other state: the tag is deterministic. Two with the same key material therefore store byte-identical objects for identical input, which allows deduplicating them on the filesystem level (e.g. with CoW/dedup tools). -The modes differ in the tag algorithm and in whether they have a key at all: +The tag is a **MAC** (HMAC-SHA256 for ``authenticated-sha256``, keyed BLAKE3 for +``authenticated-blake3``), so only somebody who has the borg key can compute it - this detects +malicious tampering, not just accidental corruption. The MAC key is derived from ``crypt_key``:: -- ``authenticated-sha256`` / ``authenticated-blake3``: the tag is a **MAC** (HMAC-SHA256 resp. - keyed BLAKE3), so only somebody who has the borg key can compute it - this detects malicious - tampering, not just accidental corruption. The MAC key is derived from ``crypt_key``:: + tag_key = sha256(crypt_key + b"borg-repoobj-mac-hmac-sha256")[:32] # authenticated-sha256 + tag_key = sha256(crypt_key + b"borg-repoobj-mac-blake3")[:32] # authenticated-blake3 - tag_key = sha256(crypt_key + b"borg-repoobj-mac-hmac-sha256")[:32] # authenticated-sha256 - tag_key = sha256(crypt_key + b"borg-repoobj-mac-blake3")[:32] # authenticated-blake3 - - It is deliberately not derived from ``id_key``: chunk ids are public, and related repositories - share the id key (see ``borg repo-create --other-repo``), which must not enable them to forge - each other's objects. ``--copy-crypt-key`` shares ``crypt_key`` and thus opts into producing - byte-identical objects across the related repositories. -- ``none-sha256`` / ``none-blake3``: there is no key at all, so the tag is an **unkeyed** hash - (plain SHA-256 resp. BLAKE3 over the same input), i.e. a checksum. It detects accidental - corruption and reads that returned the wrong bytes, but anybody who modifies an object can - recompute it - it is no protection against malicious tampering. For the same reason, the chunk - ids of these modes are unkeyed hashes of the plaintext, which makes all repositories of such a - mode dedup identically. +It is deliberately not derived from ``id_key``: chunk ids are public, and related repositories +share the id key (see ``borg repo-create --other-repo``), which must not enable them to forge +each other's objects. ``--copy-crypt-key`` shares ``crypt_key`` and thus opts into producing +byte-identical objects across the related repositories. Legacy modes ~~~~~~~~~~~~ @@ -988,7 +976,8 @@ removed in a future release. The same applies to the borg 1.x ``none`` and ``authenticated`` modes: their envelope is just the type byte followed by the payload, so nothing about an object is verified except the chunk -id over the plaintext. They were replaced by the tagged modes described above. +id over the plaintext. borg 2 has no ``none`` mode, and its ``authenticated-*`` modes use the +tagged envelope described above. All modes ~~~~~~~~~ diff --git a/docs/internals/packs.rst b/docs/internals/packs.rst index 41d3d0317..b539c2237 100644 --- a/docs/internals/packs.rst +++ b/docs/internals/packs.rst @@ -62,8 +62,8 @@ writes and ``parse()``/``parse_meta()`` accept. It binds the header's first 41 b ``encrypted_meta`` and ``encrypted_data`` as additional authenticated data (AAD: data that is authenticated together with the ciphertext, but not itself encrypted). This applies to all borg 2 modes: the AEAD encryption modes (AES-256-OCB, ChaCha20-Poly1305) authenticate it with their AEAD -tag, the ``authenticated-*`` modes with their MAC and the ``none-*`` modes with their (unkeyed) -checksum, see :ref:`tagged_envelope`. ``meta_size`` and ``data_size`` are excluded from the AAD. +tag, the ``authenticated-*`` modes with their MAC, see :ref:`tagged_envelope`. +``meta_size`` and ``data_size`` are excluded from the AAD. ``RepoObj.parse()`` reads both slots, so tampering with either size still fails the check, by changing the length of the slice being read. ``parse_meta()`` reads the metadata slot alone: it catches a changed ``meta_size`` the same way, but not a changed ``data_size``, which the repair @@ -127,10 +127,9 @@ rebuilt the index from it. Rewriting such a pack is repository-level repair, see only when it validates like any walked header. Validating needs the key, so a repair that cannot load the key walks without it. -In the ``none-*`` modes the tag is an unkeyed checksum, and in the -``authenticated-*`` modes it binds a blob to its chunk id and nothing else (see -:ref:`security_structural_auth`), so validating does not establish there that -this repository wrote the blob. Both modes also store payloads as they are, so a +In the ``authenticated-*`` modes the tag binds a blob to its chunk id and nothing +else (see :ref:`security_structural_auth`), so validating does not establish there +that this repository wrote the blob. These modes also store payloads as they are, so a backed up file can contain something that validates - the blobs of a repository sharing the key, for instance. Such a blob reads back as itself, adding a chunk nothing references, but the extent its ``data_size`` claims covers whatever diff --git a/docs/internals/security.rst b/docs/internals/security.rst index cb776335e..fcacf7a9a 100644 --- a/docs/internals/security.rst +++ b/docs/internals/security.rst @@ -82,15 +82,6 @@ to a different object ID. This holds for the AEAD encryption modes (where the AEAD tag authenticates them) as well as for the ``authenticated-*`` modes (where a MAC does, see :ref:`tagged_envelope`). -It does **not** hold for the ``none-*`` modes: they have no key, so their objects -carry an unkeyed checksum rather than a MAC, and an attacker who modifies an -object can simply recompute it. What still constrains an attacker there is the -object ID being the (unkeyed) hash of the plaintext: the content of an existing -object can not be replaced without the ID no longer matching. But the object's -metadata and the archives list are not anchored to anything secret, -so a ``none-*`` repository provides no tamper protection - only detection of -accidental corruption. - This effectively 'anchors' each archive to the key, which is controlled by the client, thereby anchoring the DAG starting from the archives list entry, making it impossible for an attacker to add or modify any part of the @@ -253,22 +244,6 @@ Different from the AEAD modes, the MAC is deterministic (a MAC needs no nonce): session key, no IV and thus no usage limit to observe, and identical input produces identical objects. -Unencrypted modes -~~~~~~~~~~~~~~~~~ - -Modes: ``--encryption none-(sha256|blake3)`` - -Supported: borg 2.0+ - -These modes have no key at all: they neither encrypt nor authenticate. Every repository -object slot carries an *unkeyed* checksum (see :ref:`tagged_envelope`), which detects -accidental corruption - bad storage hardware, a truncated write, a read that returned the -wrong bytes - before the data is used. It is not a protection against an attacker: whoever -modifies an object can recompute the checksum, and the chunk IDs are unkeyed hashes as well. - -You are advised not to use these modes. Use ``authenticated-*`` instead if you do not want -your data encrypted but do want to detect tampering; it is the same thing plus a key. - Legacy modes ~~~~~~~~~~~~ @@ -283,8 +258,8 @@ only used by these legacy modes; new repositories use ``sha256`` or ``blake3`` The borg 1.x ``none`` and ``authenticated`` modes belong here, too: their repository objects have no tag at all, so nothing about an object is verified except the chunk ID -over the plaintext - not even the object's metadata. They were replaced by the modes -described above, which cover metadata and object header as well. +over the plaintext - not even the object's metadata. borg 2 has no ``none`` mode, and its +``authenticated-*`` modes (see above) cover metadata and object header as well. borg 2.0 does not support creating new repos using these modes, but ``borg transfer`` can still read such existing repos. @@ -389,9 +364,9 @@ used: encrypted nor authenticated, so an attacker with repository access can rewrite the crypto suite. What protects against a swapped crypto suite is not this object, but: - - a swap to a suite that does not encrypt (``none-*``, but also - ``authenticated-*``: its key blob carries the same key material and no suite - name, so it loads fine) would make the client write plaintext. That is caught + - a swap to a suite that does not encrypt (``authenticated-*``: its key blob + carries the same key material and no suite name, so it loads fine) would make + the client write plaintext. That is caught by the client's security directory: it records the key type of every repository the client accessed, and borg refuses to continue with ``EncryptionMethodMismatch`` if the suite changed. A repository that does not diff --git a/docs/usage/repo-create.rst b/docs/usage/repo-create.rst index 8cd901572..cd051143d 100644 --- a/docs/usage/repo-create.rst +++ b/docs/usage/repo-create.rst @@ -11,9 +11,8 @@ Examples # Recommended AEAD cryptographic modes (key stored in the repository by default) $ borg repo-create --encryption=aes256-ocb $ borg repo-create --encryption=chacha20-poly1305 - # No encryption (not recommended) + # No encryption, only authentication (not recommended) $ borg repo-create --encryption=authenticated-sha256 - $ borg repo-create --encryption=none-sha256 # For the encrypted modes, --encryption (the cipher / AE algorithm) and --id-hash # (the id hash function) are chosen independently. --id-hash defaults to sha256; @@ -21,10 +20,9 @@ Examples $ borg repo-create --encryption=aes256-ocb --id-hash=blake3 $ borg repo-create --encryption=chacha20-poly1305 --id-hash=blake3 - # The 'authenticated-*' and 'none-*' modes name their id hash themselves, so they + # The 'authenticated-*' modes name their id hash themselves, so they # do not take a separate --id-hash. $ borg repo-create --encryption=authenticated-blake3 - $ borg repo-create --encryption=none-blake3 # Where the key is stored (--key-location) is also chosen independently. # --key-location defaults to repokey. diff --git a/docs/usage/transfer.rst b/docs/usage/transfer.rst index f1264c3a0..705a4fc2e 100644 --- a/docs/usage/transfer.rst +++ b/docs/usage/transfer.rst @@ -70,6 +70,11 @@ locations and passphrases first: # 4. Check whether we have everything (same as step 2): $ borg transfer --from-borg1 --chunker-params=$CHUNKER_PARAMS --dry-run +A Borg 1.x repository in ``none`` mode (not encrypted, unkeyed sha256 chunk IDs) is transferred +like in the second example, re-chunking with ``--chunker-params``: all Borg 2 modes use keyed +chunk IDs. Such a repository has no key material to copy, so create the new repository without +giving the Borg 1.x repository as ``--other-repo``. + Keyfile considerations when upgrading from Borg 1.x ++++++++++++++++++++++++++++++++++++++++++++++++++++ diff --git a/scripts/rest_perf_test.sh b/scripts/rest_perf_test.sh index 573b6cd8a..69cb14d3f 100755 --- a/scripts/rest_perf_test.sh +++ b/scripts/rest_perf_test.sh @@ -56,6 +56,7 @@ profile_params() { # --rest', so BORG_REMOTE_PATH is the command the server runs. export BORG_REMOTE_PATH="$VENV/bin/borg" export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes +export BORG_PASSPHRASE=rest-perf-test export BORG_DELETE_I_KNOW_WHAT_I_AM_DOING=YES RESULTS="$(mktemp -t rest_perf_results.XXXXXX)" @@ -172,7 +173,7 @@ run_profile() { run_borg repo-delete --repo "$REPO_URL" >/dev/null 2>&1 || true # start each profile with a cold pack cache (only when the opt-in cache is enabled) if [ -n "${BORG_PACKCACHE_URL:-}" ]; then rm -rf "${BORG_PACKCACHE_URL#file://}"; fi - run_borg repo-create --encryption none-sha256 --repo "$REPO_URL" + run_borg repo-create --encryption authenticated-sha256 --repo "$REPO_URL" # backup 1 (cold) run_borg create --stats --json --repo "$REPO_URL" backup1 "$DATA_DIR" diff --git a/src/borg/archiver/help_cmd.py b/src/borg/archiver/help_cmd.py index 3638c2f2b..1ac99d609 100644 --- a/src/borg/archiver/help_cmd.py +++ b/src/borg/archiver/help_cmd.py @@ -785,7 +785,7 @@ class HelpMixIn: An unknown place name is an error. An empty value (``BORG_ASSERT_ID=``) verifies at none of these places, but still where borg always verifies (see below). - Why ``read`` is not in the default: in the keyed modes, the envelope already authenticates + Why ``read`` is not in the default: the envelope already authenticates every read - the AEAD tag for the encrypted ciphersuites, the MAC for the (unencrypted) ``authenticated-*`` modes - and the chunk id is part of what that tag is computed over. So a successful decryption resp. tag check already proves that a holder of the borg key @@ -807,10 +807,8 @@ class HelpMixIn: - in ``borg check --verify-data``. That audit is what makes not verifying elsewhere defensible, so it is not configurable (there is no ``verify_data`` place name). - - for ``none-*`` mode repositories: they have no key, so nothing authenticates a read - there and their unkeyed checksums only detect accidental corruption. The id check is - therefore not optional there: it happens at every place, whatever this variable says. - Same for reading borg 1.x repositories (``borg transfer``). + - when reading borg 1.x repositories (``borg transfer --from-borg1``): their envelope does + not cover the chunk id, so the id check is not optional there. BORG_BLAKE3_MT_THRESHOLD When set to a numeric value, chunks of at least that many KiB get their id computed by multi-threaded BLAKE3, smaller ones single-threaded (default: 256, i.e. 256KiB). @@ -940,8 +938,7 @@ class HelpMixIn: Without the key, borg can not verify anything that needs it: neither the authentication tag of the repository objects nor the chunk ids. It therefore reads the repository **unverified** - a corrupted or tampered repository will - not be detected. (This only concerns the ``authenticated-*`` modes; the - ``none-*`` modes need no key and keep verifying their checksums.) + not be detected. This workaround is **only** for emergencies and **only** to extract data from an affected repository (read-only access):: diff --git a/src/borg/archiver/repo_create_cmd.py b/src/borg/archiver/repo_create_cmd.py index 6f9d332d1..a233da643 100644 --- a/src/borg/archiver/repo_create_cmd.py +++ b/src/borg/archiver/repo_create_cmd.py @@ -19,8 +19,6 @@ ENCRYPTION_DESCRIPTIONS = { "chacha20-poly1305": "ChaCha20 with Poly1305: encryption and authentication", "authenticated-sha256": "no encryption, authentication via HMAC-SHA-256", "authenticated-blake3": "no encryption, authentication via keyed BLAKE3", - "none-sha256": "no encryption, no authentication, only SHA-256 checksums", - "none-blake3": "no encryption, no authentication, only BLAKE3 checksums", } @@ -50,22 +48,21 @@ class RepoCreateMixIn: manifest = Manifest(key, repository) with Cache(repository, manifest, warn_if_unencrypted=False): pass - if key.has_secret_key: # any key-bearing suite (everything except the "none-*" modes) - logger.warning( - "\n" - "IMPORTANT: you will need both KEY AND PASSPHRASE to access this repository!\n" - "\n" - "Key storage location depends on the mode:\n" - "- repokey modes: key is stored in the repository directory.\n" - "- keyfile modes: key is stored in the home directory of this user.\n" - "\n" - "For any mode, you should:\n" - "1. Export the Borg key and store the result in a safe place:\n" - " borg key export -r REPOSITORY encrypted-key-backup\n" - " borg key export -r REPOSITORY --paper encrypted-key-backup.txt\n" - " borg key export -r REPOSITORY --qr-html encrypted-key-backup.html\n" - "2. Write down the Borg key passphrase and store it in a safe place." - ) + logger.warning( + "\n" + "IMPORTANT: you will need both KEY AND PASSPHRASE to access this repository!\n" + "\n" + "Key storage location depends on the mode:\n" + "- repokey modes: key is stored in the repository directory.\n" + "- keyfile modes: key is stored in the home directory of this user.\n" + "\n" + "For any mode, you should:\n" + "1. Export the Borg key and store the result in a safe place:\n" + " borg key export -r REPOSITORY encrypted-key-backup\n" + " borg key export -r REPOSITORY --paper encrypted-key-backup.txt\n" + " borg key export -r REPOSITORY --qr-html encrypted-key-backup.html\n" + "2. Write down the Borg key passphrase and store it in a safe place." + ) logger.warning( "\n" "Reserve some repository storage space now for emergencies like 'disk full'\n" @@ -141,9 +138,9 @@ class RepoCreateMixIn: key then, but that key is not protected: with ``repokey`` storage, anybody who can read the repository can also unlock the key, which is as good as no encryption at all (with ``keyfile`` storage, the key is only on your client, so an empty passphrase may be - acceptable if e.g. the client's disk is encrypted). Unlike with the ``none-*`` modes, - you can add a passphrase later with ``borg key change-passphrase``. ``borg repo-info`` - shows whether the key has an empty passphrase. + acceptable if e.g. the client's disk is encrypted). You can add a passphrase later with + ``borg key change-passphrase``. ``borg repo-info`` shows whether the key has an empty + passphrase. Choosing a crypto suite +++++++++++++++++++++++ @@ -157,8 +154,6 @@ class RepoCreateMixIn: - ``chacha20-poly1305``: ChaCha20 + Poly1305 (encryption + authentication). - ``authenticated-sha256`` / ``authenticated-blake3``: no encryption, but authentication (tamper detection) using HMAC-SHA-256 resp. keyed BLAKE3. - - ``none-sha256`` / ``none-blake3``: neither encryption nor authentication, only - SHA-256 resp. BLAKE3 checksums (see below). ``--id-hash`` selects the id hash function of the **encrypted** modes: @@ -180,15 +175,6 @@ class RepoCreateMixIn: This also applies to the ``authenticated-*`` modes: they do not encrypt your data, but they still have a key (used for the id hash and the authentication), so ``--key-location`` selects where that key is stored, just like for the encrypted modes. - ``--key-location`` is only ignored for the ``none-*`` modes, which have no key at all. - - The ``none-*`` modes use neither encryption nor authentication: everything in the - repository is readable by anybody, and while every repository object carries a checksum - (which detects accidental corruption, e.g. bad storage hardware), anybody who modifies an - object can just recompute that checksum. You are advised NOT to use these modes: in case - of malicious activity in the repository, they expose you to a Denial-of-Service risk (due - to how the :ref:`internals_hashindex` works) and other issues (confidentiality, - tampering, ...). If you do **not** want to encrypt the contents of your backups, but still want to detect malicious tampering, use ``--encryption authenticated-sha256`` (or ``-blake3``). These @@ -244,8 +230,8 @@ class RepoCreateMixIn: required=True, choices=encryption_argument_names(), action=Highlander, - help="select the mode: 'aes256-ocb', 'chacha20-poly1305', 'authenticated-sha256', " - "'authenticated-blake3', 'none-sha256' or 'none-blake3' **(required)**", + help="select the mode: 'aes256-ocb', 'chacha20-poly1305', 'authenticated-sha256' " + "or 'authenticated-blake3' **(required)**", ) subparser.add_argument( "-i", @@ -256,7 +242,7 @@ class RepoCreateMixIn: default=None, # None: not given. Do not default to sha256 here, see key_creator. action=Highlander, help="select the id hash function of the encrypted modes: 'sha256' or 'blake3'. " - "The 'none-*' and 'authenticated-*' modes name their hash themselves.", + "The 'authenticated-*' modes name their hash themselves.", ) subparser.add_argument( "--key-location", @@ -266,7 +252,7 @@ class RepoCreateMixIn: default="repokey", action=Highlander, help="where to store the key: 'repokey' (in the repository, default) or 'keyfile' " - "(in the local keys directory). Ignored for the ``none-*`` modes (which have no key).", + "(in the local keys directory).", ) subparser.add_argument( "--copy-crypt-key", diff --git a/src/borg/archiver/repo_info_cmd.py b/src/borg/archiver/repo_info_cmd.py index 2c5cfb2f5..07c140263 100644 --- a/src/borg/archiver/repo_info_cmd.py +++ b/src/borg/archiver/repo_info_cmd.py @@ -25,15 +25,15 @@ class RepoInfoMixIn: # the two dimensions: cipher / AE algorithm (ENC_NAME) and id hash function (IDHASH_NAME). storage = getattr(key, "storage", None) mode = {KeyBlobStorage.KEYFILE: "keyfile", KeyBlobStorage.REPO: "repokey"}.get(storage) - # the "none-*" and "authenticated-*" modes name their id hash in ENC_NAME already. + # the "authenticated-*" modes name their id hash in ENC_NAME already. suite = key.ENC_NAME if key.IDHASH_IN_ENC_NAME else "%s, %s" % (key.ENC_NAME, key.IDHASH_NAME) if key.encrypts and key.empty_passphrase: # the key is not protected by a passphrase: with repokey storage, this is as good as # no encryption at all, so make it visible here, see #9072. suite += ", empty passphrase" if not key.encrypts: - # these modes do not encrypt data; the "authenticated-*" ones (unlike "none-*") - # still have a key stored as a keyfile or repokey, so show that location if there is one. + # the "authenticated-*" modes do not encrypt data, but they have a key stored as a + # keyfile or repokey, so show that location if there is one. encryption += "No (%s, %s)" % (mode, suite) if mode else "No (%s)" % suite else: encryption += "Yes (%s, %s)" % (mode, suite) if mode else "Yes (%s)" % suite diff --git a/src/borg/constants.py b/src/borg/constants.py index 7677483d2..b6faf6cf3 100644 --- a/src/borg/constants.py +++ b/src/borg/constants.py @@ -273,12 +273,10 @@ class KeyType: # 0x50 was a borg2 beta format ("authenticated" with blake3 ids and a no-op envelope). It was # dropped before the 2.0 release, see #9104 - the byte stays reserved so it is not reused. DROPPED_BLAKE3AUTHENTICATED = 0x50 - # the "authenticated-*" and "none-*" modes: not encrypted, but every repo object slot carries - # a tag (a MAC resp. an unkeyed checksum), see MACKeyBase. + # the "authenticated-*" modes: not encrypted, but every repo object slot carries a tag (a MAC), + # see MACKeyBase. SHA256AUTHENTICATED = 0x60 BLAKE3AUTHENTICATED = 0x70 - SHA256NONE = 0x80 - BLAKE3NONE = 0x90 CACHE_TAG_NAME = "CACHEDIR.TAG" diff --git a/src/borg/crypto/key.py b/src/borg/crypto/key.py index 6ca6800ea..d8ccf083d 100644 --- a/src/borg/crypto/key.py +++ b/src/borg/crypto/key.py @@ -200,9 +200,9 @@ def key_creator(repository, args, *, other_key=None): # For the encrypted modes, the crypto suite is selected by two orthogonal dimensions: the # cipher / AE algorithm ("--encryption") and the id hash function ("--id-hash", defaulting to # sha256). For the modes that do not encrypt, the hash is not a tuning knob but *is* the - # mechanism (it authenticates resp. checksums the data), so it is part of the mode name: - # "none-sha256", "none-blake3", "authenticated-sha256", "authenticated-blake3". Giving - # "--id-hash" in addition to one of those names is only accepted if both agree. + # mechanism (it authenticates the data), so it is part of the mode name: + # "authenticated-sha256", "authenticated-blake3". Giving "--id-hash" in addition to one of + # those names is only accepted if both agree. enc = args.encryption id_hash = getattr(args, "id_hash", None) # None: not given, see the --id-hash argparse default for key in AVAILABLE_KEY_TYPES: @@ -312,22 +312,16 @@ def uses_same_chunker_secret(other_key, key): def uses_same_id_hash(other_key, key): """other_key -> key upgrade: is the id hash the same?""" # avoid breaking the deduplication by changing the id hash - old_sha256_ids = (LegacyPlaintextKey,) # unkeyed sha256 over the plaintext - new_sha256_ids = (ChecksumKey,) + # note: borg 1.x "none" (LegacyPlaintextKey) uses unkeyed sha256 ids, which no borg 2 mode has. old_hmac_sha256_ids = (AESCTRKey, LegacyAuthenticatedKey) new_hmac_sha256_ids = (AESOCBKey, CHPOKey, AuthenticatedKey) # note: we do not support blake2b for new repos, see #8867 new_blake3_ids = (Blake3AESOCBKey, Blake3CHPOKey, Blake3AuthenticatedKey) - new_unkeyed_blake3_ids = (Blake3ChecksumKey,) # unkeyed blake3 over the plaintext same_ids = ( isinstance(other_key, old_hmac_sha256_ids + new_hmac_sha256_ids) and isinstance(key, new_hmac_sha256_ids) or isinstance(other_key, new_blake3_ids) and isinstance(key, new_blake3_ids) - or isinstance(other_key, old_sha256_ids + new_sha256_ids) - and isinstance(key, new_sha256_ids) - or isinstance(other_key, new_unkeyed_blake3_ids) - and isinstance(key, new_unkeyed_blake3_ids) ) return same_ids @@ -348,7 +342,7 @@ class KeyBase: # None means "not creatable this way" (e.g. legacy read-only classes). ENC_NAME: ClassVar[str] = None # override in creatable subclasses IDHASH_NAME: ClassVar[str] = None # override in creatable subclasses (or via id-hash mix-in) - # Is the id hash part of the ENC_NAME (e.g. "none-blake3") instead of being selectable via + # Is the id hash part of the ENC_NAME (e.g. "authenticated-blake3") instead of being selectable via # "--id-hash"? True for the modes that do not encrypt: there, the hash is not a tuning knob, # it is what protects the data, so it belongs to the mode. See key_creator. IDHASH_IN_ENC_NAME: ClassVar[bool] = False @@ -377,30 +371,15 @@ class KeyBase: # id_key dummy, needs to be overwritten by subclass id_key: bytes = None - # Is assert_id() the read path's authentication mechanism for this key class? - # True (the default): the envelope does not authenticate the payload as the one written for the requested - # chunk id (there is no keyed envelope tag covering the id), so the id check must never be skipped, see - # RepoObj.parse (RepoObj1.parse for the borg 1.x classes). In borg 2, this is only the "none-*" modes: - # their decrypt() verifies just the unkeyed envelope checksum, which detects accidental corruption but - # authenticates nothing, see ChecksumKeyBase. The read-only borg 1.x classes (borg.legacy.crypto.key) - # keep the default, too - no borg 1.x envelope covers the chunk id, and for borg 1.x "authenticated", - # the keyed id hash even is the only authentication a read has: skipping it would silently demote that - # mode to "none". - # False (the AEAD ciphersuites and the "authenticated-*" modes): every read is authenticated by the keyed - # envelope tag, which covers the chunk id (it is in the AAD), independently of assert_id() - see - # AEADKeyBase.assert_id for what assert_id adds on top of that. - id_check_is_authentication: ClassVar[bool] = True - - # Does this mode encrypt the data at all? False for the modes that only tag it ("none-*", - # "authenticated-*" and their borg 1.x predecessors). Different from logically_encrypted, - # which is a per-instance property (an encrypted repo with an empty passphrase is not - # "logically" encrypted, but it does encrypt). + # Does this mode encrypt the data at all? False for the modes that only tag it + # ("authenticated-*") and for the borg 1.x "none" and "authenticated" modes. Different from + # logically_encrypted, which is a per-instance property (an encrypted repo with an empty + # passphrase is not "logically" encrypted, but it does encrypt). encrypts: ClassVar[bool] = True # Does this key class have secret key material? - # False for the "none-*" modes: their chunk ids and their envelope checksums are unkeyed, thus - # they work without any key - and the authenticated_no_key workaround (which fakes the key - # material of a key it can not unlock) must not switch off checks that need no key at all. + # False only for the read-only borg 1.x "none" mode (its chunk ids are unkeyed), so there is + # no key material a related repository could copy, see FlexiKey.create. has_secret_key: ClassVar[bool] = True # Whether this *particular instance* is encrypted from a practical point of view, @@ -733,8 +712,7 @@ class FlexiKey: key.storage = KEY_LOCATIONS.get(getattr(args, "key_location", None), cls.STORAGE) if other_key is not None: if not other_key.has_secret_key: - # the "none-*" modes (and borg 1.x "none") have no key material to copy - and they - # need none: their chunk ids are unkeyed, so they all dedup identically anyway. + # borg 1.x "none" has no key material to copy. raise Error("Copying key material from an unencrypted repository is not possible.") if isinstance(key, AESKeyBase): # user must use an AEADKeyBase subclass (AEAD modes with session keys) @@ -1084,7 +1062,7 @@ class FlexiKey: return victim -# ------------ not encrypted, but tagged: the "authenticated-*" and "none-*" modes ------------ +# ------------ not encrypted, but tagged: the "authenticated-*" modes ------------ class MACKeyBase(KeyBase): @@ -1104,7 +1082,7 @@ class MACKeyBase(KeyBase): not need any). Two repositories with the same key material therefore store byte-identical objects for identical input, which allows deduplicating them on the filesystem level. - Subclasses supply the tag algorithm (mac()) and, if they are keyed, the tag key (tag_key). + Subclasses supply the tag algorithm (mac()) and the tag key (tag_key). """ encrypts = False # these modes only tag the data, they do not encrypt it @@ -1122,7 +1100,7 @@ class MACKeyBase(KeyBase): @property def tag_key(self) -> bytes: - """The key the tag is computed with (unkeyed modes do not have one).""" + """The key the tag is computed with.""" raise NotImplementedError def init_ciphers(self, manifest_data=None): @@ -1150,9 +1128,8 @@ class MACKeyBase(KeyBase): self.assert_type(data[0], id) obj = memoryview(data) payload = obj[self.PAYLOAD_OVERHEAD :] - if self.has_secret_key and AUTHENTICATED_NO_KEY: - # we do not have the key material, so we can not verify the tag, see the - # BORG_WORKAROUNDS docs. Unkeyed modes verify even then - they need no key. + if AUTHENTICATED_NO_KEY: + # we do not have the key material, so we can not verify the tag, see the BORG_WORKAROUNDS docs. return payload header = bytes(obj[: self.HEADER_SIZE]) tag = bytes(obj[self.HEADER_SIZE : self.PAYLOAD_OVERHEAD]) @@ -1162,79 +1139,6 @@ class MACKeyBase(KeyBase): return payload -class ChecksumKeyBase(MACKeyBase): - """ - Base class of the "none-*" modes: no encryption, no key, no authentication. - - The tag is an **unkeyed** hash, i.e. a checksum: it detects accidental corruption of the - payload, the metadata or the object header (including a read that returned the wrong bytes), - but it is no protection against malicious tampering - whoever changes the object can just - recompute the checksum. Detecting that requires a secret, see the "authenticated-*" modes. - - Because there is no key, the chunk id is an unkeyed hash of the plaintext, too. Thus, all - repositories of this mode dedup identically and store byte-identical objects for identical - input, no key sharing needed. - """ - - STORAGE = KeyBlobStorage.NO_STORAGE - LOCATION_CONFIGURABLE = False - - chunk_seed = 0 - crypt_key = b"" # makes .derive_key() work, nothing secret here - id_key = b"" # makes .derive_key() work, nothing secret here - - logically_encrypted = False - has_secret_key = False - - # The checksum can be recomputed by anybody, so it is not an authentication and thus can not - # take over the read path authentication the id hash does. Different from the keyed classes - # below, verifying the chunk id must therefore never be skipped, see RepoObj.parse. - id_check_is_authentication = True - - @classmethod - def create(cls, repository, args, **kw): - logger.info( - "Encryption NOT enabled.\n" - 'Use "--encryption=aes256-ocb" (or another encrypted mode) to enable encryption.' - ) - return cls(repository) - - @classmethod - def detect(cls, repository, manifest_data, *, other=False): - return cls(repository) - - -class ChecksumKey(ChecksumKeyBase): - TYPE = KeyType.SHA256NONE - TYPES_ACCEPTABLE = {TYPE} - ENC_NAME = "none-sha256" - IDHASH_NAME = "sha256" - - def id_hash(self, data): - return sha256(data).digest() - - def mac(self, prefix, payload): - h = sha256(prefix) - h.update(payload) - return h.digest() - - -class Blake3ChecksumKey(ChecksumKeyBase): - TYPE = KeyType.BLAKE3NONE - TYPES_ACCEPTABLE = {TYPE} - ENC_NAME = "none-blake3" - IDHASH_NAME = "blake3" - - def id_hash(self, data): - return _blake3_hasher(data).digest(length=32) - - def mac(self, prefix, payload): - max_threads = blake3.AUTO if len(payload) >= get_blake3_mt_threshold() else 1 - h = blake3(prefix, max_threads=max_threads) - h.update(payload) - return h.digest(length=32) - - class AuthenticatedKeyBase(MACKeyBase, FlexiKey): """ Base class of the "authenticated-*" modes: no encryption, but real authentication. @@ -1259,11 +1163,6 @@ class AuthenticatedKeyBase(MACKeyBase, FlexiKey): # It's only authenticated, not encrypted. logically_encrypted = False - # every read is authenticated by the envelope tag (which covers the chunk id via the AAD), - # independently of assert_id() - the same reasoning as for the AEAD keys, see - # AEADKeyBase.assert_id about what verifying the chunk id adds on top of that. - id_check_is_authentication = False - # domain for deriving the envelope MAC key from crypt_key, see tag_key. It is per MAC # algorithm, so the classes below do not use the same key for different algorithms. MAC_KEY_DOMAIN: ClassVar[bytes] = None # override in subclass @@ -1359,7 +1258,7 @@ class AuthenticatedKey(ID_HMAC_SHA_256, AuthenticatedKeyBase): from ..legacy.crypto.key import AESCTRKey, Blake2AESCTRKey # noqa: F401 from ..legacy.crypto.key import Blake2AuthenticatedKey # noqa: F401 from ..legacy.crypto.key import AuthenticatedKey as LegacyAuthenticatedKey # noqa: E402 -from ..legacy.crypto.key import PlaintextKey as LegacyPlaintextKey # noqa: E402 +from ..legacy.crypto.key import PlaintextKey as LegacyPlaintextKey # noqa: E402, F401 from ..legacy.crypto.key import LEGACY_KEY_TYPES # noqa: E402 from ..legacy.crypto.key import ID_BLAKE2b_256 # noqa: F401 @@ -1435,9 +1334,6 @@ class AEADKeyBase(KeyBase): # an AEAD key may be stored as a keyfile or inside the repository (see borg key change-location). LOCATION_CONFIGURABLE = True - # the AEAD tag authenticates every read on its own, assert_id only adds the "evil client" detection below. - id_check_is_authentication = False - def assert_id(self, id, data): # Comparing the id hash here would not be needed any more for the new AEAD crypto **IF** we # could be sure that chunks were created by normal (not tampered, not evil) borg code: @@ -1456,7 +1352,7 @@ class AEADKeyBase(KeyBase): # As only that (rather special) threat is left here, this check is mostly optional for AEAD # keys (it was mandatory from #7362/#7367 until #9994): RepoObj.parse calls it at the places # BORG_ASSERT_ID lists (by default: the places that re-anchor content - check --repair, - # transfer, re-chunking), plus always in check --verify-data, see id_check_is_authentication. + # transfer, re-chunking), plus always in check --verify-data. if id and id != Manifest.MANIFEST_ID: id_computed = self.id_hash(data) if not hmac.compare_digest(id_computed, id): @@ -1610,6 +1506,4 @@ AVAILABLE_KEY_TYPES = ( # not encrypted modes (the id hash is part of the mode name, see key_creator) AuthenticatedKey, Blake3AuthenticatedKey, - ChecksumKey, - Blake3ChecksumKey, ) diff --git a/src/borg/repoobj.py b/src/borg/repoobj.py index 82d113f25..ada39c7b8 100644 --- a/src/borg/repoobj.py +++ b/src/borg/repoobj.py @@ -24,23 +24,15 @@ ASSERT_ID_PLACES = ( # not verifying elsewhere defensible, so it must not be switchable. ASSERT_ID_PLACES_MANDATORY = ("verify_data",) # borg check --verify-data # Default value of the BORG_ASSERT_ID env var: verify everywhere except on the (hot) general read -# path - there, the envelope authentication of the keyed modes already covers what a repository -# could do to us, see AEADKeyBase.assert_id. (For the modes whose id check *is* the read path -# authentication, this setting does not apply, see below.) +# path - there, the envelope authentication already covers what a repository could do to us, see +# AEADKeyBase.assert_id. BORG_ASSERT_ID_DEFAULT = ("repair", "transfer", "rechunk") # Reads through a RepoObj are attributed to this place unless a command sets another one. ASSERT_ID_PLACE_DEFAULT = "read" def get_assert_id_places(): - """Determine the configurable places that shall verify the chunk id, see the BORG_ASSERT_ID docs. - - Note: this only decides for keys that authenticate reads independently of the id hash (the AEAD - ciphersuites and the "authenticated-*" modes, see KeyBase.id_check_is_authentication) - for all - other keys, the id check is the read path authentication itself and thus always happens, no - matter what is configured here. - The same is true for ASSERT_ID_PLACES_MANDATORY. - """ + """Determine the configurable places that shall verify the chunk id, see the BORG_ASSERT_ID docs.""" value = os.environ.get("BORG_ASSERT_ID") if value is None: return frozenset(BORG_ASSERT_ID_DEFAULT) @@ -254,23 +246,16 @@ class RepoObj: compressor_cls, compression_level = Compressor.detect(compr_hdr) compressor = compressor_cls(level=compression_level) meta, data = compressor.decompress(dict(meta_compressed), data_compressed[:psize]) - # For keys where the id check is the read-path authentication (the "none-*" modes, whose - # envelope checksum is unkeyed and thus no authentication), it always has to happen - - # skipping it would remove all integrity checking from reads. - # For the keys that authenticate the envelope (the AEAD and the "authenticated-*" modes), the - # payload is already authenticated for this specific chunk id (the id is in the AAD), so the id + # All keys authenticate the envelope (the AEAD and the "authenticated-*" modes), so the + # payload is already authenticated for this specific chunk id (the id is in the AAD) and the id # check only adds detection of chunks whose plaintext does not match their id - which only an # evil/broken borg client that had the repo key could have written. Whether that extra # full-plaintext hash pass is worth it depends on the place we read at, see BORG_ASSERT_ID. place = assert_id_place if assert_id_place is not None else self.assert_id_place - assert_id = ( - self.key.id_check_is_authentication - or place in ASSERT_ID_PLACES_MANDATORY - or place in self.assert_id_places - ) + assert_id = place in ASSERT_ID_PLACES_MANDATORY or place in self.assert_id_places # the authenticated_no_key workaround fakes key material it could not unlock, so checks that - # need that key material can not work. The unkeyed modes need none, so they keep checking. - if assert_id and not (AUTHENTICATED_NO_KEY and self.key.has_secret_key): + # need that key material can not work. + if assert_id and not AUTHENTICATED_NO_KEY: self.key.assert_id(id, data) else: meta, data = None, None @@ -287,9 +272,8 @@ def object_validator(repo_objs): field outside the tag, must match csize - the data slot's payload size, recorded in the tagged metadata - plus the key's fixed envelope overhead. - In the "none-*" modes the tag is an unkeyed checksum, and in the "authenticated-*" modes it is - deterministic and binds an object to its chunk id alone. Both therefore accept an object that a - backed up file contains, at any offset in any pack. + In the "authenticated-*" modes the tag is deterministic and binds an object to its chunk id + alone. They therefore accept an object that a backed up file contains, at any offset in any pack. """ hdr_size = RepoObj.obj_header.size overhead = repo_objs.key.PAYLOAD_OVERHEAD # the envelope adds a fixed number of bytes to the payload @@ -298,10 +282,10 @@ def object_validator(repo_objs): try: meta = repo_objs.parse_meta(chunk_id, obj, ro_type=ROBJ_DONTCARE) except (IntegrityErrorBase, msgpack.UnpackException): - # arbitrary bytes fail the authentication, or the msgpack unpacking in the modes that - # authenticate without a secret key (none-*, authenticated-*) and thus accept them. + # arbitrary bytes fail the authentication - or the msgpack unpacking, if the tag is not + # verified (authenticated-* modes with the authenticated_no_key workaround). return False - # in those modes the slot can unpack to any value, so check the value: only metadata that + # without tag verification the slot can unpack to any value, so check the value: only metadata that # is unusable here gives False, any other exception propagates. if not isinstance(meta, dict): return False diff --git a/src/borg/repository.py b/src/borg/repository.py index 3ff393bc4..339ba8469 100644 --- a/src/borg/repository.py +++ b/src/borg/repository.py @@ -663,7 +663,7 @@ def superseded_gap_ranges(reader, chunks, pack_id, obj_ranges, pack_size, *, val # TODO: obj_size is verified only for objects validate accepts. A wrong obj_size # usually ends the walk at a header that does not parse, so the superseded duplicates in # the rest of the gap are kept on every rewrite. PackReader._find_header could resync to - # the next object validate accepts, but in the none-* and authenticated-* modes that can + # the next object validate accepts, but in the authenticated-* modes that can # be a copy of an object inside another object's unencrypted data, whose range can cover # bytes of the gap objects after it, which would then be dropped. offset += obj_size diff --git a/src/borg/testsuite/__init__.py b/src/borg/testsuite/__init__.py index bb432ebc1..7e1cf0204 100644 --- a/src/borg/testsuite/__init__.py +++ b/src/borg/testsuite/__init__.py @@ -271,6 +271,20 @@ def filter_xattrs(x): raise ValueError("Unsupported type: %s" % type(x)) +def make_test_key(repository=None): + """Return an "authenticated-sha256" key for tests that just need some working key. + + The key material is fixed, so the chunk ids and the stored objects are the same in every test run. + The key is not stored anywhere (no borg key, no passphrase). + """ + from ..crypto.key import AuthenticatedKey + + key = AuthenticatedKey(repository) + key.init_from_given_data(crypt_key=b"c" * 64, id_key=b"i" * 32, chunk_seed=0) + key.init_ciphers() + return key + + class BaseTestCase(unittest.TestCase): assert_in = unittest.TestCase.assertIn assert_not_in = unittest.TestCase.assertNotIn diff --git a/src/borg/testsuite/archive_test.py b/src/borg/testsuite/archive_test.py index 9f1e5296e..f98d3e161 100644 --- a/src/borg/testsuite/archive_test.py +++ b/src/borg/testsuite/archive_test.py @@ -7,10 +7,9 @@ from unittest.mock import Mock import pytest -from . import rejected_dotdot_paths, is_utime_fully_supported +from . import rejected_dotdot_paths, is_utime_fully_supported, make_test_key from ..cache import ChunkListEntry from ..constants import ROBJ_FILE_STREAM, zeros -from ..crypto.key import ChecksumKey from ..archive import Archive, CacheChunkBuffer, DownloadPipeline, RobustUnpacker, valid_msgpacked_dict from ..archive import ITEM_KEYS, Statistics from ..archive import zero_chunk_flags, zero_chunk_id, zero_chunk_ids @@ -195,7 +194,7 @@ def test_stats_as_dict(stats): ) def test_timestamp_parsing(monkeypatch, isoformat, expected): repository = Mock() - key = ChecksumKey(repository) + key = make_test_key(repository) manifest = Manifest(key, repository) a = Archive(manifest, "test", create=True) a.metadata = ArchiveItem(time=isoformat) @@ -219,7 +218,7 @@ class MockCache: def test_cache_chunk_buffer(): data = [Item(path="p1"), Item(path="p2")] cache = MockCache() - key = ChecksumKey(None) + key = make_test_key(None) chunks = CacheChunkBuffer(cache, key, None) for d in data: chunks.add(d) @@ -236,7 +235,7 @@ def test_partial_cache_chunk_buffer(): big = "0123456789abcdefghijklmnopqrstuvwxyz" * 25000 data = [Item(path="full", target=big), Item(path="partial", target=big)] cache = MockCache() - key = ChecksumKey(None) + key = make_test_key(None) chunks = CacheChunkBuffer(cache, key, None) for d in data: chunks.add(d) @@ -271,7 +270,7 @@ def test_download_pipeline_parsed_cache(): # a content data stream may reference the same chunk many times (e.g. the all-zero # chunks of a sparse file): repeated chunks shall be parsed (decrypted, authenticated, # decompressed) only once, see issue #1678. - key = ChecksumKey(None) + key = make_test_key(None) repo_objs = RepoObj(key) # note: repeated, but not all-zero data, so it is not served via the zeros shortcut chunks_data = [b"foobar" * 100, b"idletone" * 125, b"barbaz" * 100] @@ -304,7 +303,7 @@ def test_download_pipeline_parsed_cache(): def test_download_pipeline_missing_chunk(replacement_chunk): # a chunk missing in the repository is either replaced by all-zero data of the # correct size, or reported as None - and never blows up on the size check. - key = ChecksumKey(None) + key = make_test_key(None) repo_objs = RepoObj(key) data = b"foobar" * 100 id = repo_objs.id_hash(data) @@ -318,7 +317,7 @@ def test_download_pipeline_missing_chunk(replacement_chunk): def test_download_pipeline_zero_chunks_served_locally(): # repeated all-zero chunks (e.g. from the holes of a sparse file) shall be served # directly from the zeros constant, without repository access, see issue #1678. - key = ChecksumKey(None) + key = make_test_key(None) repo_objs = RepoObj(key) data = b"foobar" * 100 data_id = repo_objs.id_hash(data) @@ -358,7 +357,7 @@ def test_zero_chunk_flags(): # cheap all-zero chunk detection from the chunk ids/sizes alone: the zero chunk id # is computed for ids occurring repeatedly, while unique ids are only compared # against already memoized zero chunk ids. - key = ChecksumKey(None) + key = make_test_key(None) id_hash = key.id_hash data = b"foobar" * 100 data_id = id_hash(data) diff --git a/src/borg/testsuite/archiver/check_cmd_test.py b/src/borg/testsuite/archiver/check_cmd_test.py index bdfc14dbb..8c4caa654 100644 --- a/src/borg/testsuite/archiver/check_cmd_test.py +++ b/src/borg/testsuite/archiver/check_cmd_test.py @@ -1101,7 +1101,7 @@ def test_repair_finish_flushes_pack_writer(archivers, request): assert not repository._pack_writer._pieces # finish() stored it -@pytest.mark.parametrize("init_args", [["--encryption=aes256-ocb"], ["--encryption", "none-sha256"]]) +@pytest.mark.parametrize("init_args", [["--encryption=aes256-ocb"], ["--encryption", "authenticated-sha256"]]) def test_verify_data(archivers, request, init_args): archiver = request.getfixturevalue(archivers) if archiver.get_kind() != "local": @@ -1196,7 +1196,7 @@ def test_repair_wrong_item_metadata_chunk_content(archivers, request, monkeypatc assert "id verification failed" in output -@pytest.mark.parametrize("init_args", [["--encryption=aes256-ocb"], ["--encryption", "none-sha256"]]) +@pytest.mark.parametrize("init_args", [["--encryption=aes256-ocb"], ["--encryption", "authenticated-sha256"]]) def test_corrupted_file_chunk(archivers, request, init_args): ## like test_verify_data, but also checks a repository-only check passes after repair and a plain ## archives check reports the missing chunk. diff --git a/src/borg/testsuite/archiver/checks_test.py b/src/borg/testsuite/archiver/checks_test.py index 4831deb68..6d9431f98 100644 --- a/src/borg/testsuite/archiver/checks_test.py +++ b/src/borg/testsuite/archiver/checks_test.py @@ -5,11 +5,13 @@ import pytest from ...cache import Cache from ...constants import * # NOQA +from ...crypto.key import keyfile_format, keyfile_parse from ...helpers import Location, get_security_dir, bin_to_hex from ...helpers import EXIT_ERROR +from ...repository import Repository from .. import changedir from . import cmd, _extract_repository_id, create_test_files -from . import _set_repository_id, create_regular_file, assert_creates_file, generate_archiver_tests, RK_ENCRYPTION +from . import create_regular_file, assert_creates_file, generate_archiver_tests, RK_ENCRYPTION from . import set_empty_passphrase pytest_generate_tests = lambda metafunc: generate_archiver_tests(metafunc, kinds="local,remote") # NOQA @@ -20,6 +22,21 @@ def get_security_directory(repo_path): return get_security_dir(repository_id) +def _set_repository_id(repo_path, id): + """Change the id of the repository at repo_path and label its repokeys for the new id. + + This is what an attacker does to make their own "authenticated-*" repository (which has a borg + key, labelled with the repository id) look like the repository of somebody else. + """ + with Repository(repo_path) as repository: + keys = repository.load_keys() + repository._set_id(id) + for name, keydata in keys: + _, b64data = keyfile_parse(keydata) + repository.delete_key(name) + repository.store_key(keyfile_format(bin_to_hex(id), b64data).encode()) + + def test_repository_swap_detection(archivers, request): archiver = request.getfixturevalue(archivers) create_test_files(archiver.input_path) @@ -28,7 +45,7 @@ def test_repository_swap_detection(archivers, request): repository_id = _extract_repository_id(archiver.repository_path) cmd(archiver, "create", "test", "input") shutil.rmtree(archiver.repository_path) - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") _set_repository_id(archiver.repository_path, repository_id) assert repository_id == _extract_repository_id(archiver.repository_path) if archiver.FORK_DEFAULT: @@ -42,9 +59,9 @@ def test_repository_swap_detection2(archivers, request): archiver = request.getfixturevalue(archivers) create_test_files(archiver.input_path) original_location = archiver.repository_location - archiver.repository_location = original_location + "_unencrypted" - cmd(archiver, "repo-create", "--encryption=none-sha256") os.environ["BORG_PASSPHRASE"] = "passphrase" + archiver.repository_location = original_location + "_unencrypted" + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") archiver.repository_location = original_location + "_encrypted" cmd(archiver, "repo-create", RK_ENCRYPTION) cmd(archiver, "create", "test", "input") @@ -65,7 +82,7 @@ def test_repository_swap_detection_no_cache(archivers, request): repository_id = _extract_repository_id(archiver.repository_path) cmd(archiver, "create", "test", "input") shutil.rmtree(archiver.repository_path) - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") _set_repository_id(archiver.repository_path, repository_id) assert repository_id == _extract_repository_id(archiver.repository_path) cmd(archiver, "repo-delete", "--cache-only") @@ -80,9 +97,9 @@ def test_repository_swap_detection2_no_cache(archivers, request): archiver = request.getfixturevalue(archivers) original_location = archiver.repository_location create_test_files(archiver.input_path) - archiver.repository_location = original_location + "_unencrypted" - cmd(archiver, "repo-create", "--encryption=none-sha256") os.environ["BORG_PASSPHRASE"] = "passphrase" + archiver.repository_location = original_location + "_unencrypted" + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") archiver.repository_location = original_location + "_encrypted" cmd(archiver, "repo-create", RK_ENCRYPTION) cmd(archiver, "create", "test", "input") @@ -157,7 +174,7 @@ def test_repository_move(archivers, request, monkeypatch): def test_unknown_unencrypted(archivers, request, monkeypatch): archiver = request.getfixturevalue(archivers) - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") # Ok: repository is known cmd(archiver, "repo-info") @@ -176,7 +193,7 @@ def test_unknown_unencrypted(archivers, request, monkeypatch): output = cmd(archiver, "repo-info") # the warning says why the repository counts as unencrypted, see #9072 assert "previously unknown unencrypted repository" in output - assert "uses the none-sha256 mode, which does not encrypt the data" in output + assert "uses the authenticated-sha256 mode, which does not encrypt the data" in output def test_unknown_unencrypted_empty_passphrase(archivers, request, monkeypatch): diff --git a/src/borg/testsuite/archiver/compact_cmd_test.py b/src/borg/testsuite/archiver/compact_cmd_test.py index 531d95ce3..660cb8dcf 100644 --- a/src/borg/testsuite/archiver/compact_cmd_test.py +++ b/src/borg/testsuite/archiver/compact_cmd_test.py @@ -7,7 +7,6 @@ import pytest from ...constants import * # NOQA from ...helpers import get_cache_dir, bin_to_hex, sig_int, Error from ...hashindex import ChunkIndex -from ...crypto.key import ChecksumKey from ...repoobj import RepoObj from ...repository import Repository from ...cache import files_cache_name, discover_files_cache_names, list_chunkindex_hashes @@ -19,14 +18,15 @@ from ...archiver.compact_cmd import ArchiveGarbageCollector from ... import cache from . import cmd, create_regular_file, create_src_archive, generate_archiver_tests, open_repository, RK_ENCRYPTION from . import changedir +from .. import make_test_key from ..repository_test import H, fchunk, pdchunk pytest_generate_tests = lambda metafunc: generate_archiver_tests(metafunc, kinds="local,remote,binary") # NOQA def gc_manifest(repository): - """Return a manifest stand-in for ArchiveGarbageCollector, whose repo_objs use a none-sha256 key.""" - return SimpleNamespace(repo_objs=RepoObj(ChecksumKey(repository))) + """Return a manifest stand-in for ArchiveGarbageCollector, whose repo_objs use a test key.""" + return SimpleNamespace(repo_objs=RepoObj(make_test_key(repository))) @pytest.mark.parametrize("stats", (True, False)) diff --git a/src/borg/testsuite/archiver/create_cmd_test.py b/src/borg/testsuite/archiver/create_cmd_test.py index a40b89df7..5e99a8c75 100644 --- a/src/borg/testsuite/archiver/create_cmd_test.py +++ b/src/borg/testsuite/archiver/create_cmd_test.py @@ -141,7 +141,7 @@ def test_archived_paths(archivers, request): # no leading slash in borg archives: archived_path = posix_path.lstrip("/") create_regular_file(archiver.input_path, "test") - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", "test", "input", posix_path) # "input" directory is recursed into, "input/test" is discovered and joined by borg's recursion. # posix_path was directly given as a cli argument and should end up as archive_path in the borg archive. @@ -207,7 +207,7 @@ def test_create_duplicate_root(archivers, request): hl_b = os.path.join(path_b, "hardlink") create_regular_file(archiver.input_path, hl_a, contents=b"123456") os.link(hl_a, hl_b) - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", "test", "input", "input") # give input twice! # test if created archive has 'input' contents twice: archive_list = cmd(archiver, "list", "test", "--json-lines") @@ -223,7 +223,7 @@ def test_create_unreadable_parent(archiver): os.mkdir(root_dir) os.chmod(parent_dir, 0o111) # --x--x--x == parent dir traversable, but not readable try: - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") # issue #7746: we *can* read root_dir and we *can* traverse parent_dir, so this should work: cmd(archiver, "create", "test", root_dir) finally: diff --git a/src/borg/testsuite/archiver/disk_full_test.py b/src/borg/testsuite/archiver/disk_full_test.py index 130153e84..94829860e 100644 --- a/src/borg/testsuite/archiver/disk_full_test.py +++ b/src/borg/testsuite/archiver/disk_full_test.py @@ -47,11 +47,12 @@ def make_files(dir, count, size, rnd=True): def test_disk_full(test_pass, cmd_fixture, monkeypatch): monkeypatch.setenv("BORG_CHECK_I_KNOW_WHAT_I_AM_DOING", "YES") monkeypatch.setenv("BORG_DELETE_I_KNOW_WHAT_I_AM_DOING", "YES") + monkeypatch.setenv("BORG_PASSPHRASE", "waytooeasyonlyfortests") repo = os.path.join(DF_MOUNT, "repo") input = os.path.join(DF_MOUNT, "input") shutil.rmtree(repo, ignore_errors=True) shutil.rmtree(input, ignore_errors=True) - rc, out = cmd_fixture(f"--repo={repo}", "repo-create", "--encryption=none-sha256") + rc, out = cmd_fixture(f"--repo={repo}", "repo-create", "--encryption=authenticated-sha256") if rc != EXIT_SUCCESS: print("repo-create", rc, out) assert rc == EXIT_SUCCESS diff --git a/src/borg/testsuite/archiver/extract_cmd_test.py b/src/borg/testsuite/archiver/extract_cmd_test.py index 72c1882a7..2bfb9c4a7 100644 --- a/src/borg/testsuite/archiver/extract_cmd_test.py +++ b/src/borg/testsuite/archiver/extract_cmd_test.py @@ -536,7 +536,7 @@ def test_extract_hardlinks_twice(archivers, request): hl_b = os.path.join(path_b, "hardlink") create_regular_file(archiver.input_path, hl_a, contents=b"123456") os.link(hl_a, hl_b) - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", "test", "input", "input") # give input twice! # now test extraction with changedir("output"): @@ -777,7 +777,7 @@ def test_extract_xattrs_errors(archivers, request): create_regular_file(archiver.input_path, "file") xattr.setxattr(b"input/file", b"user.attribute", b"value") - cmd(archiver, "repo-create", "-e", "none-sha256") + cmd(archiver, "repo-create", "-e", "authenticated-sha256") cmd(archiver, "create", "test", "input") with changedir("output"): input_abspath = os.path.abspath("input/file") @@ -805,7 +805,7 @@ def test_extract_xattrs_errors(archivers, request): def test_extract_xattrs_resourcefork(archivers, request): archiver = request.getfixturevalue(archivers) create_regular_file(archiver.input_path, "file") - cmd(archiver, "repo-create", "-e", "none-sha256") + cmd(archiver, "repo-create", "-e", "authenticated-sha256") input_path = os.path.abspath("input/file") xa_key, xa_value = b"com.apple.ResourceFork", b"whatshouldbehere" # issue #7234 xattr.setxattr(input_path.encode(), xa_key, xa_value) @@ -844,7 +844,7 @@ def test_extract_restores_append_flag(archivers, request): if (platform.get_flags(src_path, st) & stat.UF_APPEND) == 0: pytest.skip("UF_APPEND not settable on this filesystem") # archive and extract - cmd(archiver, "repo-create", "-e", "none-sha256") + cmd(archiver, "repo-create", "-e", "authenticated-sha256") cmd(archiver, "create", "test", "input") with changedir("output"): cmd(archiver, "extract", "test") @@ -864,7 +864,7 @@ def test_extract_flags_errors(archivers, request): raise OSError(errno.EPERM, "EPERM") create_regular_file(archiver.input_path, "file") - cmd(archiver, "repo-create", "-e", "none-sha256") + cmd(archiver, "repo-create", "-e", "authenticated-sha256") cmd(archiver, "create", "test", "input") with changedir("output"): with patch.object(archive_module, "set_flags", patched_set_flags): @@ -914,7 +914,7 @@ def test_do_not_fail_when_percent_is_in_xattr_name(archivers, request): create_regular_file(archiver.input_path, "file") xattr.setxattr(b"input/file", b"user.attribute%p", b"value") - cmd(archiver, "repo-create", "-e", "none-sha256") + cmd(archiver, "repo-create", "-e", "authenticated-sha256") cmd(archiver, "create", "test", "input") with changedir("output"): with patch.object(xattr, "setxattr", patched_setxattr_EACCES): @@ -934,7 +934,7 @@ def test_do_not_fail_when_percent_is_in_file_name(archivers, request): os.makedirs(os.path.join(archiver.input_path, "dir%p")) xattr.setxattr(b"input/dir%p", b"user.attribute", b"value") - cmd(archiver, "repo-create", "-e", "none-sha256") + cmd(archiver, "repo-create", "-e", "authenticated-sha256") cmd(archiver, "create", "test", "input") with changedir("output"): with patch.object(xattr, "setxattr", patched_setxattr_EACCES): @@ -1162,7 +1162,7 @@ def test_extract_write_error_at_flush_is_a_warning(archivers, request): create_regular_file(archiver.input_path, "small1", size=1024) create_regular_file(archiver.input_path, "small2", size=1024) - cmd(archiver, "repo-create", "-e", "none-sha256") + cmd(archiver, "repo-create", "-e", "authenticated-sha256") cmd(archiver, "create", "test", "input") out = _extract_with_raw_file_class(archiver, NoSpaceRaw, BackupOSError) # both files got their warning, i.e. the extraction did not stop at the first one. @@ -1182,7 +1182,7 @@ def test_extract_close_error_is_a_warning(archivers, request): raise OSError(errno.EIO, "Input/output error") create_regular_file(archiver.input_path, "file1", size=1024) - cmd(archiver, "repo-create", "-e", "none-sha256") + cmd(archiver, "repo-create", "-e", "authenticated-sha256") cmd(archiver, "create", "test", "input") out = _extract_with_raw_file_class(archiver, BadCloseRaw, BackupIOError) assert f"input/file1: close: [Errno {errno.EIO}] Input/output error" in out diff --git a/src/borg/testsuite/archiver/key_cmds_test.py b/src/borg/testsuite/archiver/key_cmds_test.py index a7edfa21c..0115dff9e 100644 --- a/src/borg/testsuite/archiver/key_cmds_test.py +++ b/src/borg/testsuite/archiver/key_cmds_test.py @@ -8,7 +8,7 @@ from ...constants import * # NOQA from ...constants import KeyBlobStorage from ...crypto.key import AESOCBKey, CHPOKey, Passphrase, is_keyfile, keyfile_parse from ...crypto.key import RepoKeyNotFoundError -from ...crypto.keymanager import RepoIdMismatch, NotABorgKeyFile, UnencryptedRepo +from ...crypto.keymanager import RepoIdMismatch, NotABorgKeyFile from ...helpers import CommandError from ...helpers import bin_to_hex, hex_to_bin from ...helpers import msgpack @@ -142,24 +142,6 @@ def test_borg_key_file_env_keeps_explicit_path(archivers, request, monkeypatch): assert os.listdir(archiver.keys_path) == [] -@pytest.mark.parametrize("mode", ["none-sha256", "none-blake3"]) -def test_key_management_unavailable_for_keyless_repo(archivers, request, mode): - # the "none-*" modes have no key at all, so there is nothing to export/import. - archiver = request.getfixturevalue(archivers) - cmd(archiver, "repo-create", f"--encryption={mode}") - export_file = archiver.output_path + "/exported" - if archiver.FORK_DEFAULT: - # a forked/binary run returns the exit code, not the exception (BORG_EXIT_CODES defaults - # to "modern", so that is the specific mcode of UnencryptedRepo, not the generic EXIT_ERROR). - cmd(archiver, "key", "export", export_file, exit_code=UnencryptedRepo.exit_mcode) - cmd(archiver, "key", "import", export_file, exit_code=UnencryptedRepo.exit_mcode) - else: - with pytest.raises(UnencryptedRepo): - cmd(archiver, "key", "export", export_file) - with pytest.raises(UnencryptedRepo): - cmd(archiver, "key", "import", export_file) - - def test_key_export_keyfile(archivers, request): archiver = request.getfixturevalue(archivers) export_file = archiver.output_path + "/exported" diff --git a/src/borg/testsuite/archiver/lock_cmds_test.py b/src/borg/testsuite/archiver/lock_cmds_test.py index 6801329f4..2b937fa68 100644 --- a/src/borg/testsuite/archiver/lock_cmds_test.py +++ b/src/borg/testsuite/archiver/lock_cmds_test.py @@ -24,12 +24,13 @@ def test_with_lock(tmp_path): repo_path = tmp_path / "repo" env = os.environ.copy() env["BORG_REPO"] = Path(repo_path).as_uri() + env["BORG_PASSPHRASE"] = "waytooeasyonlyfortests" # nosec B105 # test debug output: print("sys.path: %r" % sys.path) print("PYTHONPATH: %s" % env.get("PYTHONPATH", "")) print("PATH: %s" % env.get("PATH", "")) python = sys.executable or "python3" - command0 = python, "-m", "borg", "repo-create", "--encryption=none-sha256" + command0 = python, "-m", "borg", "repo-create", "--encryption=authenticated-sha256" # Timings must be adjusted so that command1 keeps running while command2 tries to get the lock, # so that lock acquisition for command2 fails as the test expects it. lock_wait = 2 diff --git a/src/borg/testsuite/archiver/mount_cmds_test.py b/src/borg/testsuite/archiver/mount_cmds_test.py index 7b7345253..11e6b4dec 100644 --- a/src/borg/testsuite/archiver/mount_cmds_test.py +++ b/src/borg/testsuite/archiver/mount_cmds_test.py @@ -443,7 +443,7 @@ def test_migrate_lock_alive(archivers, request): # Decorate Lock.migrate_lock = write_assert_data(Lock.migrate_lock) try: - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") create_src_archive(archiver, "arch") mountpoint = os.path.join(archiver.tmpdir, "mountpoint") # In order that the decoration is kept for the borg mount process, we must not spawn, but actually fork; diff --git a/src/borg/testsuite/archiver/recreate_cmd_test.py b/src/borg/testsuite/archiver/recreate_cmd_test.py index 608199650..77829f7c9 100644 --- a/src/borg/testsuite/archiver/recreate_cmd_test.py +++ b/src/borg/testsuite/archiver/recreate_cmd_test.py @@ -67,7 +67,7 @@ def test_recreate_exclude_keep_tagged(archivers, request): @pytest.mark.skipif(not are_hardlinks_supported(), reason="hard links not supported") def test_recreate_hardlinked_tags(archivers, request): # test for issue #4911 archiver = request.getfixturevalue(archivers) - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") create_regular_file( archiver.input_path, "file1", contents=CACHE_TAG_CONTENTS ) # "wrong" filename, but correct tag contents diff --git a/src/borg/testsuite/archiver/repo_create_cmd_test.py b/src/borg/testsuite/archiver/repo_create_cmd_test.py index 9d3e6fc21..530051945 100644 --- a/src/borg/testsuite/archiver/repo_create_cmd_test.py +++ b/src/borg/testsuite/archiver/repo_create_cmd_test.py @@ -46,10 +46,8 @@ def test_repo_create_requires_encryption_option(archivers, request): # the modes that do not encrypt name their id hash themselves, --id-hash does not apply (["--encryption=authenticated-sha256"], "No (repokey, authenticated-sha256)"), (["--encryption=authenticated-blake3"], "No (repokey, authenticated-blake3)"), - (["--encryption=none-sha256"], "No (none-sha256)"), - (["--encryption=none-blake3"], "No (none-blake3)"), # giving the matching --id-hash in addition is accepted - (["--encryption=none-blake3", "--id-hash=blake3"], "No (none-blake3)"), + (["--encryption=authenticated-blake3", "--id-hash=blake3"], "No (repokey, authenticated-blake3)"), ], ) def test_repo_create_encryption_id_hash_combinations(archivers, request, extra_args, expected): @@ -60,13 +58,13 @@ def test_repo_create_encryption_id_hash_combinations(archivers, request, extra_a @pytest.mark.parametrize("mode", ["none", "authenticated"]) -def test_repo_create_rejects_bare_unencrypted_mode_names(archivers, request, mode): - # these modes always name their id hash now, e.g. "none-sha256", see #9104. +def test_repo_create_rejects_unsupported_unencrypted_mode_names(archivers, request, mode): + # there is no "none" mode, and the "authenticated-*" modes always name their id hash, see #9104. archiver = request.getfixturevalue(archivers) cmd(archiver, "repo-create", f"--encryption={mode}", exit_code=2) -@pytest.mark.parametrize("mode", ["none-sha256", "authenticated-blake3"]) +@pytest.mark.parametrize("mode", ["authenticated-sha256", "authenticated-blake3"]) def test_repo_create_rejects_conflicting_id_hash(archivers, request, mode): # the id hash of these modes is part of the mode name, so a contradicting --id-hash is an error. archiver = request.getfixturevalue(archivers) diff --git a/src/borg/testsuite/archiver/return_codes_test.py b/src/borg/testsuite/archiver/return_codes_test.py index 1a42e805c..488d4e3fc 100644 --- a/src/borg/testsuite/archiver/return_codes_test.py +++ b/src/borg/testsuite/archiver/return_codes_test.py @@ -16,7 +16,7 @@ pytest_generate_tests = lambda metafunc: generate_archiver_tests(metafunc, kinds def test_return_codes(archivers, request): archiver = request.getfixturevalue(archivers) - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", "archive", "input") with changedir("output"): cmd(archiver, "extract", "archive") diff --git a/src/borg/testsuite/archiver/tar_cmds_test.py b/src/borg/testsuite/archiver/tar_cmds_test.py index 497fd96cf..946e6cc78 100644 --- a/src/borg/testsuite/archiver/tar_cmds_test.py +++ b/src/borg/testsuite/archiver/tar_cmds_test.py @@ -120,7 +120,7 @@ def test_import_tar(archivers, request, tar_format="PAX"): archiver = request.getfixturevalue(archivers) create_test_files(archiver.input_path, create_hardlinks=False) # hard links become separate files os.unlink("input/flagfile") - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", "src", "input") cmd(archiver, "export-tar", "src", "simple.tar", f"--tar-format={tar_format}") cmd(archiver, "import-tar", "dst", "simple.tar") @@ -182,7 +182,7 @@ def test_import_tar_nfiles(archivers, request): tarinfo.type = tarfile.SYMTYPE tarinfo.linkname = "file1" tar.addfile(tarinfo) - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "import-tar", "dst", "input.tar") info = json.loads(cmd(archiver, "info", "--json", "dst")) # as with borg create, each regular file and each hardlink counts, directories/symlinks do not @@ -197,7 +197,7 @@ def test_import_tar_json(archivers, request): tarinfo = tarfile.TarInfo("dir/file1") tarinfo.size = len(data) tar.addfile(tarinfo, io.BytesIO(data)) - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") stats = json.loads(cmd(archiver, "import-tar", "--json", "dst", "input.tar"))["archive"]["stats"] assert stats["nfiles"] == 1 # fresh repository: all of the file content was new to the repository. @@ -227,7 +227,7 @@ def test_import_tar_digests(archivers, request): tarinfo = tarfile.TarInfo("dir/subdir") tarinfo.type = tarfile.DIRTYPE tar.addfile(tarinfo) - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "import-tar", "--digests=blake3", "--chunker-params=fixed,131072", "dst", "input.tar") digests = tar_item_digests(archiver, "dst") for name, data in contents.items(): @@ -257,7 +257,7 @@ def test_import_unusual_tar(archivers, request): # ./foo//bar # ./ tar_archive = os.path.join(os.path.dirname(__file__), "unusual_paths.tar") - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "import-tar", "dst", tar_archive) files = cmd(archiver, "list", "dst", "--format", "{path}{NL}").splitlines() assert set(files) == {"foobar", "bar", "foo2", "foo/bar", "."} @@ -271,7 +271,7 @@ def test_import_tar_with_dotdot(archivers, request): # Contains this file: # ../../../../etc/shadow tar_archive = os.path.join(os.path.dirname(__file__), "dotdot_path.tar") - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") with pytest.raises(ValueError, match="unexpected '..' element in path '../../../../etc/shadow'"): cmd(archiver, "import-tar", "dst", tar_archive, exit_code=2) @@ -281,7 +281,7 @@ def test_import_tar_gz(archivers, request, tar_format="GNU"): archiver = request.getfixturevalue(archivers) create_test_files(archiver.input_path, create_hardlinks=False) # hard links become separate files os.unlink("input/flagfile") - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", "src", "input") cmd(archiver, "export-tar", "src", "simple.tgz", f"--tar-format={tar_format}") cmd(archiver, "import-tar", "dst", "simple.tgz") @@ -296,7 +296,7 @@ def test_export_import_tar_zst(archivers, request, suffix): archiver = request.getfixturevalue(archivers) create_test_files(archiver.input_path, create_hardlinks=False) # hard links become separate files os.unlink("input/flagfile") - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", "src", "input") cmd(archiver, "export-tar", "src", f"simple.{suffix}") with open(f"simple.{suffix}", "rb") as fd: @@ -314,7 +314,7 @@ def test_export_import_tar_zst_mt(archivers, request, monkeypatch): monkeypatch.setattr("borg.compress._zstd_mt_workers", None) # drop the cache create_test_files(archiver.input_path, create_hardlinks=False) # hard links become separate files os.unlink("input/flagfile") - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", "src", "input") cmd(archiver, "export-tar", "src", "simple.tar.zst") cmd(archiver, "import-tar", "dst", "simple.tar.zst") @@ -345,7 +345,7 @@ def test_tar_filter_zstd_external(archivers, request): archiver = request.getfixturevalue(archivers) create_test_files(archiver.input_path, create_hardlinks=False) # hard links become separate files os.unlink("input/flagfile") - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", "src", "input") cmd(archiver, "export-tar", "src", "simple.tar.zst", "--tar-filter=zstd") with open("simple.tar.zst", "rb") as fd: @@ -375,7 +375,7 @@ def test_import_concatenated_tar_with_ignore_zeros(archivers, request): # Clean up for assert_dirs_equal. os.unlink("the_rest.tar") - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "import-tar", "--ignore-zeros", "dst", "input/concatenated.tar") # Clean up for assert_dirs_equal. os.unlink("input/concatenated.tar") @@ -400,7 +400,7 @@ def test_import_concatenated_tar_without_ignore_zeros(archivers, request): with open("the_rest.tar", "rb") as the_rest: concatenated.write(the_rest.read()) os.unlink("the_rest.tar") - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "import-tar", "dst", "input/concatenated.tar") with changedir(archiver.output_path): @@ -428,7 +428,7 @@ def test_import_tar_with_dotslash_paths(archivers, request): assert "./dir/file" in tar_content # Import the tar file into a Borg repository - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "import-tar", "dotslash", "input/dotslash.tar") # List the archive contents and verify no paths start with './' @@ -442,7 +442,7 @@ def test_roundtrip_pax_borg(archivers, request): archiver = request.getfixturevalue(archivers) create_test_files(archiver.input_path) os.remove("input/flagfile") # this would be automagically excluded due to NODUMP - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", "src", "input") cmd(archiver, "export-tar", "src", "simple.tar", "--tar-format=BORG") cmd(archiver, "import-tar", "dst", "simple.tar") @@ -459,7 +459,7 @@ def test_roundtrip_pax_xattrs(archivers, request): original_path = os.path.join(archiver.input_path, "file") xa_key, xa_value = b"user.xattrtest", b"not valid utf-8: \xff" xattr.setxattr(original_path.encode(), xa_key, xa_value) - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", "src", "input") cmd(archiver, "export-tar", "src", "xattrs.tar", "--tar-format=PAX") cmd(archiver, "import-tar", "dst", "xattrs.tar") @@ -557,7 +557,7 @@ def _create_sparse_test_input(input_path): def test_export_tar_sparse(archivers, request, chunker_params): archiver = request.getfixturevalue(archivers) contents = _create_sparse_test_input(archiver.input_path) - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", chunker_params, "test", "input") cmd(archiver, "export-tar", "test", "dense.tar") cmd(archiver, "export-tar", "--sparse", "test", "sparse.tar", "--progress") @@ -584,7 +584,7 @@ def test_export_tar_sparse(archivers, request, chunker_params): def test_export_tar_sparse_roundtrip(archivers, request, tar_format, chunker_params): archiver = request.getfixturevalue(archivers) _create_sparse_test_input(archiver.input_path) - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", chunker_params, "src", "input") cmd(archiver, "export-tar", "--sparse", f"--tar-format={tar_format}", "src", "sparse.tar") cmd(archiver, "import-tar", "dst", "sparse.tar") @@ -598,7 +598,7 @@ def test_export_tar_sparse_not_worthwhile(archivers, request): archiver = request.getfixturevalue(archivers) contents = b"X" * 448 + b"\0" * 64 create_regular_file(archiver.input_path, "tinyhole", contents=contents) - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", "--chunker-params=fixed,64", "test", "input") cmd(archiver, "export-tar", "--sparse", "test", "sparse.tar") with tarfile.open("sparse.tar") as tar: @@ -611,7 +611,7 @@ def test_export_tar_sparse_not_worthwhile(archivers, request): def test_export_tar_sparse_gnutar(archivers, request): archiver = request.getfixturevalue(archivers) _create_sparse_test_input(archiver.input_path) - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", SPARSE_CHUNKER_CDC, "test", "input") cmd(archiver, "export-tar", "--sparse", "test", "sparse.tar") with changedir("output"): @@ -629,7 +629,7 @@ def test_export_tar_sparse_hardlinks(archivers, request): contents = b"\0" * 2 * 65536 + b"data" create_regular_file(archiver.input_path, "sparse1", contents=contents) os.link(os.path.join(archiver.input_path, "sparse1"), os.path.join(archiver.input_path, "sparse2")) - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", SPARSE_CHUNKER_CDC, "src", "input") cmd(archiver, "export-tar", "--sparse", "src", "sparse.tar") with tarfile.open("sparse.tar") as tar: @@ -656,7 +656,7 @@ def test_export_tar_sparse_strip_components(archivers, request): archiver = request.getfixturevalue(archivers) contents = b"\0" * 2 * 65536 + b"end" create_regular_file(archiver.input_path, "dir/sparsefile", contents=contents) - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", SPARSE_CHUNKER_CDC, "test", "input") cmd(archiver, "export-tar", "--sparse", "--strip-components=1", "test", "sparse.tar") with tarfile.open("sparse.tar") as tar: @@ -691,7 +691,7 @@ def test_export_tar_sparse_base256_size(archivers, request, monkeypatch): pytest.skip("monkeypatching does not reach a borg binary") monkeypatch.setattr(SparseTarInfo, "octal_size_limit", 1) contents = _create_sparse_test_input(archiver.input_path) - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", SPARSE_CHUNKER_CDC, "src", "input") cmd(archiver, "export-tar", "--sparse", "src", "sparse.tar") with tarfile.open("sparse.tar") as tar: @@ -714,7 +714,7 @@ def test_export_tar_sparse_gnu_format_error(archivers, request): # --sparse requires a PAX-based tar format, the GNU format cannot store the sparse headers. archiver = request.getfixturevalue(archivers) create_regular_file(archiver.input_path, "file", contents=b"x") - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", "test", "input") if archiver.FORK_DEFAULT: output = cmd(archiver, "export-tar", "--sparse", "--tar-format=GNU", "test", "out.tar", exit_code=2) @@ -775,7 +775,7 @@ def test_acl_roundtrip(archivers, request): pytest.skip("ACLs not supported or not working correctly") # 2. Create a Borg archive - cmd(archiver, "repo-create", "--encryption=none-sha256") + cmd(archiver, "repo-create", "--encryption=authenticated-sha256") cmd(archiver, "create", "original", "input") # 3. export-tar this archive to a tar file diff --git a/src/borg/testsuite/benchmark_test.py b/src/borg/testsuite/benchmark_test.py index fa798eb17..2be77f6e8 100644 --- a/src/borg/testsuite/benchmark_test.py +++ b/src/borg/testsuite/benchmark_test.py @@ -27,7 +27,7 @@ def repo_url(request, tmpdir, monkeypatch): tmpdir.remove(rec=1) -@pytest.fixture(params=["none-sha256", "aes256-ocb"]) +@pytest.fixture(params=["authenticated-sha256", "aes256-ocb"]) def repo(request, cmd_fixture, repo_url): cmd_fixture(f"--repo={repo_url}", "repo-create", "--encryption", request.param) return repo_url diff --git a/src/borg/testsuite/cockpit_test.py b/src/borg/testsuite/cockpit_test.py index d2eeba240..be1a30d7a 100644 --- a/src/borg/testsuite/cockpit_test.py +++ b/src/borg/testsuite/cockpit_test.py @@ -15,7 +15,7 @@ except ImportError: pytestmark = pytest.mark.skipif(not have_cockpit, reason="can not import BorgCockpitApp, is textual installed?") -def test_cockpit_app_create_archive(tmp_path): +def test_cockpit_app_create_archive(tmp_path, monkeypatch): if not (is_freebsd or is_win32): pytest.skip("this slow test shall only run on FreeBSD and Windows") repo_path = tmp_path / "repo" @@ -24,7 +24,8 @@ def test_cockpit_app_create_archive(tmp_path): for i in range(5000): (input_path / f"test{i}.txt").write_text(f"content {i}") - subprocess.run(["borg", "-r", str(repo_path), "repo-create", "--encryption", "none-sha256"], check=True) + monkeypatch.setenv("BORG_PASSPHRASE", "waytooeasyonlyfortests") # also for the borg the app runs + subprocess.run(["borg", "-r", str(repo_path), "repo-create", "--encryption", "authenticated-sha256"], check=True) async def run(): app = BorgCockpitApp() diff --git a/src/borg/testsuite/crypto/crypto_test.py b/src/borg/testsuite/crypto/crypto_test.py index d071a10f9..5fb7be50c 100644 --- a/src/borg/testsuite/crypto/crypto_test.py +++ b/src/borg/testsuite/crypto/crypto_test.py @@ -9,7 +9,7 @@ from ...crypto.low_level import bytes_to_long, bytes_to_int, long_to_bytes from ...crypto.low_level import hmac_sha256, argon2_hash from ...legacy.crypto.low_level import AES from hashlib import sha256 -from ...crypto.key import CHPOKey, AESOCBKey, KeyBase, ChecksumKey +from ...crypto.key import CHPOKey, AESOCBKey, KeyBase, LegacyPlaintextKey from ...legacy.crypto.key import AESCTRKey as LegacyAESCTRKey from ...helpers import msgpack, bin_to_hex @@ -327,11 +327,11 @@ class TestDeriveKey(BaseTestCase): self.id_key = id_key def test_derive_key_with_plaintext_key(self): - """Test derive_key with ChecksumKey (empty crypt_key)""" - key = ChecksumKey(None) + """Test derive_key with the borg 1.x PlaintextKey (empty crypt_key)""" + key = LegacyPlaintextKey(None) salt, domain, size = b"salt", b"domain", 16 - # ChecksumKey has an empty crypt_key, so the derived key should be based on salt and domain only + # PlaintextKey has an empty crypt_key, so the derived key should be based on salt and domain only derived_key = key.derive_key(salt=salt, domain=domain, size=size) expected = sha256(b"" + salt + domain).digest()[:size] self.assert_equal(derived_key, expected) diff --git a/src/borg/testsuite/crypto/key_test.py b/src/borg/testsuite/crypto/key_test.py index 930e6f1ea..63252c809 100644 --- a/src/borg/testsuite/crypto/key_test.py +++ b/src/borg/testsuite/crypto/key_test.py @@ -6,7 +6,7 @@ from unittest.mock import MagicMock import pytest from ...crypto.key import BLAKE3_MT_THRESHOLD_KIB, get_blake3_mt_threshold -from ...crypto.key import ChecksumKey, Blake3ChecksumKey, keyfile_parse +from ...crypto.key import keyfile_parse from ...crypto.key import AuthenticatedKey, Blake3AuthenticatedKey from ...crypto.key import AESCTRKey, Blake2AESCTRKey, Blake2AuthenticatedKey from ...crypto.key import LegacyAuthenticatedKey @@ -91,8 +91,6 @@ class TestKey: # keyfile and repokey are no longer separate classes (storage is a per-key property), # so each crypto suite appears once here. # not encrypted, but tagged - ChecksumKey, - Blake3ChecksumKey, AuthenticatedKey, Blake3AuthenticatedKey, # legacy crypto (read-only, borg 1.x) @@ -132,21 +130,6 @@ class TestKey: # is storage-agnostic now and always probes repo candidates, even for keyfile keys. return getattr(self, "key_data", b"") - def test_none_sha256(self): - key = ChecksumKey.create(None, None) - chunk = b"foo" - id = key.id_hash(chunk) - # the chunk id of the "none-*" modes is the plain (unkeyed) hash of the chunk - assert bin_to_hex(id) == "2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae" - assert chunk == key.decrypt(id, key.encrypt(id, chunk)) - - def test_none_blake3(self): - key = Blake3ChecksumKey.create(None, None) - chunk = b"foo" - id = key.id_hash(chunk) - assert bin_to_hex(id) == "04e0bb39f30b1a3feb89f536c93be15055482df748674b00d26e5a75777702e9" - assert chunk == key.decrypt(id, key.encrypt(id, chunk)) - def test_keyfile(self, monkeypatch, keys_dir): monkeypatch.setenv("BORG_PASSPHRASE", "test") key = AESCTRKey.create(self.MockRepository(), self.kf_args()) @@ -401,9 +384,7 @@ class TestMACEnvelope: CRYPT_KEY = bytes(range(64)) ID_KEY = bytes(range(100, 132)) - ALL_CLASSES = (ChecksumKey, Blake3ChecksumKey, AuthenticatedKey, Blake3AuthenticatedKey) - KEYED_CLASSES = (AuthenticatedKey, Blake3AuthenticatedKey) - UNKEYED_CLASSES = (ChecksumKey, Blake3ChecksumKey) + ALL_CLASSES = (AuthenticatedKey, Blake3AuthenticatedKey) class MockRepository: id = bytes(32) @@ -411,8 +392,7 @@ class TestMACEnvelope: def make_key(self, cls, crypt_key=None): key = cls(self.MockRepository()) - if cls.has_secret_key: - key.init_from_given_data(crypt_key=crypt_key or self.CRYPT_KEY, id_key=self.ID_KEY, chunk_seed=0) + key.init_from_given_data(crypt_key=crypt_key or self.CRYPT_KEY, id_key=self.ID_KEY, chunk_seed=0) return key @pytest.fixture(params=ALL_CLASSES) @@ -439,8 +419,6 @@ class TestMACEnvelope: def test_format_is_stable(self, cls): # golden vectors: these bytes must not change silently, they are an on-disk format. expected = { - ChecksumKey: "80001ab01692fad0f0b89f27983cbf51b1bd20ab73de736aea83363b749ee6ba314f", - Blake3ChecksumKey: "90008dec559eda6de95536f198f6b54b3b2728678c94ca1b3f0b77cf37e5b11c9b7c", AuthenticatedKey: "600033ecaaf8c34d4142fa502278986c8f49efbcbf879de16d3c2767e1252bfb1994", Blake3AuthenticatedKey: "70006de48e2139f8995790ec81b256e87f40ef5810d140d28396a8d07734ab05a358", }[cls] @@ -449,18 +427,8 @@ class TestMACEnvelope: envelope = key.encrypt(key.id_hash(plaintext), plaintext, aad=b"") assert bin_to_hex(envelope) == expected + bin_to_hex(plaintext) - @pytest.mark.parametrize("cls", UNKEYED_CLASSES) - def test_unkeyed_modes_are_repo_independent(self, cls): - # no key material at all, so any two repositories of such a mode store identical objects - # for identical input - that is what allows deduplicating them on the filesystem level. - key1, key2 = self.make_key(cls), self.make_key(cls) - payload = b"payload" - id = key1.id_hash(payload) - assert key1.id_hash(payload) == key2.id_hash(payload) - assert key1.encrypt(id, payload, aad=b"aad") == key2.encrypt(id, payload, aad=b"aad") - - @pytest.mark.parametrize("cls", KEYED_CLASSES) - def test_keyed_modes_depend_on_crypt_key(self, cls): + @pytest.mark.parametrize("cls", ALL_CLASSES) + def test_tag_depends_on_crypt_key(self, cls): # the tag key is derived from crypt_key: same crypt_key -> same objects (that is what # "repo-create --other-repo --copy-crypt-key" gives), different crypt_key -> different tag. payload = b"payload" @@ -512,20 +480,9 @@ class TestMACEnvelope: assert not key.logically_encrypted assert key.IDHASH_IN_ENC_NAME assert key.ENC_NAME.endswith("-" + key.IDHASH_NAME) - if cls in self.KEYED_CLASSES: - assert key.has_secret_key - # the envelope tag authenticates every read, so verifying the chunk id on top of that - # is optional (see BORG_ASSERT_ID), like for the AEAD modes. - assert not key.id_check_is_authentication - assert key.STORAGE == KeyBlobStorage.REPO - assert key.LOCATION_CONFIGURABLE - else: - assert not key.has_secret_key - # the checksum can be recomputed by anybody, thus it is no authentication: the chunk id - # check is the only one left and must never be skipped. - assert key.id_check_is_authentication - assert key.STORAGE == KeyBlobStorage.NO_STORAGE - assert not key.LOCATION_CONFIGURABLE + assert key.has_secret_key + assert key.STORAGE == KeyBlobStorage.REPO + assert key.LOCATION_CONFIGURABLE @pytest.mark.parametrize("cls", ALL_CLASSES) def test_type_bytes_are_distinct(self, cls): @@ -535,7 +492,7 @@ class TestMACEnvelope: assert identify_key(envelope) is cls def test_authenticated_no_key_workaround(self, monkeypatch): - # without the key material, the keyed modes can not verify the tag - but they can still + # without the key material, these modes can not verify the tag - but they can still # read the data (that is the point of the workaround, see BORG_WORKAROUNDS). key = self.make_key(AuthenticatedKey) payload = b"payload" @@ -545,7 +502,7 @@ class TestMACEnvelope: monkeypatch.setattr("borg.crypto.key.AUTHENTICATED_NO_KEY", True) assert key.decrypt(id, bytes(envelope), aad=b"aad") == payload - @pytest.mark.parametrize("cls", KEYED_CLASSES) + @pytest.mark.parametrize("cls", ALL_CLASSES) def test_authenticated_no_key_key_gone(self, cls, monkeypatch, tmp_path): # a completely lost borg key (no repokey object, no keyfile) is also covered by the # workaround: detect() proceeds with fake key material instead of failing, see #10238. @@ -568,18 +525,6 @@ class TestMACEnvelope: # decrypt() skips the verification, so reading works. assert key.decrypt(id, envelope, aad=b"aad") == payload - def test_unkeyed_modes_verify_despite_workaround(self, monkeypatch): - # the unkeyed modes need no key material, so the workaround must not switch their - # checksum verification off. - monkeypatch.setattr("borg.crypto.key.AUTHENTICATED_NO_KEY", True) - key = self.make_key(ChecksumKey) - payload = b"payload" - id = key.id_hash(payload) - envelope = bytearray(key.encrypt(id, payload, aad=b"aad")) - envelope[5] ^= 1 - with pytest.raises(IntegrityError): - key.decrypt(id, bytes(envelope), aad=b"aad") - @pytest.mark.parametrize("cls", (LegacyAuthenticatedKey, Blake2AuthenticatedKey)) def test_legacy_authenticated_no_key_key_gone(cls, monkeypatch, tmp_path): @@ -674,5 +619,5 @@ def test_key_class_of_needs_key_info(): key_class_of(repository) with pytest.raises(RepositoryKeyInfoMissing): key_factory(repository) - repository = MagicMock(encryption="none-sha256", id_hash="sha256") - assert key_class_of(repository) is ChecksumKey + repository = MagicMock(encryption="authenticated-sha256", id_hash="sha256") + assert key_class_of(repository) is AuthenticatedKey diff --git a/src/borg/testsuite/repoobj_test.py b/src/borg/testsuite/repoobj_test.py index bd582a374..60a78a8f3 100644 --- a/src/borg/testsuite/repoobj_test.py +++ b/src/borg/testsuite/repoobj_test.py @@ -1,8 +1,8 @@ import pytest from ..constants import ROBJ_FILE_STREAM, ROBJ_ARCHIVE_META -from ..crypto.key import AESOCBKey, ChecksumKey, AuthenticatedKey, CHPOKey, LegacyPlaintextKey -from ..helpers import CompressionSpec, msgpack +from ..crypto.key import AESOCBKey, AuthenticatedKey, CHPOKey, LegacyAuthenticatedKey, LegacyPlaintextKey +from ..helpers import CompressionSpec from ..helpers.errors import Error, IntegrityError from ..repository import Repository from ..repoobj import ( @@ -16,6 +16,7 @@ from ..repoobj import ( ) from ..legacy.repoobj import RepoObj1 from ..compress import LZ4 +from . import make_test_key # offsets of object header fields. CHUNK_ID_OFFSET = len(OBJ_MAGIC) + 1 # after the magic and the version byte @@ -30,8 +31,8 @@ def repository(tmpdir): @pytest.fixture def key(repository): - # "none-sha256" mode: not encrypted and not authenticated, but checksummed. - return ChecksumKey(repository) + # "authenticated-sha256" mode: not encrypted, but the envelope is MAC-authenticated. + return make_test_key(repository) @pytest.fixture @@ -40,15 +41,6 @@ def legacy_key(repository): return LegacyPlaintextKey(repository) -@pytest.fixture -def authenticated_key(repository): - # "authenticated-sha256" mode: not encrypted, but the envelope is MAC-authenticated. - key = AuthenticatedKey(repository) - key.init_from_random_data() - key.init_ciphers() - return key - - @pytest.fixture def aead_key(repository): key = CHPOKey(repository) @@ -100,10 +92,13 @@ def test_format_parse_roundtrip_borg1(legacy_key): # legacy assert edata.startswith(bytes((key.TYPE, compressor.ID, compressor.level))) -def test_borg1_borg2_transition(legacy_key, key): +def test_borg1_borg2_transition(repository, key): # Borg transfer reads Borg 1.x repository objects (without decompressing them), # and writes Borg 2 repository objects (providing already-compressed data to avoid recompression). - # The borg 1.x "none" and the borg 2 "none-sha256" mode use the same (unkeyed sha256) chunk ids. + # The borg 1.x "authenticated" and the borg 2 "authenticated-sha256" mode use the same (hmac-sha256) + # chunk ids, if the borg 2 repository is a related one, i.e. has the same id key. + legacy_key = LegacyAuthenticatedKey(repository) + legacy_key.id_key = key.id_key meta = {} # borg1 does not support this kind of metadata data = b"foobar" * 10 len_data = len(data) @@ -195,10 +190,10 @@ def _tamper(cdata, offset): return bytes(tampered) -@pytest.fixture(params=["key", "authenticated_key", "aead_key"]) +@pytest.fixture(params=["key", "aead_key"]) def protected_key(request): - # every borg 2 mode protects the object with a tag over the payload and the AAD - an unkeyed - # checksum for "none-*", a MAC for "authenticated-*", the AEAD tag for the encrypted modes. + # every borg 2 mode protects the object with a tag over the payload and the AAD - a MAC for + # "authenticated-*", the AEAD tag for the encrypted modes. return request.getfixturevalue(request.param) @@ -269,29 +264,6 @@ def test_tampered_data_detected(protected_key): repo_objs.parse(id, tampered, ro_type=ROBJ_FILE_STREAM) -def test_checksum_mode_does_not_authenticate(key): - # The honest limit of the "none-*" modes: their tag is an unkeyed checksum, so somebody who - # modifies an object can just recompute it. Detecting that needs a secret - the - # "authenticated-*" and the encrypted modes have one, this mode does not. - repo_objs = RepoObj(key) - data = b"foobar" * 10 - id = repo_objs.id_hash(data) - cdata = repo_objs.format(id, {"custom": "something"}, data, ro_type=ROBJ_FILE_STREAM) - - hdr_size = RepoObj.obj_header.size - hdr = RepoObj.ObjHeader(*RepoObj.obj_header.unpack(cdata[:hdr_size])) - header_aad = OBJ_MAGIC + bytes([OBJ_VERSION]) + id - # rewrite the meta slot with attacker-chosen content and a recomputed checksum - forged_meta = dict(repo_objs.parse_meta(id, cdata, ro_type=ROBJ_FILE_STREAM), custom="forged") - forged_meta_slot = key.encrypt(id, msgpack.packb(forged_meta), aad=header_aad + b"M") - forged = ( - RepoObj.obj_header.pack(hdr.magic, hdr.version, hdr.chunk_id, len(forged_meta_slot), hdr.data_size) - + forged_meta_slot - + cdata[hdr_size + hdr.meta_size :] - ) - assert repo_objs.parse_meta(id, forged, ro_type=ROBJ_FILE_STREAM)["custom"] == "forged" - - def test_header_aad_tamper_detected_at_key_layer(protected_key): # Calls key.encrypt()/key.decrypt() directly with header_aad, to check that every byte of # header_aad (magic, version, chunk_id) is authenticated, not just chunk_id. @@ -440,27 +412,11 @@ def test_assert_id_places_env_var_mandatory_place(aead_key, monkeypatch): assert "verify_data: always verifies, can not be configured" in str(exc_info.value) -def test_assert_id_never_skipped_for_unauthenticated_key(key, monkeypatch): - # ChecksumKey ("none-sha256" mode): the envelope checksum is unkeyed and thus no authentication, - # so the id check is the only integrity check reads have and must happen no matter what the - # user configured. - monkeypatch.setenv("BORG_ASSERT_ID", "") # verify nowhere - but this is not switchable off - assert key.id_check_is_authentication - repo_objs = RepoObj(key) - id = repo_objs.id_hash(b"foobar" * 10) - cdata = wrong_content_object(repo_objs, id) - - for place in ASSERT_ID_PLACES: - with pytest.raises(IntegrityError): - repo_objs.parse(id, cdata, ro_type=ROBJ_FILE_STREAM, assert_id_place=place) - - -def test_assert_id_configurable_for_authenticated_key(authenticated_key, monkeypatch): +def test_assert_id_configurable_for_authenticated_key(key, monkeypatch): # AuthenticatedKey ("authenticated-sha256"): the envelope MAC authenticates every read (the # chunk id is in the AAD), so verifying the id on top of that is configurable, like for AEAD. monkeypatch.setenv("BORG_ASSERT_ID", "") # verify at none of the configurable places - assert not authenticated_key.id_check_is_authentication - repo_objs = RepoObj(authenticated_key) + repo_objs = RepoObj(key) id = repo_objs.id_hash(b"foobar" * 10) cdata = wrong_content_object(repo_objs, id) @@ -481,14 +437,13 @@ def validator_input(repo_objs, data): return chunk_id, obj[: hdr_size + meta_size] -@pytest.mark.parametrize("key_class", [ChecksumKey, AuthenticatedKey, CHPOKey, AESOCBKey]) +@pytest.mark.parametrize("key_class", [AuthenticatedKey, CHPOKey, AESOCBKey]) def test_object_validator_checks_the_sizes_for_every_envelope(key_class): # data_size == csize + the envelope overhead must hold for each key family; changing meta_size # or data_size must fail validation. key = key_class(None) - if hasattr(key, "init_from_random_data"): - key.init_from_random_data() - key.init_ciphers() + key.init_from_random_data() + key.init_ciphers() repo_objs = RepoObj(key) chunk_id, head = validator_input(repo_objs, b"payload" * 100) validate = object_validator(repo_objs) @@ -501,9 +456,9 @@ def test_object_validator_checks_the_sizes_for_every_envelope(key_class): @pytest.mark.parametrize("meta", [b"not a mapping", 42, [1, 2, 3], {}, {"csize": "17"}, {"csize": True}]) def test_object_validator_rejects_metadata_without_a_usable_csize(monkeypatch, meta): - # in the modes that authenticate without a secret key the metadata slot can unpack to any - # value. Each of these must come back as "invalid", not as an exception out of the validator. - repo_objs = RepoObj(ChecksumKey(None)) + # if the tag is not verified (authenticated_no_key workaround), the metadata slot can unpack to + # any value. Each of these must come back as "invalid", not as an exception out of the validator. + repo_objs = RepoObj(make_test_key()) chunk_id, head = validator_input(repo_objs, b"payload" * 100) monkeypatch.setattr(repo_objs, "parse_meta", lambda *args, **kwargs: meta) assert not object_validator(repo_objs)(chunk_id, head) @@ -512,7 +467,7 @@ def test_object_validator_rejects_metadata_without_a_usable_csize(monkeypatch, m def test_object_validator_propagates_an_unexpected_exception(monkeypatch): # answering "invalid" for a bug would answer it for every object of every pack, so anything # other than a failed authentication or unpacking reaches the caller. - repo_objs = RepoObj(ChecksumKey(None)) + repo_objs = RepoObj(make_test_key()) chunk_id, head = validator_input(repo_objs, b"payload" * 100) def parse_meta_raising_valueerror(*args, **kwargs): @@ -528,7 +483,7 @@ def test_object_validator_accepts_every_compression(compression): # data_size == csize + the envelope overhead is what pins data_size, so every compressor must # record the whole payload it produced as csize. The obfuscating one pads the payload and # records the padded size. - repo_objs = RepoObj(ChecksumKey(None)) + repo_objs = RepoObj(make_test_key()) repo_objs.compressor = CompressionSpec(compression).compressor chunk_id, head = validator_input(repo_objs, b"payload" * 100) assert object_validator(repo_objs)(chunk_id, head) diff --git a/src/borg/testsuite/repository_test.py b/src/borg/testsuite/repository_test.py index 80be75677..7e498131a 100644 --- a/src/borg/testsuite/repository_test.py +++ b/src/borg/testsuite/repository_test.py @@ -14,12 +14,13 @@ from .. import repository as repository_module from ..cache import chunkindex_is_invalid, delete_chunkindex_from_repo, write_chunkindex_invalid from ..compress import CNONE from ..constants import MAX_CLOCK_SKEW, ROBJ_FILE_STREAM -from ..crypto.key import CHPOKey, ChecksumKey +from ..crypto.key import CHPOKey from ..helpers import IntegrityError, Location, bin_to_hex from ..hashindex import ChunkIndex, ChunkIndexEntry from ..repository import Repository, MAX_DATA_SIZE, MAX_VALIDATED_META_SIZE, propagate_rsh, rest_serve_command from ..repository import PackWriter, PackReader, PackTracker, superseded_gap_ranges from ..repoobj import RepoObj, OBJ_MAGIC, OBJ_VERSION, object_validator +from . import make_test_key from .hashindex_test import H from .repoobj_test import CHUNK_ID_OFFSET, DATA_SIZE_OFFSET, META_SIZE_OFFSET @@ -2346,10 +2347,10 @@ def test_pack_reader_resync_recovers_from_size_past_the_pack_end(): ] -def none_repo_objs(): - # a RepoObj with a "none-*" key and no compression: it formats real objects (tagged metadata +def plain_repo_objs(): + # a RepoObj with an "authenticated-*" key and no compression: it formats real objects (tagged metadata # slot, csize) and stores their payload as it is. - repo_objs = RepoObj(ChecksumKey(None)) + repo_objs = RepoObj(make_test_key(None)) repo_objs.compressor = CNONE() return repo_objs @@ -2363,7 +2364,7 @@ def real_chunk(repo_objs, data): def test_pack_reader_reads_no_further_than_the_pack_end(): # the walk asks for META_READ_SIZE bytes per header, so the last objects of a pack have fewer # bytes left than that. Every read stays inside the pack and asks for at least a header. - repo_objs = none_repo_objs() + repo_objs = plain_repo_objs() id1, obj1 = real_chunk(repo_objs, b"payload-one") id2, obj2 = real_chunk(repo_objs, b"payload-two") pack = bytes(obj1) + bytes(obj2) @@ -2386,7 +2387,7 @@ def test_pack_reader_resync_rejects_a_header_with_a_wrong_data_size(shape): # pack leaves the header parseable, so the walk checks it against csize from the tagged metadata # and drops obj1. The shapes are where the wrong size points: into obj3, exactly onto obj3's # header, and back into obj1 itself. - repo_objs = none_repo_objs() + repo_objs = plain_repo_objs() _, obj1 = real_chunk(repo_objs, b"A" * 100) id2, obj2 = real_chunk(repo_objs, b"B" * 100) id3, obj3 = real_chunk(repo_objs, b"C" * 100) @@ -2407,7 +2408,7 @@ def test_pack_reader_resync_rejects_a_header_with_a_wrong_data_size(shape): def test_pack_reader_resync_rejects_a_header_with_a_wrong_meta_size(): # the metadata slot's tag covers the slot, so a wrong meta_size fails it. - repo_objs = none_repo_objs() + repo_objs = plain_repo_objs() _, obj1 = real_chunk(repo_objs, b"A" * 100) id2, obj2 = real_chunk(repo_objs, b"B" * 100) obj1 = bytearray(obj1) @@ -2418,7 +2419,7 @@ def test_pack_reader_resync_rejects_a_header_with_a_wrong_meta_size(): def test_pack_reader_resync_rejects_a_header_with_a_wrong_chunk_id(): # the chunk id is part of the AAD of the metadata slot's tag. - repo_objs = none_repo_objs() + repo_objs = plain_repo_objs() _, obj1 = real_chunk(repo_objs, b"A" * 100) id2, obj2 = real_chunk(repo_objs, b"B" * 100) obj1 = bytearray(obj1) @@ -2428,10 +2429,10 @@ def test_pack_reader_resync_rejects_a_header_with_a_wrong_chunk_id(): def test_pack_reader_resync_starts_past_the_last_validated_object(): - # "none-*" mode, no compression: obj1's payload holds a complete, intact object - a user backed + # "authenticated-*" mode, no compression: obj1's payload holds a complete, intact object - a user backed # up such bytes. obj2's header is broken. The walk validated obj1, so the scan starts past it # and never sees the decoy; obj1 is kept. - repo_objs = none_repo_objs() + repo_objs = plain_repo_objs() _, decoy = real_chunk(repo_objs, b"decoy") id1, obj1 = real_chunk(repo_objs, b"A" * 30 + decoy + b"A" * 30) assert decoy in obj1 # the decoy is in the pack verbatim @@ -2528,13 +2529,14 @@ def test_pack_reader_resync_accepts_an_object_with_corrupt_data(tmp_path): repo_objs.parse(real_id, bytes(obj2), ro_type=ROBJ_FILE_STREAM) -def test_pack_reader_resync_rejects_damaged_user_content_without_a_key(tmp_path): - # In "none-*" mode with no compression, user content lands in the pack as it is, so a backed up - # file can contain something shaped like an object. The metadata slot's checksum covers the +def test_pack_reader_resync_rejects_damaged_user_content_without_encryption(tmp_path): + # In "authenticated-*" mode with no compression, user content lands in the pack as it is, so a + # backed up file can contain something shaped like an object. The metadata slot's tag covers the # object header, so damaged candidate bytes are still ruled out - what these modes can not rule - # out is an intact object put into a file on purpose, there being no secret to tell them apart. + # out is an intact object (written with the same key) put into a file on purpose: the tag is + # deterministic and binds an object to its chunk id alone. repository = Repository(str(tmp_path / "repo"), create=True) - repo_objs = RepoObj(ChecksumKey(repository)) + repo_objs = RepoObj(make_test_key(repository)) repo_objs.compressor = CNONE() decoy = bytearray(repo_objs.format(repo_objs.id_hash(b"decoy"), {}, b"decoy", ro_type=ROBJ_FILE_STREAM)) hdr = RepoObj.ObjHeader(*RepoObj.obj_header.unpack(bytes(decoy[: RepoObj.obj_header.size])))