borgbackup/borg
1
0
Fork 0
mirror of https://github.com/borgbackup/borg.git synced 2026-10-12 02:11:36 +00:00
Code Issues Packages Projects Releases Wiki Activity

remove the none-sha256 and none-blake3 encryption modes

Browse Source 
Every borg 2 mode has a borg key now: the AEAD modes (aes256-ocb,
chacha20-poly1305) and the authenticated-sha256 / authenticated-blake3
modes. There is no compatibility with repositories older borg 2 betas
created in a none-* mode.

- crypto.key: remove ChecksumKeyBase, ChecksumKey, Blake3ChecksumKey and
  the SHA256NONE / BLAKE3NONE key types.
- remove KeyBase.id_check_is_authentication: all borg 2 envelopes
  authenticate the chunk id, so RepoObj.parse verifies the id only at
  the BORG_ASSERT_ID places and the mandatory ones.
- repo-create: always show the key / passphrase hint.
- a borg 1.x "none" repository has unkeyed chunk ids, which no borg 2
  mode has, so transferring it needs --chunker-params (docs updated).
- tests: use authenticated-sha256 where none-sha256 was used as the
  cheap default mode, add make_test_key() for tests needing some key.
  The repository swap detection tests now re-label the repokey of the
  attacker's repository for the faked repository id.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
Thomas Waldmann
2026-09-18 01:17:09 +02:00
co-authored by Claude Fable 5.1
parent 9614abb2c4
commit b724ece6b8
35 changed files with 262 additions and 522 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.rst
+1 -2
View File
@@ -81,8 +81,7 @@ Main features
authenticity.
If you do not need confidentiality, there are also modes that only
authenticate (``authenticated-sha256`` / ``authenticated-blake3``) and
modes that do neither (``none-sha256`` / ``none-blake3``).
authenticate (``authenticated-sha256`` / ``authenticated-blake3``).
**Hashing**
You can choose between HMAC-SHA256 and Blake3.
docs/internals/data-structures.rst
+17 -28
View File
@@ -122,15 +122,11 @@ function is selected via ``borg repo-create --id-hash``, independently of
``--encryption``. It is a keyed MAC over the plaintext (keyed by ``id_key``):
``sha256`` selects HMAC-SHA256, ``blake3`` selects a keyed BLAKE3.
For the modes **without encryption**, the id hash is what protects the data, so
it is part of the mode name and not separately selectable (giving ``--id-hash``
in addition is only accepted if it agrees with the mode name):
- ``authenticated-sha256`` / ``authenticated-blake3`` have key material and thus
use the same keyed MACs as the encrypted modes.
- ``none-sha256`` / ``none-blake3`` have no key at all, so the id is a plain
SHA-256 resp. BLAKE3 hash of the plaintext. All repositories of such a mode
therefore deduplicate identically, see :ref:`tagged_envelope`.
For the modes **without encryption** (``authenticated-sha256`` /
``authenticated-blake3``), the id hash is what protects the data, so it is part
of the mode name and not separately selectable (giving ``--id-hash`` in addition
is only accepted if it agrees with the mode name). These modes have key material
and thus use the same keyed MACs as the encrypted modes, see :ref:`tagged_envelope`.
As the id / key is used for deduplication, id_hash must be a cryptographically
strong hash or MAC.
@@ -934,7 +930,7 @@ session key changes also keep the total advantage low over the lifetime of a bor
Modes without encryption
~~~~~~~~~~~~~~~~~~~~~~~~
The ``authenticated-*`` and ``none-*`` modes do not encrypt: the payload of a repository
The ``authenticated-*`` modes do not encrypt: the payload of a repository
object slot (the compressed chunk data resp. the packed metadata, see `Repository objects`_)
is stored as-is. Every slot still carries a 32 byte tag::
@@ -957,25 +953,17 @@ There is no nonce, no session and no other state: the tag is deterministic. Two
with the same key material therefore store byte-identical objects for identical input, which
allows deduplicating them on the filesystem level (e.g. with CoW/dedup tools).
The modes differ in the tag algorithm and in whether they have a key at all:
The tag is a **MAC** (HMAC-SHA256 for ``authenticated-sha256``, keyed BLAKE3 for
``authenticated-blake3``), so only somebody who has the borg key can compute it - this detects
malicious tampering, not just accidental corruption. The MAC key is derived from ``crypt_key``::
- ``authenticated-sha256`` / ``authenticated-blake3``: the tag is a **MAC** (HMAC-SHA256 resp.
keyed BLAKE3), so only somebody who has the borg key can compute it - this detects malicious
tampering, not just accidental corruption. The MAC key is derived from ``crypt_key``::
tag_key = sha256(crypt_key + b"borg-repoobj-mac-hmac-sha256")[:32] # authenticated-sha256
tag_key = sha256(crypt_key + b"borg-repoobj-mac-blake3")[:32] # authenticated-blake3
tag_key = sha256(crypt_key + b"borg-repoobj-mac-hmac-sha256")[:32] # authenticated-sha256
tag_key = sha256(crypt_key + b"borg-repoobj-mac-blake3")[:32] # authenticated-blake3
It is deliberately not derived from ``id_key``: chunk ids are public, and related repositories
share the id key (see ``borg repo-create --other-repo``), which must not enable them to forge
each other's objects. ``--copy-crypt-key`` shares ``crypt_key`` and thus opts into producing
byte-identical objects across the related repositories.
- ``none-sha256`` / ``none-blake3``: there is no key at all, so the tag is an **unkeyed** hash
(plain SHA-256 resp. BLAKE3 over the same input), i.e. a checksum. It detects accidental
corruption and reads that returned the wrong bytes, but anybody who modifies an object can
recompute it - it is no protection against malicious tampering. For the same reason, the chunk
ids of these modes are unkeyed hashes of the plaintext, which makes all repositories of such a
mode dedup identically.
It is deliberately not derived from ``id_key``: chunk ids are public, and related repositories
share the id key (see ``borg repo-create --other-repo``), which must not enable them to forge
each other's objects. ``--copy-crypt-key`` shares ``crypt_key`` and thus opts into producing
byte-identical objects across the related repositories.
Legacy modes
~~~~~~~~~~~~
@@ -988,7 +976,8 @@ removed in a future release.
The same applies to the borg 1.x ``none`` and ``authenticated`` modes: their envelope is just
the type byte followed by the payload, so nothing about an object is verified except the chunk
id over the plaintext. They were replaced by the tagged modes described above.
id over the plaintext. borg 2 has no ``none`` mode, and its ``authenticated-*`` modes use the
tagged envelope described above.
All modes
~~~~~~~~~
docs/internals/packs.rst
+5 -6
View File
@@ -62,8 +62,8 @@ writes and ``parse()``/``parse_meta()`` accept. It binds the header's first 41 b
``encrypted_meta`` and ``encrypted_data`` as additional authenticated data (AAD: data that is
authenticated together with the ciphertext, but not itself encrypted). This applies to all borg 2
modes: the AEAD encryption modes (AES-256-OCB, ChaCha20-Poly1305) authenticate it with their AEAD
tag, the ``authenticated-*`` modes with their MAC and the ``none-*`` modes with their (unkeyed)
checksum, see :ref:`tagged_envelope`. ``meta_size`` and ``data_size`` are excluded from the AAD.
tag, the ``authenticated-*`` modes with their MAC, see :ref:`tagged_envelope`.
``meta_size`` and ``data_size`` are excluded from the AAD.
``RepoObj.parse()`` reads both slots, so tampering with either size still fails the check, by
changing the length of the slice being read. ``parse_meta()`` reads the metadata slot alone: it
catches a changed ``meta_size`` the same way, but not a changed ``data_size``, which the repair
@@ -127,10 +127,9 @@ rebuilt the index from it. Rewriting such a pack is repository-level repair, see
only when it validates like any walked header. Validating needs the key, so a
repair that cannot load the key walks without it.
In the ``none-*`` modes the tag is an unkeyed checksum, and in the
``authenticated-*`` modes it binds a blob to its chunk id and nothing else (see
:ref:`security_structural_auth`), so validating does not establish there that
this repository wrote the blob. Both modes also store payloads as they are, so a
In the ``authenticated-*`` modes the tag binds a blob to its chunk id and nothing
else (see :ref:`security_structural_auth`), so validating does not establish there
that this repository wrote the blob. These modes also store payloads as they are, so a
backed up file can contain something that validates - the blobs of a repository
sharing the key, for instance. Such a blob reads back as itself, adding a chunk
nothing references, but the extent its ``data_size`` claims covers whatever
docs/internals/security.rst
+5 -30
View File
@@ -82,15 +82,6 @@ to a different object ID. This holds for the AEAD encryption modes (where the
AEAD tag authenticates them) as well as for the ``authenticated-*`` modes
(where a MAC does, see :ref:`tagged_envelope`).
It does **not** hold for the ``none-*`` modes: they have no key, so their objects
carry an unkeyed checksum rather than a MAC, and an attacker who modifies an
object can simply recompute it. What still constrains an attacker there is the
object ID being the (unkeyed) hash of the plaintext: the content of an existing
object can not be replaced without the ID no longer matching. But the object's
metadata and the archives list are not anchored to anything secret,
so a ``none-*`` repository provides no tamper protection - only detection of
accidental corruption.
This effectively 'anchors' each archive to the key, which is controlled by the
client, thereby anchoring the DAG starting from the archives list entry,
making it impossible for an attacker to add or modify any part of the
@@ -253,22 +244,6 @@ Different from the AEAD modes, the MAC is deterministic (a MAC needs no nonce):
session key, no IV and thus no usage limit to observe, and identical input produces identical
objects.
Unencrypted modes
~~~~~~~~~~~~~~~~~
Modes: ``--encryption none-(sha256|blake3)``
Supported: borg 2.0+
These modes have no key at all: they neither encrypt nor authenticate. Every repository
object slot carries an *unkeyed* checksum (see :ref:`tagged_envelope`), which detects
accidental corruption - bad storage hardware, a truncated write, a read that returned the
wrong bytes - before the data is used. It is not a protection against an attacker: whoever
modifies an object can recompute the checksum, and the chunk IDs are unkeyed hashes as well.
You are advised not to use these modes. Use ``authenticated-*`` instead if you do not want
your data encrypted but do want to detect tampering; it is the same thing plus a key.
Legacy modes
~~~~~~~~~~~~
@@ -283,8 +258,8 @@ only used by these legacy modes; new repositories use ``sha256`` or ``blake3``
The borg 1.x ``none`` and ``authenticated`` modes belong here, too: their repository
objects have no tag at all, so nothing about an object is verified except the chunk ID
over the plaintext - not even the object's metadata. They were replaced by the modes
described above, which cover metadata and object header as well.
over the plaintext - not even the object's metadata. borg 2 has no ``none`` mode, and its
``authenticated-*`` modes (see above) cover metadata and object header as well.
borg 2.0 does not support creating new repos using these modes,
but ``borg transfer`` can still read such existing repos.
@@ -389,9 +364,9 @@ used:
encrypted nor authenticated, so an attacker with repository access can rewrite the
crypto suite. What protects against a swapped crypto suite is not this object, but:
- a swap to a suite that does not encrypt (``none-*``, but also
``authenticated-*``: its key blob carries the same key material and no suite
name, so it loads fine) would make the client write plaintext. That is caught
- a swap to a suite that does not encrypt (``authenticated-*``: its key blob
carries the same key material and no suite name, so it loads fine) would make
the client write plaintext. That is caught
by the client's security directory: it records the key type of every
repository the client accessed, and borg refuses to continue with
``EncryptionMethodMismatch`` if the suite changed. A repository that does not
docs/usage/repo-create.rst
+2 -4
View File
@@ -11,9 +11,8 @@ Examples
# Recommended AEAD cryptographic modes (key stored in the repository by default)
$ borg repo-create --encryption=aes256-ocb
$ borg repo-create --encryption=chacha20-poly1305
# No encryption (not recommended)
# No encryption, only authentication (not recommended)
$ borg repo-create --encryption=authenticated-sha256
$ borg repo-create --encryption=none-sha256
# For the encrypted modes, --encryption (the cipher / AE algorithm) and --id-hash
# (the id hash function) are chosen independently. --id-hash defaults to sha256;
@@ -21,10 +20,9 @@ Examples
$ borg repo-create --encryption=aes256-ocb --id-hash=blake3
$ borg repo-create --encryption=chacha20-poly1305 --id-hash=blake3
# The 'authenticated-*' and 'none-*' modes name their id hash themselves, so they
# The 'authenticated-*' modes name their id hash themselves, so they
# do not take a separate --id-hash.
$ borg repo-create --encryption=authenticated-blake3
$ borg repo-create --encryption=none-blake3
# Where the key is stored (--key-location) is also chosen independently.
# --key-location defaults to repokey.
docs/usage/transfer.rst
+5
View File
@@ -70,6 +70,11 @@ locations and passphrases first:
# 4. Check whether we have everything (same as step 2):
$ borg transfer --from-borg1 --chunker-params=$CHUNKER_PARAMS --dry-run
A Borg 1.x repository in ``none`` mode (not encrypted, unkeyed sha256 chunk IDs) is transferred
like in the second example, re-chunking with ``--chunker-params``: all Borg 2 modes use keyed
chunk IDs. Such a repository has no key material to copy, so create the new repository without
giving the Borg 1.x repository as ``--other-repo``.
Keyfile considerations when upgrading from Borg 1.x
++++++++++++++++++++++++++++++++++++++++++++++++++++
scripts/rest_perf_test.sh
+2 -1
View File
@@ -56,6 +56,7 @@ profile_params() {
# --rest', so BORG_REMOTE_PATH is the command the server runs.
export BORG_REMOTE_PATH="$VENV/bin/borg"
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes
export BORG_PASSPHRASE=rest-perf-test
export BORG_DELETE_I_KNOW_WHAT_I_AM_DOING=YES
RESULTS="$(mktemp -t rest_perf_results.XXXXXX)"
@@ -172,7 +173,7 @@ run_profile() {
run_borg repo-delete --repo "$REPO_URL" >/dev/null 2>&1 || true
# start each profile with a cold pack cache (only when the opt-in cache is enabled)
if [ -n "${BORG_PACKCACHE_URL:-}" ]; then rm -rf "${BORG_PACKCACHE_URL#file://}"; fi
run_borg repo-create --encryption none-sha256 --repo "$REPO_URL"
run_borg repo-create --encryption authenticated-sha256 --repo "$REPO_URL"
# backup 1 (cold)
run_borg create --stats --json --repo "$REPO_URL" backup1 "$DATA_DIR"
src/borg/archiver/help_cmd.py
+4 -7
View File
@@ -785,7 +785,7 @@ class HelpMixIn:
An unknown place name is an error. An empty value (``BORG_ASSERT_ID=``) verifies at none of
these places, but still where borg always verifies (see below).
Why ``read`` is not in the default: in the keyed modes, the envelope already authenticates
Why ``read`` is not in the default: the envelope already authenticates
every read - the AEAD tag for the encrypted ciphersuites, the MAC for the (unencrypted)
``authenticated-*`` modes - and the chunk id is part of what that tag is computed over. So
a successful decryption resp. tag check already proves that a holder of the borg key
@@ -807,10 +807,8 @@ class HelpMixIn:
- in ``borg check --verify-data``. That audit is what makes not verifying elsewhere
defensible, so it is not configurable (there is no ``verify_data`` place name).
- for ``none-*`` mode repositories: they have no key, so nothing authenticates a read
there and their unkeyed checksums only detect accidental corruption. The id check is
therefore not optional there: it happens at every place, whatever this variable says.
Same for reading borg 1.x repositories (``borg transfer``).
- when reading borg 1.x repositories (``borg transfer --from-borg1``): their envelope does
not cover the chunk id, so the id check is not optional there.
BORG_BLAKE3_MT_THRESHOLD
When set to a numeric value, chunks of at least that many KiB get their id computed by
multi-threaded BLAKE3, smaller ones single-threaded (default: 256, i.e. 256KiB).
@@ -940,8 +938,7 @@ class HelpMixIn:
Without the key, borg can not verify anything that needs it: neither the
authentication tag of the repository objects nor the chunk ids. It therefore
reads the repository **unverified** - a corrupted or tampered repository will
not be detected. (This only concerns the ``authenticated-*`` modes; the
``none-*`` modes need no key and keep verifying their checksums.)
not be detected.
This workaround is **only** for emergencies and **only** to extract data
from an affected repository (read-only access)::
src/borg/archiver/repo_create_cmd.py
+22 -36
View File
@@ -19,8 +19,6 @@ ENCRYPTION_DESCRIPTIONS = {
"chacha20-poly1305": "ChaCha20 with Poly1305: encryption and authentication",
"authenticated-sha256": "no encryption, authentication via HMAC-SHA-256",
"authenticated-blake3": "no encryption, authentication via keyed BLAKE3",
"none-sha256": "no encryption, no authentication, only SHA-256 checksums",
"none-blake3": "no encryption, no authentication, only BLAKE3 checksums",
}
@@ -50,22 +48,21 @@ class RepoCreateMixIn:
manifest = Manifest(key, repository)
with Cache(repository, manifest, warn_if_unencrypted=False):
pass
if key.has_secret_key: # any key-bearing suite (everything except the "none-*" modes)
logger.warning(
"\n"
"IMPORTANT: you will need both KEY AND PASSPHRASE to access this repository!\n"
"\n"
"Key storage location depends on the mode:\n"
"- repokey modes: key is stored in the repository directory.\n"
"- keyfile modes: key is stored in the home directory of this user.\n"
"\n"
"For any mode, you should:\n"
"1. Export the Borg key and store the result in a safe place:\n"
" borg key export -r REPOSITORY encrypted-key-backup\n"
" borg key export -r REPOSITORY --paper encrypted-key-backup.txt\n"
" borg key export -r REPOSITORY --qr-html encrypted-key-backup.html\n"
"2. Write down the Borg key passphrase and store it in a safe place."
)
logger.warning(
"\n"
"IMPORTANT: you will need both KEY AND PASSPHRASE to access this repository!\n"
"\n"
"Key storage location depends on the mode:\n"
"- repokey modes: key is stored in the repository directory.\n"
"- keyfile modes: key is stored in the home directory of this user.\n"
"\n"
"For any mode, you should:\n"
"1. Export the Borg key and store the result in a safe place:\n"
" borg key export -r REPOSITORY encrypted-key-backup\n"
" borg key export -r REPOSITORY --paper encrypted-key-backup.txt\n"
" borg key export -r REPOSITORY --qr-html encrypted-key-backup.html\n"
"2. Write down the Borg key passphrase and store it in a safe place."
)
logger.warning(
"\n"
"Reserve some repository storage space now for emergencies like 'disk full'\n"
@@ -141,9 +138,9 @@ class RepoCreateMixIn:
key then, but that key is not protected: with ``repokey`` storage, anybody who can read
the repository can also unlock the key, which is as good as no encryption at all (with
``keyfile`` storage, the key is only on your client, so an empty passphrase may be
acceptable if e.g. the client's disk is encrypted). Unlike with the ``none-*`` modes,
you can add a passphrase later with ``borg key change-passphrase``. ``borg repo-info``
shows whether the key has an empty passphrase.
acceptable if e.g. the client's disk is encrypted). You can add a passphrase later with
``borg key change-passphrase``. ``borg repo-info`` shows whether the key has an empty
passphrase.
Choosing a crypto suite
+++++++++++++++++++++++
@@ -157,8 +154,6 @@ class RepoCreateMixIn:
- ``chacha20-poly1305``: ChaCha20 + Poly1305 (encryption + authentication).
- ``authenticated-sha256`` / ``authenticated-blake3``: no encryption, but authentication
(tamper detection) using HMAC-SHA-256 resp. keyed BLAKE3.
- ``none-sha256`` / ``none-blake3``: neither encryption nor authentication, only
SHA-256 resp. BLAKE3 checksums (see below).
``--id-hash`` selects the id hash function of the **encrypted** modes:
@@ -180,15 +175,6 @@ class RepoCreateMixIn:
This also applies to the ``authenticated-*`` modes: they do not encrypt your data, but they
still have a key (used for the id hash and the authentication), so ``--key-location``
selects where that key is stored, just like for the encrypted modes.
``--key-location`` is only ignored for the ``none-*`` modes, which have no key at all.
The ``none-*`` modes use neither encryption nor authentication: everything in the
repository is readable by anybody, and while every repository object carries a checksum
(which detects accidental corruption, e.g. bad storage hardware), anybody who modifies an
object can just recompute that checksum. You are advised NOT to use these modes: in case
of malicious activity in the repository, they expose you to a Denial-of-Service risk (due
to how the :ref:`internals_hashindex` works) and other issues (confidentiality,
tampering, ...).
If you do **not** want to encrypt the contents of your backups, but still want to detect
malicious tampering, use ``--encryption authenticated-sha256`` (or ``-blake3``). These
@@ -244,8 +230,8 @@ class RepoCreateMixIn:
required=True,
choices=encryption_argument_names(),
action=Highlander,
help="select the mode: 'aes256-ocb', 'chacha20-poly1305', 'authenticated-sha256', "
"'authenticated-blake3', 'none-sha256' or 'none-blake3' **(required)**",
help="select the mode: 'aes256-ocb', 'chacha20-poly1305', 'authenticated-sha256' "
"or 'authenticated-blake3' **(required)**",
)
subparser.add_argument(
"-i",
@@ -256,7 +242,7 @@ class RepoCreateMixIn:
default=None, # None: not given. Do not default to sha256 here, see key_creator.
action=Highlander,
help="select the id hash function of the encrypted modes: 'sha256' or 'blake3'. "
"The 'none-*' and 'authenticated-*' modes name their hash themselves.",
"The 'authenticated-*' modes name their hash themselves.",
)
subparser.add_argument(
"--key-location",
@@ -266,7 +252,7 @@ class RepoCreateMixIn:
default="repokey",
action=Highlander,
help="where to store the key: 'repokey' (in the repository, default) or 'keyfile' "
"(in the local keys directory). Ignored for the ``none-*`` modes (which have no key).",
"(in the local keys directory).",
)
subparser.add_argument(
"--copy-crypt-key",
src/borg/archiver/repo_info_cmd.py
+3 -3
View File
@@ -25,15 +25,15 @@ class RepoInfoMixIn:
# the two dimensions: cipher / AE algorithm (ENC_NAME) and id hash function (IDHASH_NAME).
storage = getattr(key, "storage", None)
mode = {KeyBlobStorage.KEYFILE: "keyfile", KeyBlobStorage.REPO: "repokey"}.get(storage)
# the "none-*" and "authenticated-*" modes name their id hash in ENC_NAME already.
# the "authenticated-*" modes name their id hash in ENC_NAME already.
suite = key.ENC_NAME if key.IDHASH_IN_ENC_NAME else "%s, %s" % (key.ENC_NAME, key.IDHASH_NAME)
if key.encrypts and key.empty_passphrase:
# the key is not protected by a passphrase: with repokey storage, this is as good as
# no encryption at all, so make it visible here, see #9072.
suite += ", empty passphrase"
if not key.encrypts:
# these modes do not encrypt data; the "authenticated-*" ones (unlike "none-*")
# still have a key stored as a keyfile or repokey, so show that location if there is one.
# the "authenticated-*" modes do not encrypt data, but they have a key stored as a
# keyfile or repokey, so show that location if there is one.
encryption += "No (%s, %s)" % (mode, suite) if mode else "No (%s)" % suite
else:
encryption += "Yes (%s, %s)" % (mode, suite) if mode else "Yes (%s)" % suite
src/borg/constants.py
+2 -4
View File
@@ -273,12 +273,10 @@ class KeyType:
# 0x50 was a borg2 beta format ("authenticated" with blake3 ids and a no-op envelope). It was
# dropped before the 2.0 release, see #9104 - the byte stays reserved so it is not reused.
DROPPED_BLAKE3AUTHENTICATED = 0x50
# the "authenticated-*" and "none-*" modes: not encrypted, but every repo object slot carries
# a tag (a MAC resp. an unkeyed checksum), see MACKeyBase.
# the "authenticated-*" modes: not encrypted, but every repo object slot carries a tag (a MAC),
# see MACKeyBase.
SHA256AUTHENTICATED = 0x60
BLAKE3AUTHENTICATED = 0x70
SHA256NONE = 0x80
BLAKE3NONE = 0x90
CACHE_TAG_NAME = "CACHEDIR.TAG"
src/borg/crypto/key.py
+19 -125
View File
@@ -200,9 +200,9 @@ def key_creator(repository, args, *, other_key=None):
# For the encrypted modes, the crypto suite is selected by two orthogonal dimensions: the
# cipher / AE algorithm ("--encryption") and the id hash function ("--id-hash", defaulting to
# sha256). For the modes that do not encrypt, the hash is not a tuning knob but *is* the
# mechanism (it authenticates resp. checksums the data), so it is part of the mode name:
# "none-sha256", "none-blake3", "authenticated-sha256", "authenticated-blake3". Giving
# "--id-hash" in addition to one of those names is only accepted if both agree.
# mechanism (it authenticates the data), so it is part of the mode name:
# "authenticated-sha256", "authenticated-blake3". Giving "--id-hash" in addition to one of
# those names is only accepted if both agree.
enc = args.encryption
id_hash = getattr(args, "id_hash", None) # None: not given, see the --id-hash argparse default
for key in AVAILABLE_KEY_TYPES:
@@ -312,22 +312,16 @@ def uses_same_chunker_secret(other_key, key):
def uses_same_id_hash(other_key, key):
"""other_key -> key upgrade: is the id hash the same?"""
# avoid breaking the deduplication by changing the id hash
old_sha256_ids = (LegacyPlaintextKey,) # unkeyed sha256 over the plaintext
new_sha256_ids = (ChecksumKey,)
# note: borg 1.x "none" (LegacyPlaintextKey) uses unkeyed sha256 ids, which no borg 2 mode has.
old_hmac_sha256_ids = (AESCTRKey, LegacyAuthenticatedKey)
new_hmac_sha256_ids = (AESOCBKey, CHPOKey, AuthenticatedKey)
# note: we do not support blake2b for new repos, see #8867
new_blake3_ids = (Blake3AESOCBKey, Blake3CHPOKey, Blake3AuthenticatedKey)
new_unkeyed_blake3_ids = (Blake3ChecksumKey,) # unkeyed blake3 over the plaintext
same_ids = (
isinstance(other_key, old_hmac_sha256_ids + new_hmac_sha256_ids)
and isinstance(key, new_hmac_sha256_ids)
or isinstance(other_key, new_blake3_ids)
and isinstance(key, new_blake3_ids)
or isinstance(other_key, old_sha256_ids + new_sha256_ids)
and isinstance(key, new_sha256_ids)
or isinstance(other_key, new_unkeyed_blake3_ids)
and isinstance(key, new_unkeyed_blake3_ids)
)
return same_ids
@@ -348,7 +342,7 @@ class KeyBase:
# None means "not creatable this way" (e.g. legacy read-only classes).
ENC_NAME: ClassVar[str] = None # override in creatable subclasses
IDHASH_NAME: ClassVar[str] = None # override in creatable subclasses (or via id-hash mix-in)
# Is the id hash part of the ENC_NAME (e.g. "none-blake3") instead of being selectable via
# Is the id hash part of the ENC_NAME (e.g. "authenticated-blake3") instead of being selectable via
# "--id-hash"? True for the modes that do not encrypt: there, the hash is not a tuning knob,
# it is what protects the data, so it belongs to the mode. See key_creator.
IDHASH_IN_ENC_NAME: ClassVar[bool] = False
@@ -377,30 +371,15 @@ class KeyBase:
# id_key dummy, needs to be overwritten by subclass
id_key: bytes = None
# Is assert_id() the read path's authentication mechanism for this key class?
# True (the default): the envelope does not authenticate the payload as the one written for the requested
# chunk id (there is no keyed envelope tag covering the id), so the id check must never be skipped, see
# RepoObj.parse (RepoObj1.parse for the borg 1.x classes). In borg 2, this is only the "none-*" modes:
# their decrypt() verifies just the unkeyed envelope checksum, which detects accidental corruption but
# authenticates nothing, see ChecksumKeyBase. The read-only borg 1.x classes (borg.legacy.crypto.key)
# keep the default, too - no borg 1.x envelope covers the chunk id, and for borg 1.x "authenticated",
# the keyed id hash even is the only authentication a read has: skipping it would silently demote that
# mode to "none".
# False (the AEAD ciphersuites and the "authenticated-*" modes): every read is authenticated by the keyed
# envelope tag, which covers the chunk id (it is in the AAD), independently of assert_id() - see
# AEADKeyBase.assert_id for what assert_id adds on top of that.
id_check_is_authentication: ClassVar[bool] = True
# Does this mode encrypt the data at all? False for the modes that only tag it ("none-*",
# "authenticated-*" and their borg 1.x predecessors). Different from logically_encrypted,
# which is a per-instance property (an encrypted repo with an empty passphrase is not
# "logically" encrypted, but it does encrypt).
# Does this mode encrypt the data at all? False for the modes that only tag it
# ("authenticated-*") and for the borg 1.x "none" and "authenticated" modes. Different from
# logically_encrypted, which is a per-instance property (an encrypted repo with an empty
# passphrase is not "logically" encrypted, but it does encrypt).
encrypts: ClassVar[bool] = True
# Does this key class have secret key material?
# False for the "none-*" modes: their chunk ids and their envelope checksums are unkeyed, thus
# they work without any key - and the authenticated_no_key workaround (which fakes the key
# material of a key it can not unlock) must not switch off checks that need no key at all.
# False only for the read-only borg 1.x "none" mode (its chunk ids are unkeyed), so there is
# no key material a related repository could copy, see FlexiKey.create.
has_secret_key: ClassVar[bool] = True
# Whether this *particular instance* is encrypted from a practical point of view,
@@ -733,8 +712,7 @@ class FlexiKey:
key.storage = KEY_LOCATIONS.get(getattr(args, "key_location", None), cls.STORAGE)
if other_key is not None:
if not other_key.has_secret_key:
# the "none-*" modes (and borg 1.x "none") have no key material to copy - and they
# need none: their chunk ids are unkeyed, so they all dedup identically anyway.
# borg 1.x "none" has no key material to copy.
raise Error("Copying key material from an unencrypted repository is not possible.")
if isinstance(key, AESKeyBase):
# user must use an AEADKeyBase subclass (AEAD modes with session keys)
@@ -1084,7 +1062,7 @@ class FlexiKey:
return victim
# ------------ not encrypted, but tagged: the "authenticated-*" and "none-*" modes ------------
# ------------ not encrypted, but tagged: the "authenticated-*" modes ------------
class MACKeyBase(KeyBase):
@@ -1104,7 +1082,7 @@ class MACKeyBase(KeyBase):
not need any). Two repositories with the same key material therefore store byte-identical
objects for identical input, which allows deduplicating them on the filesystem level.
Subclasses supply the tag algorithm (mac()) and, if they are keyed, the tag key (tag_key).
Subclasses supply the tag algorithm (mac()) and the tag key (tag_key).
"""
encrypts = False # these modes only tag the data, they do not encrypt it
@@ -1122,7 +1100,7 @@ class MACKeyBase(KeyBase):
@property
def tag_key(self) -> bytes:
"""The key the tag is computed with (unkeyed modes do not have one)."""
"""The key the tag is computed with."""
raise NotImplementedError
def init_ciphers(self, manifest_data=None):
@@ -1150,9 +1128,8 @@ class MACKeyBase(KeyBase):
self.assert_type(data[0], id)
obj = memoryview(data)
payload = obj[self.PAYLOAD_OVERHEAD :]
if self.has_secret_key and AUTHENTICATED_NO_KEY:
# we do not have the key material, so we can not verify the tag, see the
# BORG_WORKAROUNDS docs. Unkeyed modes verify even then - they need no key.
if AUTHENTICATED_NO_KEY:
# we do not have the key material, so we can not verify the tag, see the BORG_WORKAROUNDS docs.
return payload
header = bytes(obj[: self.HEADER_SIZE])
tag = bytes(obj[self.HEADER_SIZE : self.PAYLOAD_OVERHEAD])
@@ -1162,79 +1139,6 @@ class MACKeyBase(KeyBase):
return payload
class ChecksumKeyBase(MACKeyBase):
"""
Base class of the "none-*" modes: no encryption, no key, no authentication.
The tag is an **unkeyed** hash, i.e. a checksum: it detects accidental corruption of the
payload, the metadata or the object header (including a read that returned the wrong bytes),
but it is no protection against malicious tampering - whoever changes the object can just
recompute the checksum. Detecting that requires a secret, see the "authenticated-*" modes.
Because there is no key, the chunk id is an unkeyed hash of the plaintext, too. Thus, all
repositories of this mode dedup identically and store byte-identical objects for identical
input, no key sharing needed.
"""
STORAGE = KeyBlobStorage.NO_STORAGE
LOCATION_CONFIGURABLE = False
chunk_seed = 0
crypt_key = b"" # makes .derive_key() work, nothing secret here
id_key = b"" # makes .derive_key() work, nothing secret here
logically_encrypted = False
has_secret_key = False
# The checksum can be recomputed by anybody, so it is not an authentication and thus can not
# take over the read path authentication the id hash does. Different from the keyed classes
# below, verifying the chunk id must therefore never be skipped, see RepoObj.parse.
id_check_is_authentication = True
@classmethod
def create(cls, repository, args, **kw):
logger.info(
"Encryption NOT enabled.\n"
'Use "--encryption=aes256-ocb" (or another encrypted mode) to enable encryption.'
)
return cls(repository)
@classmethod
def detect(cls, repository, manifest_data, *, other=False):
return cls(repository)
class ChecksumKey(ChecksumKeyBase):
TYPE = KeyType.SHA256NONE
TYPES_ACCEPTABLE = {TYPE}
ENC_NAME = "none-sha256"
IDHASH_NAME = "sha256"
def id_hash(self, data):
return sha256(data).digest()
def mac(self, prefix, payload):
h = sha256(prefix)
h.update(payload)
return h.digest()
class Blake3ChecksumKey(ChecksumKeyBase):
TYPE = KeyType.BLAKE3NONE
TYPES_ACCEPTABLE = {TYPE}
ENC_NAME = "none-blake3"
IDHASH_NAME = "blake3"
def id_hash(self, data):
return _blake3_hasher(data).digest(length=32)
def mac(self, prefix, payload):
max_threads = blake3.AUTO if len(payload) >= get_blake3_mt_threshold() else 1
h = blake3(prefix, max_threads=max_threads)
h.update(payload)
return h.digest(length=32)
class AuthenticatedKeyBase(MACKeyBase, FlexiKey):
"""
Base class of the "authenticated-*" modes: no encryption, but real authentication.
@@ -1259,11 +1163,6 @@ class AuthenticatedKeyBase(MACKeyBase, FlexiKey):
# It's only authenticated, not encrypted.
logically_encrypted = False
# every read is authenticated by the envelope tag (which covers the chunk id via the AAD),
# independently of assert_id() - the same reasoning as for the AEAD keys, see
# AEADKeyBase.assert_id about what verifying the chunk id adds on top of that.
id_check_is_authentication = False
# domain for deriving the envelope MAC key from crypt_key, see tag_key. It is per MAC
# algorithm, so the classes below do not use the same key for different algorithms.
MAC_KEY_DOMAIN: ClassVar[bytes] = None # override in subclass
@@ -1359,7 +1258,7 @@ class AuthenticatedKey(ID_HMAC_SHA_256, AuthenticatedKeyBase):
from ..legacy.crypto.key import AESCTRKey, Blake2AESCTRKey # noqa: F401
from ..legacy.crypto.key import Blake2AuthenticatedKey # noqa: F401
from ..legacy.crypto.key import AuthenticatedKey as LegacyAuthenticatedKey # noqa: E402
from ..legacy.crypto.key import PlaintextKey as LegacyPlaintextKey # noqa: E402
from ..legacy.crypto.key import PlaintextKey as LegacyPlaintextKey # noqa: E402, F401
from ..legacy.crypto.key import LEGACY_KEY_TYPES # noqa: E402
from ..legacy.crypto.key import ID_BLAKE2b_256 # noqa: F401
@@ -1435,9 +1334,6 @@ class AEADKeyBase(KeyBase):
# an AEAD key may be stored as a keyfile or inside the repository (see borg key change-location).
LOCATION_CONFIGURABLE = True
# the AEAD tag authenticates every read on its own, assert_id only adds the "evil client" detection below.
id_check_is_authentication = False
def assert_id(self, id, data):
# Comparing the id hash here would not be needed any more for the new AEAD crypto **IF** we
# could be sure that chunks were created by normal (not tampered, not evil) borg code:
@@ -1456,7 +1352,7 @@ class AEADKeyBase(KeyBase):
# As only that (rather special) threat is left here, this check is mostly optional for AEAD
# keys (it was mandatory from #7362/#7367 until #9994): RepoObj.parse calls it at the places
# BORG_ASSERT_ID lists (by default: the places that re-anchor content - check --repair,
# transfer, re-chunking), plus always in check --verify-data, see id_check_is_authentication.
# transfer, re-chunking), plus always in check --verify-data.
if id and id != Manifest.MANIFEST_ID:
id_computed = self.id_hash(data)
if not hmac.compare_digest(id_computed, id):
@@ -1610,6 +1506,4 @@ AVAILABLE_KEY_TYPES = (
# not encrypted modes (the id hash is part of the mode name, see key_creator)
AuthenticatedKey,
Blake3AuthenticatedKey,
ChecksumKey,
Blake3ChecksumKey,
)
src/borg/repoobj.py
+13 -29
View File
@@ -24,23 +24,15 @@ ASSERT_ID_PLACES = (
# not verifying elsewhere defensible, so it must not be switchable.
ASSERT_ID_PLACES_MANDATORY = ("verify_data",) # borg check --verify-data
# Default value of the BORG_ASSERT_ID env var: verify everywhere except on the (hot) general read
# path - there, the envelope authentication of the keyed modes already covers what a repository
# could do to us, see AEADKeyBase.assert_id. (For the modes whose id check *is* the read path
# authentication, this setting does not apply, see below.)
# path - there, the envelope authentication already covers what a repository could do to us, see
# AEADKeyBase.assert_id.
BORG_ASSERT_ID_DEFAULT = ("repair", "transfer", "rechunk")
# Reads through a RepoObj are attributed to this place unless a command sets another one.
ASSERT_ID_PLACE_DEFAULT = "read"
def get_assert_id_places():
"""Determine the configurable places that shall verify the chunk id, see the BORG_ASSERT_ID docs.
Note: this only decides for keys that authenticate reads independently of the id hash (the AEAD
ciphersuites and the "authenticated-*" modes, see KeyBase.id_check_is_authentication) - for all
other keys, the id check is the read path authentication itself and thus always happens, no
matter what is configured here.
The same is true for ASSERT_ID_PLACES_MANDATORY.
"""
"""Determine the configurable places that shall verify the chunk id, see the BORG_ASSERT_ID docs."""
value = os.environ.get("BORG_ASSERT_ID")
if value is None:
return frozenset(BORG_ASSERT_ID_DEFAULT)
@@ -254,23 +246,16 @@ class RepoObj:
compressor_cls, compression_level = Compressor.detect(compr_hdr)
compressor = compressor_cls(level=compression_level)
meta, data = compressor.decompress(dict(meta_compressed), data_compressed[:psize])
# For keys where the id check is the read-path authentication (the "none-*" modes, whose
# envelope checksum is unkeyed and thus no authentication), it always has to happen -
# skipping it would remove all integrity checking from reads.
# For the keys that authenticate the envelope (the AEAD and the "authenticated-*" modes), the
# payload is already authenticated for this specific chunk id (the id is in the AAD), so the id
# All keys authenticate the envelope (the AEAD and the "authenticated-*" modes), so the
# payload is already authenticated for this specific chunk id (the id is in the AAD) and the id
# check only adds detection of chunks whose plaintext does not match their id - which only an
# evil/broken borg client that had the repo key could have written. Whether that extra
# full-plaintext hash pass is worth it depends on the place we read at, see BORG_ASSERT_ID.
place = assert_id_place if assert_id_place is not None else self.assert_id_place
assert_id = (
self.key.id_check_is_authentication
or place in ASSERT_ID_PLACES_MANDATORY
or place in self.assert_id_places
)
assert_id = place in ASSERT_ID_PLACES_MANDATORY or place in self.assert_id_places
# the authenticated_no_key workaround fakes key material it could not unlock, so checks that
# need that key material can not work. The unkeyed modes need none, so they keep checking.
if assert_id and not (AUTHENTICATED_NO_KEY and self.key.has_secret_key):
# need that key material can not work.
if assert_id and not AUTHENTICATED_NO_KEY:
self.key.assert_id(id, data)
else:
meta, data = None, None
@@ -287,9 +272,8 @@ def object_validator(repo_objs):
field outside the tag, must match csize - the data slot's payload size, recorded in the tagged
metadata - plus the key's fixed envelope overhead.
In the "none-*" modes the tag is an unkeyed checksum, and in the "authenticated-*" modes it is
deterministic and binds an object to its chunk id alone. Both therefore accept an object that a
backed up file contains, at any offset in any pack.
In the "authenticated-*" modes the tag is deterministic and binds an object to its chunk id
alone. They therefore accept an object that a backed up file contains, at any offset in any pack.
"""
hdr_size = RepoObj.obj_header.size
overhead = repo_objs.key.PAYLOAD_OVERHEAD # the envelope adds a fixed number of bytes to the payload
@@ -298,10 +282,10 @@ def object_validator(repo_objs):
try:
meta = repo_objs.parse_meta(chunk_id, obj, ro_type=ROBJ_DONTCARE)
except (IntegrityErrorBase, msgpack.UnpackException):
# arbitrary bytes fail the authentication, or the msgpack unpacking in the modes that
# authenticate without a secret key (none-*, authenticated-*) and thus accept them.
# arbitrary bytes fail the authentication - or the msgpack unpacking, if the tag is not
# verified (authenticated-* modes with the authenticated_no_key workaround).
return False
# in those modes the slot can unpack to any value, so check the value: only metadata that
# without tag verification the slot can unpack to any value, so check the value: only metadata that
# is unusable here gives False, any other exception propagates.
if not isinstance(meta, dict):
return False
src/borg/repository.py
+1 -1
View File
@@ -663,7 +663,7 @@ def superseded_gap_ranges(reader, chunks, pack_id, obj_ranges, pack_size, *, val
# TODO: obj_size is verified only for objects validate accepts. A wrong obj_size
# usually ends the walk at a header that does not parse, so the superseded duplicates in
# the rest of the gap are kept on every rewrite. PackReader._find_header could resync to
# the next object validate accepts, but in the none-* and authenticated-* modes that can
# the next object validate accepts, but in the authenticated-* modes that can
# be a copy of an object inside another object's unencrypted data, whose range can cover
# bytes of the gap objects after it, which would then be dropped.
offset += obj_size
src/borg/testsuite/__init__.py
+14
View File
@@ -271,6 +271,20 @@ def filter_xattrs(x):
raise ValueError("Unsupported type: %s" % type(x))
def make_test_key(repository=None):
"""Return an "authenticated-sha256" key for tests that just need some working key.
The key material is fixed, so the chunk ids and the stored objects are the same in every test run.
The key is not stored anywhere (no borg key, no passphrase).
"""
from ..crypto.key import AuthenticatedKey
key = AuthenticatedKey(repository)
key.init_from_given_data(crypt_key=b"c" * 64, id_key=b"i" * 32, chunk_seed=0)
key.init_ciphers()
return key
class BaseTestCase(unittest.TestCase):
assert_in = unittest.TestCase.assertIn
assert_not_in = unittest.TestCase.assertNotIn
src/borg/testsuite/archive_test.py
+8 -9
View File
@@ -7,10 +7,9 @@ from unittest.mock import Mock
import pytest
from . import rejected_dotdot_paths, is_utime_fully_supported
from . import rejected_dotdot_paths, is_utime_fully_supported, make_test_key
from ..cache import ChunkListEntry
from ..constants import ROBJ_FILE_STREAM, zeros
from ..crypto.key import ChecksumKey
from ..archive import Archive, CacheChunkBuffer, DownloadPipeline, RobustUnpacker, valid_msgpacked_dict
from ..archive import ITEM_KEYS, Statistics
from ..archive import zero_chunk_flags, zero_chunk_id, zero_chunk_ids
@@ -195,7 +194,7 @@ def test_stats_as_dict(stats):
)
def test_timestamp_parsing(monkeypatch, isoformat, expected):
repository = Mock()
key = ChecksumKey(repository)
key = make_test_key(repository)
manifest = Manifest(key, repository)
a = Archive(manifest, "test", create=True)
a.metadata = ArchiveItem(time=isoformat)
@@ -219,7 +218,7 @@ class MockCache:
def test_cache_chunk_buffer():
data = [Item(path="p1"), Item(path="p2")]
cache = MockCache()
key = ChecksumKey(None)
key = make_test_key(None)
chunks = CacheChunkBuffer(cache, key, None)
for d in data:
chunks.add(d)
@@ -236,7 +235,7 @@ def test_partial_cache_chunk_buffer():
big = "0123456789abcdefghijklmnopqrstuvwxyz" * 25000
data = [Item(path="full", target=big), Item(path="partial", target=big)]
cache = MockCache()
key = ChecksumKey(None)
key = make_test_key(None)
chunks = CacheChunkBuffer(cache, key, None)
for d in data:
chunks.add(d)
@@ -271,7 +270,7 @@ def test_download_pipeline_parsed_cache():
# a content data stream may reference the same chunk many times (e.g. the all-zero
# chunks of a sparse file): repeated chunks shall be parsed (decrypted, authenticated,
# decompressed) only once, see issue #1678.
key = ChecksumKey(None)
key = make_test_key(None)
repo_objs = RepoObj(key)
# note: repeated, but not all-zero data, so it is not served via the zeros shortcut
chunks_data = [b"foobar" * 100, b"idletone" * 125, b"barbaz" * 100]
@@ -304,7 +303,7 @@ def test_download_pipeline_parsed_cache():
def test_download_pipeline_missing_chunk(replacement_chunk):
# a chunk missing in the repository is either replaced by all-zero data of the
# correct size, or reported as None - and never blows up on the size check.
key = ChecksumKey(None)
key = make_test_key(None)
repo_objs = RepoObj(key)
data = b"foobar" * 100
id = repo_objs.id_hash(data)
@@ -318,7 +317,7 @@ def test_download_pipeline_missing_chunk(replacement_chunk):
def test_download_pipeline_zero_chunks_served_locally():
# repeated all-zero chunks (e.g. from the holes of a sparse file) shall be served
# directly from the zeros constant, without repository access, see issue #1678.
key = ChecksumKey(None)
key = make_test_key(None)
repo_objs = RepoObj(key)
data = b"foobar" * 100
data_id = repo_objs.id_hash(data)
@@ -358,7 +357,7 @@ def test_zero_chunk_flags():
# cheap all-zero chunk detection from the chunk ids/sizes alone: the zero chunk id
# is computed for ids occurring repeatedly, while unique ids are only compared
# against already memoized zero chunk ids.
key = ChecksumKey(None)
key = make_test_key(None)
id_hash = key.id_hash
data = b"foobar" * 100
data_id = id_hash(data)
src/borg/testsuite/archiver/check_cmd_test.py
+2 -2
View File
@@ -1101,7 +1101,7 @@ def test_repair_finish_flushes_pack_writer(archivers, request):
assert not repository._pack_writer._pieces # finish() stored it
@pytest.mark.parametrize("init_args", [["--encryption=aes256-ocb"], ["--encryption", "none-sha256"]])
@pytest.mark.parametrize("init_args", [["--encryption=aes256-ocb"], ["--encryption", "authenticated-sha256"]])
def test_verify_data(archivers, request, init_args):
archiver = request.getfixturevalue(archivers)
if archiver.get_kind() != "local":
@@ -1196,7 +1196,7 @@ def test_repair_wrong_item_metadata_chunk_content(archivers, request, monkeypatc
assert "id verification failed" in output
@pytest.mark.parametrize("init_args", [["--encryption=aes256-ocb"], ["--encryption", "none-sha256"]])
@pytest.mark.parametrize("init_args", [["--encryption=aes256-ocb"], ["--encryption", "authenticated-sha256"]])
def test_corrupted_file_chunk(archivers, request, init_args):
## like test_verify_data, but also checks a repository-only check passes after repair and a plain
## archives check reports the missing chunk.
src/borg/testsuite/archiver/checks_test.py
+26 -9
View File
@@ -5,11 +5,13 @@ import pytest
from ...cache import Cache
from ...constants import * # NOQA
from ...crypto.key import keyfile_format, keyfile_parse
from ...helpers import Location, get_security_dir, bin_to_hex
from ...helpers import EXIT_ERROR
from ...repository import Repository
from .. import changedir
from . import cmd, _extract_repository_id, create_test_files
from . import _set_repository_id, create_regular_file, assert_creates_file, generate_archiver_tests, RK_ENCRYPTION
from . import create_regular_file, assert_creates_file, generate_archiver_tests, RK_ENCRYPTION
from . import set_empty_passphrase
pytest_generate_tests = lambda metafunc: generate_archiver_tests(metafunc, kinds="local,remote") # NOQA
@@ -20,6 +22,21 @@ def get_security_directory(repo_path):
return get_security_dir(repository_id)
def _set_repository_id(repo_path, id):
"""Change the id of the repository at repo_path and label its repokeys for the new id.
This is what an attacker does to make their own "authenticated-*" repository (which has a borg
key, labelled with the repository id) look like the repository of somebody else.
"""
with Repository(repo_path) as repository:
keys = repository.load_keys()
repository._set_id(id)
for name, keydata in keys:
_, b64data = keyfile_parse(keydata)
repository.delete_key(name)
repository.store_key(keyfile_format(bin_to_hex(id), b64data).encode())
def test_repository_swap_detection(archivers, request):
archiver = request.getfixturevalue(archivers)
create_test_files(archiver.input_path)
@@ -28,7 +45,7 @@ def test_repository_swap_detection(archivers, request):
repository_id = _extract_repository_id(archiver.repository_path)
cmd(archiver, "create", "test", "input")
shutil.rmtree(archiver.repository_path)
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
_set_repository_id(archiver.repository_path, repository_id)
assert repository_id == _extract_repository_id(archiver.repository_path)
if archiver.FORK_DEFAULT:
@@ -42,9 +59,9 @@ def test_repository_swap_detection2(archivers, request):
archiver = request.getfixturevalue(archivers)
create_test_files(archiver.input_path)
original_location = archiver.repository_location
archiver.repository_location = original_location + "_unencrypted"
cmd(archiver, "repo-create", "--encryption=none-sha256")
os.environ["BORG_PASSPHRASE"] = "passphrase"
archiver.repository_location = original_location + "_unencrypted"
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
archiver.repository_location = original_location + "_encrypted"
cmd(archiver, "repo-create", RK_ENCRYPTION)
cmd(archiver, "create", "test", "input")
@@ -65,7 +82,7 @@ def test_repository_swap_detection_no_cache(archivers, request):
repository_id = _extract_repository_id(archiver.repository_path)
cmd(archiver, "create", "test", "input")
shutil.rmtree(archiver.repository_path)
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
_set_repository_id(archiver.repository_path, repository_id)
assert repository_id == _extract_repository_id(archiver.repository_path)
cmd(archiver, "repo-delete", "--cache-only")
@@ -80,9 +97,9 @@ def test_repository_swap_detection2_no_cache(archivers, request):
archiver = request.getfixturevalue(archivers)
original_location = archiver.repository_location
create_test_files(archiver.input_path)
archiver.repository_location = original_location + "_unencrypted"
cmd(archiver, "repo-create", "--encryption=none-sha256")
os.environ["BORG_PASSPHRASE"] = "passphrase"
archiver.repository_location = original_location + "_unencrypted"
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
archiver.repository_location = original_location + "_encrypted"
cmd(archiver, "repo-create", RK_ENCRYPTION)
cmd(archiver, "create", "test", "input")
@@ -157,7 +174,7 @@ def test_repository_move(archivers, request, monkeypatch):
def test_unknown_unencrypted(archivers, request, monkeypatch):
archiver = request.getfixturevalue(archivers)
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
# Ok: repository is known
cmd(archiver, "repo-info")
@@ -176,7 +193,7 @@ def test_unknown_unencrypted(archivers, request, monkeypatch):
output = cmd(archiver, "repo-info")
# the warning says why the repository counts as unencrypted, see #9072
assert "previously unknown unencrypted repository" in output
assert "uses the none-sha256 mode, which does not encrypt the data" in output
assert "uses the authenticated-sha256 mode, which does not encrypt the data" in output
def test_unknown_unencrypted_empty_passphrase(archivers, request, monkeypatch):
src/borg/testsuite/archiver/compact_cmd_test.py
+3 -3
View File
@@ -7,7 +7,6 @@ import pytest
from ...constants import * # NOQA
from ...helpers import get_cache_dir, bin_to_hex, sig_int, Error
from ...hashindex import ChunkIndex
from ...crypto.key import ChecksumKey
from ...repoobj import RepoObj
from ...repository import Repository
from ...cache import files_cache_name, discover_files_cache_names, list_chunkindex_hashes
@@ -19,14 +18,15 @@ from ...archiver.compact_cmd import ArchiveGarbageCollector
from ... import cache
from . import cmd, create_regular_file, create_src_archive, generate_archiver_tests, open_repository, RK_ENCRYPTION
from . import changedir
from .. import make_test_key
from ..repository_test import H, fchunk, pdchunk
pytest_generate_tests = lambda metafunc: generate_archiver_tests(metafunc, kinds="local,remote,binary") # NOQA
def gc_manifest(repository):
"""Return a manifest stand-in for ArchiveGarbageCollector, whose repo_objs use a none-sha256 key."""
return SimpleNamespace(repo_objs=RepoObj(ChecksumKey(repository)))
"""Return a manifest stand-in for ArchiveGarbageCollector, whose repo_objs use a test key."""
return SimpleNamespace(repo_objs=RepoObj(make_test_key(repository)))
@pytest.mark.parametrize("stats", (True, False))
src/borg/testsuite/archiver/create_cmd_test.py
+3 -3
View File
@@ -141,7 +141,7 @@ def test_archived_paths(archivers, request):
# no leading slash in borg archives:
archived_path = posix_path.lstrip("/")
create_regular_file(archiver.input_path, "test")
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", "test", "input", posix_path)
# "input" directory is recursed into, "input/test" is discovered and joined by borg's recursion.
# posix_path was directly given as a cli argument and should end up as archive_path in the borg archive.
@@ -207,7 +207,7 @@ def test_create_duplicate_root(archivers, request):
hl_b = os.path.join(path_b, "hardlink")
create_regular_file(archiver.input_path, hl_a, contents=b"123456")
os.link(hl_a, hl_b)
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", "test", "input", "input") # give input twice!
# test if created archive has 'input' contents twice:
archive_list = cmd(archiver, "list", "test", "--json-lines")
@@ -223,7 +223,7 @@ def test_create_unreadable_parent(archiver):
os.mkdir(root_dir)
os.chmod(parent_dir, 0o111) # --x--x--x == parent dir traversable, but not readable
try:
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
# issue #7746: we *can* read root_dir and we *can* traverse parent_dir, so this should work:
cmd(archiver, "create", "test", root_dir)
finally:
src/borg/testsuite/archiver/disk_full_test.py
+2 -1
View File
@@ -47,11 +47,12 @@ def make_files(dir, count, size, rnd=True):
def test_disk_full(test_pass, cmd_fixture, monkeypatch):
monkeypatch.setenv("BORG_CHECK_I_KNOW_WHAT_I_AM_DOING", "YES")
monkeypatch.setenv("BORG_DELETE_I_KNOW_WHAT_I_AM_DOING", "YES")
monkeypatch.setenv("BORG_PASSPHRASE", "waytooeasyonlyfortests")
repo = os.path.join(DF_MOUNT, "repo")
input = os.path.join(DF_MOUNT, "input")
shutil.rmtree(repo, ignore_errors=True)
shutil.rmtree(input, ignore_errors=True)
rc, out = cmd_fixture(f"--repo={repo}", "repo-create", "--encryption=none-sha256")
rc, out = cmd_fixture(f"--repo={repo}", "repo-create", "--encryption=authenticated-sha256")
if rc != EXIT_SUCCESS:
print("repo-create", rc, out)
assert rc == EXIT_SUCCESS
src/borg/testsuite/archiver/extract_cmd_test.py
+9 -9
View File
@@ -536,7 +536,7 @@ def test_extract_hardlinks_twice(archivers, request):
hl_b = os.path.join(path_b, "hardlink")
create_regular_file(archiver.input_path, hl_a, contents=b"123456")
os.link(hl_a, hl_b)
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", "test", "input", "input") # give input twice!
# now test extraction
with changedir("output"):
@@ -777,7 +777,7 @@ def test_extract_xattrs_errors(archivers, request):
create_regular_file(archiver.input_path, "file")
xattr.setxattr(b"input/file", b"user.attribute", b"value")
cmd(archiver, "repo-create", "-e", "none-sha256")
cmd(archiver, "repo-create", "-e", "authenticated-sha256")
cmd(archiver, "create", "test", "input")
with changedir("output"):
input_abspath = os.path.abspath("input/file")
@@ -805,7 +805,7 @@ def test_extract_xattrs_errors(archivers, request):
def test_extract_xattrs_resourcefork(archivers, request):
archiver = request.getfixturevalue(archivers)
create_regular_file(archiver.input_path, "file")
cmd(archiver, "repo-create", "-e", "none-sha256")
cmd(archiver, "repo-create", "-e", "authenticated-sha256")
input_path = os.path.abspath("input/file")
xa_key, xa_value = b"com.apple.ResourceFork", b"whatshouldbehere" # issue #7234
xattr.setxattr(input_path.encode(), xa_key, xa_value)
@@ -844,7 +844,7 @@ def test_extract_restores_append_flag(archivers, request):
if (platform.get_flags(src_path, st) & stat.UF_APPEND) == 0:
pytest.skip("UF_APPEND not settable on this filesystem")
# archive and extract
cmd(archiver, "repo-create", "-e", "none-sha256")
cmd(archiver, "repo-create", "-e", "authenticated-sha256")
cmd(archiver, "create", "test", "input")
with changedir("output"):
cmd(archiver, "extract", "test")
@@ -864,7 +864,7 @@ def test_extract_flags_errors(archivers, request):
raise OSError(errno.EPERM, "EPERM")
create_regular_file(archiver.input_path, "file")
cmd(archiver, "repo-create", "-e", "none-sha256")
cmd(archiver, "repo-create", "-e", "authenticated-sha256")
cmd(archiver, "create", "test", "input")
with changedir("output"):
with patch.object(archive_module, "set_flags", patched_set_flags):
@@ -914,7 +914,7 @@ def test_do_not_fail_when_percent_is_in_xattr_name(archivers, request):
create_regular_file(archiver.input_path, "file")
xattr.setxattr(b"input/file", b"user.attribute%p", b"value")
cmd(archiver, "repo-create", "-e", "none-sha256")
cmd(archiver, "repo-create", "-e", "authenticated-sha256")
cmd(archiver, "create", "test", "input")
with changedir("output"):
with patch.object(xattr, "setxattr", patched_setxattr_EACCES):
@@ -934,7 +934,7 @@ def test_do_not_fail_when_percent_is_in_file_name(archivers, request):
os.makedirs(os.path.join(archiver.input_path, "dir%p"))
xattr.setxattr(b"input/dir%p", b"user.attribute", b"value")
cmd(archiver, "repo-create", "-e", "none-sha256")
cmd(archiver, "repo-create", "-e", "authenticated-sha256")
cmd(archiver, "create", "test", "input")
with changedir("output"):
with patch.object(xattr, "setxattr", patched_setxattr_EACCES):
@@ -1162,7 +1162,7 @@ def test_extract_write_error_at_flush_is_a_warning(archivers, request):
create_regular_file(archiver.input_path, "small1", size=1024)
create_regular_file(archiver.input_path, "small2", size=1024)
cmd(archiver, "repo-create", "-e", "none-sha256")
cmd(archiver, "repo-create", "-e", "authenticated-sha256")
cmd(archiver, "create", "test", "input")
out = _extract_with_raw_file_class(archiver, NoSpaceRaw, BackupOSError)
# both files got their warning, i.e. the extraction did not stop at the first one.
@@ -1182,7 +1182,7 @@ def test_extract_close_error_is_a_warning(archivers, request):
raise OSError(errno.EIO, "Input/output error")
create_regular_file(archiver.input_path, "file1", size=1024)
cmd(archiver, "repo-create", "-e", "none-sha256")
cmd(archiver, "repo-create", "-e", "authenticated-sha256")
cmd(archiver, "create", "test", "input")
out = _extract_with_raw_file_class(archiver, BadCloseRaw, BackupIOError)
assert f"input/file1: close: [Errno {errno.EIO}] Input/output error" in out
src/borg/testsuite/archiver/key_cmds_test.py
+1 -19
View File
@@ -8,7 +8,7 @@ from ...constants import * # NOQA
from ...constants import KeyBlobStorage
from ...crypto.key import AESOCBKey, CHPOKey, Passphrase, is_keyfile, keyfile_parse
from ...crypto.key import RepoKeyNotFoundError
from ...crypto.keymanager import RepoIdMismatch, NotABorgKeyFile, UnencryptedRepo
from ...crypto.keymanager import RepoIdMismatch, NotABorgKeyFile
from ...helpers import CommandError
from ...helpers import bin_to_hex, hex_to_bin
from ...helpers import msgpack
@@ -142,24 +142,6 @@ def test_borg_key_file_env_keeps_explicit_path(archivers, request, monkeypatch):
assert os.listdir(archiver.keys_path) == []
@pytest.mark.parametrize("mode", ["none-sha256", "none-blake3"])
def test_key_management_unavailable_for_keyless_repo(archivers, request, mode):
# the "none-*" modes have no key at all, so there is nothing to export/import.
archiver = request.getfixturevalue(archivers)
cmd(archiver, "repo-create", f"--encryption={mode}")
export_file = archiver.output_path + "/exported"
if archiver.FORK_DEFAULT:
# a forked/binary run returns the exit code, not the exception (BORG_EXIT_CODES defaults
# to "modern", so that is the specific mcode of UnencryptedRepo, not the generic EXIT_ERROR).
cmd(archiver, "key", "export", export_file, exit_code=UnencryptedRepo.exit_mcode)
cmd(archiver, "key", "import", export_file, exit_code=UnencryptedRepo.exit_mcode)
else:
with pytest.raises(UnencryptedRepo):
cmd(archiver, "key", "export", export_file)
with pytest.raises(UnencryptedRepo):
cmd(archiver, "key", "import", export_file)
def test_key_export_keyfile(archivers, request):
archiver = request.getfixturevalue(archivers)
export_file = archiver.output_path + "/exported"
src/borg/testsuite/archiver/lock_cmds_test.py
+2 -1
View File
@@ -24,12 +24,13 @@ def test_with_lock(tmp_path):
repo_path = tmp_path / "repo"
env = os.environ.copy()
env["BORG_REPO"] = Path(repo_path).as_uri()
env["BORG_PASSPHRASE"] = "waytooeasyonlyfortests" # nosec B105
# test debug output:
print("sys.path: %r" % sys.path)
print("PYTHONPATH: %s" % env.get("PYTHONPATH", ""))
print("PATH: %s" % env.get("PATH", ""))
python = sys.executable or "python3"
command0 = python, "-m", "borg", "repo-create", "--encryption=none-sha256"
command0 = python, "-m", "borg", "repo-create", "--encryption=authenticated-sha256"
# Timings must be adjusted so that command1 keeps running while command2 tries to get the lock,
# so that lock acquisition for command2 fails as the test expects it.
lock_wait = 2
src/borg/testsuite/archiver/mount_cmds_test.py
+1 -1
View File
@@ -443,7 +443,7 @@ def test_migrate_lock_alive(archivers, request):
# Decorate
Lock.migrate_lock = write_assert_data(Lock.migrate_lock)
try:
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
create_src_archive(archiver, "arch")
mountpoint = os.path.join(archiver.tmpdir, "mountpoint")
# In order that the decoration is kept for the borg mount process, we must not spawn, but actually fork;
src/borg/testsuite/archiver/recreate_cmd_test.py
+1 -1
View File
@@ -67,7 +67,7 @@ def test_recreate_exclude_keep_tagged(archivers, request):
@pytest.mark.skipif(not are_hardlinks_supported(), reason="hard links not supported")
def test_recreate_hardlinked_tags(archivers, request): # test for issue #4911
archiver = request.getfixturevalue(archivers)
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
create_regular_file(
archiver.input_path, "file1", contents=CACHE_TAG_CONTENTS
) # "wrong" filename, but correct tag contents
src/borg/testsuite/archiver/repo_create_cmd_test.py
+4 -6
View File
@@ -46,10 +46,8 @@ def test_repo_create_requires_encryption_option(archivers, request):
# the modes that do not encrypt name their id hash themselves, --id-hash does not apply
(["--encryption=authenticated-sha256"], "No (repokey, authenticated-sha256)"),
(["--encryption=authenticated-blake3"], "No (repokey, authenticated-blake3)"),
(["--encryption=none-sha256"], "No (none-sha256)"),
(["--encryption=none-blake3"], "No (none-blake3)"),
# giving the matching --id-hash in addition is accepted
(["--encryption=none-blake3", "--id-hash=blake3"], "No (none-blake3)"),
(["--encryption=authenticated-blake3", "--id-hash=blake3"], "No (repokey, authenticated-blake3)"),
],
)
def test_repo_create_encryption_id_hash_combinations(archivers, request, extra_args, expected):
@@ -60,13 +58,13 @@ def test_repo_create_encryption_id_hash_combinations(archivers, request, extra_a
@pytest.mark.parametrize("mode", ["none", "authenticated"])
def test_repo_create_rejects_bare_unencrypted_mode_names(archivers, request, mode):
# these modes always name their id hash now, e.g. "none-sha256", see #9104.
def test_repo_create_rejects_unsupported_unencrypted_mode_names(archivers, request, mode):
# there is no "none" mode, and the "authenticated-*" modes always name their id hash, see #9104.
archiver = request.getfixturevalue(archivers)
cmd(archiver, "repo-create", f"--encryption={mode}", exit_code=2)
@pytest.mark.parametrize("mode", ["none-sha256", "authenticated-blake3"])
@pytest.mark.parametrize("mode", ["authenticated-sha256", "authenticated-blake3"])
def test_repo_create_rejects_conflicting_id_hash(archivers, request, mode):
# the id hash of these modes is part of the mode name, so a contradicting --id-hash is an error.
archiver = request.getfixturevalue(archivers)
src/borg/testsuite/archiver/return_codes_test.py
+1 -1
View File
@@ -16,7 +16,7 @@ pytest_generate_tests = lambda metafunc: generate_archiver_tests(metafunc, kinds
def test_return_codes(archivers, request):
archiver = request.getfixturevalue(archivers)
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", "archive", "input")
with changedir("output"):
cmd(archiver, "extract", "archive")
src/borg/testsuite/archiver/tar_cmds_test.py
+24 -24
View File
@@ -120,7 +120,7 @@ def test_import_tar(archivers, request, tar_format="PAX"):
archiver = request.getfixturevalue(archivers)
create_test_files(archiver.input_path, create_hardlinks=False) # hard links become separate files
os.unlink("input/flagfile")
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", "src", "input")
cmd(archiver, "export-tar", "src", "simple.tar", f"--tar-format={tar_format}")
cmd(archiver, "import-tar", "dst", "simple.tar")
@@ -182,7 +182,7 @@ def test_import_tar_nfiles(archivers, request):
tarinfo.type = tarfile.SYMTYPE
tarinfo.linkname = "file1"
tar.addfile(tarinfo)
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "import-tar", "dst", "input.tar")
info = json.loads(cmd(archiver, "info", "--json", "dst"))
# as with borg create, each regular file and each hardlink counts, directories/symlinks do not
@@ -197,7 +197,7 @@ def test_import_tar_json(archivers, request):
tarinfo = tarfile.TarInfo("dir/file1")
tarinfo.size = len(data)
tar.addfile(tarinfo, io.BytesIO(data))
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
stats = json.loads(cmd(archiver, "import-tar", "--json", "dst", "input.tar"))["archive"]["stats"]
assert stats["nfiles"] == 1
# fresh repository: all of the file content was new to the repository.
@@ -227,7 +227,7 @@ def test_import_tar_digests(archivers, request):
tarinfo = tarfile.TarInfo("dir/subdir")
tarinfo.type = tarfile.DIRTYPE
tar.addfile(tarinfo)
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "import-tar", "--digests=blake3", "--chunker-params=fixed,131072", "dst", "input.tar")
digests = tar_item_digests(archiver, "dst")
for name, data in contents.items():
@@ -257,7 +257,7 @@ def test_import_unusual_tar(archivers, request):
# ./foo//bar
# ./
tar_archive = os.path.join(os.path.dirname(__file__), "unusual_paths.tar")
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "import-tar", "dst", tar_archive)
files = cmd(archiver, "list", "dst", "--format", "{path}{NL}").splitlines()
assert set(files) == {"foobar", "bar", "foo2", "foo/bar", "."}
@@ -271,7 +271,7 @@ def test_import_tar_with_dotdot(archivers, request):
# Contains this file:
# ../../../../etc/shadow
tar_archive = os.path.join(os.path.dirname(__file__), "dotdot_path.tar")
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
with pytest.raises(ValueError, match="unexpected '..' element in path '../../../../etc/shadow'"):
cmd(archiver, "import-tar", "dst", tar_archive, exit_code=2)
@@ -281,7 +281,7 @@ def test_import_tar_gz(archivers, request, tar_format="GNU"):
archiver = request.getfixturevalue(archivers)
create_test_files(archiver.input_path, create_hardlinks=False) # hard links become separate files
os.unlink("input/flagfile")
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", "src", "input")
cmd(archiver, "export-tar", "src", "simple.tgz", f"--tar-format={tar_format}")
cmd(archiver, "import-tar", "dst", "simple.tgz")
@@ -296,7 +296,7 @@ def test_export_import_tar_zst(archivers, request, suffix):
archiver = request.getfixturevalue(archivers)
create_test_files(archiver.input_path, create_hardlinks=False) # hard links become separate files
os.unlink("input/flagfile")
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", "src", "input")
cmd(archiver, "export-tar", "src", f"simple.{suffix}")
with open(f"simple.{suffix}", "rb") as fd:
@@ -314,7 +314,7 @@ def test_export_import_tar_zst_mt(archivers, request, monkeypatch):
monkeypatch.setattr("borg.compress._zstd_mt_workers", None) # drop the cache
create_test_files(archiver.input_path, create_hardlinks=False) # hard links become separate files
os.unlink("input/flagfile")
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", "src", "input")
cmd(archiver, "export-tar", "src", "simple.tar.zst")
cmd(archiver, "import-tar", "dst", "simple.tar.zst")
@@ -345,7 +345,7 @@ def test_tar_filter_zstd_external(archivers, request):
archiver = request.getfixturevalue(archivers)
create_test_files(archiver.input_path, create_hardlinks=False) # hard links become separate files
os.unlink("input/flagfile")
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", "src", "input")
cmd(archiver, "export-tar", "src", "simple.tar.zst", "--tar-filter=zstd")
with open("simple.tar.zst", "rb") as fd:
@@ -375,7 +375,7 @@ def test_import_concatenated_tar_with_ignore_zeros(archivers, request):
# Clean up for assert_dirs_equal.
os.unlink("the_rest.tar")
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "import-tar", "--ignore-zeros", "dst", "input/concatenated.tar")
# Clean up for assert_dirs_equal.
os.unlink("input/concatenated.tar")
@@ -400,7 +400,7 @@ def test_import_concatenated_tar_without_ignore_zeros(archivers, request):
with open("the_rest.tar", "rb") as the_rest:
concatenated.write(the_rest.read())
os.unlink("the_rest.tar")
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "import-tar", "dst", "input/concatenated.tar")
with changedir(archiver.output_path):
@@ -428,7 +428,7 @@ def test_import_tar_with_dotslash_paths(archivers, request):
assert "./dir/file" in tar_content
# Import the tar file into a Borg repository
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "import-tar", "dotslash", "input/dotslash.tar")
# List the archive contents and verify no paths start with './'
@@ -442,7 +442,7 @@ def test_roundtrip_pax_borg(archivers, request):
archiver = request.getfixturevalue(archivers)
create_test_files(archiver.input_path)
os.remove("input/flagfile") # this would be automagically excluded due to NODUMP
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", "src", "input")
cmd(archiver, "export-tar", "src", "simple.tar", "--tar-format=BORG")
cmd(archiver, "import-tar", "dst", "simple.tar")
@@ -459,7 +459,7 @@ def test_roundtrip_pax_xattrs(archivers, request):
original_path = os.path.join(archiver.input_path, "file")
xa_key, xa_value = b"user.xattrtest", b"not valid utf-8: \xff"
xattr.setxattr(original_path.encode(), xa_key, xa_value)
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", "src", "input")
cmd(archiver, "export-tar", "src", "xattrs.tar", "--tar-format=PAX")
cmd(archiver, "import-tar", "dst", "xattrs.tar")
@@ -557,7 +557,7 @@ def _create_sparse_test_input(input_path):
def test_export_tar_sparse(archivers, request, chunker_params):
archiver = request.getfixturevalue(archivers)
contents = _create_sparse_test_input(archiver.input_path)
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", chunker_params, "test", "input")
cmd(archiver, "export-tar", "test", "dense.tar")
cmd(archiver, "export-tar", "--sparse", "test", "sparse.tar", "--progress")
@@ -584,7 +584,7 @@ def test_export_tar_sparse(archivers, request, chunker_params):
def test_export_tar_sparse_roundtrip(archivers, request, tar_format, chunker_params):
archiver = request.getfixturevalue(archivers)
_create_sparse_test_input(archiver.input_path)
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", chunker_params, "src", "input")
cmd(archiver, "export-tar", "--sparse", f"--tar-format={tar_format}", "src", "sparse.tar")
cmd(archiver, "import-tar", "dst", "sparse.tar")
@@ -598,7 +598,7 @@ def test_export_tar_sparse_not_worthwhile(archivers, request):
archiver = request.getfixturevalue(archivers)
contents = b"X" * 448 + b"\0" * 64
create_regular_file(archiver.input_path, "tinyhole", contents=contents)
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", "--chunker-params=fixed,64", "test", "input")
cmd(archiver, "export-tar", "--sparse", "test", "sparse.tar")
with tarfile.open("sparse.tar") as tar:
@@ -611,7 +611,7 @@ def test_export_tar_sparse_not_worthwhile(archivers, request):
def test_export_tar_sparse_gnutar(archivers, request):
archiver = request.getfixturevalue(archivers)
_create_sparse_test_input(archiver.input_path)
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", SPARSE_CHUNKER_CDC, "test", "input")
cmd(archiver, "export-tar", "--sparse", "test", "sparse.tar")
with changedir("output"):
@@ -629,7 +629,7 @@ def test_export_tar_sparse_hardlinks(archivers, request):
contents = b"\0" * 2 * 65536 + b"data"
create_regular_file(archiver.input_path, "sparse1", contents=contents)
os.link(os.path.join(archiver.input_path, "sparse1"), os.path.join(archiver.input_path, "sparse2"))
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", SPARSE_CHUNKER_CDC, "src", "input")
cmd(archiver, "export-tar", "--sparse", "src", "sparse.tar")
with tarfile.open("sparse.tar") as tar:
@@ -656,7 +656,7 @@ def test_export_tar_sparse_strip_components(archivers, request):
archiver = request.getfixturevalue(archivers)
contents = b"\0" * 2 * 65536 + b"end"
create_regular_file(archiver.input_path, "dir/sparsefile", contents=contents)
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", SPARSE_CHUNKER_CDC, "test", "input")
cmd(archiver, "export-tar", "--sparse", "--strip-components=1", "test", "sparse.tar")
with tarfile.open("sparse.tar") as tar:
@@ -691,7 +691,7 @@ def test_export_tar_sparse_base256_size(archivers, request, monkeypatch):
pytest.skip("monkeypatching does not reach a borg binary")
monkeypatch.setattr(SparseTarInfo, "octal_size_limit", 1)
contents = _create_sparse_test_input(archiver.input_path)
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", SPARSE_CHUNKER_CDC, "src", "input")
cmd(archiver, "export-tar", "--sparse", "src", "sparse.tar")
with tarfile.open("sparse.tar") as tar:
@@ -714,7 +714,7 @@ def test_export_tar_sparse_gnu_format_error(archivers, request):
# --sparse requires a PAX-based tar format, the GNU format cannot store the sparse headers.
archiver = request.getfixturevalue(archivers)
create_regular_file(archiver.input_path, "file", contents=b"x")
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", "test", "input")
if archiver.FORK_DEFAULT:
output = cmd(archiver, "export-tar", "--sparse", "--tar-format=GNU", "test", "out.tar", exit_code=2)
@@ -775,7 +775,7 @@ def test_acl_roundtrip(archivers, request):
pytest.skip("ACLs not supported or not working correctly")
# 2. Create a Borg archive
cmd(archiver, "repo-create", "--encryption=none-sha256")
cmd(archiver, "repo-create", "--encryption=authenticated-sha256")
cmd(archiver, "create", "original", "input")
# 3. export-tar this archive to a tar file
src/borg/testsuite/benchmark_test.py
+1 -1
View File
@@ -27,7 +27,7 @@ def repo_url(request, tmpdir, monkeypatch):
tmpdir.remove(rec=1)
@pytest.fixture(params=["none-sha256", "aes256-ocb"])
@pytest.fixture(params=["authenticated-sha256", "aes256-ocb"])
def repo(request, cmd_fixture, repo_url):
cmd_fixture(f"--repo={repo_url}", "repo-create", "--encryption", request.param)
return repo_url
src/borg/testsuite/cockpit_test.py
+3 -2
View File
@@ -15,7 +15,7 @@ except ImportError:
pytestmark = pytest.mark.skipif(not have_cockpit, reason="can not import BorgCockpitApp, is textual installed?")
def test_cockpit_app_create_archive(tmp_path):
def test_cockpit_app_create_archive(tmp_path, monkeypatch):
if not (is_freebsd or is_win32):
pytest.skip("this slow test shall only run on FreeBSD and Windows")
repo_path = tmp_path / "repo"
@@ -24,7 +24,8 @@ def test_cockpit_app_create_archive(tmp_path):
for i in range(5000):
(input_path / f"test{i}.txt").write_text(f"content {i}")
subprocess.run(["borg", "-r", str(repo_path), "repo-create", "--encryption", "none-sha256"], check=True)
monkeypatch.setenv("BORG_PASSPHRASE", "waytooeasyonlyfortests") # also for the borg the app runs
subprocess.run(["borg", "-r", str(repo_path), "repo-create", "--encryption", "authenticated-sha256"], check=True)
async def run():
app = BorgCockpitApp()
src/borg/testsuite/crypto/crypto_test.py
+4 -4
View File
@@ -9,7 +9,7 @@ from ...crypto.low_level import bytes_to_long, bytes_to_int, long_to_bytes
from ...crypto.low_level import hmac_sha256, argon2_hash
from ...legacy.crypto.low_level import AES
from hashlib import sha256
from ...crypto.key import CHPOKey, AESOCBKey, KeyBase, ChecksumKey
from ...crypto.key import CHPOKey, AESOCBKey, KeyBase, LegacyPlaintextKey
from ...legacy.crypto.key import AESCTRKey as LegacyAESCTRKey
from ...helpers import msgpack, bin_to_hex
@@ -327,11 +327,11 @@ class TestDeriveKey(BaseTestCase):
self.id_key = id_key
def test_derive_key_with_plaintext_key(self):
"""Test derive_key with ChecksumKey (empty crypt_key)"""
key = ChecksumKey(None)
"""Test derive_key with the borg 1.x PlaintextKey (empty crypt_key)"""
key = LegacyPlaintextKey(None)
salt, domain, size = b"salt", b"domain", 16
# ChecksumKey has an empty crypt_key, so the derived key should be based on salt and domain only
# PlaintextKey has an empty crypt_key, so the derived key should be based on salt and domain only
derived_key = key.derive_key(salt=salt, domain=domain, size=size)
expected = sha256(b"" + salt + domain).digest()[:size]
self.assert_equal(derived_key, expected)
src/borg/testsuite/crypto/key_test.py
+12 -67
View File
@@ -6,7 +6,7 @@ from unittest.mock import MagicMock
import pytest
from ...crypto.key import BLAKE3_MT_THRESHOLD_KIB, get_blake3_mt_threshold
from ...crypto.key import ChecksumKey, Blake3ChecksumKey, keyfile_parse
from ...crypto.key import keyfile_parse
from ...crypto.key import AuthenticatedKey, Blake3AuthenticatedKey
from ...crypto.key import AESCTRKey, Blake2AESCTRKey, Blake2AuthenticatedKey
from ...crypto.key import LegacyAuthenticatedKey
@@ -91,8 +91,6 @@ class TestKey:
# keyfile and repokey are no longer separate classes (storage is a per-key property),
# so each crypto suite appears once here.
# not encrypted, but tagged
ChecksumKey,
Blake3ChecksumKey,
AuthenticatedKey,
Blake3AuthenticatedKey,
# legacy crypto (read-only, borg 1.x)
@@ -132,21 +130,6 @@ class TestKey:
# is storage-agnostic now and always probes repo candidates, even for keyfile keys.
return getattr(self, "key_data", b"")
def test_none_sha256(self):
key = ChecksumKey.create(None, None)
chunk = b"foo"
id = key.id_hash(chunk)
# the chunk id of the "none-*" modes is the plain (unkeyed) hash of the chunk
assert bin_to_hex(id) == "2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae"
assert chunk == key.decrypt(id, key.encrypt(id, chunk))
def test_none_blake3(self):
key = Blake3ChecksumKey.create(None, None)
chunk = b"foo"
id = key.id_hash(chunk)
assert bin_to_hex(id) == "04e0bb39f30b1a3feb89f536c93be15055482df748674b00d26e5a75777702e9"
assert chunk == key.decrypt(id, key.encrypt(id, chunk))
def test_keyfile(self, monkeypatch, keys_dir):
monkeypatch.setenv("BORG_PASSPHRASE", "test")
key = AESCTRKey.create(self.MockRepository(), self.kf_args())
@@ -401,9 +384,7 @@ class TestMACEnvelope:
CRYPT_KEY = bytes(range(64))
ID_KEY = bytes(range(100, 132))
ALL_CLASSES = (ChecksumKey, Blake3ChecksumKey, AuthenticatedKey, Blake3AuthenticatedKey)
KEYED_CLASSES = (AuthenticatedKey, Blake3AuthenticatedKey)
UNKEYED_CLASSES = (ChecksumKey, Blake3ChecksumKey)
ALL_CLASSES = (AuthenticatedKey, Blake3AuthenticatedKey)
class MockRepository:
id = bytes(32)
@@ -411,8 +392,7 @@ class TestMACEnvelope:
def make_key(self, cls, crypt_key=None):
key = cls(self.MockRepository())
if cls.has_secret_key:
key.init_from_given_data(crypt_key=crypt_key or self.CRYPT_KEY, id_key=self.ID_KEY, chunk_seed=0)
key.init_from_given_data(crypt_key=crypt_key or self.CRYPT_KEY, id_key=self.ID_KEY, chunk_seed=0)
return key
@pytest.fixture(params=ALL_CLASSES)
@@ -439,8 +419,6 @@ class TestMACEnvelope:
def test_format_is_stable(self, cls):
# golden vectors: these bytes must not change silently, they are an on-disk format.
expected = {
ChecksumKey: "80001ab01692fad0f0b89f27983cbf51b1bd20ab73de736aea83363b749ee6ba314f",
Blake3ChecksumKey: "90008dec559eda6de95536f198f6b54b3b2728678c94ca1b3f0b77cf37e5b11c9b7c",
AuthenticatedKey: "600033ecaaf8c34d4142fa502278986c8f49efbcbf879de16d3c2767e1252bfb1994",
Blake3AuthenticatedKey: "70006de48e2139f8995790ec81b256e87f40ef5810d140d28396a8d07734ab05a358",
}[cls]
@@ -449,18 +427,8 @@ class TestMACEnvelope:
envelope = key.encrypt(key.id_hash(plaintext), plaintext, aad=b"")
assert bin_to_hex(envelope) == expected + bin_to_hex(plaintext)
@pytest.mark.parametrize("cls", UNKEYED_CLASSES)
def test_unkeyed_modes_are_repo_independent(self, cls):
# no key material at all, so any two repositories of such a mode store identical objects
# for identical input - that is what allows deduplicating them on the filesystem level.
key1, key2 = self.make_key(cls), self.make_key(cls)
payload = b"payload"
id = key1.id_hash(payload)
assert key1.id_hash(payload) == key2.id_hash(payload)
assert key1.encrypt(id, payload, aad=b"aad") == key2.encrypt(id, payload, aad=b"aad")
@pytest.mark.parametrize("cls", KEYED_CLASSES)
def test_keyed_modes_depend_on_crypt_key(self, cls):
@pytest.mark.parametrize("cls", ALL_CLASSES)
def test_tag_depends_on_crypt_key(self, cls):
# the tag key is derived from crypt_key: same crypt_key -> same objects (that is what
# "repo-create --other-repo --copy-crypt-key" gives), different crypt_key -> different tag.
payload = b"payload"
@@ -512,20 +480,9 @@ class TestMACEnvelope:
assert not key.logically_encrypted
assert key.IDHASH_IN_ENC_NAME
assert key.ENC_NAME.endswith("-" + key.IDHASH_NAME)
if cls in self.KEYED_CLASSES:
assert key.has_secret_key
# the envelope tag authenticates every read, so verifying the chunk id on top of that
# is optional (see BORG_ASSERT_ID), like for the AEAD modes.
assert not key.id_check_is_authentication
assert key.STORAGE == KeyBlobStorage.REPO
assert key.LOCATION_CONFIGURABLE
else:
assert not key.has_secret_key
# the checksum can be recomputed by anybody, thus it is no authentication: the chunk id
# check is the only one left and must never be skipped.
assert key.id_check_is_authentication
assert key.STORAGE == KeyBlobStorage.NO_STORAGE
assert not key.LOCATION_CONFIGURABLE
assert key.has_secret_key
assert key.STORAGE == KeyBlobStorage.REPO
assert key.LOCATION_CONFIGURABLE
@pytest.mark.parametrize("cls", ALL_CLASSES)
def test_type_bytes_are_distinct(self, cls):
@@ -535,7 +492,7 @@ class TestMACEnvelope:
assert identify_key(envelope) is cls
def test_authenticated_no_key_workaround(self, monkeypatch):
# without the key material, the keyed modes can not verify the tag - but they can still
# without the key material, these modes can not verify the tag - but they can still
# read the data (that is the point of the workaround, see BORG_WORKAROUNDS).
key = self.make_key(AuthenticatedKey)
payload = b"payload"
@@ -545,7 +502,7 @@ class TestMACEnvelope:
monkeypatch.setattr("borg.crypto.key.AUTHENTICATED_NO_KEY", True)
assert key.decrypt(id, bytes(envelope), aad=b"aad") == payload
@pytest.mark.parametrize("cls", KEYED_CLASSES)
@pytest.mark.parametrize("cls", ALL_CLASSES)
def test_authenticated_no_key_key_gone(self, cls, monkeypatch, tmp_path):
# a completely lost borg key (no repokey object, no keyfile) is also covered by the
# workaround: detect() proceeds with fake key material instead of failing, see #10238.
@@ -568,18 +525,6 @@ class TestMACEnvelope:
# decrypt() skips the verification, so reading works.
assert key.decrypt(id, envelope, aad=b"aad") == payload
def test_unkeyed_modes_verify_despite_workaround(self, monkeypatch):
# the unkeyed modes need no key material, so the workaround must not switch their
# checksum verification off.
monkeypatch.setattr("borg.crypto.key.AUTHENTICATED_NO_KEY", True)
key = self.make_key(ChecksumKey)
payload = b"payload"
id = key.id_hash(payload)
envelope = bytearray(key.encrypt(id, payload, aad=b"aad"))
envelope[5] ^= 1
with pytest.raises(IntegrityError):
key.decrypt(id, bytes(envelope), aad=b"aad")
@pytest.mark.parametrize("cls", (LegacyAuthenticatedKey, Blake2AuthenticatedKey))
def test_legacy_authenticated_no_key_key_gone(cls, monkeypatch, tmp_path):
@@ -674,5 +619,5 @@ def test_key_class_of_needs_key_info():
key_class_of(repository)
with pytest.raises(RepositoryKeyInfoMissing):
key_factory(repository)
repository = MagicMock(encryption="none-sha256", id_hash="sha256")
assert key_class_of(repository) is ChecksumKey
repository = MagicMock(encryption="authenticated-sha256", id_hash="sha256")
assert key_class_of(repository) is AuthenticatedKey
src/borg/testsuite/repoobj_test.py
+23 -68
View File
@@ -1,8 +1,8 @@
import pytest
from ..constants import ROBJ_FILE_STREAM, ROBJ_ARCHIVE_META
from ..crypto.key import AESOCBKey, ChecksumKey, AuthenticatedKey, CHPOKey, LegacyPlaintextKey
from ..helpers import CompressionSpec, msgpack
from ..crypto.key import AESOCBKey, AuthenticatedKey, CHPOKey, LegacyAuthenticatedKey, LegacyPlaintextKey
from ..helpers import CompressionSpec
from ..helpers.errors import Error, IntegrityError
from ..repository import Repository
from ..repoobj import (
@@ -16,6 +16,7 @@ from ..repoobj import (
)
from ..legacy.repoobj import RepoObj1
from ..compress import LZ4
from . import make_test_key
# offsets of object header fields.
CHUNK_ID_OFFSET = len(OBJ_MAGIC) + 1 # after the magic and the version byte
@@ -30,8 +31,8 @@ def repository(tmpdir):
@pytest.fixture
def key(repository):
# "none-sha256" mode: not encrypted and not authenticated, but checksummed.
return ChecksumKey(repository)
# "authenticated-sha256" mode: not encrypted, but the envelope is MAC-authenticated.
return make_test_key(repository)
@pytest.fixture
@@ -40,15 +41,6 @@ def legacy_key(repository):
return LegacyPlaintextKey(repository)
@pytest.fixture
def authenticated_key(repository):
# "authenticated-sha256" mode: not encrypted, but the envelope is MAC-authenticated.
key = AuthenticatedKey(repository)
key.init_from_random_data()
key.init_ciphers()
return key
@pytest.fixture
def aead_key(repository):
key = CHPOKey(repository)
@@ -100,10 +92,13 @@ def test_format_parse_roundtrip_borg1(legacy_key): # legacy
assert edata.startswith(bytes((key.TYPE, compressor.ID, compressor.level)))
def test_borg1_borg2_transition(legacy_key, key):
def test_borg1_borg2_transition(repository, key):
# Borg transfer reads Borg 1.x repository objects (without decompressing them),
# and writes Borg 2 repository objects (providing already-compressed data to avoid recompression).
# The borg 1.x "none" and the borg 2 "none-sha256" mode use the same (unkeyed sha256) chunk ids.
# The borg 1.x "authenticated" and the borg 2 "authenticated-sha256" mode use the same (hmac-sha256)
# chunk ids, if the borg 2 repository is a related one, i.e. has the same id key.
legacy_key = LegacyAuthenticatedKey(repository)
legacy_key.id_key = key.id_key
meta = {} # borg1 does not support this kind of metadata
data = b"foobar" * 10
len_data = len(data)
@@ -195,10 +190,10 @@ def _tamper(cdata, offset):
return bytes(tampered)
@pytest.fixture(params=["key", "authenticated_key", "aead_key"])
@pytest.fixture(params=["key", "aead_key"])
def protected_key(request):
# every borg 2 mode protects the object with a tag over the payload and the AAD - an unkeyed
# checksum for "none-*", a MAC for "authenticated-*", the AEAD tag for the encrypted modes.
# every borg 2 mode protects the object with a tag over the payload and the AAD - a MAC for
# "authenticated-*", the AEAD tag for the encrypted modes.
return request.getfixturevalue(request.param)
@@ -269,29 +264,6 @@ def test_tampered_data_detected(protected_key):
repo_objs.parse(id, tampered, ro_type=ROBJ_FILE_STREAM)
def test_checksum_mode_does_not_authenticate(key):
# The honest limit of the "none-*" modes: their tag is an unkeyed checksum, so somebody who
# modifies an object can just recompute it. Detecting that needs a secret - the
# "authenticated-*" and the encrypted modes have one, this mode does not.
repo_objs = RepoObj(key)
data = b"foobar" * 10
id = repo_objs.id_hash(data)
cdata = repo_objs.format(id, {"custom": "something"}, data, ro_type=ROBJ_FILE_STREAM)
hdr_size = RepoObj.obj_header.size
hdr = RepoObj.ObjHeader(*RepoObj.obj_header.unpack(cdata[:hdr_size]))
header_aad = OBJ_MAGIC + bytes([OBJ_VERSION]) + id
# rewrite the meta slot with attacker-chosen content and a recomputed checksum
forged_meta = dict(repo_objs.parse_meta(id, cdata, ro_type=ROBJ_FILE_STREAM), custom="forged")
forged_meta_slot = key.encrypt(id, msgpack.packb(forged_meta), aad=header_aad + b"M")
forged = (
RepoObj.obj_header.pack(hdr.magic, hdr.version, hdr.chunk_id, len(forged_meta_slot), hdr.data_size)
+ forged_meta_slot
+ cdata[hdr_size + hdr.meta_size :]
)
assert repo_objs.parse_meta(id, forged, ro_type=ROBJ_FILE_STREAM)["custom"] == "forged"
def test_header_aad_tamper_detected_at_key_layer(protected_key):
# Calls key.encrypt()/key.decrypt() directly with header_aad, to check that every byte of
# header_aad (magic, version, chunk_id) is authenticated, not just chunk_id.
@@ -440,27 +412,11 @@ def test_assert_id_places_env_var_mandatory_place(aead_key, monkeypatch):
assert "verify_data: always verifies, can not be configured" in str(exc_info.value)
def test_assert_id_never_skipped_for_unauthenticated_key(key, monkeypatch):
# ChecksumKey ("none-sha256" mode): the envelope checksum is unkeyed and thus no authentication,
# so the id check is the only integrity check reads have and must happen no matter what the
# user configured.
monkeypatch.setenv("BORG_ASSERT_ID", "") # verify nowhere - but this is not switchable off
assert key.id_check_is_authentication
repo_objs = RepoObj(key)
id = repo_objs.id_hash(b"foobar" * 10)
cdata = wrong_content_object(repo_objs, id)
for place in ASSERT_ID_PLACES:
with pytest.raises(IntegrityError):
repo_objs.parse(id, cdata, ro_type=ROBJ_FILE_STREAM, assert_id_place=place)
def test_assert_id_configurable_for_authenticated_key(authenticated_key, monkeypatch):
def test_assert_id_configurable_for_authenticated_key(key, monkeypatch):
# AuthenticatedKey ("authenticated-sha256"): the envelope MAC authenticates every read (the
# chunk id is in the AAD), so verifying the id on top of that is configurable, like for AEAD.
monkeypatch.setenv("BORG_ASSERT_ID", "") # verify at none of the configurable places
assert not authenticated_key.id_check_is_authentication
repo_objs = RepoObj(authenticated_key)
repo_objs = RepoObj(key)
id = repo_objs.id_hash(b"foobar" * 10)
cdata = wrong_content_object(repo_objs, id)
@@ -481,14 +437,13 @@ def validator_input(repo_objs, data):
return chunk_id, obj[: hdr_size + meta_size]
@pytest.mark.parametrize("key_class", [ChecksumKey, AuthenticatedKey, CHPOKey, AESOCBKey])
@pytest.mark.parametrize("key_class", [AuthenticatedKey, CHPOKey, AESOCBKey])
def test_object_validator_checks_the_sizes_for_every_envelope(key_class):
# data_size == csize + the envelope overhead must hold for each key family; changing meta_size
# or data_size must fail validation.
key = key_class(None)
if hasattr(key, "init_from_random_data"):
key.init_from_random_data()
key.init_ciphers()
key.init_from_random_data()
key.init_ciphers()
repo_objs = RepoObj(key)
chunk_id, head = validator_input(repo_objs, b"payload" * 100)
validate = object_validator(repo_objs)
@@ -501,9 +456,9 @@ def test_object_validator_checks_the_sizes_for_every_envelope(key_class):
@pytest.mark.parametrize("meta", [b"not a mapping", 42, [1, 2, 3], {}, {"csize": "17"}, {"csize": True}])
def test_object_validator_rejects_metadata_without_a_usable_csize(monkeypatch, meta):
# in the modes that authenticate without a secret key the metadata slot can unpack to any
# value. Each of these must come back as "invalid", not as an exception out of the validator.
repo_objs = RepoObj(ChecksumKey(None))
# if the tag is not verified (authenticated_no_key workaround), the metadata slot can unpack to
# any value. Each of these must come back as "invalid", not as an exception out of the validator.
repo_objs = RepoObj(make_test_key())
chunk_id, head = validator_input(repo_objs, b"payload" * 100)
monkeypatch.setattr(repo_objs, "parse_meta", lambda *args, **kwargs: meta)
assert not object_validator(repo_objs)(chunk_id, head)
@@ -512,7 +467,7 @@ def test_object_validator_rejects_metadata_without_a_usable_csize(monkeypatch, m
def test_object_validator_propagates_an_unexpected_exception(monkeypatch):
# answering "invalid" for a bug would answer it for every object of every pack, so anything
# other than a failed authentication or unpacking reaches the caller.
repo_objs = RepoObj(ChecksumKey(None))
repo_objs = RepoObj(make_test_key())
chunk_id, head = validator_input(repo_objs, b"payload" * 100)
def parse_meta_raising_valueerror(*args, **kwargs):
@@ -528,7 +483,7 @@ def test_object_validator_accepts_every_compression(compression):
# data_size == csize + the envelope overhead is what pins data_size, so every compressor must
# record the whole payload it produced as csize. The obfuscating one pads the payload and
# records the padded size.
repo_objs = RepoObj(ChecksumKey(None))
repo_objs = RepoObj(make_test_key())
repo_objs.compressor = CompressionSpec(compression).compressor
chunk_id, head = validator_input(repo_objs, b"payload" * 100)
assert object_validator(repo_objs)(chunk_id, head)
src/borg/testsuite/repository_test.py
+17 -15
View File
@@ -14,12 +14,13 @@ from .. import repository as repository_module
from ..cache import chunkindex_is_invalid, delete_chunkindex_from_repo, write_chunkindex_invalid
from ..compress import CNONE
from ..constants import MAX_CLOCK_SKEW, ROBJ_FILE_STREAM
from ..crypto.key import CHPOKey, ChecksumKey
from ..crypto.key import CHPOKey
from ..helpers import IntegrityError, Location, bin_to_hex
from ..hashindex import ChunkIndex, ChunkIndexEntry
from ..repository import Repository, MAX_DATA_SIZE, MAX_VALIDATED_META_SIZE, propagate_rsh, rest_serve_command
from ..repository import PackWriter, PackReader, PackTracker, superseded_gap_ranges
from ..repoobj import RepoObj, OBJ_MAGIC, OBJ_VERSION, object_validator
from . import make_test_key
from .hashindex_test import H
from .repoobj_test import CHUNK_ID_OFFSET, DATA_SIZE_OFFSET, META_SIZE_OFFSET
@@ -2346,10 +2347,10 @@ def test_pack_reader_resync_recovers_from_size_past_the_pack_end():
]
def none_repo_objs():
# a RepoObj with a "none-*" key and no compression: it formats real objects (tagged metadata
def plain_repo_objs():
# a RepoObj with an "authenticated-*" key and no compression: it formats real objects (tagged metadata
# slot, csize) and stores their payload as it is.
repo_objs = RepoObj(ChecksumKey(None))
repo_objs = RepoObj(make_test_key(None))
repo_objs.compressor = CNONE()
return repo_objs
@@ -2363,7 +2364,7 @@ def real_chunk(repo_objs, data):
def test_pack_reader_reads_no_further_than_the_pack_end():
# the walk asks for META_READ_SIZE bytes per header, so the last objects of a pack have fewer
# bytes left than that. Every read stays inside the pack and asks for at least a header.
repo_objs = none_repo_objs()
repo_objs = plain_repo_objs()
id1, obj1 = real_chunk(repo_objs, b"payload-one")
id2, obj2 = real_chunk(repo_objs, b"payload-two")
pack = bytes(obj1) + bytes(obj2)
@@ -2386,7 +2387,7 @@ def test_pack_reader_resync_rejects_a_header_with_a_wrong_data_size(shape):
# pack leaves the header parseable, so the walk checks it against csize from the tagged metadata
# and drops obj1. The shapes are where the wrong size points: into obj3, exactly onto obj3's
# header, and back into obj1 itself.
repo_objs = none_repo_objs()
repo_objs = plain_repo_objs()
_, obj1 = real_chunk(repo_objs, b"A" * 100)
id2, obj2 = real_chunk(repo_objs, b"B" * 100)
id3, obj3 = real_chunk(repo_objs, b"C" * 100)
@@ -2407,7 +2408,7 @@ def test_pack_reader_resync_rejects_a_header_with_a_wrong_data_size(shape):
def test_pack_reader_resync_rejects_a_header_with_a_wrong_meta_size():
# the metadata slot's tag covers the slot, so a wrong meta_size fails it.
repo_objs = none_repo_objs()
repo_objs = plain_repo_objs()
_, obj1 = real_chunk(repo_objs, b"A" * 100)
id2, obj2 = real_chunk(repo_objs, b"B" * 100)
obj1 = bytearray(obj1)
@@ -2418,7 +2419,7 @@ def test_pack_reader_resync_rejects_a_header_with_a_wrong_meta_size():
def test_pack_reader_resync_rejects_a_header_with_a_wrong_chunk_id():
# the chunk id is part of the AAD of the metadata slot's tag.
repo_objs = none_repo_objs()
repo_objs = plain_repo_objs()
_, obj1 = real_chunk(repo_objs, b"A" * 100)
id2, obj2 = real_chunk(repo_objs, b"B" * 100)
obj1 = bytearray(obj1)
@@ -2428,10 +2429,10 @@ def test_pack_reader_resync_rejects_a_header_with_a_wrong_chunk_id():
def test_pack_reader_resync_starts_past_the_last_validated_object():
# "none-*" mode, no compression: obj1's payload holds a complete, intact object - a user backed
# "authenticated-*" mode, no compression: obj1's payload holds a complete, intact object - a user backed
# up such bytes. obj2's header is broken. The walk validated obj1, so the scan starts past it
# and never sees the decoy; obj1 is kept.
repo_objs = none_repo_objs()
repo_objs = plain_repo_objs()
_, decoy = real_chunk(repo_objs, b"decoy")
id1, obj1 = real_chunk(repo_objs, b"A" * 30 + decoy + b"A" * 30)
assert decoy in obj1 # the decoy is in the pack verbatim
@@ -2528,13 +2529,14 @@ def test_pack_reader_resync_accepts_an_object_with_corrupt_data(tmp_path):
repo_objs.parse(real_id, bytes(obj2), ro_type=ROBJ_FILE_STREAM)
def test_pack_reader_resync_rejects_damaged_user_content_without_a_key(tmp_path):
# In "none-*" mode with no compression, user content lands in the pack as it is, so a backed up
# file can contain something shaped like an object. The metadata slot's checksum covers the
def test_pack_reader_resync_rejects_damaged_user_content_without_encryption(tmp_path):
# In "authenticated-*" mode with no compression, user content lands in the pack as it is, so a
# backed up file can contain something shaped like an object. The metadata slot's tag covers the
# object header, so damaged candidate bytes are still ruled out - what these modes can not rule
# out is an intact object put into a file on purpose, there being no secret to tell them apart.
# out is an intact object (written with the same key) put into a file on purpose: the tag is
# deterministic and binds an object to its chunk id alone.
repository = Repository(str(tmp_path / "repo"), create=True)
repo_objs = RepoObj(ChecksumKey(repository))
repo_objs = RepoObj(make_test_key(repository))
repo_objs.compressor = CNONE()
decoy = bytearray(repo_objs.format(repo_objs.id_hash(b"decoy"), {}, b"decoy", ro_type=ROBJ_FILE_STREAM))
hdr = RepoObj.ObjHeader(*RepoObj.obj_header.unpack(bytes(decoy[: RepoObj.obj_header.size])))