mypy checks the tests against low_level.pyi, so the test hook the
chunked AEAD update introduced has to be declared there, too.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
EVP_EncryptUpdate() / EVP_DecryptUpdate() take the input length as a
C int, and the AEAD ciphers passed the whole message in one call, with
all lengths held in C ints: a message of 2 GiB or more failed with
SystemError (negative size for the result allocation), and above 4 GiB
the payload was silently truncated to its length modulo 2^32.
Feed the input (data as well as AAD) to OpenSSL in chunks of at most
1 GiB and keep every byte count in a Py_ssize_t. Both AEAD modes (OCB,
chacha20-poly1305) stream, so chunked calls give the same result as a
single one; the chunk size is a module variable so a test can shrink it
and cover chunk boundaries that fall inside a cipher block. A second
test round-trips a 2 GiB message, skipped unless BORG_TESTS_BIG_MEMORY
is set.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Remove stale index entries of all written packs before indexing any object, so the result
does not depend on the pack order. Look up each pack once, update the docs.
- the envelope AAD also holds the repository id and a tag that tells a full
object name and the namespace of a hashed-name object (index/) apart, so
objects of another repository using the same key material, or envelopes
made for the other kind of name, fail the authentication.
- index fragments are not hashed on load to verify their names: the
authentication already proves their content (borg check still verifies
the names). The error message says "authentication failed".
- a corrupt chunk index (a fragment that fails the authentication or does
not deserialize) aborts every command that needs it, with the new
CorruptChunkIndexFragment error (rc 94): run "borg check --repair".
Only the commands that rewrite the whole index anyway, under an exclusive
lock (borg check --repair, borg compact, borg repo-compress), rebuild it
from the packs; borg compact --dry-run aborts.
- borg check: such a fragment is an index error, --repair rebuilds the
index from the packs. A check without --repair never rebuilds the index,
and the index cross-check is skipped if the store hash check already found
the index corrupt. Help text updated.
- remember the plaintext hash of the fragments read in a session and do not
store a fragment with known content again (unless forced), so e.g. a
crashed repack converges again.
- debug get-obj/put-obj/delete-obj: load the key via with_repository(),
including the security checks.
- tests: open_repository() always loads the real key, more tests; docs.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Describe the store object envelope and update what the store sees of the
index/ and cache/ objects, see #9819, #10235.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
cache/checked-packs and cache/referenced-by-archive.* are now protected like
the objects in the packs (encrypted and authenticated in the encrypting modes,
authenticated only in the authenticated-* modes), instead of having a store
hash appended.
The envelope binds each object to its name: a references cache copied to
another archive's name is ignored and rebuilt by scanning that archive.
PackTracker takes the repository instead of its store.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The index/ fragments are now protected like the objects in the packs:
encrypted and authenticated in the encrypting modes, authenticated only in
the authenticated-* modes. A fragment's name is the store hash of the stored
envelope, so borg check keeps verifying index/ by name, also server-side.
As the envelope is randomized in the encrypting modes, there is no "fragment
with this content already exists" upload skip any more.
A fragment that does not match its name or fails the authentication (e.g.
a plaintext fragment written by an older borg 2 beta) is corrupt: the chunk
index is rebuilt from the packs.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
key_factory, key_creator and Manifest set the key as the repository key,
so the index/ and cache/ store objects can be protected by it.
borg repo-create writes the empty chunk index after the key was created,
Repository.create() does not write one any more.
borg debug get-obj, put-obj and delete-obj load the key, as they access
the chunk index; delete-obj without a key aborts now instead of keeping the
superseded gap bytes.
Tests: repositories opened by the tests get a key (a test key, or the real
key via open_repository).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The index/ and cache/ store objects are to be protected by the repository
key, like the objects in the packs. These methods wrap a store object in the
key's envelope (encrypted and authenticated in the encrypting modes,
authenticated only in the authenticated-* modes), binding the object name
(or, for content-addressed objects, the namespace) into the envelope as AAD.
Without a key, they fail with the new Repository.KeyRequired error (rc 28).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A new passphrase (e.g. for borg key change-passphrase or borg key add)
can now also be given via a passcommand or a file descriptor, not only
via BORG_NEW_PASSPHRASE: the BORG_NEW_* variables form a group like
BORG_* and BORG_OTHER_* (mutually exclusive within the group,
independent of the other groups), checked before the documented
BORG_PASSPHRASE / BORG_PASSCOMMAND / BORG_PASSPHRASE_FD fallback,
which stays as it is.
Adapted from borgbackup/borg#8995 (the group selection was rewritten:
the original selected the NEW passcommand on the wrong flag, which
broke BORG_OTHER_PASSCOMMAND and never used BORG_NEW_PASSCOMMAND).
Co-authored-by: Łukasz Stelmach <[email protected]>
Co-Authored-By: Claude Fable 5 <[email protected]>
An interrupted --repair index rebuild no longer reports the index as rebuilt, finish() warns
that its rebuild can not be interrupted, and ChunkIndexRebuildInterrupted is listed in frontends.rst.
New deployment chapter deriving the borg passphrase from a YubiKey OTP
slot's HMAC-SHA1 challenge-response via BORG_PASSCOMMAND: only a random
challenge is stored, the passphrase itself is computed by the token on
every unlock (the long-proven KeePassXC-style mechanism, as also
documented for restic in the Arch wiki).
Covers programming a slot with and without touch (touchless =
unattended backups, --touch = presence-gated interactive use),
redundant twin YubiKeys via a shared slot secret, adding the derived
passphrase as an additional borg key to an existing repository, the
paper copy of the derived passphrase as the recovery path, and honest
caveats: Yubico-proprietary (not on other vendors' keys nor the
FIDO2-only Security Key series - the age guide covers those), two
scarce slots, device-bound vs presence-gated threat model, macOS Input
Monitoring permission, and ykman instead of the deprecated ykpers
tooling.
Co-Authored-By: Claude Fable 5 <[email protected]>
New deployment chapter showing how to combine BORG_PASSCOMMAND with
age: a strong random borg passphrase stored age-encrypted, decrypted on
demand - and, via the age plugin ecosystem, bound to hardware: any
FIDO2 security key (age-plugin-fido2-hmac, touch-gated by CTAP spec,
interactive use), YubiKey PIV (age-plugin-yubikey, policies allow
silent unattended use), TPM 2.0 (age-plugin-tpm) and Apple Secure
Enclave (age-plugin-se) for machine-bound unattended backups.
The guide starts with a plain no-hardware example to introduce the
pattern, gives an at-a-glance table of what works unattended vs. what
requires user presence, covers multi-recipient redundancy (several
YubiKeys / TPM unlocking the same passphrase), the offline paper copy
as the recovery path, revocation (change-passphrase, not just
re-encrypting) and an honest threat-model note (at-rest protection;
presence-gated options prevent silent unlocking, device-bound ones do
not).
This is the supported way to use hardware-backed key protection with
borg, instead of borg-native hardware integrations. See the discussion
in borgbackup/borg#8995 / borgbackup/borg#10399.
Co-Authored-By: Claude Fable 5 <[email protected]>
Assert the passphrase error text in test_check_aborts_on_wrong_passphrase,
as test_check_aborts_without_key_info does for its error, and in both tests
assert that neither the repository check nor the archive consistency check
started, instead of the looser "Starting" not in output.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
All tests in repo_space_cmd_test.py reserve space, which makes borg
generate 64 MiB objects via os.urandom. os.urandom is currently
extremely slow on Haiku, so these tests run into the 300s test timeout
there.
Co-Authored-By: Claude Fable 5 <[email protected]>
build_chunkindex_from_repo() reads every pack of the repository when it has
to do a slow rebuild. On a large repository that takes a long time, and it
did not look at sig_int, so a Ctrl-C during it had no visible effect until
it was done.
It now takes an interruptible kwarg, default False. With it set, the pack
loop checks sig_int before reading the next pack, discards the index it
built so far and raises ChunkIndexRebuildInterrupted. A partial index is
never returned nor stored: it lacks chunks that are still in the packs, so
a check would report them as lost and --repair would drop the archives'
references to them.
Interruptibility is opt-in per call site because one caller must not have
it. ArchiveChecker.finish() rebuilds the index with sig_int already set
(check() calls it after the interrupt, #9850) and then calls
delete_chunkindex_invalid(), which declares the stored index to match the
packs. That only holds once every pack was indexed, so that rebuild has to
run to completion.
The two call sites that do get interruptible=True are the rebuild in
ArchiveChecker.check(), which runs before any archive is looked at, and the
--repair index rebuild in Repository.check(). The latter catches the
exception, warns, and leaves index_repaired False, so the existing summary
reports "Interrupted full repository check, index still corrupt so far."
and check() returns False. The corrupt index/ fragments are still there and
the next use rebuilds from the packs.
The index object verification loop in Repository.check() stays
uninterruptible on purpose: index_errors decides whether the packs are
verified and whether the index is rebuilt, so breaking out of it early
makes an interrupted check on a fully corrupt index report no problems. It
is bounded by the index size, not by the repository size.
Also update the check epilog: it still described a SIGINT-deaf key recovery
scan in ArchiveChecker.make_key(), which does not exist anymore since
Regular files were listed with the path stored in the archive, all other
member types with the raw tar member name, e.g. "./dir" instead of "dir".
Show the stored path for all of them.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The BORG tar format transfers the hlid in the BORG.item.meta pax header, so
hard links do not become separate files there. Keep them in the test input,
so the test also covers hard links plus stripping for the pax header path.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Regular files were listed with their stripped path, all other member types
with the unstripped tar member name. Show the stripped path for all of them.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The path elements are counted on the normalized member name, so a leading
"./" is not a path element. GNU tar and bsdtar count it, so the same
tarball needs a count that is higher by one there.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Remove the given number of leading path elements from the tar member
names when creating the archive items; members whose path has fewer or
equally many elements are silently skipped (same semantics as extract /
export-tar). Hard link targets are stripped accordingly so they keep
referencing their target's chunks; symbolic link targets are content and
are left unchanged. Also works for BORG format tars, where the items are
restored from pax headers.
Co-Authored-By: Claude Fable 5 <[email protected]>
test_extract ran "borg extract" with the cwd set to the tmpdir that the
repo_url fixture also uses for the repository, the keys dir and the cache dir.
That directory is therefore never empty, so since extract refuses to extract
into a non-empty directory (see #10057), all 8 parametrisations failed with
rc 33 (ExtractionDirNotEmpty) already on the first (and, with the pedantic
defaults rounds=1/iterations=1/warmup_rounds=0, only) call.
Extract into a dedicated, per-round empty subdirectory created by pedantic's
setup hook, which is not timed. Using --continue instead would keep the test
green, but it makes extract stat and compare the files that are already there,
which is not the extract that this benchmark is supposed to measure. The
per-round directory also keeps the benchmark correct if rounds or warmup_rounds
are ever raised above the defaults.
The other benchmark tests are not affected: they do not write into the cwd.
Co-Authored-By: Claude Opus 5 <[email protected]>
Extracting into a non-empty directory replaces existing files that are in
the way and leaves all other existing files as they are, so the result is a
mix of existing and extracted files. Accidentally running "borg extract" in
a directory that is in use (like a home directory) thus replaced current
files by their archived state and blended the archive into the directory.
Now borg aborts with Archive.ExtractionDirNotEmpty (rc 33) before writing
anything if the current directory is not empty, unless --continue is given.
This also applies when only a part of the archive is extracted. --dry-run
and --stdout do not write into the directory, so they are not checked.
No new option: --continue already is the way to extract into a directory
that has contents. Its help and the extract docs now describe what it does
(skip files that are fully extracted already, replace other existing files)
and that a damaged file with unchanged size and mtime must be removed
before it can be extracted again.
Tests that extracted repeatedly into the same directory now use --continue
or a fresh directory.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
So the borg mount tests also run on Windows and the Windows binary
can mount (if WinFsp is installed).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
mfusepy already knows how to talk to the FUSE 2 API of WinFsp, so this is
about the borg side:
- mountpoint: WinFsp mounts on an unused drive (X:) or creates (and later
removes) the mountpoint directory, so it must not exist yet.
- there is no fork: borg mount always stays in the foreground. Ctrl-C or
Ctrl-Break unmount. borg umount is not supported there.
- WinFsp ignores the "ro" mount option, a read-only volume has to be
requested from the init callback (FSP_FUSE_CAP_READ_ONLY).
- WinFsp makes a security descriptor from uid / gid / mode and checks all
access against it. The archived uids / gids do not mean anything on
Windows, so the uid and gid mount options are left to WinFsp and default
to -1 (the user who mounts).
- volname gives the volume label (default: borgfs).
- st_birthtime is given to mfusepy, WinFsp uses it as the creation time.
- readlink of something that is not a symlink fails with EINVAL (WinFsp
tries that for "/" to find out whether symlinks are supported).
Tests: on Windows, fuse_mount runs borg mount as a child process and stops
it like Ctrl-Break does. WinFsp does not support hard link counts. With
mfusepy 3.1.1, symlinks show up as regular files, because WinFsp has getdir
and readlink in a different order in struct fuse_operations than libfuse has.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
TW