The hash function that names store objects (packs/, index/, keys/, locks/),
that is appended as a checksum to cache/checked-packs and the reference
caches, and that names keyfiles and the files cache suffix, is now defined
in exactly one place: store_hash() in crypto/key.py, together with
STORE_HASH_NAME (its name for borgstore) and STORE_HASH_SIZE. Every user
calls store_hash(data).digest() or .hexdigest() and refers to it as the
"store hash" in code comments and docs, so the hash function can be
changed there without touching anything else. The tests use store_hash()
as well.
The docs state which hash function it is (blake3) in one place, the store
hash section of data-structures.rst, and refer to it by that name
everywhere else.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Every object whose name is a digest of its content is now named by the
256 bit blake3 hash instead of sha256: packs/, index/ fragments, keys/
(repokey borg keys) and locks/ in the repository, automatically placed
keyfiles (the key id is the same hash) and the files cache suffix.
One helper, crypto.key.blake3_256(), computes the hash for all of them
and hashes big inputs (packs, index fragments) multi-threaded from the
existing blake3 threshold on; Blake3ChecksumKey.id_hash reuses it.
Repository.NAME_HASH names the algorithm for borgstore's hash() and
defrag(), which already support blake3.
Not changed: checksums that are appended to content rather than used as
a name (checked-packs tracker, archive reference caches) and the archive
ids, which are the key mode's id hash.
Not backwards compatible with repositories and files caches written by
earlier borg2 betas.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
borg called platformdirs with the appname only, so on Windows the
appauthor component defaulted to the appname and all default
directories ended up under a doubled AppData\...\borg\borg path.
Pass appauthor=False everywhere (no vendor directory level; on
Linux/macOS the appauthor is unused, so nothing changes there) and
roaming=True for the config dir, restoring the layout the CHANGES
upgrade notes had documented as intended all along:
- config (incl. keys): C:\Users\<user>\AppData\Roaming\borg
(follows the user's roaming profile in domain environments)
- data (incl. security state): C:\Users\<user>\AppData\Local\borg
- cache: C:\Users\<user>\AppData\Local\borg\Cache
- runtime: C:\Users\<user>\AppData\Local\Temp\borg
This matches what comparable tools do on Windows (pip, rclone and
kopia keep the configuration in the roaming AppData and machine
state/caches in the local AppData, all without a vendor directory).
The cache keeps platformdirs' opinionated "Cache" subdir so the
CACHEDIR.TAG written into the cache root stays out of the data dir.
Only Windows is affected. The first Windows binary was only released
with 2.0.0b23, so beta users needing to migrate should be rare: move
the keys from the old %LOCALAPPDATA%\borg\borg\keys to the new
location (or set the BORG_* dir env vars to the old paths); cache and
security state are recreated as needed.
Docs: update the environment help topic (and the generated
environment.rst.inc), FAQ, key/transfer usage docs and CHANGES.
Co-Authored-By: Claude Fable 5 <[email protected]>
All command lines were executed against scratch repositories and all
output samples regenerated from real runs (prettified, not invented):
- recreate: the example used the removed --recompress; show
borg repo-compress instead; refresh the borg info samples.
- diff: default/text output has modified:/added:/removed: labels and
mtime/ctime brackets now; the JSON-lines example used borg1 keys
and lacked --json-lines on the command; add --content-only.
- mount: example passed the repository as a positional (it would have
been taken as the mountpoint); use -r.
- tar: the archive-copy loop was a bash syntax error and used
borg list with repo-list-only format keys; replaced with a working
while-read loop over borg repo-list (verified end-to-end, identical
archive contents in the destination repo).
- key: regenerate repo-create/change-passphrase samples (hex-id named
keyfiles, Key location line, no "Synchronizing index"); the import
example silently did nothing without --key-location=keyfile; fix
the Windows keys path (AppData\Local, not Roaming); explain why the
keyfile name changes on passphrase change.
- transfer: the borg1->borg2 upgrade steps were missing --from-borg1
and used a ssh:// destination URL (rejected for current repos);
fix the Windows keys path.
- notes: --noatime and --nobsdflags no longer exist (atime is opt-in
via --atime); stray "|" removed.
- compact: segments wording removed.
- info/repo-info/repo-delete/list/find: output samples regenerated
(new labels/order, UTC offsets, repo version line, prompt text;
isomtime format and symlink size corrected in list).
- general: fix broken inline markup (stray backtick).
- serve: the authorized_keys forced command needs --rest (it is
deliberately not client-suppliable); document the actual client
argument allowlist including --backend.
Co-Authored-By: Claude Fable 5 <[email protected]>
"--encryption=none" and "--encryption=authenticated" have not existed since
the id hash became part of the mode name for the modes that do not encrypt:
they are now "none-sha256"/"none-blake3" and "authenticated-sha256"/
"authenticated-blake3". The examples in docs/usage/repo-create.rst still used
the old names, so copying them gave an argparse error.
The "authenticated --id-hash=blake3" example was doubly wrong: those modes
name their id hash themselves and reject a separate --id-hash. Replaced it
with plain "authenticated-blake3"/"none-blake3" examples and dropped the
now-wrong claim that the non-encrypting modes only support sha256.
Also:
- "--encryption=aes-ocb" (docs/usage/key.rst, docs/quickstart_example.rst.inc)
is not a valid mode either, it is "aes256-ocb".
- scripts/rest_perf_test.sh used "--encryption none".
- regenerated the affected man pages; borg-repo-create(1) additionally picked
up the already-fixed --key-location help text.
All examples were verified by actually running them.
Rename chunks cache / cached chunks index to chunks index or index, renaming parameters/variables where appropriate to avoid 'cache' terminology for index.
The multi-key feature (#9743) added the "borg key add", "borg key list"
and "borg key remove" subcommands but never wired up their docs:
- scripts/make.py: map key_add/key_list/key_remove to the "key" usage
group, so build_man can locate their examples (it previously aborted
with FileNotFoundError: docs/usage/key_add.rst).
- docs/usage/key.rst: include the three new generated snippets so they
show up on the HTML key page.
- regenerate the affected docs: new key_{add,list,remove}.rst.inc usage
snippets, new borg-key-{add,list,remove}.1 man pages, and borg-key.1
(SYNOPSIS + SEE ALSO now reference the new subcommands).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Borg used to read the manifest's key-type byte and then look for the key in
exactly one place (keyfile or repokey) depending on the key class that byte
selected. As a result every crypto suite was duplicated into a keyfile class
and a repokey class that differed only in TYPE, NAME, ARG_NAME and STORAGE.
Now key *location* is independent of the type byte: detection tries keyfiles
first and repokeys afterwards until a passphrase unlocks a key. The type byte
still selects the crypto suite (id hash, MAC, cipher) to instantiate. Where a
key is stored (keyfile vs repokey) is therefore a per-key property
(self.storage), not a separate class, so a repository may even hold a mix of
keyfile- and repo-stored borg keys.
With storage decoupled from class identity, the keyfile/repokey class pairs
collapse into one class per crypto suite:
- modern AEAD: AESOCBKey, CHPOKey, Blake3AESOCBKey, Blake3CHPOKey
- legacy borg 1.x (read-only): AESCTRKey, Blake2AESCTRKey
There is now exactly one type byte per modern crypto suite (the old separate
repokey type bytes 0x11/0x21/0x31/0x41 were removed; borg2 is beta and only
needs to read repos it created). identify_key() matches on TYPES_ACCEPTABLE.
CLI: --encryption selects only the crypto suite (aes-ocb, chacha20-poly1305,
blake3-aes-ocb, blake3-chacha20-poly1305, authenticated*, none); the storage
location is chosen with the new --key-location=repokey|keyfile (default
repokey). The old combined modes (repokey-aes-ocb etc.) were removed.
borg key import also gained --key-location. borg key change-location no longer
swaps key classes or rewrites the manifest; it just re-saves the unlocked key
at the new location.
Keyfile removal (key remove, change-location) now overwrites the keyfile with
random data via secure_erase() before unlinking, consistent with save().
borg 1.x legacy read compatibility is preserved (the legacy class merge is a
behavior-preserving rename; the legacy type bytes incl. PASSPHRASE stay in
TYPES_ACCEPTABLE).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Running 'borg key import' on a keyfile repository with the BORG_KEY_FILE
environment variable set works correctly if the BORG_KEY_FILE file
already exists. However, the command crashes if the BORG_KEY_FILE file
does not exist:
$ BORG_KEY_FILE=newborgkey borg key import /home/strager/borg-backups/straglum borgkey
Local Exception
Traceback (most recent call last):
[snip]
File "[snip]/borg/crypto/key.py", line 713, in sanity_check
with open(filename, 'rb') as fd:
FileNotFoundError: [Errno 2] No such file or directory: '[snip]/newborgkey'
Platform: Linux straglum 5.0.0-25-generic #26~18.04.1-Ubuntu SMP Thu Aug 1 13:51:02 UTC 2019 x86_64
Linux: debian buster/sid
Borg: 1.1.11 Python: CPython 3.7.7 msgpack: 0.5.6
PID: 15306 CWD: /home/strager/Projects/borg
sys.argv: ['[snip]/borg', 'key', 'import', '/home/strager/borg-backups/straglum', 'borgkey']
SSH_ORIGINAL_COMMAND: None
Make 'borg key import' not require the BORG_KEY_FILE file to already
exist.
This commit does not change the behavior of 'borg key import' without
BORG_KEY_FILE. This commit also does not change the behavior of 'borg
key import' on a repokey repository.
