Special tags like @PROT are kept. --clear can be combined with --add
to replace the normal tags, it is mutually exclusive with --set.
Previously, a bare "--set" cleared all tags, but that is not possible
anymore since --set now takes exactly one tag.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
They used nargs="*" and thus swallowed the archive NAME if it was given
after them, e.g. "borg tag --set keep aid:1ddaae55" failed.
Now each option takes exactly one tag and can be given multiple times,
e.g. "borg tag --add a --add b NAME".
Also add examples to the "borg tag" help.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
ArchiveInfo now carries the parsed ArchiveItem (as an attribute, not a
tuple field, so equality and hashing are unchanged). Archive accepts an
ArchiveInfo instead of a name or id; if it carries the metadata, the
archive is neither looked up again (1 list + 1 load) nor its metadata
loaded again (1 load). Without it, Archive looks the archive up by id as
before, e.g. for the archives of borg 1.x repositories.
ArchiveFormatter (repo-list, prune --list, ...), the per-archive loops
of info, list, tag, find, compact, recreate and analyze, and transfer
(for the source archives) pass the ArchiveInfo. repo-list with the
default format for 1000 archives with 50 ms emulated store latency takes
2.1 s instead of 238 s.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Listing the archives (repo-list and every archive match) read each archive
metadata object with its own repository.get, i.e. one store roundtrip per
archive. Archive metadata objects are small and usually each in a tiny pack
of its own, which is the case Store.gather is made for.
Repository.gather_many reads objects in batches (at most 1000 objects, or
until 16 MiB) with one store.gather per batch, keeping the order of the ids.
Objects in cached packs are sliced from the cache; unknown or still buffered
ids and batches hitting a missing or truncated pack fall back to get().
Archives._infos uses it: repo-list --short for 1000 archives with 50 ms
emulated store latency takes 2.0 s instead of 56 s.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
borgstore 0.7.0 adds Store.gather (many byte ranges with one call, one
roundtrip for REST) and fixes partial loads with a negative offset from
cached namespaces.
Store.defrag now requires all sources to be items of the given namespace,
which borg's defrag calls (hex pack names in "packs") already satisfy.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
In borg 2, the positional argument of borg key import is the key backup
path, not the repository. Like most of the docs, assume BORG_REPO is set.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Update the data structures and security docs: the lock objects are in the
store object envelope, locking needs the key (it is loaded before waiting for
the lock), unreadable lock objects are treated as exclusive locks, and what
the store still sees about the locks. Add a change log entry.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The lock objects below locks/ were stored as plaintext JSON, named by the
store hash of that JSON. Anybody with read access to the store could see
which hosts and processes use the repository, and as the content is easy to
guess, even the names alone could reveal it.
Now the lock content is stored in the repository key's store object
envelope, like the index/ and cache/ objects (encrypted and authenticated in
the encrypting modes, authenticated only in the authenticated-* modes), bound
to the repository id and the locks namespace. The name is the store hash of
the envelope. Lock objects are encrypted with encrypt_oneshot(), as a
LockRefresher thread may refresh the lock while the main thread encrypts.
storelocking stays independent of the crypto code: Lock gets seal/unseal
callables, Repository provides them. A Lock keeps using the key it was made
with, so our own lock objects stay readable for us.
A lock object that can not be unsealed (corrupt, tampered with, written by
an older borg 2 beta, ...) is treated as a foreign exclusive lock (fail
closed): it only expires if the store's clock confirms that it was not
written for longer than the stale timeout; without store-side mtimes, only
break-lock removes it. borg warns once about it.
break-lock now loads the key, too (it shows the lock owners with --info).
repo-delete (also with --force) needs the key, as locking needs it.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Repository.open() now loads the key (via key_factory, or a key_loader given
to Repository) before it acquires the lock, see Repository.acquire_lock().
This is a preparation for protecting the lock objects with the key (#8386),
and it also means that borg asks for the passphrase before waiting for the
lock (and no lock is held while the user types the passphrase).
- key_factory() returns the repository's key if it was loaded already, so
commands loading the key later (Manifest.load, check, ...) do not ask for
the passphrase again.
- the "other" repository (--other-repo) gets a key_loader using the
BORG_OTHER_* settings.
- repo-create locks after it created the key and before it writes the config
(until then, the store is not a repository for other clients).
- tests opening a Repository directly: set a key (or key_loader) before the
lock is taken, or open it with lock=False.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
AEADKeyBase.encrypt() uses the key's current session: the shared cipher and
its IV counter. Two threads calling it concurrently could encrypt two messages
with the same IV, which must never happen with AES-OCB or ChaCha20-Poly1305.
encrypt_oneshot() encrypts in a fresh session (new random session id, IV 0)
and neither uses nor changes the key's current session, so it is safe to call
from any thread. decrypt() reads the result like any other envelope.
This is for rare, small objects that may get written from a background
thread, like the repository lock objects (a LockRefresher thread refreshes
the lock). For the MAC-only (authenticated-*) keys, encrypt() has no state and
encrypt_oneshot() is the same.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
RepoObj.parse_header(buf) checks the header length, OBJ_MAGIC and the
version and returns (ObjHeader, None) or (None, problem). parse_meta(),
parse(), object_validator, PackReader._parse_header and
Repository.get(read_data=False) use it instead of unpacking the header
themselves; each keeps only its own size checks.
The error messages now agree: a wrong magic is "no object header" and a
wrong version is "unsupported object version N" everywhere.
get(read_data=False) now rejects a bad magic or version before it uses
meta_size to size its read.
A rewritten copy of a corrupt pack gets a new id that passes borg check. repo-compress keeps these packs unchanged and warns to run "borg check --repair --verify-data".
A rewritten or merged copy of a corrupt pack gets a new id that passes borg check. compact keeps these packs unchanged and warns to run "borg check --repair --verify-data".
borg check crashed with an uncaught IntegrityError (or ObjectNotFound)
when one of an archive's item_ptrs chunks (the list of item metadata
chunk ids) was corrupted (or missing).
The archives check now reports such a chunk as an error and continues
with the item metadata chunk ids from the remaining item_ptrs chunks.
With --repair, the archive is rebuilt from the items it could read.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
borg 1.x protected the key of the authenticated and authenticated-blake2
modes with the passphrase (pbkdf2, "sha256" algorithm), like the key of
the encrypted modes. The legacy authenticated key classes lacked
Pbkdf2FileMixin, so borg 2 could not decrypt that key and --from-borg1
(e.g. borg transfer) failed with "passphrase is incorrect".
The test uses keys exported from borg 1.4.5 authenticated and
authenticated-blake2 repositories.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Opening a borg 1.x repository without --from-borg1 crashed with a traceback while
loading the config/config store object, because a borg 1.x repository has a config
*file* where borg 2 has the config/ namespace (NotADirectoryError for a local
repository, HTTP 500 via ssh://; InvalidRepository on Windows).
borgstore 0.6.4 reports a name below an object as ObjectNotFound on all platforms,
so require it. If config/config is not found, look at "config" via the store
backend and raise Repository.LegacyRepository (rc 29) if it is a file, telling the
user to use --from-borg1 (e.g. with borg transfer).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
--from-borg1 via ssh:// (e.g. borg transfer from a borg 1.x repository)
failed with a borg 1.x "borg serve" on the remote host:
- the client called info() after open, which borg 1.x servers do not
have ("Borg server is too old for info"),
- then close() sent the close RPC, which borg 1.x servers do not have
either ("RPC method close is not valid"), masking the first error,
- get_manifest is also missing on borg 1.x servers,
- borg 1.x servers pack bytes into the msgpack "raw" type, which the
client unpacks to str, so e.g. the repository id and object data
arrived as str.
For borg 1.x servers, the client now uses repository version 1 (the only
version borg 1.x serves), does not send the close RPC (borg 1.x closes
the repository when stdin gets closed), gets the manifest object via get
and converts the bytes results back to bytes.
Tested with a real borg 1.4.5 "borg serve": repo-list, version,
repo-create --other-repo and transfer with --from-borg1, repokey and
keyfile mode. The testsuite emulates a borg 1.x server.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Describe ssh:// repository URLs for current repositories (served by a
remote "borg serve --rest") and for legacy borg 1.x repositories (with
--from-borg1, served by a remote "borg serve") and add a change log
entry.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
If the first request to the remote "borg serve" fails (e.g. a borg 1.x
borg serve does not know --rest, or borg is not installed there), raise
an Error showing the server's stderr and a hint (BORG_REMOTE_PATH,
--from-borg1) instead of a "Local Exception" traceback.
Repository.close() does not raise from closing the store while unwinding
an error anymore (like the lock release before it): a dead store fails
to close, too, which masked the original error.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
ssh:// now addresses a current repository via REST over stdio over ssh
(served by a remote "borg serve --rest"); with --from-borg1, it still
addresses a legacy borg 1.x repository via the legacy RPC protocol.
The testsuite uses ssh://__testsuite__/path (like the legacy client) to
run "borg serve --rest" locally instead of via ssh.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
illumos does not overcommit memory: fork() must reserve swap for the
child's entire address space. The big pytest-xdist workers therefore
sporadically failed to spawn subprocesses (borg serve, for the
remote_archiver tests) with OSError: [Errno 12] "Not enough space"
(illumos' strerror for ENOMEM) when the image's small swap was
exhausted - the swap-backed tmpfs /tmp and ZFS-ARC-squeezed availrmem
eat into the same reservation pool. Seen e.g. in the omniOS job of
runs 35858263044 (PR #10411) and 35913850908 (master).
Add a 4 GiB sparse swap zvol before running the tests so fork()
reservations have room; being sparse, it takes no pool space unless
actually paged to. Also log swap -lh / swap -sh for diagnosability.
Co-Authored-By: Claude Fable 5 <[email protected]>
