RobustUnpacker's resync heuristic looked for a msgpack map whose first key
is one of the known item keys. As items are packed with sorted keys, an
item written by a newer borg whose alphabetically first key is unknown to
this version would not have been found as a resync point after damage, and
its items would have been lost by check --repair.
valid_msgpacked_dict() now accepts any first key that is a short msgpack
str looking like an item key (lowercase identifier: [a-z0-9_]+), without a
list of known keys. Keys with other characters, non-str keys and empty
keys are still rejected, so the scan does not attempt an unpack at every
map-shaped byte offset of damaged data.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The manifest carried the union of all item keys ever written to the repo, so
that an older borg's check would not reject items with keys a newer borg
introduced: check treated an item with a key outside that list as invalid,
and --repair dropped it from the rebuilt archive.
That mechanism was fragile: check --repair rebuilds a lost manifest from
the running version's hardcoded list, and a later --repair on an older
version would then delete every item with newer keys. It also caught next
to nothing in borg 2, where the item stream is authenticated as a whole.
Now check only requires the "path" and "mtime" keys to be present, keeps
items with unknown keys as they are (Item preserves them, so --repair
writes them back unchanged) and warns once per archive about the unknown
keys, as they were possibly written by a newer borg. The resync heuristic
of RobustUnpacker uses the hardcoded ITEM_KEYS.
The manifest does not store item keys anymore; a config["item_keys"] list
(older borg 2) or a top-level "item_keys" list (borg 1.x) is accepted and
ignored on load, and the next manifest write drops it. The manifest content
is now just the version and the (optional) feature flags.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Without the timestamp, the manifest content only changes when a newer borg
version adds item keys (or when feature flags get set), so rewriting it
after every archive operation just stored the same content again under a
new ciphertext.
Manifest.write() now compares the packed manifest with what load() read
from the repository and skips the store write if they are equal. The
callers stay as they are: repo-create and the check --repair rebuild
still always write (nothing was loaded), and an archive operation writes
only if there is something new to store, e.g. after a borg upgrade or to
drop a legacy "timestamp" entry.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Since the manifest timestamp replay check is gone (#10371), the timestamp
had no remaining purpose except being shown as "last_modified" in the JSON
output of repo-info, repo-list and create. There is no repository-level
modification time anymore, so that key is removed from the JSON output
(and its docs) instead of replacing it with a poor substitute.
The manifest keeps version 2. A "timestamp" entry, as present in borg 1.x
manifests and in manifests written by older borg 2 versions, is still
accepted (ManifestItem keeps the key as legacy) and ignored when reading;
the next manifest write drops it.
Note: older borg 2 versions fail with a TypeError when accessing a
repository whose manifest has no timestamp (betas are for new repos only).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The security directory used to record the manifest timestamp of the last
access and refused access (RepositoryReplay / RepositoryIDNotUnique) when
the repository's manifest was older than that.
The check was introduced for borg 1.x, where a rolled-back repository could
make the client reuse AES-CTR counter values and where a local chunks cache
newer than the repository would produce archives referencing missing chunks.
Neither applies to borg 2:
- the AEAD modes derive a fresh session key from a random 192-bit session id
per invocation, with IVs counted from zero per session, so no repository
state can cause nonce reuse.
- the chunks index is loaded from the repository itself and files cache
entries referencing unknown chunks are dropped, so a restored older copy
of the repository does not corrupt new archives.
What the check still detected was only a wholesale rollback that includes
the manifest: the archive list lives in archives/ pointers now, so a server
could roll back or drop archives while keeping the current manifest anyway.
In practice it mostly hit users restoring an older repository copy (and some
read-only filesystem situations, #7853) and locked them out.
The manifest timestamp itself stays: it is still written monotonically and
shown by "borg repo-info" as last_modified.
The RepositoryReplay and RepositoryIDNotUnique classes are kept (never
raised) so that their return codes 63 and 64 stay reserved.
Also removes the manifest-timestamp file from the security directory (a
leftover one is ignored), the FAQ entry about the removed warning and the
data-structures doc claim that the timestamp prevents logical replay.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The assert was added in 2017 (6c2c51939) when the archives dict was
stored inside the manifest object and the limited msgpack unpacker
capped its map length. In borg 2 the manifest's "archives" dict is
always empty: each archive has its own archives/<hexid> store entry.
Archives.count() has no cache and lists the archives/ namespace via
borgstore (which deliberately bypasses its cache), so every manifest
write, i.e. every create/delete/undelete/prune/rename/recreate/check,
paid a full store listing (a directory scan plus a stat() per entry
locally, a round trip over ssh/rest) just for this assert.
The only remaining user of MAX_ARCHIVES is prune, which uses it to
size the zero-padding of its per-archive counter keys, so move the
constant there and replace its comment, which still described the
removed manifest unpacker limit.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Review feedback on the shared chunk index:
- Repository.chunks: the docstring said the index is always built lazily and the setter
comment listed the callers that install one; compact now does that too, say so.
- compact_cmd.get_repository_chunks(): the comment claimed the sharing ends when
compact_packs() invalidates the index, but on a dry run and when there is nothing to
compact nothing is invalidated and the index stays shared until close(), which then
persists only what the build flagged F_NEW - nothing after reading the fragments,
everything after a slow rebuild from the packs. Describe all of it.
- cleanup_files_cache(): it runs after save_chunk_index() only when the store changed, so
"may run after".
- test_compact_builds_the_chunk_index_only_once() covered one path through
compact_packs(). Parametrize it over the four: a pack rewrite, a dry run, a slow
rebuild from the packs with no index fragments in the repo, and a merge of tiny packs.
Each builds the index once now; against master's compact they build it 3, 2, 3 and 3
times, so each scenario fails there on its own.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The REST backend's open() does not contact the server, so for rest:// a
missing repository is only detected by the first request (loading
config/readme), which fails with BackendDoesNotExist (HTTP 410) instead of
ObjectNotFound. That escaped Repository.open() as a "Local Exception"
with rc 2, while a missing local repository gives Repository.DoesNotExist
with rc 13 - which e.g. borgmatic relies on to decide whether to run
repo-create.
Co-Authored-By: Claude Opus 5 <[email protected]>
"borg compact" built the chunk index up to three times per run and kept two
copies of it in memory at the same time:
1. get_repository_chunks() builds compact's own index (it needs the usage flags).
2. analyze_archives() reads the archive metadata objects, and resolving their pack
locations lazily builds a second, identical index inside the Repository.
3. compact_packs() calls delete_chunkindex_from_repo(), which drops the repository's
in-memory index, so the archive listing in cleanup_files_cache() built a third one.
The chunk index is the biggest structure borg keeps in memory (about 80 bytes per
chunk plus hash table overhead), so a large repository paid for that twice over.
Share compact's index with the repository while the repository reads through it:
get_repository_chunks() installs it as repository.chunks, so analyze_archives()
resolves pack locations through that same index. The repository only reads pack
locations and F_PENDING from an index, never the F_USED flags or the sizes compact
tracks for --stats, and stored index fragments zero the flags either way - so the
index the repository would have built is the very same one.
The sharing ends in compact_packs(): delete_chunkindex_from_repo() (#9748) drops the
repository's reference before the first store change, and compact does not hand it
back. Nothing needs it there - compact_pack() and merge_packs() get chunks=self.chunks,
--stats and save_chunk_index() use self.chunks directly. So while the persisted index
is gone, the repository holds no in-memory index that Repository.close() could persist
on an aborted run, and save_chunk_index() can empty the index in place with clear=True.
cleanup_files_cache() used to list the archives itself, after save_chunk_index() had
cleared the index, which is what built the third index. It now reuses the names
analyze_archives() already collected, so it needs no repository access at all - and
every archive's metadata is read once per compact run instead of twice.
Verified on the same repository: master and this produce byte-identical repositories,
identical persisted index/* fragments (completed, dry-run, no-op, interrupted and
slow-rebuild runs alike) and identical output, with 3 index builds before and 1 after.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Co-Authored-By: Claude Opus 5 <[email protected]>
The checked-packs tracker (cache/checked-packs) and the per-archive
reference caches (cache/referenced-by-archive.*) carry a checksum of
their content appended to it. Use blake3_256() for it, like for the
object names, so sha256 is not used for any store object checksum any
more. An object with an old checksum fails verification and is rebuilt.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Object names are hex, so give the naming code a hex helper instead of
converting the bytes helper's result at every call site: the keyfile,
repokey, lock, index fragment and files cache naming now use
blake3_256_hex(). The tests use blake3(...).hexdigest() directly.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Every object whose name is a digest of its content is now named by the
256 bit blake3 hash instead of sha256: packs/, index/ fragments, keys/
(repokey borg keys) and locks/ in the repository, automatically placed
keyfiles (the key id is the same hash) and the files cache suffix.
One helper, crypto.key.blake3_256(), computes the hash for all of them
and hashes big inputs (packs, index fragments) multi-threaded from the
existing blake3 threshold on; Blake3ChecksumKey.id_hash reuses it.
Repository.NAME_HASH names the algorithm for borgstore's hash() and
defrag(), which already support blake3.
Not changed: checksums that are appended to content rather than used as
a name (checked-packs tracker, archive reference caches) and the archive
ids, which are the key mode's id hash.
Not backwards compatible with repositories and files caches written by
earlier borg2 betas.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Format version 0x01 was only ever written by borg 2.0.0b22 (before that
there were no packs, b23 and later write 0x02). We don't owe compatibility
across betas, so parse()/parse_meta() now accept only OBJ_VERSION_HEADER_AAD.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The byte counts of a "modified" file were derived from the difference of the
two chunk id SETS, which loses everything about order and multiplicity: a file
whose chunks were only reordered was reported as modified with 0 B added and
0 B removed, and duplicating a chunk added no bytes at all.
Align the two chunk lists as sequences instead (difflib.SequenceMatcher, the
way a text diff aligns lines) and count the chunks that are not part of the
alignment: the ones of archive1 as removed, the ones of archive2 as added.
Insertions, removals, moves and duplicated chunks are now all reflected by the
byte counts.
The common prefix and suffix of the two lists are stripped first. That is what
makes the usual cases cheap (e.g. a file that was appended to) and it also
keeps the matcher away from the long runs of identical chunks it is slow on.
SequenceMatcher still degrades to quadratic runtime on chunk lists that repeat
the same chunk id very often (sparse files, VM images with big all-zero
ranges), so the alignment is skipped above MAX_ALIGN_CHUNKS / MAX_ALIGN_WORK
and the chunk ids are only counted then, which is what borg did before.
Co-Authored-By: Claude Opus 5 <[email protected]>
The S3 tests download a MinIO server (and the mc client) from dl.min.io. Those URLs
now answer HTTP 410 Gone, so "curl -fsSL" exits 22 and the step fails before borg is
even installed. That takes down three native_tests jobs, both oldglibc_binary jobs
(their S3 smoke test needs the same server), the codecov upload (no tests ran, so no
coverage) and, via fail-fast, the rest of the native_tests matrix.
MinIO stopped publishing binaries on GitHub as well: the releases from
RELEASE.2025-10-15T17-29-55Z on carry no assets. So pin the last releases that do
publish them - minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z -
and check each download against the sha256 published next to it.
Release and sums live in the workflow-level env block, shared by the two steps that
download MinIO, so a bump is a one-place change.
Verified: the amd64 and arm64 binaries match the pinned sums, and running the patched
step in a linux/arm64 container downloads both, starts the server and creates the
bucket.
Co-Authored-By: Claude Opus 5 <[email protected]>
- repo-info: append "empty passphrase" to the Encrypted: line if the key was unlocked with an
empty passphrase (for repokey and keyfile storage). The key now remembers this in the new
empty_passphrase attribute, which (unlike logically_encrypted) is independent of the storage.
- security: the "Attempting to access a previously unknown unencrypted repository" warning now
says why the repository counts as unencrypted - a mode that does not encrypt the data
(none-* / authenticated-*) or a repokey with an empty passphrase. Previously, users with an
encrypted repository could not tell why borg called it unencrypted, as repo-info said
"Encrypted: Yes".
- docs (repo-create, quickstart, repo-info): an empty passphrase still encrypts with a random
key, when it may be acceptable (keyfile storage), and that a passphrase can be added later
with "borg key change-passphrase" (unlike with the none-* modes).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Rebuilding the chunks index from the packs walks every object header. With
neither a validate callable nor drop_corrupt_tail, iter_headers raises
IntegrityError at a corrupt header, which reached the user as a traceback with
rc 90.
Catch it in build_chunkindex_from_repo and raise CorruptPack (rc 93, no
traceback) instead, telling the user to run "borg check --repair". Continuing
past the header is not an option there: the resulting index would be incomplete.
Only borg check passes a validator (and, when repairing, drop_corrupt_tail) and
gets past it.
ArchiveChecker builds its own chunks index with a validate callable, but
repository.get() resolves chunk locations through repository.chunks, a separate
index that is built lazily on first access. That rebuild had no validator and no
drop_corrupt_tail, so a check aborted at a corrupt object header it had already
resynced past (or dropped the tail at) when building the checker index - with
--repair as well as without it, where the check stopped halfway through its
diagnosis and told the user to run the command they were running.
Add Repository.chunkindex_validate and Repository.chunkindex_drop_corrupt_tail,
passed on to the lazy rebuild, and set them in ArchiveChecker.check(): the
validator always, so both modes resync the same way, and drop_corrupt_tail only
when repairing, so a check that just diagnoses still never drops a pack tail.
When the chunk ids can not be compared (the archives were created with different
--chunker-params), ItemDiff._content_diff() emitted a "modified" content change
unconditionally, and equal() returned False on the ctime/mtime difference before
ever comparing the content, so a file that was only touched showed up as
"modified: (can't get size)" in the default output mode (--content-only was
correct). Hardlink siblings whose ctime changed were affected the same way.
Now _content_diff() compares the sizes and then the content itself (once - the
chunk iterators can be consumed only once) and emits the count-less "modified"
change only if the content really differs; _content_equal() reuses that result.
Also: cover the default output mode with different chunker params in the tests,
add a changelog entry.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Add a note to the borg extract help/docs and a FAQ entry: restoring the owner
and group of files, device nodes, privileged file flags and privileged xattrs
needs root privileges. When running as a normal user, borg silently keeps the
extracted files owned by the invoking user (no warning, exit code unaffected),
skips privileged flags, and skips privileged xattrs and device nodes with a
warning. --numeric-ids does not change that.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
ItemDiff._content_diff() computed the added/removed byte counts from the set
differences of the chunk ids of both items, and actual_change() in diff_cmd.py
dropped every "modified" change with 0 B added and 0 B removed - a filter meant
for a file that was only touched (same chunk list, only ctime/mtime differ).
A file whose content changed by reordering (or duplicating) its existing chunks
has equal chunk id sets, so its content change was dropped as well: the default
output showed only the time changes, --content-only printed an item with an
empty "changes" list and --stats did not count it.
Now _content_diff() emits no content change when the chunk lists are equal and
a "modified" change with the (possibly 0/0) counts when they differ, so the
0/0 filter is gone.
Also document where the byte counts come from (diff epilog, frontends.rst) and
add a test for the reordered-chunks case.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The "Archive Differencing" section of internals/frontends.rst already
describes the ctime/mtime changes introduced by #7335 (and the --stats
line), so #7486 was in substance done, but the section still had two
claims that do not match the current output:
- the intro only mentioned file contents and user/group/mode as what an
ItemDiff can be about, although timestamps and the presence of an item
(added/removed directories, links, devices, fifos) are reported as well.
- the "changes" example claimed two changes for a file whose contents and
ownership changed, but an ownership change is reported as "changed
owner" plus "changed user" and/or "changed group", so that gives at
least three. Use the common contents + mtime + ctime case instead,
which also matches the example output shown below it.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
#10346 did this only in the oldglibc_binary job (glibc239 binaries). The
Linux binaries from native_tests (ubuntu-26.04, glibc243) still bundled
two OpenSSLs: the system's libcrypto, which borg's crypto extension uses
and PyInstaller bundles, and the copy statically linked into the
manylinux wheel of cryptography (a paramiko dependency, sftp extra).
Build cryptography from source against the system OpenSSL there, too,
for the Linux "binary" matrix entries - the runner image has the Rust
toolchain. The step checks that the extension links libcrypto
dynamically and loads the same OpenSSL version as the system's openssl
program; the smoke test of the built binary checks that the bundled
extension links the bundled libcrypto, and prints the bundle sizes.
The stripping of the debug symbols and the trimming of the botocore
models from #10346 live in the shared PyInstaller spec, so they already
apply to these binaries. See #10345.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The Linux binaries were about 70 MiB as a single file and unpacked to
152 MiB at every start, see #10345. Three changes:
Strip the debug symbols on Linux: pythons built with the default CFLAGS
carry -g, and the spec had strip=False everywhere, so about half of the
Linux binaries was debug info. Enable PyInstaller's strip for the
collected shared libraries on Linux (only there: on macOS it would
interfere with code signing, and it is not recommended on Windows).
Python tracebacks do not need the symbols, only gdb would. Measured on
the glibc239 x86_64 bundle: 152 -> 78 MiB unpacked, 69.6 -> 43.4 MiB
compressed.
Bundle only the botocore service models borg needs: botocore (boto3, s3
extra) ships the JSON service models of all ~430 AWS services, 23 MiB,
about 14 MiB in the compressed binary. borg only ever creates an s3
client - also for S3-compatible services like MinIO or Backblaze B2,
which just get a different endpoint URL - so the spec keeps only the s3
model, the shared top-level files and the models the credential
providers may need (sts, sso, sso-oidc).
In the oldglibc_binary job, build cryptography (a paramiko dependency
from the sftp extra) from source against the OpenSSL built there,
instead of using the manylinux wheel that statically links its own copy
of OpenSSL - the binary contained two OpenSSLs. The runner image has the
Rust toolchain this needs.
The job now checks that the bundled OpenSSL and Python libraries are the
ones built there by their GNU build IDs (the files are stripped now),
that cryptography links and loads that OpenSSL, that no bundled shared
library still carries debug info, that the other botocore models are
gone, prints the bundle sizes, and smoke-tests the frozen binary against
a MinIO server - the test suite's S3 test runs borg from the venv, not
the binary.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The Linux binaries from native_tests are built on ubuntu-26.04 and thus
need glibc 2.43, and the x86_64 one does not run on CPUs below x86-64-v3
("Illegal instruction"). The new oldglibc_binary job builds on
ubuntu-24.04 and ubuntu-24.04-arm (glibc 2.39, the oldest GitHub-hosted
Ubuntu image, with its baseline x86-64 toolchain and packages) and
produces borg-linux-glibc239-{x86_64,arm64}-gh.
Fixes#10342.
Ubuntu 24.04 only has OpenSSL 3.0 and Python 3.12 (and setup-python's
3.14 for it links the system OpenSSL 3.0), so the job builds OpenSSL
3.5.8 (no-tests, no-docs - the OpenSSL test suite is very slow and is
never run) and Python 3.14.7 from source, pinned by version and sha256,
links Python and borg's crypto extension against that OpenSSL, and
PyInstaller bundles those libraries. OpenSSL is built with an rpath to
its prefix, so that its libssl and the openssl program do not pick up
the system's OpenSSL 3.0 libraries, which have the same SONAMEs. Python
is a PGO build like the setup-python ones; its PGO training run (a
subset of python's test suite) runs in parallel worker processes. The
job checks that the ssl module, borg's crypto extension and the frozen
binary really use the OpenSSL and Python built there and that no
bundled ELF file needs a newer glibc than the binary's name promises.
The binary is built on every run of the job and the test suite runs
against it. On tags, the binaries get a provenance attestation and the
release job (which now depends on this job) attaches them and their
sigstore bundles to the GitHub release.
Also: scripts/glibc_check.py no longer crashes on an ELF file without
versioned glibc symbols.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Only BORG_PASSPHRASE was removed from the environment given to subprocesses
(the command given in BORG_PASSCOMMAND, --paths-from-command / --content-from-command commands, tar
filter commands, borg with-lock commands, fusermount/umount, fakeroot
detection, ssh for legacy borg 1.x repos). Now also remove:
- BORG_NEW_PASSPHRASE, BORG_OTHER_PASSPHRASE: secrets, like BORG_PASSPHRASE.
- BORG_PASSCOMMAND, BORG_OTHER_PASSCOMMAND: often embed secrets or paths to them.
- BORG_PASSPHRASE_FD, BORG_OTHER_PASSPHRASE_FD: the file descriptors are not
inherited by the subprocess (Popen closes them), so the numbers would be
dangling and misleading there.
- BORGSTORE_REST_PASSWORD: borgstore's REST authentication password.
The other borg env vars (BORG_REPO, BORG_KEY_FILE, ...) are not secrets and
with-lock scripts may rely on them, so they are kept.
Also document this in the BORG_PASSPHRASE help text and add tests.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
