The security directory used to record the manifest timestamp of the last
access and refused access (RepositoryReplay / RepositoryIDNotUnique) when
the repository's manifest was older than that.
The check was introduced for borg 1.x, where a rolled-back repository could
make the client reuse AES-CTR counter values and where a local chunks cache
newer than the repository would produce archives referencing missing chunks.
Neither applies to borg 2:
- the AEAD modes derive a fresh session key from a random 192-bit session id
per invocation, with IVs counted from zero per session, so no repository
state can cause nonce reuse.
- the chunks index is loaded from the repository itself and files cache
entries referencing unknown chunks are dropped, so a restored older copy
of the repository does not corrupt new archives.
What the check still detected was only a wholesale rollback that includes
the manifest: the archive list lives in archives/ pointers now, so a server
could roll back or drop archives while keeping the current manifest anyway.
In practice it mostly hit users restoring an older repository copy (and some
read-only filesystem situations, #7853) and locked them out.
The manifest timestamp itself stays: it is still written monotonically and
shown by "borg repo-info" as last_modified.
The RepositoryReplay and RepositoryIDNotUnique classes are kept (never
raised) so that their return codes 63 and 64 stay reserved.
Also removes the manifest-timestamp file from the security directory (a
leftover one is ignored), the FAQ entry about the removed warning and the
data-structures doc claim that the timestamp prevents logical replay.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The assert was added in 2017 (6c2c51939) when the archives dict was
stored inside the manifest object and the limited msgpack unpacker
capped its map length. In borg 2 the manifest's "archives" dict is
always empty: each archive has its own archives/<hexid> store entry.
Archives.count() has no cache and lists the archives/ namespace via
borgstore (which deliberately bypasses its cache), so every manifest
write, i.e. every create/delete/undelete/prune/rename/recreate/check,
paid a full store listing (a directory scan plus a stat() per entry
locally, a round trip over ssh/rest) just for this assert.
The only remaining user of MAX_ARCHIVES is prune, which uses it to
size the zero-padding of its per-archive counter keys, so move the
constant there and replace its comment, which still described the
removed manifest unpacker limit.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The REST backend's open() does not contact the server, so for rest:// a
missing repository is only detected by the first request (loading
config/readme), which fails with BackendDoesNotExist (HTTP 410) instead of
ObjectNotFound. That escaped Repository.open() as a "Local Exception"
with rc 2, while a missing local repository gives Repository.DoesNotExist
with rc 13 - which e.g. borgmatic relies on to decide whether to run
repo-create.
Co-Authored-By: Claude Opus 5 <[email protected]>
The byte counts of a "modified" file were derived from the difference of the
two chunk id SETS, which loses everything about order and multiplicity: a file
whose chunks were only reordered was reported as modified with 0 B added and
0 B removed, and duplicating a chunk added no bytes at all.
Align the two chunk lists as sequences instead (difflib.SequenceMatcher, the
way a text diff aligns lines) and count the chunks that are not part of the
alignment: the ones of archive1 as removed, the ones of archive2 as added.
Insertions, removals, moves and duplicated chunks are now all reflected by the
byte counts.
The common prefix and suffix of the two lists are stripped first. That is what
makes the usual cases cheap (e.g. a file that was appended to) and it also
keeps the matcher away from the long runs of identical chunks it is slow on.
SequenceMatcher still degrades to quadratic runtime on chunk lists that repeat
the same chunk id very often (sparse files, VM images with big all-zero
ranges), so the alignment is skipped above MAX_ALIGN_CHUNKS / MAX_ALIGN_WORK
and the chunk ids are only counted then, which is what borg did before.
Co-Authored-By: Claude Opus 5 <[email protected]>
The S3 tests download a MinIO server (and the mc client) from dl.min.io. Those URLs
now answer HTTP 410 Gone, so "curl -fsSL" exits 22 and the step fails before borg is
even installed. That takes down three native_tests jobs, both oldglibc_binary jobs
(their S3 smoke test needs the same server), the codecov upload (no tests ran, so no
coverage) and, via fail-fast, the rest of the native_tests matrix.
MinIO stopped publishing binaries on GitHub as well: the releases from
RELEASE.2025-10-15T17-29-55Z on carry no assets. So pin the last releases that do
publish them - minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z -
and check each download against the sha256 published next to it.
Release and sums live in the workflow-level env block, shared by the two steps that
download MinIO, so a bump is a one-place change.
Verified: the amd64 and arm64 binaries match the pinned sums, and running the patched
step in a linux/arm64 container downloads both, starts the server and creates the
bucket.
Co-Authored-By: Claude Opus 5 <[email protected]>
- repo-info: append "empty passphrase" to the Encrypted: line if the key was unlocked with an
empty passphrase (for repokey and keyfile storage). The key now remembers this in the new
empty_passphrase attribute, which (unlike logically_encrypted) is independent of the storage.
- security: the "Attempting to access a previously unknown unencrypted repository" warning now
says why the repository counts as unencrypted - a mode that does not encrypt the data
(none-* / authenticated-*) or a repokey with an empty passphrase. Previously, users with an
encrypted repository could not tell why borg called it unencrypted, as repo-info said
"Encrypted: Yes".
- docs (repo-create, quickstart, repo-info): an empty passphrase still encrypts with a random
key, when it may be acceptable (keyfile storage), and that a passphrase can be added later
with "borg key change-passphrase" (unlike with the none-* modes).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Rebuilding the chunks index from the packs walks every object header. With
neither a validate callable nor drop_corrupt_tail, iter_headers raises
IntegrityError at a corrupt header, which reached the user as a traceback with
rc 90.
Catch it in build_chunkindex_from_repo and raise CorruptPack (rc 93, no
traceback) instead, telling the user to run "borg check --repair". Continuing
past the header is not an option there: the resulting index would be incomplete.
Only borg check passes a validator (and, when repairing, drop_corrupt_tail) and
gets past it.
ArchiveChecker builds its own chunks index with a validate callable, but
repository.get() resolves chunk locations through repository.chunks, a separate
index that is built lazily on first access. That rebuild had no validator and no
drop_corrupt_tail, so a check aborted at a corrupt object header it had already
resynced past (or dropped the tail at) when building the checker index - with
--repair as well as without it, where the check stopped halfway through its
diagnosis and told the user to run the command they were running.
Add Repository.chunkindex_validate and Repository.chunkindex_drop_corrupt_tail,
passed on to the lazy rebuild, and set them in ArchiveChecker.check(): the
validator always, so both modes resync the same way, and drop_corrupt_tail only
when repairing, so a check that just diagnoses still never drops a pack tail.
When the chunk ids can not be compared (the archives were created with different
--chunker-params), ItemDiff._content_diff() emitted a "modified" content change
unconditionally, and equal() returned False on the ctime/mtime difference before
ever comparing the content, so a file that was only touched showed up as
"modified: (can't get size)" in the default output mode (--content-only was
correct). Hardlink siblings whose ctime changed were affected the same way.
Now _content_diff() compares the sizes and then the content itself (once - the
chunk iterators can be consumed only once) and emits the count-less "modified"
change only if the content really differs; _content_equal() reuses that result.
Also: cover the default output mode with different chunker params in the tests,
add a changelog entry.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Add a note to the borg extract help/docs and a FAQ entry: restoring the owner
and group of files, device nodes, privileged file flags and privileged xattrs
needs root privileges. When running as a normal user, borg silently keeps the
extracted files owned by the invoking user (no warning, exit code unaffected),
skips privileged flags, and skips privileged xattrs and device nodes with a
warning. --numeric-ids does not change that.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
ItemDiff._content_diff() computed the added/removed byte counts from the set
differences of the chunk ids of both items, and actual_change() in diff_cmd.py
dropped every "modified" change with 0 B added and 0 B removed - a filter meant
for a file that was only touched (same chunk list, only ctime/mtime differ).
A file whose content changed by reordering (or duplicating) its existing chunks
has equal chunk id sets, so its content change was dropped as well: the default
output showed only the time changes, --content-only printed an item with an
empty "changes" list and --stats did not count it.
Now _content_diff() emits no content change when the chunk lists are equal and
a "modified" change with the (possibly 0/0) counts when they differ, so the
0/0 filter is gone.
Also document where the byte counts come from (diff epilog, frontends.rst) and
add a test for the reordered-chunks case.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The "Archive Differencing" section of internals/frontends.rst already
describes the ctime/mtime changes introduced by #7335 (and the --stats
line), so #7486 was in substance done, but the section still had two
claims that do not match the current output:
- the intro only mentioned file contents and user/group/mode as what an
ItemDiff can be about, although timestamps and the presence of an item
(added/removed directories, links, devices, fifos) are reported as well.
- the "changes" example claimed two changes for a file whose contents and
ownership changed, but an ownership change is reported as "changed
owner" plus "changed user" and/or "changed group", so that gives at
least three. Use the common contents + mtime + ctime case instead,
which also matches the example output shown below it.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
#10346 did this only in the oldglibc_binary job (glibc239 binaries). The
Linux binaries from native_tests (ubuntu-26.04, glibc243) still bundled
two OpenSSLs: the system's libcrypto, which borg's crypto extension uses
and PyInstaller bundles, and the copy statically linked into the
manylinux wheel of cryptography (a paramiko dependency, sftp extra).
Build cryptography from source against the system OpenSSL there, too,
for the Linux "binary" matrix entries - the runner image has the Rust
toolchain. The step checks that the extension links libcrypto
dynamically and loads the same OpenSSL version as the system's openssl
program; the smoke test of the built binary checks that the bundled
extension links the bundled libcrypto, and prints the bundle sizes.
The stripping of the debug symbols and the trimming of the botocore
models from #10346 live in the shared PyInstaller spec, so they already
apply to these binaries. See #10345.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The Linux binaries were about 70 MiB as a single file and unpacked to
152 MiB at every start, see #10345. Three changes:
Strip the debug symbols on Linux: pythons built with the default CFLAGS
carry -g, and the spec had strip=False everywhere, so about half of the
Linux binaries was debug info. Enable PyInstaller's strip for the
collected shared libraries on Linux (only there: on macOS it would
interfere with code signing, and it is not recommended on Windows).
Python tracebacks do not need the symbols, only gdb would. Measured on
the glibc239 x86_64 bundle: 152 -> 78 MiB unpacked, 69.6 -> 43.4 MiB
compressed.
Bundle only the botocore service models borg needs: botocore (boto3, s3
extra) ships the JSON service models of all ~430 AWS services, 23 MiB,
about 14 MiB in the compressed binary. borg only ever creates an s3
client - also for S3-compatible services like MinIO or Backblaze B2,
which just get a different endpoint URL - so the spec keeps only the s3
model, the shared top-level files and the models the credential
providers may need (sts, sso, sso-oidc).
In the oldglibc_binary job, build cryptography (a paramiko dependency
from the sftp extra) from source against the OpenSSL built there,
instead of using the manylinux wheel that statically links its own copy
of OpenSSL - the binary contained two OpenSSLs. The runner image has the
Rust toolchain this needs.
The job now checks that the bundled OpenSSL and Python libraries are the
ones built there by their GNU build IDs (the files are stripped now),
that cryptography links and loads that OpenSSL, that no bundled shared
library still carries debug info, that the other botocore models are
gone, prints the bundle sizes, and smoke-tests the frozen binary against
a MinIO server - the test suite's S3 test runs borg from the venv, not
the binary.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The Linux binaries from native_tests are built on ubuntu-26.04 and thus
need glibc 2.43, and the x86_64 one does not run on CPUs below x86-64-v3
("Illegal instruction"). The new oldglibc_binary job builds on
ubuntu-24.04 and ubuntu-24.04-arm (glibc 2.39, the oldest GitHub-hosted
Ubuntu image, with its baseline x86-64 toolchain and packages) and
produces borg-linux-glibc239-{x86_64,arm64}-gh.
Fixes#10342.
Ubuntu 24.04 only has OpenSSL 3.0 and Python 3.12 (and setup-python's
3.14 for it links the system OpenSSL 3.0), so the job builds OpenSSL
3.5.8 (no-tests, no-docs - the OpenSSL test suite is very slow and is
never run) and Python 3.14.7 from source, pinned by version and sha256,
links Python and borg's crypto extension against that OpenSSL, and
PyInstaller bundles those libraries. OpenSSL is built with an rpath to
its prefix, so that its libssl and the openssl program do not pick up
the system's OpenSSL 3.0 libraries, which have the same SONAMEs. Python
is a PGO build like the setup-python ones; its PGO training run (a
subset of python's test suite) runs in parallel worker processes. The
job checks that the ssl module, borg's crypto extension and the frozen
binary really use the OpenSSL and Python built there and that no
bundled ELF file needs a newer glibc than the binary's name promises.
The binary is built on every run of the job and the test suite runs
against it. On tags, the binaries get a provenance attestation and the
release job (which now depends on this job) attaches them and their
sigstore bundles to the GitHub release.
Also: scripts/glibc_check.py no longer crashes on an ELF file without
versioned glibc symbols.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Only BORG_PASSPHRASE was removed from the environment given to subprocesses
(the command given in BORG_PASSCOMMAND, --paths-from-command / --content-from-command commands, tar
filter commands, borg with-lock commands, fusermount/umount, fakeroot
detection, ssh for legacy borg 1.x repos). Now also remove:
- BORG_NEW_PASSPHRASE, BORG_OTHER_PASSPHRASE: secrets, like BORG_PASSPHRASE.
- BORG_PASSCOMMAND, BORG_OTHER_PASSCOMMAND: often embed secrets or paths to them.
- BORG_PASSPHRASE_FD, BORG_OTHER_PASSPHRASE_FD: the file descriptors are not
inherited by the subprocess (Popen closes them), so the numbers would be
dangling and misleading there.
- BORGSTORE_REST_PASSWORD: borgstore's REST authentication password.
The other borg env vars (BORG_REPO, BORG_KEY_FILE, ...) are not secrets and
with-lock scripts may rely on them, so they are kept.
Also document this in the BORG_PASSPHRASE help text and add tests.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
hlfuse.FUSE(self, mountpoint, options, ...) passed the option list as mfusepy's
third positional parameter, which is raw_fi. Consequences:
- the libfuse options were never applied: the mount was rw instead of ro,
without default_permissions, and user-given -o options like allow_other
were silently ignored (fsname only looked right because it defaulted to the
operations class name, which is "borgfs").
- raw_fi was (accidentally) true, so open/read/release received the
fuse_file_info struct - which they expect - but getattr with a file handle
(kernel fgetattr, e.g. for a read reaching EOF after the attribute timeout)
received the struct too and used it as a dict key: TypeError -> EINVAL.
Visible as "cat: ...: Invalid argument" for any file that takes more than
a second (the default attr_timeout) to read.
Now convert the option list to keyword arguments, request raw_fi explicitly
and take the file handle from the fuse_file_info struct in getattr. Add a
regression test that checks the mount is read-only and reads a file to EOF
after the attribute timeout.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Rename the "Added chunk volume" / "Removed chunk volume" summary lines to
"Added size" / "Removed size" and the JSON keys added_chunk_volume /
removed_chunk_volume to size_added / size_removed. These totals are the sums
over all reported items of what the per-path output calls "added" / "removed"
and what --sort-by size_added / size_removed sorts by, so use that existing
vocabulary instead of introducing a new one. #10327 was not released yet, so
nothing depends on the old names.
Document the --stats summary line of borg diff --json-lines in
docs/internals/frontends.rst, which describes the JSON Lines output for
frontends but did not mention it.
Also add a granularity_sleep() between the two archives of the diff --stats
tests: test_stats_counts_metadata_only_changes expects the input directory to
be reported as changed (mtime/ctime), which fails on filesystems with
whole-second timestamps (reproduced on HFS+: "Changed items: 2" instead of 3)
when both archives are created within the same second. The other tests in that
file already sleep there.
Follow-up to #10327 (#796).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
For a BackupWarning, the args of the warning include the caught BackupError
exception. Storing that in the global warnings list kept the exception - and
thus its traceback and the frames (with all their locals) of the code that
failed - alive for the rest of the borg run, one per warning: for a failed
file in borg extract, that is the extract_item frame including the chunk
data that was being written. borg extract of 300 files of 2.5 MB whose
writes all fail peaked at 876 MiB RSS instead of 409 MiB, and on a full
disk every remaining file adds another chunk.
Store the exception's message text instead, which is what formatting the
warning message with the exception gives anyway.
Add a regression test (the exception wrapped by a BackupWarning must be
freed after print_warning_instance()) and a unit test for the get_ec()
warnings logic.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
TW