Same as BORG_MOUNT_ARCHIVE_DIR_FORMAT, but usable from fstab and autofs
entries; if both are given, the mount option is used. As mount options
are separated by commas, the format can not contain a comma.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
acquire() creates its lock object, waits for the race recheck delay and
lists the lock objects again to detect other clients racing for the lock.
With list-after-write consistency (local filesystems, sftp, rest, AWS S3,
MinIO), at least the client that created its lock object last sees the
other's lock in its second listing, so the delay is not needed for
correctness and the 0.01s default stays.
Stores that only list a new object after a lag (e.g. NFS clients caching
directory listings, some cloud storages via rclone) need a delay of at
least that lag. That lag is a property of the storage, not of the request
latency, so it is not measured, but set via BORG_LOCK_RECHECK_DELAY.
Invalid values abort with an error.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Describe ssh:// repository URLs for current repositories (served by a
remote "borg serve --rest") and for legacy borg 1.x repositories (with
--from-borg1, served by a remote "borg serve") and add a change log
entry.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Extracting into a non-empty directory replaces existing files that are in
the way and leaves all other existing files as they are, so the result is a
mix of existing and extracted files. Accidentally running "borg extract" in
a directory that is in use (like a home directory) thus replaced current
files by their archived state and blended the archive into the directory.
Now borg aborts with Archive.ExtractionDirNotEmpty (rc 33) before writing
anything if the current directory is not empty, unless --continue is given.
This also applies when only a part of the archive is extracted. --dry-run
and --stdout do not write into the directory, so they are not checked.
No new option: --continue already is the way to extract into a directory
that has contents. Its help and the extract docs now describe what it does
(skip files that are fully extracted already, replace other existing files)
and that a damaged file with unchanged size and mtime must be removed
before it can be extracted again.
Tests that extracted repeatedly into the same directory now use --continue
or a fresh directory.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Every borg 2 mode has a borg key now: the AEAD modes (aes256-ocb,
chacha20-poly1305) and the authenticated-sha256 / authenticated-blake3
modes. There is no compatibility with repositories older borg 2 betas
created in a none-* mode.
- crypto.key: remove ChecksumKeyBase, ChecksumKey, Blake3ChecksumKey and
the SHA256NONE / BLAKE3NONE key types.
- remove KeyBase.id_check_is_authentication: all borg 2 envelopes
authenticate the chunk id, so RepoObj.parse verifies the id only at
the BORG_ASSERT_ID places and the mandatory ones.
- repo-create: always show the key / passphrase hint.
- a borg 1.x "none" repository has unkeyed chunk ids, which no borg 2
mode has, so transferring it needs --chunker-params (docs updated).
- tests: use authenticated-sha256 where none-sha256 was used as the
cheap default mode, add make_test_key() for tests needing some key.
The repository swap detection tests now re-label the repokey of the
attacker's repository for the faked repository id.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Opening a repository loaded four store objects: config/readme (sanity check),
config/version, config/id and config/manifest (only read for its key type
byte since the manifest lost all other content). Now there is one plain text
INI object, config/config, e.g.:
# This is a Borg Backup repository.
# See https://borgbackup.readthedocs.io/
[repository]
version = 5
id = <64 hex digits>
encryption = aes256-ocb
id_hash = sha256
The crypto suite of the key is recorded by the --encryption / --id-hash
names, so key_factory() selects the key class from the config without
reading any repository object. Both names are recorded or none; a config
with one but not the other is invalid, an unsupported suite raises
InvalidRepositoryConfig. The repository version is 5; only version 5 is
accepted, there is no code to read the old layout and no migration (betas
are for new repositories only).
The manifest object is gone for borg 2 repositories: Manifest is now only
the in-memory container for key, repo_objs, repository and archives; its
write() and all callers, get_manifest()/put_manifest(), the manifest checks
and rebuild in "borg check", "borg debug dump-manifest", the key type
detection from stored objects (key_from_repository) and the ROBJ_MANIFEST
type are removed. borg 1.x repositories are still read via their manifest
(legacy_key_factory), as it holds their archives list. NoManifestError is
kept (never raised) so that rc 26 stays reserved.
Repository.create() creates all namespace directories in advance
(store.create_levels()) and writes the config (version, id, no key info)
right away, so a repository created via the Python API is an openable
key/value store. "borg repo-create" alone defers the config write
(create_config=False) until after the key was created, so nothing that
looks like a repository exists until repo-create succeeds: any failure or
interruption before that destroys the store (and the keyfile, in keyfile
mode), and a failure inside create() destroys the store as well.
A store without repository config is reported as not a valid repository
(InvalidRepository; DoesNotExist is only for a missing store backend, the
rest:// case from #10365). repo-create on it raises IncompleteRepository
(rc 11, "has no repository config"), as the store backend refuses any
non-empty directory. "borg repo-delete --force" destroys such a store, so
that ssh/rest/s3 users can remove it without other access to the storage,
but only if it has the packs, archives, index and config namespaces every
store borg created has: a data directory, a home directory or a borg 1.x
repository is never destroyed by borg. key_factory() on a config without
key info raises RepositoryKeyInfoMissing (rc 54).
The cache config's [cache]/[integrity] cross-check (manifest id) could
never detect anything and is dropped.
Docs: new "Repository config" section replaces the manifest section
(including how to recreate a lost config by hand, as check --repair can
not), the layout list, packs and security internals (where the protection
against a swapped crypto suite lives), repo-info example, error list and the
remaining manifest mentions are updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The hash function that names store objects (packs/, index/, keys/, locks/),
that is appended as a checksum to cache/checked-packs and the reference
caches, and that names keyfiles and the files cache suffix, is now defined
in exactly one place: store_hash() in crypto/key.py, together with
STORE_HASH_NAME (its name for borgstore) and STORE_HASH_SIZE. Every user
calls store_hash(data).digest() or .hexdigest() and refers to it as the
"store hash" in code comments and docs, so the hash function can be
changed there without touching anything else. The tests use store_hash()
as well.
The docs state which hash function it is (blake3) in one place, the store
hash section of data-structures.rst, and refer to it by that name
everywhere else.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Every object whose name is a digest of its content is now named by the
256 bit blake3 hash instead of sha256: packs/, index/ fragments, keys/
(repokey borg keys) and locks/ in the repository, automatically placed
keyfiles (the key id is the same hash) and the files cache suffix.
One helper, crypto.key.blake3_256(), computes the hash for all of them
and hashes big inputs (packs, index fragments) multi-threaded from the
existing blake3 threshold on; Blake3ChecksumKey.id_hash reuses it.
Repository.NAME_HASH names the algorithm for borgstore's hash() and
defrag(), which already support blake3.
Not changed: checksums that are appended to content rather than used as
a name (checked-packs tracker, archive reference caches) and the archive
ids, which are the key mode's id hash.
Not backwards compatible with repositories and files caches written by
earlier borg2 betas.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- repo-info: append "empty passphrase" to the Encrypted: line if the key was unlocked with an
empty passphrase (for repokey and keyfile storage). The key now remembers this in the new
empty_passphrase attribute, which (unlike logically_encrypted) is independent of the storage.
- security: the "Attempting to access a previously unknown unencrypted repository" warning now
says why the repository counts as unencrypted - a mode that does not encrypt the data
(none-* / authenticated-*) or a repokey with an empty passphrase. Previously, users with an
encrypted repository could not tell why borg called it unencrypted, as repo-info said
"Encrypted: Yes".
- docs (repo-create, quickstart, repo-info): an empty passphrase still encrypts with a random
key, when it may be acceptable (keyfile storage), and that a passphrase can be added later
with "borg key change-passphrase" (unlike with the none-* modes).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Statistics.usize (the size of the chunks that were new to the repository) is
accumulated while an archive is created and printed as "Deduplicated size" in
the text --stats block, but Statistics.as_dict() did not include it, so the
--json output (which replaces the text block) had no way to get it.
as_dict() now reports it as "deduplicated_size", as borg 1.x did (there it was
the deduplicated compressed size, in borg 2 it is the deduplicated original
size, see the 2.0.0a2 changelog). This covers borg create --json, borg
import-tar --json and the archive_progress JSON lines.
Archive.calc_stats() (used by borg info) can not compute the deduplicated size
of an existing archive (that is expensive, see borg analyze), so it now marks
usize as unknown (None) and as_dict() leaves the key out there instead of
reporting a misleading 0.
Also fix the create help text: borg 2 prints a single stats block, there is no
"This Archive" / "All archives" table anymore.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
An empty value made borg fail:
$ BORG_ZSTD_MT_WORKERS= borg benchmark cpu --compressing
Error: BORG_ZSTD_MT_WORKERS must be an integer, but is: ''
That is unhelpful - an env var that is set to an empty string is usually
meant to be "not configured", e.g. when a wrapper script or a systemd unit
passes a variable through unconditionally. Use the defaults in that case
(whitespace-only values are treated the same way).
Co-Authored-By: Claude Opus 5 <[email protected]>
The blockwise kernel tested its 8 lanes with a loop accumulating
`hit |= ((c ^ s[k]) & M[k]) == 0`. clang lowers that into a chain of
conditional increments; gcc >= 12 at -O2 vectorises it and reloads the
eight 8-byte stores of s[] as 16-byte vectors, which fails store-to-load
forwarding on every block - the "blockwise 8x slower than scalar" seen on
Zen CPUs in #10160. Written as a short-circuit chain it compiles to eight
test-and-branch pairs that leave the loop at the first hit.
Measured at the kernel level (bit-identical cut points and hash state):
- Apple M3 Pro, Apple clang 17 -O3: blockwise 2450 -> 2950 MB/s of scanned
data, i.e. +24% over the shipped aarch64 default, the NEON kernel (2370)
- Zen 4, gcc 14 -O2 (Debian flags): blockwise 766 -> 1707 MB/s (2.2x);
the x86-64 default stays the sequential loop (2390)
So the aarch64 default becomes blockwise: the NEON kernel already lost to
the shipped blockwise kernel there (the file's own note said so while
bz64_kernel_default() still picked neon), and it loses more now. It stays
selectable via BORG_BUZHASH64_KERNEL=neon. The default tables in the tests
are now per chunker family, and the env var docs say which kernel each
family defaults to on aarch64.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
borg called platformdirs with the appname only, so on Windows the
appauthor component defaulted to the appname and all default
directories ended up under a doubled AppData\...\borg\borg path.
Pass appauthor=False everywhere (no vendor directory level; on
Linux/macOS the appauthor is unused, so nothing changes there) and
roaming=True for the config dir, restoring the layout the CHANGES
upgrade notes had documented as intended all along:
- config (incl. keys): C:\Users\<user>\AppData\Roaming\borg
(follows the user's roaming profile in domain environments)
- data (incl. security state): C:\Users\<user>\AppData\Local\borg
- cache: C:\Users\<user>\AppData\Local\borg\Cache
- runtime: C:\Users\<user>\AppData\Local\Temp\borg
This matches what comparable tools do on Windows (pip, rclone and
kopia keep the configuration in the roaming AppData and machine
state/caches in the local AppData, all without a vendor directory).
The cache keeps platformdirs' opinionated "Cache" subdir so the
CACHEDIR.TAG written into the cache root stays out of the data dir.
Only Windows is affected. The first Windows binary was only released
with 2.0.0b23, so beta users needing to migrate should be rare: move
the keys from the old %LOCALAPPDATA%\borg\borg\keys to the new
location (or set the BORG_* dir env vars to the old paths); cache and
security state are recreated as needed.
Docs: update the environment help topic (and the generated
environment.rst.inc), FAQ, key/transfer usage docs and CHANGES.
Co-Authored-By: Claude Fable 5 <[email protected]>
The samples in repo-list.rst, copy.rst and rename.rst predate the tags
column of the default repo-list format, so their column spacing was
wrong. Regenerated from real runs of each page's own scenario.
Co-Authored-By: Claude Fable 5 <[email protected]>
Add the missing http(s):// section (borgstore REST server, HTTP Basic
auth via URL or BORGSTORE_REST_USERNAME/PASSWORD, URL path only for
reverse-proxy prefixes - unlike rest:// it is not a filesystem path).
Qualify file:// as absolute-paths-only and state explicitly that
ssh:// works only for legacy borg 1.x repositories.
Co-Authored-By: Claude Fable 5 <[email protected]>
The sample listing still showed borg1-era content-hash suffixes
(doc.cda00bc9.txt); the versions view names files with a sequential
5-digit version (doc.00001.txt). Sample regenerated from a real
mount -o versions run.
Co-Authored-By: Claude Fable 5 <[email protected]>
All verified against the implementation (and at runtime where feasible):
- passphrase env vars: they are mutually exclusive (borg errors when
more than one is set), they do not take precedence over each other.
- authenticated_no_key: no repo config file / "key = anything" in
borg2; use borg repo-delete / repo-create instead of delete/init.
- BORG_BASE_DIR: no $HOME/~$USER/~ guessing anymore.
- Building section: the BORG_*_PREFIX vars are used unconditionally
(no header probing, no bundled fallback); document all four
including BORG_OPENSSL_NAME and the OpenBSD case.
- patterns: the "-" prefix recurses (it is "!" that matches the
no-recurse behavior of --exclude/--exclude-from); pattern options
are applied in strict command-line order, not staged.
- placeholders: no --name option; add missing {unixtime} and {uuid4}.
- drop ``...`` markup inside patterns code-block comments (rendered
literally everywhere).
- environment topic: add BORG_DIFF_FORMAT, BORG_JSON_INDENT,
BORG_STORE_CACHE, BORG_PACK_CACHE_SIZE,
BORG_MOUNT_DATA_CACHE_ENTRIES, and the pack tuning/debugging
variables BORG_PACK_MAX_SIZE, BORG_PACK_MAX_COUNT, BORG_PACK_ASYNC
and BORG_PACK_TRACE.
- fix the dead freedesktop basedir-spec link.
Regenerate docs/usage/help.rst.inc and
docs/usage/general/environment.rst.inc accordingly.
Co-Authored-By: Claude Fable 5 <[email protected]>
All command lines were executed against scratch repositories and all
output samples regenerated from real runs (prettified, not invented):
- recreate: the example used the removed --recompress; show
borg repo-compress instead; refresh the borg info samples.
- diff: default/text output has modified:/added:/removed: labels and
mtime/ctime brackets now; the JSON-lines example used borg1 keys
and lacked --json-lines on the command; add --content-only.
- mount: example passed the repository as a positional (it would have
been taken as the mountpoint); use -r.
- tar: the archive-copy loop was a bash syntax error and used
borg list with repo-list-only format keys; replaced with a working
while-read loop over borg repo-list (verified end-to-end, identical
archive contents in the destination repo).
- key: regenerate repo-create/change-passphrase samples (hex-id named
keyfiles, Key location line, no "Synchronizing index"); the import
example silently did nothing without --key-location=keyfile; fix
the Windows keys path (AppData\Local, not Roaming); explain why the
keyfile name changes on passphrase change.
- transfer: the borg1->borg2 upgrade steps were missing --from-borg1
and used a ssh:// destination URL (rejected for current repos);
fix the Windows keys path.
- notes: --noatime and --nobsdflags no longer exist (atime is opt-in
via --atime); stray "|" removed.
- compact: segments wording removed.
- info/repo-info/repo-delete/list/find: output samples regenerated
(new labels/order, UTC offsets, repo version line, prompt text;
isomtime format and symlink size corrected in list).
- general: fix broken inline markup (stray backtick).
- serve: the authorized_keys forced command needs --rest (it is
deliberately not client-suppliable); document the actual client
argument allowlist including --backend.
Co-Authored-By: Claude Fable 5 <[email protected]>
- quickstart: no segments in borg2 (packs/unreferenced-objects
wording); backend list is file:/rest: plus sftp/s3/b2; fix the
example script's exit-code handling for BORG_EXIT_CODES=modern
(highest-rc-wins ranked warnings 100..127 above errors 3..99 and
the rc==1 warning branch was dead) with a severity classifier;
regenerate the --stats and repo-list samples from real runs; make
the prose repo path match the script's rest:// URL; typos; correct
the repository model description (no manifest tracking blocks).
- repository-urls: use rest:// in the BORG_REPO example (ssh:// is
legacy-only, as the same file says).
- file-systems: rewrite for packs (chunks are grouped into pack files
of up to ~50 MB; compact works at pack granularity).
- resources: borg repo-delete (not "delete repo", not server-side);
replace the false "single-threaded, max 100% of one core" claim
with the real multi-threaded cases (zstd, blake3, pack writer).
- positional-arguments: drop borg1 repo::archive wording.
- config: fix the --print_config example (must precede the
subcommand) and the related prose.
- usage_general.rst.inc (man intro): add the archive-specification
and config includes that were only added to the usage variant when
the two files diverged (oversight from the #4587 split; the
archive-specification include uses :start-after: to avoid a
duplicate label).
- fix dead links: IEC binary prefixes (wikipedia anchor moved),
paperkey.html in the borg 1.x changelog.
- misc: update benchmark-crud commands to borg2 (with a note that the
numbers are borg1-era); add a preamble to create_chunker-params
noting it predates borg2/fastcdc.
Co-Authored-By: Claude Fable 5 <[email protected]>
The generated borgfs.1 was corrupted: jsonargparse injects --config and
--print_config into the borgfs parser (a top-level parser), and
--print_config's "[=flags]" metavar is invalid rST option-list syntax,
so the man page embedded a docutils WARNING and an unformatted blob of
options. Hide jsonargparse-injected actions (detected structurally by
their module, so future injected options stay filtered) and borg's
top-level-only --cockpit from the borgfs docs via help=SUPPRESS, which
all make.py writers already honor. -V/--version stays documented, as
in borg 1.x. Regenerate borgfs.1 accordingly (now also showing the
REPOSITORY positional).
Remove docs/usage/borgfs.rst{,.inc}: build_usage intentionally stopped
generating a borgfs HTML page long ago (#3404), so the leftover
.rst.inc still documented the borg 1.x CLI (REPOSITORY_OR_ARCHIVE,
--prefix, ...) and was published as an orphan page. Nothing references
these files; borgfs is documented on the "borg mount" page.
Co-Authored-By: Claude Fable 5 <[email protected]>
borgfs *is* the "borg mount" command, so the "mount:" section of the
config file(s) applies to it - but as borgfs has no subcommands, its
mount options live at the top level. Lift the keys of the "mount:"
section to the top level for borgfs (a section key wins over the same
top-level key); all other subcommand sections are still ignored.
Implemented as an opt-in adopt_config_sections attribute next to the
existing ignore_unknown_config_keys mechanism; the borg parser is
unaffected.
Co-Authored-By: Claude Fable 5 <[email protected]>
The jsonargparse migration broke borgfs completely:
- it never mounted: get_func() fell back to printing the main help
(rc 0) because the borgfs parser has no subcommand and nothing routed
it to do_mount. get_func() now dispatches to do_mount for borgfs.
- it lost its fstab calling convention: mount(8) invokes
"borgfs <repo> <mountpoint> -o <options>", but the parser only had
MOUNTPOINT [PATH...]. borgfs now accepts an optional REPOSITORY
first positional again, mapped to where -r/--repo lands, so
with_repository works unchanged. -r/--repo and BORG_REPO still work.
- any default.yaml with per-subcommand sections (exactly what the docs
recommend) aborted borgfs with e.g. "Option 'create.output_filter'
is not accepted", because borgfs is a top-level parser sharing the
default config file but has no subcommands. The borgfs parser now
opts into dropping unknown config keys via jsonargparse's own
strip_unknown(); the borg parser still rejects unknown keys.
Add parser-level tests for the dispatch, the optional positional, and
the default.yaml tolerance, plus a regression guard asserting that
"borg mount" grows no repository positional.
Co-Authored-By: Claude Fable 5 <[email protected]>
The example showed the borg 1.x output (just name and time). borg 2 lists
archive ID, time, name, user and host, see docs/usage/repo-list.rst.
Also use different archive IDs before and after the rename: renaming rewrites
the archive metadata, so the archive ID changes, as the command description
already says.
Co-Authored-By: Claude Fable 5 <[email protected]>
Copying is cheap and fast: no file content is read or written, only a new
archive metadata object is created. Like any deduplicated archives, the two
archives share their data, so a copy needs almost no additional repository
space.
The copy is an independent archive: deleting either of the two archives
keeps the other one intact, because "borg compact" only frees chunks that no
remaining archive references.
Archive.copy() is Archive.rename() without removing the original archive
entry. Copying an archive to its own name is refused: the new metadata would
be identical, thus have the same archive ID, and no second archive would be
created. NEWNAME may well be the name of some other existing archive, the
copy then just joins that archive series.
OLDNAME must match precisely one archive, so it accepts an archive name (if
it is unique) or an archive ID like "aid:d34db33f".
Co-Authored-By: Claude Fable 5 <[email protected]>
A "rest://host/" URL (no repository path) did not match rest_re, fell through
to local_re and became the local directory "./rest:/host", which then failed
with a confusing "Repository ... does not exist" message. Only "ssh://" and
"file://" were excluded from local_re, "rest://" was not.
Exclude all schemes borg parses itself from local_re, in the opaque form too,
so no "rest:...", "ssh:..." or "file:..." input can silently be taken for a
local path, and tell the accepted forms when such a URL does not parse.
A local path that really starts with such a prefix can still be given as
"./rest:..." or as an absolute path.
"--encryption=none" and "--encryption=authenticated" have not existed since
the id hash became part of the mode name for the modes that do not encrypt:
they are now "none-sha256"/"none-blake3" and "authenticated-sha256"/
"authenticated-blake3". The examples in docs/usage/repo-create.rst still used
the old names, so copying them gave an argparse error.
The "authenticated --id-hash=blake3" example was doubly wrong: those modes
name their id hash themselves and reject a separate --id-hash. Replaced it
with plain "authenticated-blake3"/"none-blake3" examples and dropped the
now-wrong claim that the non-encrypting modes only support sha256.
Also:
- "--encryption=aes-ocb" (docs/usage/key.rst, docs/quickstart_example.rst.inc)
is not a valid mode either, it is "aes256-ocb".
- scripts/rest_perf_test.sh used "--encryption none".
- regenerated the affected man pages; borg-repo-create(1) additionally picked
up the already-fixed --key-location help text.
All examples were verified by actually running them.
Add an "Incremental transfer" section to the transfer command help:
transfer skips archives that are already present in the destination
repo (name+timestamp or name+id) and only stores chunks the destination
does not have yet, so it can be run repeatedly (e.g. from a cron job).
Also mention that interrupting a transfer is safe, that transfer never
deletes anything in the destination repo, and add a pointer from the
related FAQ entry.
Co-Authored-By: Claude Opus 5 <[email protected]>
The help text claimed the default was "{archive:<36} {time} [{id}]{NL}",
while the code default is the wider template with id/time/archive/tags/
username/hostname/comment.
Define the default once as FORMAT_DEFAULT and use it both for the
BORG_REPO_LIST_FORMAT fallback and in the --format help text, so the two
can not drift apart again.
Co-Authored-By: Claude Opus 5 <[email protected]>
Four issues reported for the borg2 completions in #3086:
- -r/--repo and --other-repo completed nothing. They now complete
directories - a local repository is one, and a remote location like
ssh://... simply does not match any.
- The generated script hardcoded "borg": it registered its completions
for that command and, worse, ran "borg repo-list" to look up archive
names and tags. A borg 2 installed as "borg2" next to a borg 1.x
therefore asked the borg 1.x for the archives of a borg 2 repository,
which of course completed nothing. It now uses the command borg was
invoked as, as long as that is a borg* name - borg is also invoked in
ways that give no usable command name (python -m borg, the test
suite), and then "borg" stays the right guess.
- Completing "borg repo-create --encryption=" showed the same help text,
listing all the modes, next to each of the modes. zsh and fish now get
a short description of what that one mode does; bash and tcsh have no
completion descriptions, so they just complete the mode names.
- The help texts are written in RST and the shells show them verbatim as
the completion description, markup included ("**(required)**",
"``--format``"). Strip the inline markup for the completions.
The encryption modes need shtab >= 1.11.0: only from that release does a
.complete function take precedence over an argument's choices. Older
shtab offers both, so bash would complete every mode twice. That release
also has the fish path-completion fix borg was working around, so drop
the local FILE override again.
Also fix two stale encryption mode names: --encryption=repokey-aes-ocb
in the "Encryption NOT enabled" hint and the "none" mode in the
--key-location help are both from before #9104. The latter has to say
``none-*``: a bare "none-*" starts RST emphasis, and the docs are built
with -W.
Co-Authored-By: Claude Opus 5 <[email protected]>
Until now every chunker ran its simplest implementation unless a
BORG_*_KERNEL env var named another one, because which kernel is fastest
is not predictable from the instruction set. It is predictable per
platform though, now that #10160 has benchmarks from a range of CPUs, so
choose accordingly instead of leaving speed on the table by default:
- fastcdc / buzhash64: neon on aarch64, scalar (the plain sequential
loop, which the compiler folds into a single instruction there) on
x86-64, blockwise elsewhere.
- toeplitz-aes / rabin-aes / goldilocks-aes: wider is simply faster
here, so vaes, else aes-ni on x86-64, aes-arm64 on aarch64, and evp
where there is no AES hardware path.
Both fall back where a build or a CPU does not have the preferred
kernel. An env var still means exactly what it says: a kernel that
cannot run here is an error, never a silent fallback.
On an Apple M3 this makes the default chunkers 1.6x (goldilocks-aes) to
2.1x (buzhash64) faster.
With "--digests ALGOS", borg computes hash digests over the full content of each
regular file and stores them in the item's new "digests" dict (hash algorithm name
-> digest), e.g. --digests=blake3 gives {"blake3": ...} (32 bytes). ALGOS is a
comma-separated list of hash algorithm names - the same algorithms that are available
as "borg list" format keys - or "none". Default: none, digests are opt-in.
Unlike the chunk ids, such a digest depends neither on the chunker nor on borg's key,
so it is the same digest that e.g. "b3sum" computes for that file. "borg list --format
{blake3}" uses a stored digest if the item has one, instead of reading the file content
from the repository.
The content is hashed while it is read and processed anyway: the first chunk inline,
from the second (big enough) chunk on by a background thread. Our hash implementations
release the GIL, so the hashing can overlap with chunking, id hashing, compression and
encryption instead of adding to their runtime. ContentDigester.update() takes ownership
of the chunk data and releases it after hashing, so nothing is copied for this and the
memory is still freed timely.
Measured (12 cores, medians of 5 runs, relative to --digests=none, which is as fast as
borg without this):
1GiB new 1GiB known 20000 small files
--digests=blake3 free free +6..12%
--digests=blake3,sha256 free +10% +16%
--digests=blake3,sha256,sha512 +7% +60% +24%
One background thread hashes all requested algorithms one after the other, so their
combined throughput is the harmonic sum, while many small files are hashed inline by
the main thread. Hashing is free exactly while it is faster than the rest of the
pipeline - a single algorithm is, several may not be. That is why digests are opt-in.
Digests are not computed if borg does not read the full content: an unchanged file gets
them from the files cache and additional hard links from the first hard link, but
"borg create --reuse-from" items do not have digests. Files that already are in the
files cache without digests get them when they change (and thus are read again), so
changing --digests only affects the files that borg reads.
On Windows, \\server\share\repo was accepted by the Location parser and then
failed deep inside borgstore with "Invalid or unsupported Backend Storage URL:
file://server/share/repo" (Path.as_uri() makes the server the URL authority).
UNC paths are not supported (see #3141), so reject every local path that
starts with "//" (after resolving it) in Location.parse(), on all platforms,
with a message that names the workaround: mount the share (Windows: map it to
a drive letter with "net use") and use the mounted path.
This also removes the "//" exclusion from local_re (bare //server/share/path
now gets the same message instead of a bare "Invalid location format") and
drops the old test_smb, which asserted that file:////server/share/path is
accepted on non-Windows platforms - that only ever meant something on Cygwin
and was never tested there.
Also document this in the repository URLs docs section.
When showing a whole repository, the archive directories were named like
the archives, with -{id:.8} appended to names that are not unique (the
archives of a series). Now the BORG_MOUNT_ARCHIVE_DIR_FORMAT environment
variable gives the format of these names, using the placeholders of
borg repo-list --format (default: {name}, the same naming as before).
Names that are still not unique (or empty, "." or "..") get -{id:.8}
appended, "/" and NUL are replaced, an invalid format string is an error.
As this is implemented in the shared ArchiveVFS, it applies to borg mount
and borg webdav alike.
The "--map requires exactly one input path" and "the reference archive must
contain exactly one file item" paragraphs interrupted the description of the
map file format. Move both to the end of the Input maps section and rebuild
the generated usage docs / man page.
Converts thin_dump XML (full mode: allocation map of one thin LV) and
thin_delta XML (delta mode: differences between two thin snapshots)
into the borg create --map format. The docstring documents the full
workflow including the reserve/release_metadata_snap steps and the
snapshot discipline needed for --reuse-from.
Also add a docs section pointing from the --read-special LVM example
to the input map based approach.
Co-Authored-By: Claude Fable 5 <[email protected]>
For input map ranges marked "same", reuse the chunks of the reference
archive's item instead of reading the input. Reference chunks that only
partially overlap "same" ranges are re-read completely, so the result
is correct with any chunker; with the fixed chunker, read windows align
with the reference chunk grid. Reference chunks missing from the repo
are read again (like the files cache does on a lost chunk).
Together with --map, this implements efficient incremental backups of
block device snapshots: only changed ranges need to be read, e.g. as
reported by thin_delta for LVM thin volume snapshots.
Co-Authored-By: Claude Fable 5 <[email protected]>
An input map describes the content ranges of the single input file:
data ranges are read and stored, zero ranges are stored as holes
without reading them. Primary use case: backing up snapshots of large
(esp. thin-provisioned) block devices, where the storage layer knows
which ranges are in use.
Co-Authored-By: Claude Fable 5 <[email protected]>
