blake3 1.0.9 did not build on windows/msys2/mingw, so it was excluded
there. 1.0.10 is out, so drop the win32-specific requirement line and
require >= 1.0.10 everywhere.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
msgpack 1.2.3 brings fixes relevant to borg's usage of the C extension:
memory leaks when map key validation or container hooks fail
(GHSA-j586-36cw-2gc2) and a use-after-free when cleaning up an
incomplete Unpacker.skip().
The C Unpacker now raises ValueError when switching between unpack,
skip, header readers or read_bytes() while an object is incomplete.
borg only uses feed() plus unpack()/iteration (and RobustUnpacker
creates a fresh Unpacker on each resync attempt), so this does not
affect borg.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
borgstore 0.7.0 adds Store.gather (many byte ranges with one call, one
roundtrip for REST) and fixes partial loads with a negative offset from
cached namespaces.
Store.defrag now requires all sources to be items of the given namespace,
which borg's defrag calls (hex pack names in "packs") already satisfy.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Opening a borg 1.x repository without --from-borg1 crashed with a traceback while
loading the config/config store object, because a borg 1.x repository has a config
*file* where borg 2 has the config/ namespace (NotADirectoryError for a local
repository, HTTP 500 via ssh://; InvalidRepository on Windows).
borgstore 0.6.4 reports a name below an object as ObjectNotFound on all platforms,
so require it. If config/config is not found, look at "config" via the store
backend and raise Repository.LegacyRepository (rc 29) if it is a file, telling the
user to use --from-borg1 (e.g. with borg transfer).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
zoneinfo finds no tz database on haiku, so every "date:" pattern with a
named timezone fails with ZoneInfoNotFoundError. Same situation as on
windows, so pull in the tzdata package there, too.
Co-Authored-By: Claude Opus 5 <[email protected]>
msgpack 1.2.2 only brings bug fixes relevant to borg's usage:
correct Timestamp.from_datetime for datetimes far from the epoch,
reject out-of-range timestamp nanoseconds, copy ExtraData.extra out of
a temporary contiguous buffer, and a new RuntimeError guard against
re-entrant Unpacker.feed() (borg never feeds from inside a hook).
Co-Authored-By: Claude Opus 5 <[email protected]>
The [tool.setuptools_scm.tag] table only exists since setuptools-scm 10.1
(vcs-versioning 2.0.0). Older versions pass the whole [tool.setuptools_scm]
table as kwargs to their Configuration class, so an unknown "tag" key is a
hard TypeError - breaking builds against the setuptools-scm versions that
distributions ship, see https://bugs.debian.org/1144303 .
Setting it was only about silencing the FutureWarning about the coming
default change anyway: strict tag matching just requires a dot in the tag,
which all borg release tags have, so both settings select the same tags.
Co-Authored-By: Claude Opus 5 <[email protected]>
Four issues reported for the borg2 completions in #3086:
- -r/--repo and --other-repo completed nothing. They now complete
directories - a local repository is one, and a remote location like
ssh://... simply does not match any.
- The generated script hardcoded "borg": it registered its completions
for that command and, worse, ran "borg repo-list" to look up archive
names and tags. A borg 2 installed as "borg2" next to a borg 1.x
therefore asked the borg 1.x for the archives of a borg 2 repository,
which of course completed nothing. It now uses the command borg was
invoked as, as long as that is a borg* name - borg is also invoked in
ways that give no usable command name (python -m borg, the test
suite), and then "borg" stays the right guess.
- Completing "borg repo-create --encryption=" showed the same help text,
listing all the modes, next to each of the modes. zsh and fish now get
a short description of what that one mode does; bash and tcsh have no
completion descriptions, so they just complete the mode names.
- The help texts are written in RST and the shells show them verbatim as
the completion description, markup included ("**(required)**",
"``--format``"). Strip the inline markup for the completions.
The encryption modes need shtab >= 1.11.0: only from that release does a
.complete function take precedence over an argument's choices. Older
shtab offers both, so bash would complete every mode twice. That release
also has the fish path-completion fix borg was working around, so drop
the local FILE override again.
Also fix two stale encryption mode names: --encryption=repokey-aes-ocb
in the "Encryption NOT enabled" hint and the "none" mode in the
--key-location help are both from before #9104. The latter has to say
``none-*``: a bare "none-*" starts RST emphasis, and the docs are built
with -W.
Co-Authored-By: Claude Opus 5 <[email protected]>
- src/borg/crypto/low_level.pyx: implement `argon2_hash` using OpenSSL's
`EVP_KDF` API for ARGON2 (requires OpenSSL >= 3.2.0).
- src/borg/crypto/key.py: switch to the native `argon2_hash` implementation,
removing `argon2-cffi` dependency.
- setup.py: require OpenSSL >= 3.2.0 for the crypto extension to ensure
ARGON2 KDF support is available.
- pyproject.toml: drop `argon2-cffi` dependency.
- docs: update installation requirements and security documentation to
reflect the transition to OpenSSL for Argon2.
The lanes are computed in parallel via OpenSSL's thread pool: OpenSSL's
Argon2 only uses threads if OSSL_set_max_threads() enabled the thread pool
and a "threads" parameter > 1 is given. Passing threads=1 would compute the
4 lanes of our default parallelism=4 sequentially, and unlocking would get
slower than it was with argon2-cffi, which used real threads for the lanes.
The thread count only affects speed, never the derived key (only the lanes
parameter does), so this is fully compatible with existing keys.
Apple M-series (OpenSSL 3.6.3), borg defaults (t=3, m=64 MiB, p=4):
threads=1: 83 ms, threads=4: 29 ms (2.8x faster).
Co-Authored-By: Claude Opus 5 <[email protected]>
write_raw_lock duplicated _create_lock's wire format (field layout, timestamp
format, sha256 key, store path), so a future format change (e.g. AEAD lock
objects) would have made the skew tests silently keep writing the old format.
_create_lock gained an explicit content timestamp parameter instead; the
helper keeps only the store-side mtime override. The free_pid fixture was a
verbatim copy of the one in fslocking_test.py - import it like
platform_test.py does (incl. the same per-file F811 ignore).
Co-Authored-By: Claude Fable 5 <[email protected]>
Lock staleness was judged by comparing a lock's content timestamp
(stamped by its writer's clock) against the reader's local clock.
A client whose clock runs >30 min ahead would thus kill another
client's healthy lock and could then e.g. run compact deleting
chunks the victim still references - a finished archive referencing
deleted chunks.
Fix: a lock may only be considered stale by age if it looks stale in
BOTH clock domains:
- writer/local clock domain: local now vs. lock content timestamp
(the pre-existing rule), AND
- store clock domain: store "now" vs. the lock object's store-side
mtime (new, using borgstore's ItemInfo.mtime).
Store "now" is derived from our own lock object's mtime plus elapsed
monotonic time, so all store-domain comparisons happen within the
store's own clock domain: neither the clients' nor the store's
absolute clock error matters. A client without an own lock object
defers the kill until acquire() has created one (a listing made
right afterwards confirms or vetoes the candidates).
Store-side mtimes are advisory only: they can veto a kill, but they
can never cause one on their own, so a hostile or broken store gains
no new capabilities (it can already delete locks or serve fabricated
fresh ones - lock objects are unauthenticated). For the same reason,
the process_alive() check now runs *first* and is never vetoed by
store timestamps: if the lock owner is a process on our own machine
and it is dead, we know that locally, and a store serving bogus,
always-fresh mtimes must not be able to keep an abandoned lock alive
forever and block us.
Backends without store-side mtimes (e.g. rclone: mtime == 0) keep
the previous behavior.
Additionally, since each lock object carries two timestamps of the
same write instant (content time = writer clock, mtime = store
clock), the writers' per-store clock offsets are comparable: on
acquire, borg now warns (once) if another active client's clock is
skewed by more than MAX_MUTUAL_CLOCK_SKEW (5 min) against ours -
diagnosis only, never an abort, so spoofed store timestamps cannot
block backups.
The manifest-timestamp behavior is intentionally unchanged.
ItemInfo.mtime requires borgstore 0.6.1, so the borgstore
requirement is bumped accordingly.
Co-Authored-By: Claude Fable 5 <[email protected]>
argon2-cffi-bindings 26.1.0 switched its build backend from setuptools to
scikit-build-core, which asks for cmake as a dynamic build requirement when
it does not find one on PATH. There are no cmake wheels for the BSDs and
OmniOS, so pip builds cmake from its sdist there - that takes about half an
hour and then fails on NetBSD, OpenBSD and OmniOS, so those VM test jobs
never get as far as running any test. On OmniOS the bootstrapped cmake has
no TLS support and can not even download the CMake tarball. FreeBSD gets
through it, but pays ~44 minutes for the privilege.
25.1.0 builds with setuptools + cffi and needs no cmake: it installs from
its sdist in seconds. It also still ships abi3 wheels for the platforms that
have them.
This is meant to be temporary: argon2 is moving to OpenSSL, which removes
the argon2-cffi dependency (and this pin) completely, see #9602.
A full index rewrite used to build one sorted Python list of all keys
(~90 bytes per key, e.g. ~9 GB for a 100M chunks repo). As the keys are
uniformly distributed hash digests, partition them by their leading key
bits into 2 ** prefix_bits similarly sized, disjoint sets and select /
sort / write one partition's keys at a time, so the extra memory needed
is bounded by one partition (~ one fragment, <= 32 MB worth of keys),
independent of the index size.
Selecting a partition is a cheap, C-level filtering scan of the hash
table (borghash 0.2.0 items(prefix_bits=..., prefix=...)), so runtime
stays unchanged (measured: 21s for a 5M-entry full rewrite either way,
peak extra memory 480 MB -> 85 MB).
Partition membership and prefix_bits (chosen from the entry count) only
depend on the selected entries, so identical entry sets still produce
identical fragments (writing/repacking stays idempotent and convergent),
and as the prefix compares the keys' leading bits, ascending prefixes
yield the same globally sorted key sequence as one all-keys sort did.
ChunkIndex gets a new_count property (count of F_NEW entries, maintained
incrementally) so the incremental write path can pick prefix_bits
without an extra counting pass; as a bonus, a nothing-new incremental
write now skips the full table scan entirely.
Requires borghash ~= 0.2.0.
Co-Authored-By: Claude Fable 5 <[email protected]>
fish completions are now generated by shtab, like the bash, zsh and tcsh
ones, extended with a fish preamble that provides the same dynamic
completions the bash/zsh scripts have (archive names, aid: archive IDs,
tags, sort keys, files cache modes, compression specs, chunker params,
relative time markers, timestamps, file sizes, help topics).
This needs shtab >= 1.9.3, which brings the fixed fish backend (see
tqdm/shtab#227, #228, #229, #230 and #231), so the pin was bumped.
Arguments taking a filesystem path are now typed, so that all shells can
complete them: such a path was either untyped or just a str, so the
shells relied on their default completion (bash/zsh/tcsh) or offered
nothing at all (fish). FilesystemPathSpec is used for arguments taking a
file (--exclude-from, --patterns-from, TARFILE, the borg debug file
arguments, borg key export/import PATH), the new FilesystemDirSpec for
arguments taking a directory (MOUNTPOINT, borg benchmark crud PATH).
Besides the completions, they now also reject empty paths and slashify
them on Windows, like the other filesystem path arguments do.
Note that borg key export/import PATH completed directories before
(it takes a file) and borg benchmark crud PATH completed files (it takes
a directory) - both are fixed by this.
shtab's fish FILE completion needs a little help to also complete
absolute paths, see tqdm/shtab#245 (fix: tqdm/shtab#246).
The completion script is now always generated for the "borg" command,
no matter how borg was invoked while generating it - completions
generated by "python -m borg completion ..." were useless before, as
the shells would complete a "__main__.py" command.
The hand-written, unmaintained fish completions in
scripts/shell_completions/ are removed, as are their tests.
Co-Authored-By: Claude Fable 5 <[email protected]>
rabin-aes, goldilocks-aes and toeplitz-aes differed only in their rolling
universal hash, but each carried its own copy of the AES layer, the
two-lane scan structure, the chunker class and the test suite - about 79%
of the goldilocks/toeplitz code was line-identical to rabin-aes.
Three shared pieces replace the copies:
- phte_core.h: AES-128 key expansion, endian-explicit LE helpers, hardware
detection, single-block AES helpers and the context base (round keys, EVP
context, path selection).
- phte_scan.h: the scan itself as a compile-time template. Each kernel
defines its rolling functions and PH_PREFIX/PH_CTX/PH_ROLL/PH_ROLL2/
PH_DIGEST64, then includes it, so the rolling code is still expanded
inline into fully specialized scan loops - no function pointers, no
cross-kernel calls, identical code generation.
- phte_chunker.pyx/.pxd: ChunkerPHTE, the shared chunker driver (buffering,
min/max clamping, normalized chunking, scan loop, sparse handling). The
subclasses implement _scan()/_digest64() as Cython cdef methods, i.e. one
vtable call per scan span (up to a whole chunk) and per chunk start, never
per byte.
Likewise for the tests: the behavior every one of these chunkers must have
(kernel equivalence, size distribution, shift resilience, param parsing,
fuzzing) now runs parametrized over all three in phte_chunkers_test.py; the
per-chunker modules keep only their frozen golden chunk points and the
tests of their own hash arithmetic and key derivation.
Net -2335 lines. Each kernel .c is now 84-125 lines (was 523-564) and each
chunker .pyx no longer contains the ~250-line class body.
Verified unchanged behavior and performance:
- all three frozen golden chunk-point tests unchanged, 508 tests pass,
mypy/black/ruff clean, e2e create+extract with all three chunkers OK;
- C-level A/B of the production scan (same harness, HEAD kernels vs these,
64 MiB, best of 8): rabin 683.6 -> 682.5, goldilocks 331.8 -> 331.8,
toeplitz 687.4 -> 694.7 MB/s hw; EVP paths 376.6 -> 376.6, 251 -> 250,
394.2 -> 394.0 - all within +-1%;
- disassembly of ra_scan keeps the identical AES and table-lookup structure
(853 vs 850 instructions; the delta is prologue register shuffling caused
by the changed struct field offsets).
Also excludes *.pxd from the wheel (build-time only, borg's first .pxd).
Co-Authored-By: Claude Fable 5 <[email protected]>
Both generate broken zsh completions: shtab replaced escape_zsh() with
shlex.quote() for the zsh help text, but still interpolates the result inside
a double-quoted _arguments spec. shlex.quote returns a self-contained shell
word (single-quoted, with ' written as '"'"'), which is only valid at top
level, so the quote parity flips and the rest of the spec goes unquoted.
Every description with a space gained stray literal quotes, and an apostrophe
anywhere left an unterminated quote that cascaded to shtab's own
default='*::: :->{name}', whose -> then parsed as a redirection:
$ borg completion zsh | zsh -n
(stdin):715: parse error near `>'
Our --sort-by help for `borg diff` contains '>size_added,path', which is what
tripped test_zsh_completion_syntax. Reported as tqdm/shtab#224, fix proposed
in tqdm/shtab#225; drop the exclusions once a release carries it.
Only CI jobs with a cold .tox cache saw this (macOS 15 first), since shtab was
unpinned above 1.8.0 -- cached envs kept the working 1.8.1.
Fixes#9982
- fix a TZ leak that broke later tests. The local-timezone test set
TZ + tzset() but restored only the env var, corrupting the runner's local time. Restore through a set_timezone fixture that re-runs tzset() on teardown.
- In parsing, reject surrounding whitespace and trailing newlines. Use \Z instead
of $ and drop the redundant strip() so 'date:2025\n' no longer parses.
- add tzdata dependency on Windows so [Region/City] zones resolve there.
- allow a space in place of T as the date-time separator and rewrite the date:
helptext to be clearer, separating exact from interval forms and adding a note on DST/UTC.
Port of borgbackup/borg#9790 to master.
Note: the PR's second commit (Archive.delete: don't reuse msgpack Unpacker
after an unpacking failure) does not apply here - on master Archive.delete
no longer unpacks item metadata (it just removes the archive and lets
"borg compact" reclaim space), so the reuse-after-failure code path does
not exist. The streaming RobustUnpacker already creates a fresh Unpacker
on resync.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Remove the dead BORG_TESTONLY_SHA256_PACK_ID tox env and CI job, fix the packs.rst pack-id docs, and reword comments/tests to describe sha256 pack naming instead of the removed shortcut.
Fixes#9448.
borg mount forks into a background daemon, so coverage was missing the process that actually handles the FUSE requests, leaving fuse.py and hlfuse.py at 0%.
Enable coverage patches:
patch = ["subprocess", "_exit"]
This lets coverage follow spawned subprocesses and still record data for paths that terminate via os._exit().
moved --junitxml parameter to the tox configuration.
haiku: add coverage params to pytest invocation (tox not used there).
vm_tests: add test_results and coverage uploads.
hard-code coverage.xml as coverage filename
"*/borg/fuse.py" - suspect, let's try what happens if we do not omit.
"*/borg/support/*" - directory does not exist anymore.
"*/borg/hash_sizes.py" - does not exist anymore.
