During a refresh the listing contains our old and our new lock object; a
storage clock step between their writes makes the old one look skewed against
our new anchor, which produced a misleading "clock skew with <our own host>"
warning right after the (correct) storage clock step warning. Skip all lock
objects carrying our own id in the skew check, not just the current one.
Co-Authored-By: Claude Fable 5 <[email protected]>
Explain lock objects, acquiring, staleness (both clock domains, store "now",
advisory-only store timestamps, deferral, mtime-less backends), the clock skew
warning and refreshing where the code lives; the internals docs keep a summary
and point there for the details.
Co-Authored-By: Claude Fable 5 <[email protected]>
When the mtime of a freshly created lock object is harvested, compare it with
what the previous anchor extrapolates for that instant: if our own wall and
monotonic clocks agree about the elapsed time (no suspend, no local clock step)
but the storage's mtimes do not, the storage's clock jumped. Warn once.
Diagnostic only - a backward step larger than the stale timeout can defeat the
store-domain cross-check in _is_stale_lock (documented residual risk); this at
least names the cause when a vanished lock gets investigated.
Co-Authored-By: Claude Fable 5 <[email protected]>
refresh() listed first and created the new lock object afterwards, so its stale
sweep judged other locks with the store-clock anchor of the previous refresh, up
to refresh_td (15min) old. A storage clock that stepped back within that window
by more than the stale timeout could make the store-domain cross-check wrongly
confirm a skewed peer's healthy lock as stale - the hazard the cross-check exists
to prevent. Now refresh() creates the new lock object first and lists afterwards:
the listing harvests a seconds-old anchor before the sweep runs, shrinking the
exposure from 15 minutes to the write/list latency. Our old lock object is exempt
from the sweep during the refresh (it may legitimately be older than the stale
timeout, e.g. after a suspend, see #9883).
Co-Authored-By: Claude Fable 5 <[email protected]>
write_raw_lock duplicated _create_lock's wire format (field layout, timestamp
format, sha256 key, store path), so a future format change (e.g. AEAD lock
objects) would have made the skew tests silently keep writing the old format.
_create_lock gained an explicit content timestamp parameter instead; the
helper keeps only the store-side mtime override. The free_pid fixture was a
verbatim copy of the one in fslocking_test.py - import it like
platform_test.py does (incl. the same per-file F811 ignore).
Co-Authored-By: Claude Fable 5 <[email protected]>
The anchor is a store-clock calibration, not a property of the lock object,
so deleting our transient lock does not invalidate it. Keeping it lets an
acquire that is blocked by a healthy-but-skewed lock veto the kill on the
first listing of every retry (2 store round-trips) instead of re-running the
defer/create/veto/delete cycle (7 round-trips plus lock churn) each time.
Safe: store times stay veto-only, so a kept anchor can never cause a kill.
To bound the mis-veto window of an anchor frozen by a suspend, _store_now
now refuses anchors older than the stale timeout, with age measured by our
wall clock (which, unlike time.monotonic(), keeps counting while suspended);
this also defuses the leftover-anchor hazard of the break_lock and
refresh-abort paths. Also document the accepted residual risk of a store
clock stepping backwards by more than the stale timeout.
Co-Authored-By: Claude Fable 5 <[email protected]>
borg with-lock runs its LockRefresher thread without serialization against
the main thread, and terminate()'s bounded join can leave a wedged refresh()
running while the main thread releases. The anchor state (store key, content
timestamp, store mtime, monotonic) was spread over separate attributes, so
such an interleaving could raise KeyError (my_lock_key rebound between the
harvest's membership test and subscript), raise TypeError (fields nulled
between _store_now's guard and use), or silently pair one lock object's
mtime with another's monotonic/content timestamp, skewing _store_now and
_mutual_skew by up to the refresh interval. A LockAnchor namedtuple replaced
as a whole plus single-read locals makes every observed state internally
consistent; the harvest only updates an anchor still describing the same
lock object.
Co-Authored-By: Claude Fable 5 <[email protected]>
The stale-veto path warned unconditionally, but a veto is not by itself
evidence of skew: after a suspend, our store "now" estimate lags (monotonic
clock stood still), so a genuinely stale foreign lock of a perfectly synced
client gets vetoed and produced a bogus "clock skew of ~0s" warning. The
per-listing skew check in _get_locks already covers the vetoed lock with a
proper magnitude gate, so the veto-path warning (and with it the optional-skew
calling convention of _warn_clock_skew) can just go away.
Co-Authored-By: Claude Fable 5 <[email protected]>
The skew check only ran on acquire success, but the listing that satisfies
an exclusive acquire can only contain our own lock, so exclusive commands
could never warn about a moderately skewed peer. Checking each listing in
_get_locks also warns when acquire times out on a skewed peer's lock, and
makes the last_seen_locks replay machinery unnecessary.
Co-Authored-By: Claude Fable 5 <[email protected]>
Lock staleness was judged by comparing a lock's content timestamp
(stamped by its writer's clock) against the reader's local clock.
A client whose clock runs >30 min ahead would thus kill another
client's healthy lock and could then e.g. run compact deleting
chunks the victim still references - a finished archive referencing
deleted chunks.
Fix: a lock may only be considered stale by age if it looks stale in
BOTH clock domains:
- writer/local clock domain: local now vs. lock content timestamp
(the pre-existing rule), AND
- store clock domain: store "now" vs. the lock object's store-side
mtime (new, using borgstore's ItemInfo.mtime).
Store "now" is derived from our own lock object's mtime plus elapsed
monotonic time, so all store-domain comparisons happen within the
store's own clock domain: neither the clients' nor the store's
absolute clock error matters. A client without an own lock object
defers the kill until acquire() has created one (a listing made
right afterwards confirms or vetoes the candidates).
Store-side mtimes are advisory only: they can veto a kill, but they
can never cause one on their own, so a hostile or broken store gains
no new capabilities (it can already delete locks or serve fabricated
fresh ones - lock objects are unauthenticated). For the same reason,
the process_alive() check now runs *first* and is never vetoed by
store timestamps: if the lock owner is a process on our own machine
and it is dead, we know that locally, and a store serving bogus,
always-fresh mtimes must not be able to keep an abandoned lock alive
forever and block us.
Backends without store-side mtimes (e.g. rclone: mtime == 0) keep
the previous behavior.
Additionally, since each lock object carries two timestamps of the
same write instant (content time = writer clock, mtime = store
clock), the writers' per-store clock offsets are comparable: on
acquire, borg now warns (once) if another active client's clock is
skewed by more than MAX_MUTUAL_CLOCK_SKEW (5 min) against ours -
diagnosis only, never an abort, so spoofed store timestamps cannot
block backups.
The manifest-timestamp behavior is intentionally unchanged.
ItemInfo.mtime requires borgstore 0.6.1, so the borgstore
requirement is bumped accordingly.
Co-Authored-By: Claude Fable 5 <[email protected]>
Until now every chunker ran its simplest implementation unless a
BORG_*_KERNEL env var named another one, because which kernel is fastest
is not predictable from the instruction set. It is predictable per
platform though, now that #10160 has benchmarks from a range of CPUs, so
choose accordingly instead of leaving speed on the table by default:
- fastcdc / buzhash64: neon on aarch64, scalar (the plain sequential
loop, which the compiler folds into a single instruction there) on
x86-64, blockwise elsewhere.
- toeplitz-aes / rabin-aes / goldilocks-aes: wider is simply faster
here, so vaes, else aes-ni on x86-64, aes-arm64 on aarch64, and evp
where there is no AES hardware path.
Both fall back where a build or a CPU does not have the preferred
kernel. An env var still means exactly what it says: a kernel that
cannot run here is an error, never a silent fallback.
On an Apple M3 this makes the default chunkers 1.6x (goldilocks-aes) to
2.1x (buzhash64) faster.
The inverted canary added in #10097 watched for actions/setup-python
gaining Python 3.12 support on the ubuntu-26.04 image. It went red on
2026-08-15 and 2026-08-16: the "Try to set up Python 3.12" step now
succeeds, which was the whole point of the probe. Its job is done, so
remove it.
Older Pythons are therefore no longer categorically unavailable on
ubuntu-26.04, so the matrix can gain multi-version coverage there once
we move to those runners.
Co-Authored-By: Claude Opus 5 <[email protected]>
With "--digests ALGOS", borg computes hash digests over the full content of each
regular file and stores them in the item's new "digests" dict (hash algorithm name
-> digest), e.g. --digests=blake3 gives {"blake3": ...} (32 bytes). ALGOS is a
comma-separated list of hash algorithm names - the same algorithms that are available
as "borg list" format keys - or "none". Default: none, digests are opt-in.
Unlike the chunk ids, such a digest depends neither on the chunker nor on borg's key,
so it is the same digest that e.g. "b3sum" computes for that file. "borg list --format
{blake3}" uses a stored digest if the item has one, instead of reading the file content
from the repository.
The content is hashed while it is read and processed anyway: the first chunk inline,
from the second (big enough) chunk on by a background thread. Our hash implementations
release the GIL, so the hashing can overlap with chunking, id hashing, compression and
encryption instead of adding to their runtime. ContentDigester.update() takes ownership
of the chunk data and releases it after hashing, so nothing is copied for this and the
memory is still freed timely.
Measured (12 cores, medians of 5 runs, relative to --digests=none, which is as fast as
borg without this):
1GiB new 1GiB known 20000 small files
--digests=blake3 free free +6..12%
--digests=blake3,sha256 free +10% +16%
--digests=blake3,sha256,sha512 +7% +60% +24%
One background thread hashes all requested algorithms one after the other, so their
combined throughput is the harmonic sum, while many small files are hashed inline by
the main thread. Hashing is free exactly while it is faster than the rest of the
pipeline - a single algorithm is, several may not be. That is why digests are opt-in.
Digests are not computed if borg does not read the full content: an unchanged file gets
them from the files cache and additional hard links from the first hard link, but
"borg create --reuse-from" items do not have digests. Files that already are in the
files cache without digests get them when they change (and thus are read again), so
changing --digests only affects the files that borg reads.
Without os.readv (win32) we read via os.read and copy into the scan buffer.
os.read allocates a buffer of the requested size, no matter how few bytes it
then returns, so requesting the whole free scan buffer (up to 8MiB for the
default max chunk size) is expensive for small files: one allocation of that
size per read call, several calls per file.
Measured with readv disabled on CPython, backing up 20000 small files (82.6MiB
of data): 468GiB requested in 60002 os.read calls, 1.42s spent in os.read and
4.04s total. Capping the request at 256KiB - same call count, same data - gives
0.06s in os.read and 2.42s total, close to the 2.36s of the readv path.
Chunking a 2GiB file is unaffected (the big reads there return what they ask
for). pypy showed the same effect much more strongly until it gained os.readv.
Co-Authored-By: Claude Opus 5 <[email protected]>
borg works on pypy3.11: tested with a py3.11 nightly build (523222a2c5cc,
PyPy 8.0.0-alpha0), where the full test suite passes (2588 passed, 1 xfailed)
and CPython is unaffected.
The pypy bugs found while doing this are all fixed in pypy now, so no
workarounds are left. What remains are test expectations for behaviour that
is not a pypy bug:
- platformflags: add is_pypy.
- item_test: xfail test_unknown_property - setting undeclared attributes on
cdef class instances is not blocked under cpyext.
- msgpack_test: pypy only has the pure-python msgpack, so expect it to be slow.
- lock_cmds_test: ignore stderr warnings unrelated to lock handling (the
pure-python msgpack one and cython's collection_type one), and use
sys.executable instead of "python3" (robustness, not pypy-specific).
Co-Authored-By: Claude Opus 5 <[email protected]>
is_utime_fully_supported() and is_birthtime_fully_supported() probed via
os.utime, which can neither set the timestamps of a symlink itself nor the
birthtime on win32 - so they returned False there and all the tests they
guard were skipped. Probe via platform.set_times() instead, which does the
right thing on all platforms.
Also: a FILETIME has a resolution of 100ns, use that for st_mtime_ns_round
on win32 (rather than the 1s fallback).
Co-Authored-By: Claude Opus 5 <[email protected]>
On Windows, stat().st_ctime is the file *creation* time and not the POSIX
"metadata change time", and Python 3.12 deprecated it for that reason.
Thus, do not archive it as item.ctime any more (there simply is no ctime
on Windows), but archive it as item.birthtime - which is also what
Python >= 3.12 gives us via st_birthtime_ns.
Co-Authored-By: Claude Opus 5 <[email protected]>
Previously, a failure to set the archived flags was silently ignored.
Now it emits a warning and sets the warning exit code, like xattr
errors already do.
- platform.base.set_flags (used on NetBSD and generic POSIX) now also does
the masked read-modify-write cycle instead of setting the archived value
as-is, see #9039. The flag masks are defined once in platform.base and
shared by the freebsd/darwin implementations.
- if setting the flags fails with EPERM (e.g. super-user-only SF_* flags
while not running as root, or immutable/append-only on Linux without
CAP_LINUX_IMMUTABLE), retry influencing only the unprivileged flags, so
e.g. the nodump flag still gets restored, see #1345.
- macOS: tolerate EOPNOTSUPP like the other platforms already do.
- linux: remove the unused FS_COMPR_FL declaration.
On Windows, \\server\share\repo was accepted by the Location parser and then
failed deep inside borgstore with "Invalid or unsupported Backend Storage URL:
file://server/share/repo" (Path.as_uri() makes the server the URL authority).
UNC paths are not supported (see #3141), so reject every local path that
starts with "//" (after resolving it) in Location.parse(), on all platforms,
with a message that names the workaround: mount the share (Windows: map it to
a drive letter with "net use") and use the mounted path.
This also removes the "//" exclusion from local_re (bare //server/share/path
now gets the same message instead of a bare "Invalid location format") and
drops the old test_smb, which asserted that file:////server/share/path is
accepted on non-Windows platforms - that only ever meant something on Cygwin
and was never tested there.
Also document this in the repository URLs docs section.
When showing a whole repository, the archive directories were named like
the archives, with -{id:.8} appended to names that are not unique (the
archives of a series). Now the BORG_MOUNT_ARCHIVE_DIR_FORMAT environment
variable gives the format of these names, using the placeholders of
borg repo-list --format (default: {name}, the same naming as before).
Names that are still not unique (or empty, "." or "..") get -{id:.8}
appended, "/" and NUL are replaced, an invalid format string is an error.
As this is implemented in the shared ArchiveVFS, it applies to borg mount
and borg webdav alike.
- fuse_impl: record the import error of each FUSE implementation that
failed to load. also catch non-ImportError failures: mfusepy raises
OSError at import time if libfuse is missing (and AttributeError /
NotImplementedError in other cases), which previously crashed borg
with a traceback instead of a clean error.
- borg mount: show the recorded errors, so users see the actual cause
(e.g. a missing shared library) instead of just "no FUSE support".
- sysinfo: report mfusepy as the active FUSE implementation (it showed
"fuse: None" although mounting worked) and include the recorded
import errors if no implementation loaded, so debug output pasted
into issues shows why.
- docs: note that third-party binaries might lack FUSE support.
Co-Authored-By: Claude Fable 5 <[email protected]>
The "--map requires exactly one input path" and "the reference archive must
contain exactly one file item" paragraphs interrupted the description of the
map file format. Move both to the end of the Input maps section and rebuild
the generated usage docs / man page.
Converts thin_dump XML (full mode: allocation map of one thin LV) and
thin_delta XML (delta mode: differences between two thin snapshots)
into the borg create --map format. The docstring documents the full
workflow including the reserve/release_metadata_snap steps and the
snapshot discipline needed for --reuse-from.
Also add a docs section pointing from the --read-special LVM example
to the input map based approach.
Co-Authored-By: Claude Fable 5 <[email protected]>
For input map ranges marked "same", reuse the chunks of the reference
archive's item instead of reading the input. Reference chunks that only
partially overlap "same" ranges are re-read completely, so the result
is correct with any chunker; with the fixed chunker, read windows align
with the reference chunk grid. Reference chunks missing from the repo
are read again (like the files cache does on a lost chunk).
Together with --map, this implements efficient incremental backups of
block device snapshots: only changed ranges need to be read, e.g. as
reported by thin_delta for LVM thin volume snapshots.
Co-Authored-By: Claude Fable 5 <[email protected]>
An input map describes the content ranges of the single input file:
data ranges are read and stored, zero ranges are stored as holes
without reading them. Primary use case: backing up snapshots of large
(esp. thin-provisioned) block devices, where the storage layer knows
which ranges are in use.
Co-Authored-By: Claude Fable 5 <[email protected]>
argon2-cffi-bindings 26.1.0 switched its build backend from setuptools to
scikit-build-core, which asks for cmake as a dynamic build requirement when
it does not find one on PATH. There are no cmake wheels for the BSDs and
OmniOS, so pip builds cmake from its sdist there - that takes about half an
hour and then fails on NetBSD, OpenBSD and OmniOS, so those VM test jobs
never get as far as running any test. On OmniOS the bootstrapped cmake has
no TLS support and can not even download the CMake tarball. FreeBSD gets
through it, but pays ~44 minutes for the privilege.
25.1.0 builds with setuptools + cffi and needs no cmake: it installs from
its sdist in seconds. It also still ships abi3 wheels for the platforms that
have them.
This is meant to be temporary: argon2 is moving to OpenSSL, which removes
the argon2-cffi dependency (and this pin) completely, see #9602.
FileReader used "blockify_gen is None" for both "not started yet" and
"exhausted": _fill_buffer() cleared it on StopIteration, and read() /
readinto() take a cleared generator as their cue to build a fresh
FileFMAPReader.blockify(), which replays the fmap from its start. dread()
reads sequentially and ignores its offset argument, so such a replay
continued at the current file position and appended duplicate data. An
fmap that does not start at 0 seeks back to its start on every replay and
never terminates at all.
This stayed invisible as long as every fmap reached EOF, because the
replay then hits a 0-byte read immediately: that is true for sparsemap()
and for "create --map", which always maps the whole file. "create --map
--reuse-from" is the first caller that reads only parts of a file.
Track exhaustion in its own flag and never restart the generator.
blockify() also has to stop treating a short read as EOF, or dropping the
restart truncates the input instead: a file object may return less than
requested without being at EOF, see buzhash_self_test test_small_reads,
which passed only because of the restart.
Thomas Waldmann