swidth_slice does not split a wide (e.g. CJK) character, so each of the two
slices around the "..." can be one cell short of its width. The truncated
result was then up to 2 cells narrower than the space, and nothing padded it.
borg create --progress and borg extract --progress repaint their line with a
carriage return and rely on the padding to overwrite the previous line: with
wide characters in the paths, the last characters of a longer previous line
stayed visible.
Backport of #10480.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
msgpack 1.2.3 brings fixes relevant to borg's usage of the C extension:
memory leaks when map key validation or container hooks fail
(GHSA-j586-36cw-2gc2) and a use-after-free when cleaning up an
incomplete Unpacker.skip().
The C Unpacker now raises ValueError when switching between unpack,
skip, header readers or read_bytes() while an object is incomplete.
borg only uses feed() plus unpack()/iteration (and RobustUnpacker
creates a fresh Unpacker on each resync attempt), so this does not
affect borg.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
RemoteRepository.call_many took the next call from the front of a list
(calls.pop(0), self.preload_ids.pop(0)), which moves all the remaining
entries each time. For a single item with millions of chunks (preload +
get_many over all of its chunk ids), that made extract from a remote
repository quadratic in the chunk count of the item.
Use collections.deque with popleft() for calls, waiting_for and
preload_ids.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The "About modes without encryption" paragraph claimed that only modes
other than "none" let borg detect accidental corruption or malicious
tampering. That is only half true: "none" mode still verifies the CRC32
of every repository object and the SHA-256 chunk IDs (e.g. on extract or
with "borg check --verify-data"), so accidental corruption is detected.
What "none" mode cannot detect is malicious tampering, because none of
these checks involve a secret key.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
An archive made with "borg create ARCHIVE ." has an item whose path is
".", so extract_item computed <dest>/. and the "remove what is there
before (re)creating the item" block called os.rmdir(<dest>/.) on it.
posix requires rmdir() to fail with EINVAL for a path whose last
component is ".", so everywhere else this was a silently ignored no-op.
Haiku's dir_remove() strips a trailing "/." from the path instead and
then removed the extraction directory, which is empty at that point -
after which the extraction failed with B_ENTRY_NOT_FOUND for every item.
There is nothing to remove for such a path anyway (the item is the
directory we are extracting into), so skip the block.
Co-Authored-By: Claude Opus 5 <[email protected]>
The overread check does not work on haiku: it builds its trap via
ctypes.util.find_library('c'), which haiku answers by walking
os.environ['LIBRARY_PATH'] - unset in a non-interactive shell, so the
check subprocess dies with a KeyError and the test reports an overread
that never happened.
Co-Authored-By: Claude Opus 5 <[email protected]>
Backport of 2fa73c95e from master.
kill_stale_lock() only expected FileNotFoundError from os.listdir(). Haiku
throws EBUSY there while the lock directory is concurrently replaced via
rename(), and sometimes "bad data" (B_BAD_DATA) instead, so the exception
escaped ExclusiveLock.acquire() and the lock race test saw it as unclean
concurrency handling.
Do not enumerate error codes: not being able to read the lock directory
means we can not call the lock stale, whatever the reason.
Co-Authored-By: Claude Opus 5 <[email protected]>
Backport of 74d943092 and 9004e3c22 from master.
cross-platform-actions 1.5.0 adds support for Haiku R1/beta6, so use that
instead of r1beta5 (2024), which was too painful to work with, see #9463.
The r1beta6 image already has git, pkgconfig, lz4 and openssl3, and it
comes with a usable python3, so it does not need a python or cffi package
installed anymore (1.4 does not use argon2-cffi).
Also, from master:
The VM corrupts data every now and then: rustc crashed on crate metadata
it had written itself ("assertion failed: bytes[len] == STR_SENTINEL")
and pip found sha256 mismatches reading big wheels back from its own
cache. So build outside /boot/system/cache/tmp (haiku's /tmp), run one
rustc at a time and retry a pip install with a purged cache, so a damaged
file does not fail the whole job.
Co-Authored-By: Claude Opus 5 <[email protected]>
- pyproject.toml: add the "Programming Language :: Python :: 3.15" classifier
- tox: add 315 to env_list and the py315-{none,fuse2,fuse3} env sections
- CI: add an ubuntu-26.04 / 3.15-dev / py315-fuse3 entry to both native_tests
matrices (PR and push)
3.15 is still a prerelease, so the CI entry is marked "allow-failure": true and
the job gets continue-on-error: ${{ matrix.allow-failure || false }} - a broken
prerelease python (or a dependency without 3.15 wheels / not compiling for 3.15
yet) then does not fail the build. Remove the "allow-failure" entries once 3.15
is final.
Verified locally on macOS/arm64 with 3.15.0rc1: the C/Cython extensions build
and the full test suite passes.
Co-Authored-By: Claude Opus 5 <[email protected]>
msgpack 1.2.2 only brings bug fixes relevant to borg's usage:
correct Timestamp.from_datetime for datetimes far from the epoch,
reject out-of-range timestamp nanoseconds, copy ExtraData.extra out of
a temporary contiguous buffer, and a new RuntimeError guard against
re-entrant Unpacker.feed() (borg never feeds from inside a hook).
Co-Authored-By: Claude Opus 5 <[email protected]>
The chunks cache sync emitted 2 INFO lines per archive:
- warm chunks.archive.d (the usual case):
"Reading cached archive chunk index for <name> ..." plus
"Merging into master chunks index ..."
- cache rebuild:
"Fetching and building archive index for <name> ..." plus
"Merging into master chunks index ..."
On repositories with many archives this floods the output of anybody
running with -v / --info, without telling them much - "Merging into
master chunks index ..." does not even name the archive.
Log these at debug level, so --info now only shows the summary:
Synchronizing chunks cache...
Archives: %d, w/ cached Idx: %d, w/ outdated Idx: %d, w/o cached Idx: %d.
Done.
Per-archive feedback for interactive use is still available via
--progress (the cache.sync progress indicator names each archive), and
borg 2 already logs all of this at debug level.
While a command runs, borg installs a SIGINT handler (SigIntManager) that only
remembers that Ctrl-C was pressed, so that a running operation like borg create
can still be finished in an orderly way. That handler is also active while borg
waits for input from the user - so a Ctrl-C, or a SIGINT sent by a frontend, was
just remembered while input() / getpass() kept waiting.
While waiting for an answer or a passphrase there is nothing to finish, so
temporarily install the raising SIGINT handler around the input call - for the
y/n questions in yes() and for the passphrase prompt.
Both abort right away with rc 130 now, which is what an interactive user expects
and what frontends like Pika Backup need for their abort feature. Terminal echo
is not left disabled: getpass() restores the termios settings in a finally
clause, also when the read raises.
borg repo-create keeps cleaning up after itself: an interrupt at the "Enter new
passphrase" prompt runs into the existing
`except (EOFError, KeyboardInterrupt): repository.destroy()`, so neither a
repository directory nor a key file is left behind (verified for the repokey and
the keyfile case, at the first and at the confirmation prompt).
The tests use their own SigIntManager instance rather than the global sig_int,
which can only be entered once per process. They send the signal with
signal.raise_signal(): os.kill() would send it to the process and some kernels
(e.g. NetBSD) then deliver it to another thread if there is one - the main
thread would only run the handler later, after the prompt restored the previous
handler, and the test would flap.
Co-Authored-By: Claude Opus 5 <[email protected]>
Like for create/import-tar/delete/prune, --quick-stats skips the
repository-wide "All archives" and chunk index statistics.
Computing them means walking the metadata of ALL archives in the
repository (plus a chunk index summary), so they are much more expensive
than what is shown about the archive itself - especially for repos with
many archives, for remote repos (one round trip per archive) and for
repos still containing pre-1.2 archives (no size/nfiles metadata, so
their item metadata stream must be read).
- borg info ARCHIVE --quick-stats: only show the archive's own stats.
- borg info REPO --quick-stats: only show repository id, location,
encryption, cache and security dir.
- with --json, the top-level "cache" object is omitted, so serializing
it does not trigger the stats computation.
Note: this does not make the per-archive "Deduplicated size" cheaper,
that one still needs to read the archive's item metadata stream.
Co-Authored-By: Claude Opus 5 <[email protected]>
borg issued a single read()/write() call for the whole bucket array, so a
repository index larger than 2GiB meant one syscall transferring >2^31 bytes.
The code assumed that a buffered reader would split such a request into several
underlying reads. That holds on Linux, where the kernel caps every read at
MAX_RW_COUNT (0x7ffff000) so a huge request always short-reads and CPython's
BufferedReader loops. It does not hold everywhere: on FreeBSD IOSIZE_MAX is
SSIZE_MAX, so the whole multi-GB request goes to one read(2).
On the FreeBSD+ZFS servers of some storage providers, such huge single reads
have been observed to return a bogus, too-large byte count, which CPython then
rejects:
OSError: raw readinto() returned invalid length 4945346542
(should have been between 0 and 2756575232)
Every reported bad return value N satisfies N = M - 18 + k*65536, where M is the
requested length and 18 is sizeof(HashHeader), and every reported M was > 2^31.
The root cause is below borg (OS or filesystem) and is still unidentified, so
this is a workaround rather than a fix: it keeps borg out of the size range in
which the problem has been observed. Failures stay loud - short reads and short
writes are still errors.
The buckets are read into a single preallocated bytes object, filled slice by
slice, so peak memory is unchanged; reading slices into a list and joining them
afterwards would have doubled it for a multi-GB index.
Verified on FreeBSD 15.1 + ZFS against a real 4.76GB index: what used to be one
read of 4763418624 bytes is now 17 reads of 256MiB plus one of 200015872, the
index round-trips byte-identically, and the testsuite passes.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01FepzD7Uz7AmSGJE7T7PzQk
Determining the fqdn issues a DNS query. That happened at import time, so
EVERY borg process paid for it - which can take very long on hosts whose
own hostname does not resolve (~35s per process on github's macOS runners,
which made the macOS test jobs take hours: each remote test spawns ~10
borg serve / client processes).
Replace the module-level hostname / fqdn / hostid attributes with
get_hostname() / get_fqdn() / get_hostid() functions. A cached
_gethostname() keeps all derived values consistent for the process
lifetime; get_fqdn() is additionally cached because it issues the DNS
query. Processes that never use them no longer resolve anything at all,
on any platform. get_hostname() never needs DNS, and a set BORG_HOST_ID
avoids the fqdn lookup entirely.
Co-Authored-By: Claude Opus 5 <[email protected]>
RemoteRepository.close() closed stdin and stdout and then did a bare
p.wait(). It abandoned the select loop that was draining the child's
stderr, but kept the read end of that pipe open - so if the child still
had more than a pipe buffer (64kiB) of stderr output to write, it
blocked writing to that pipe while borg blocked in wait4() waiting for
it to exit. Neither side could make progress and there was no timeout,
terminate() or kill() fallback, so the process pair hung forever.
close() now drains the remote's stderr until EOF - the remote's error
messages shall get to the client - and only then waits for the child,
with a timeout and SIGTERM / SIGKILL escalation, so that it can not
block unboundedly on a child that is blocked on a pipe we own.
The folding implementation loads full 16 byte vectors from 16 byte
aligned addresses, but it was also used for the bytes in front of the
first alignment boundary and for inputs shorter than 16 bytes. That
caused two defects:
- For inputs of 4..15 bytes it loaded 16 bytes starting at src, reading
up to 12 bytes past the end of the buffer. Where those bytes fall into
an unmapped page, this segfaults. repository.py _read() calls
crc32(memoryview(header)[4:]), which is a 5 byte view, so this was on
the hot path for every segment entry read and is the cause of the
intermittent NetBSD test suite segfaults tracked in #5922.
fixes#10149
- In the alignment prologue XOR_INITIAL() xored the 32 bit initial value
into the low 4 bytes of the vector, but partial_fold() then kept only
algn_diff bytes of it. For algn_diff of 1..3 the upper bytes of the
initial value were silently discarded, giving a wrong checksum.
fixes#10150
Hand both cases to the table driven implementation instead, so the
folding code only ever sees full aligned blocks and the initial value is
always folded into a complete 16 byte vector. This is the structure
zlib-ng uses as well, see crc32_copy_small() in their
arch/x86/crc32_pclmulqdq_tpl.h.
It is also faster for the sizes borg actually uses. Measured on an amd64
Zen 3 machine, ns per call:
len before after
5 9.45 5.00
9 9.60 7.90
16 7.42 6.70
41 11.68 9.28
4096 248.88 246.30
Only 12..15 byte inputs get slower (9.6 -> 13.8 ns), and borg passes 5
and 37 byte headers.
Tests: test_crc32 only swept start offsets 0..3. CPython's bytes payloads
are 16 byte aligned, so those cover alignments 0, 15, 14 and 13 and can
never reach the broken 1..3 - widen the sweep to 0..15.
Add three more tests, all comparing against zlib.crc32:
- test_crc32_random_slices takes memoryview slices of a 1 MiB buffer at
random, unaligned start and end offsets, which gives large slices with
an unaligned head at every 16 byte alignment.
- test_crc32_short_slices walks every end offset from start to start+50
for 10 random starts, covering the empty input, the scalar path, the
short path and the folding path at arbitrary alignments.
- test_crc32_clmul_no_overread puts the buffer flush against a PROT_NONE
guard page. It runs in a subprocess, so a regression fails the test
rather than killing the test runner with a SIGSEGV.
All four fail against the unpatched implementation and pass with this
change; they add about 0.4s to the test suite.
Co-Authored-By: Claude Opus 5 <[email protected]>
create processes directory entries in inode order (scandir_inorder), so
which of the two hardlinks becomes the hardlink master depends on the
inode numbers the filesystem assigned to the dirs a and b. On linux
ext4 these follow creation order, but e.g. FreeBSD ufs dirpref spreads
new dirs over cylinder groups, so b may get a lower inode than a - then
b/hardlink became the master and diff of only input/b succeeded (rc 0)
instead of warning about the excluded hardlink source (rc 1).
Determine master/slave from the actual dir inode numbers and exclude
the master's dir, making the test work on any filesystem. Also drop the
FreeBSD/NetBSD skip: this failure was traversal-order dependence, not
the ctime quirks of #9147/#9153.
Co-Authored-By: Claude Fable 5 <[email protected]>
Backport of #10075 (master): the pytest tmp dirs (and the borg test repos
in them) were on OpenBSD's slow FFS disk. OpenBSD 7.9 has no python 3.14
package, so it cannot get the faster coverage core - this is the
equivalent speedup for that job.
Co-Authored-By: Claude Fable 5 <[email protected]>
Backport from master: the cross-platform-actions default VM has 2 cpus,
so half of the runner's 4 vcpus sat idle while pytest -n auto only had
2 workers.
Co-Authored-By: Claude Fable 5 <[email protected]>
Backport of #10133 (master). On python 3.14, coverage.py uses its
sys.monitoring based core, which measures with ~zero tracing overhead,
while the C tracer core used on older pythons roughly doubles the test
suite's user CPU time. On master this took the NetBSD pytest phase from
972s (3.13) down to 627s (3.14).
pkgsrc for NetBSD 11.0 ships python314 and py314-pip/tox/virtualenv.
Co-Authored-By: Claude Fable 5 <[email protected]>
Backport of #10135 (master). The runner hostname is a long, unresolvable
.local name and macOS needs ~35s to give up resolving it. borg resolves
the fqdn at import time (platform/base.py), so every spawned borg process
paid those 35s - the 1.4-maint macOS jobs took ~4h47m (intel) and the
arm one often even longer.
Co-Authored-By: Claude Fable 5 <[email protected]>
ubuntu 22.04 GHA runners are being phased out; master already runs
everything on 24.04. Moves the CodeQL job, the lint job and the
py310-fuse2 test job over.
The remaining ubuntu-22.04 jobs (x86_64 and arm64) build the release
binaries and must stay there for glibc 2.35 compatibility.
Co-Authored-By: Claude Opus 5 <[email protected]>
Image available since cross-platform-actions/openbsd-builder v0.14.0,
which is included in the cross-platform-actions/action v1.4.0 we use.
7.9's default python3 is 3.13 (was: 3.12 on 7.7), so switch the tox env
accordingly. The packaged openssl is 3.5 now (was: 3.4).
Co-Authored-By: Claude Opus 5 <[email protected]>
If no --chunker-params was given, borg recreate does not rechunk, so the
target archive keeps the source archive's chunks. But it still recorded
the default chunker params in the target archive's metadata, so e.g.
"borg recreate --comment ..." of an archive created with non-default
chunker params claimed the default params afterwards.
Now the target archive inherits the source archive's chunker params
(if they are known) unless we actually rechunkify.
Co-Authored-By: Claude Opus 5 <[email protected]>
get_write_fd only created the segment dir for the first segment of a dir, so
writing a segment file into a dir that is missing (e.g. removed by a fsck)
crashed with FileNotFoundError. This also happened to 'borg check --repair'
when it wanted to add a commit tag to segment <index transaction id> + 1,
so repair died on the damage it is supposed to repair.
get_fd let a FileNotFoundError escape when a segment file the repository index
still refers to was gone, giving a traceback instead of telling what is wrong.
It now raises an IntegrityError naming the missing segment file and pointing at
borg check --repair. As the archives check catches IntegrityError, 'borg check
--verify-data' now reports (and with --repair deletes) the affected chunks
instead of crashing.
Co-Authored-By: Claude Opus 5 <[email protected]>
/tmp is a tmpfs without EAs, we can't use that for our tests. On
NetBSD 11 the root filesystem has extended attributes enabled, so a
plain directory there works and we do not need to build, attach and
mount a separate FFS image anymore.
Co-Authored-By: Claude Opus 5 <[email protected]>
TW