Since the PyPy nightly of 2026-10-11 (PyPy 8.0.2-alpha0, rev 544bd2f768f9), 12 tests
in archive_test.py (test_missing_chunk, test_corrupt_chunk, test_resync_*) fail with:
BufferError: Existing exports of data: object cannot be re-sized
at `del self._buffered_data[:drop_len]` in RobustUnpacker._search().
_search() scanned the buffer inside `with memoryview(self._buffered_data) as data:`
and called valid_msgpacked_dict(data[offset:]) for every offset, which also sliced the
key out of that view. CPython frees these unreferenced child views immediately, so the
bytearray is exportless again as soon as the with block releases the parent view.
PyPy has no refcounting: the child views stay alive until the GC runs. PyPy 8.0.0 did
not count child views as exports of the bytearray, the new nightly does (like CPython
always did), so the resize after the with block fails.
Fix: valid_msgpacked_dict(d, offset=0) only indexes d, it never slices it, and _search()
passes (data, offset) instead of a slice. Together with the explicitly released slice in
_try_item_start(), no unreleased view of _buffered_data exists anymore when it is
resized. The scan loop also gets a bit faster on CPython (no slice object per offset).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The "repository was previously located at" check compared path strings only,
so reaching the same local repository directory via a symlink, a bind mount or
a different letter case on a case-insensitive filesystem looked like a
relocation. Answering yes rewrote the stored location, so alternating between
the paths kept triggering the prompt (breaking e.g. unattended prune).
If the strings differ, but both are local paths referring to the same existing
directory (same st_dev/st_ino, st_ino != 0), do not warn. The stored location
and all displayed paths stay as they were. Remote locations are not affected,
and a real move/rename (old path gone) still triggers the warning.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The files cache (file sizes, timestamps, inode numbers, chunk lists and
--digests content digests) was stored as plaintext msgpack in the local
cache directory, protected only by a checksum without a key, so anyone
who could read the cache directory could read it, and anyone who could
write it could make borg skip changed files.
It is now written as a sealed stream: encrypted and authenticated with
the repository key (authenticated only in the authenticated-* modes),
bound to the repository id and the file name of the files cache. A files
cache that fails the verification is ignored with a warning (the exit
code is not changed) and rebuilt from the previous archive.
The checksums in the [integrity] section of the cache config are not
needed any more, so the cache config is no longer rewritten on every
cache close().
The FAQ about cache security now describes what is protected and what is not.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A sealed stream cuts a byte stream into 1 MiB frames and puts every
frame into the key's envelope (encrypted and authenticated in the
encrypting modes, authenticated only in the authenticated-* modes),
so neither the writer nor the reader needs memory for more than a frame.
The AAD of a frame is a caller-given context plus the frame number and
a "last frame" flag, so the reader detects changed, reordered, dropped
or duplicated frames, truncation and trailing data. Every frame is
sealed with encrypt_oneshot(), so sealed streams are thread-safe.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Same as BORG_MOUNT_ARCHIVE_DIR_FORMAT, but usable from fstab and autofs
entries; if both are given, the mount option is used. As mount options
are separated by commas, the format can not contain a comma.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
borg create --sparse seeks over holes, but the archive still does not
record the hole layout, so extract --sparse can not reproduce it exactly.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Repository.__init__ now parses and validates both variables, so a bad
value fails before the store is created, opened or locked. Each must be
a positive integer, and BORG_PACK_MAX_SIZE must be below the new
MAX_PACK_SIZE_LIMIT of 2126512128 (2 GiB - 20 MiB). PackWriter.add keeps
the object that crosses the cap, so packs stay below 2 GiB, clear of OS
bugs with files of 2 GiB or more. A rejected value raises Error with the
variable name and the value.
With only BORG_PACK_MAX_COUNT set, the writer still gets
MAX_PACK_SIZE_LIMIT - 1 as its byte ceiling. pack_max_size returns the
user's size, or DEFAULT_PACK_MAX_SIZE when that variable is unset, so
compaction keeps targeting that value.
The environment-variable help and the packs internals page state the
limit, and repository tests cover rejected values (also on repository
creation), the size one below the limit, count-only mode, the unset
defaults, and both variables set together.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019wSaUMZrtm6XSzwEcxavkv
GIMP source (editable text layers) and a preview of the chosen
rectangular design "C" from the ticket, plus a README with the
font, colours and printing notes.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
acquire() creates its lock object, waits for the race recheck delay and
lists the lock objects again to detect other clients racing for the lock.
With list-after-write consistency (local filesystems, sftp, rest, AWS S3,
MinIO), at least the client that created its lock object last sees the
other's lock in its second listing, so the delay is not needed for
correctness and the 0.01s default stays.
Stores that only list a new object after a lag (e.g. NFS clients caching
directory listings, some cloud storages via rclone) need a delay of at
least that lag. That lag is a property of the storage, not of the request
latency, so it is not measured, but set via BORG_LOCK_RECHECK_DELAY.
Invalid values abort with an error.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
- pass 1 walks the gaps of the packs that have any and counts their superseded duplicates as
reclaimable. A pack holding only unused indexed objects and superseded duplicates is deleted.
The walk refreshes the lock per pack, shows progress and stops on Ctrl-C.
- compact_pack drops the superseded duplicates found in pass 1 (superseded_ranges).
- superseded_gap_ranges reads GAP_READ_SIZE bytes after a small object, so small objects share a
store request, and META_READ_SIZE bytes after a large one.
- compact and repo-compress check the index entries of every pack. A pack in which they overlap
or reach past the end of the file is treated as untrusted. compact logs it as an error and
keeps it unchanged.
- a tiny pack that the all-packs gate keeps from being dropped or rewritten is a merge candidate.
An archive's item metadata chunks are written into the same packs as the
file content chunks (one ChunkBuffer flush per 8 MiB of items). Reading
them via get_many loaded each whole 50 MB pack just to take a few MB of
item metadata out of it: e.g. "borg list" of an archive with 3 MB of item
metadata loaded 49.6 MB.
DownloadPipeline.fetch_many now uses gather_many for ROBJ_ARCHIVE_STREAM
(unpack_many / iter_items: list, diff, extract, mount, compact, recreate,
transfer, analyze, ...), as do archive_get_items for the item_ptrs and the
archives check for the item metadata chunks. File content is still read
via get_many (whole packs).
The legacy repositories get a gather_many that just calls get_many, so
transfer from a borg 1.x repository still works.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Correct what borg compact reclaims after a crash, how compact/repo-compress
rewrite index/, when the index is rebuilt from the packs and when the repair
walk is used. Add pack writer and read path docs.
Without the object, the commands silently fell back to the built-in defaults, so
anybody with write access to the store could remove a repository default (e.g. an
"obfuscate" compression) unnoticed: the key protects the content of the object,
not its presence.
Now "borg repo-create" always writes config/defaults (empty if no default was
given), and a missing object is an error (Repository.DefaultsMissing, rc 34) like
one that fails the authentication: the commands using the defaults refuse to run.
"borg check" reports such an object, "borg check --repair" replaces it by empty
defaults, so the repository can be used again (with the built-in defaults).
with_repository reads the defaults only after the security checks (assert_secure).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
"borg repo-create --compression SPEC --chunker-params PARAMS" stores repository
defaults in the new config/defaults store object.
Compression: the commands that compress data (create, recreate, import-tar,
transfer, repo-compress, debug put-obj) use the repository default if no
compression was given via the command line, the environment or default.yaml;
without a repository default, they use lz4 as before.
Chunker params: "--chunker-params default" now means the repository default,
else the built-in CHUNKER_PARAMS: ChunkerParams("default") returns the
DEFAULT_CHUNKER_PARAMS marker and with_repository replaces it by the real
chunker params. borg create and borg import-tar use the marker as their parser
default, so they use the repository default if no chunker params were given.
borg recreate and borg transfer still only rechunk if --chunker-params is
given; with "default", they rechunk to the repository default.
config/defaults is stored in the key's store object envelope, so it is
authenticated: nobody without the key can change the defaults (e.g. remove an
"obfuscate" compression) without being noticed. config/config can not hold
them, as it is read before the key is known and is not authenticated.
config/defaults is read at most once per Repository object.
borg repo-info shows the defaults (also in --json).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
borgstore 0.7.0 adds Store.gather (many byte ranges with one call, one
roundtrip for REST) and fixes partial loads with a negative offset from
cached namespaces.
Store.defrag now requires all sources to be items of the given namespace,
which borg's defrag calls (hex pack names in "packs") already satisfy.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Update the data structures and security docs: the lock objects are in the
store object envelope, locking needs the key (it is loaded before waiting for
the lock), unreadable lock objects are treated as exclusive locks, and what
the store still sees about the locks. Add a change log entry.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
borg 1.x protected the key of the authenticated and authenticated-blake2
modes with the passphrase (pbkdf2, "sha256" algorithm), like the key of
the encrypted modes. The legacy authenticated key classes lacked
Pbkdf2FileMixin, so borg 2 could not decrypt that key and --from-borg1
(e.g. borg transfer) failed with "passphrase is incorrect".
The test uses keys exported from borg 1.4.5 authenticated and
authenticated-blake2 repositories.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Opening a borg 1.x repository without --from-borg1 crashed with a traceback while
loading the config/config store object, because a borg 1.x repository has a config
*file* where borg 2 has the config/ namespace (NotADirectoryError for a local
repository, HTTP 500 via ssh://; InvalidRepository on Windows).
borgstore 0.6.4 reports a name below an object as ObjectNotFound on all platforms,
so require it. If config/config is not found, look at "config" via the store
backend and raise Repository.LegacyRepository (rc 29) if it is a file, telling the
user to use --from-borg1 (e.g. with borg transfer).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
--from-borg1 via ssh:// (e.g. borg transfer from a borg 1.x repository)
failed with a borg 1.x "borg serve" on the remote host:
- the client called info() after open, which borg 1.x servers do not
have ("Borg server is too old for info"),
- then close() sent the close RPC, which borg 1.x servers do not have
either ("RPC method close is not valid"), masking the first error,
- get_manifest is also missing on borg 1.x servers,
- borg 1.x servers pack bytes into the msgpack "raw" type, which the
client unpacks to str, so e.g. the repository id and object data
arrived as str.
For borg 1.x servers, the client now uses repository version 1 (the only
version borg 1.x serves), does not send the close RPC (borg 1.x closes
the repository when stdin gets closed), gets the manifest object via get
and converts the bytes results back to bytes.
Tested with a real borg 1.4.5 "borg serve": repo-list, version,
repo-create --other-repo and transfer with --from-borg1, repokey and
keyfile mode. The testsuite emulates a borg 1.x server.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Describe ssh:// repository URLs for current repositories (served by a
remote "borg serve --rest") and for legacy borg 1.x repositories (with
--from-borg1, served by a remote "borg serve") and add a change log
entry.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
TW