The generated borgfs.1 was corrupted: jsonargparse injects --config and
--print_config into the borgfs parser (a top-level parser), and
--print_config's "[=flags]" metavar is invalid rST option-list syntax,
so the man page embedded a docutils WARNING and an unformatted blob of
options. Hide jsonargparse-injected actions (detected structurally by
their module, so future injected options stay filtered) and borg's
top-level-only --cockpit from the borgfs docs via help=SUPPRESS, which
all make.py writers already honor. -V/--version stays documented, as
in borg 1.x. Regenerate borgfs.1 accordingly (now also showing the
REPOSITORY positional).
Remove docs/usage/borgfs.rst{,.inc}: build_usage intentionally stopped
generating a borgfs HTML page long ago (#3404), so the leftover
.rst.inc still documented the borg 1.x CLI (REPOSITORY_OR_ARCHIVE,
--prefix, ...) and was published as an orphan page. Nothing references
these files; borgfs is documented on the "borg mount" page.
Co-Authored-By: Claude Fable 5 <[email protected]>
Copying is cheap and fast: no file content is read or written, only a new
archive metadata object is created. Like any deduplicated archives, the two
archives share their data, so a copy needs almost no additional repository
space.
The copy is an independent archive: deleting either of the two archives
keeps the other one intact, because "borg compact" only frees chunks that no
remaining archive references.
Archive.copy() is Archive.rename() without removing the original archive
entry. Copying an archive to its own name is refused: the new metadata would
be identical, thus have the same archive ID, and no second archive would be
created. NEWNAME may well be the name of some other existing archive, the
copy then just joins that archive series.
OLDNAME must match precisely one archive, so it accepts an archive name (if
it is unique) or an archive ID like "aid:d34db33f".
Co-Authored-By: Claude Fable 5 <[email protected]>
"--encryption=none" and "--encryption=authenticated" have not existed since
the id hash became part of the mode name for the modes that do not encrypt:
they are now "none-sha256"/"none-blake3" and "authenticated-sha256"/
"authenticated-blake3". The examples in docs/usage/repo-create.rst still used
the old names, so copying them gave an argparse error.
The "authenticated --id-hash=blake3" example was doubly wrong: those modes
name their id hash themselves and reject a separate --id-hash. Replaced it
with plain "authenticated-blake3"/"none-blake3" examples and dropped the
now-wrong claim that the non-encrypting modes only support sha256.
Also:
- "--encryption=aes-ocb" (docs/usage/key.rst, docs/quickstart_example.rst.inc)
is not a valid mode either, it is "aes256-ocb".
- scripts/rest_perf_test.sh used "--encryption none".
- regenerated the affected man pages; borg-repo-create(1) additionally picked
up the already-fixed --key-location help text.
All examples were verified by actually running them.
The "--map requires exactly one input path" and "the reference archive must
contain exactly one file item" paragraphs interrupted the description of the
map file format. Move both to the end of the Input maps section and rebuild
the generated usage docs / man page.
Converts thin_dump XML (full mode: allocation map of one thin LV) and
thin_delta XML (delta mode: differences between two thin snapshots)
into the borg create --map format. The docstring documents the full
workflow including the reserve/release_metadata_snap steps and the
snapshot discipline needed for --reuse-from.
Also add a docs section pointing from the --read-special LVM example
to the input map based approach.
Co-Authored-By: Claude Fable 5 <[email protected]>
For input map ranges marked "same", reuse the chunks of the reference
archive's item instead of reading the input. Reference chunks that only
partially overlap "same" ranges are re-read completely, so the result
is correct with any chunker; with the fixed chunker, read windows align
with the reference chunk grid. Reference chunks missing from the repo
are read again (like the files cache does on a lost chunk).
Together with --map, this implements efficient incremental backups of
block device snapshots: only changed ranges need to be read, e.g. as
reported by thin_delta for LVM thin volume snapshots.
Co-Authored-By: Claude Fable 5 <[email protected]>
An input map describes the content ranges of the single input file:
data ranges are read and stored, zero ranges are stored as holes
without reading them. Primary use case: backing up snapshots of large
(esp. thin-provisioned) block devices, where the storage layer knows
which ranges are in use.
Co-Authored-By: Claude Fable 5 <[email protected]>
The help text still claimed sparse hole detection is "supported only by
fixed chunker", which is stale: --sparse is passed to every chunker via
get_chunker(), and ChunkerBase/FileFMAPReader implement sparse processing
generically (build the hole map via SEEK_HOLE/SEEK_DATA, seek over holes
instead of reading them, process their content as all-zero).
Verified for all chunkers (buzhash, buzhash64, fastcdc, rabin-aes,
goldilocks-aes, toeplitz-aes, fixed) by chunking a real sparse file with
sparse=True and a hole-poisoned file with an explicit fmap: content is
reproduced exactly and hole ranges are seeked over, not read.
Also move the sparse processing note in docs/internals/data-structures.rst
from the "fixed" chunker section to the chunker overview, and regenerate
the affected generated docs (docs/usage/create.rst.inc, borg-create.1).
Co-Authored-By: Claude Fable 5 <[email protected]>
If a recursion root (a path given on the command line or via a patterns
file) is a symlink, borg now follows it and archives what it points to,
using the path as given.
Symlinks encountered while recursing are not affected, they are archived
as symlinks as before. Symlinks given via --paths-from-* are never
followed either.
Archiving the symlink target under the symlink's path also keeps the files
cache working if the symlink target changes its name (e.g. a 'current'
symlink pointing to the latest of a series of directories).
A recursion root that is a symlink with a non-existing target is skipped
with a warning (new BackupBrokenSymlinkError, rc 112).
As a followed symlink must be opened without O_NOFOLLOW, add flags_dir_follow
and flags_normal_follow and let the callers choose the flags.
borg find iterates over the archives selected by the usual archive
filter options (all archives by default) from newest to oldest, over
all items of each archive, and outputs one line per item matching the
given PATHs (paths or patterns, matching like in borg list /
borg extract).
Output lines are formatted like borg list default output, but are
prefixed with a short archive ID and the archive name. As the
archives of a series all share the same name, only the archive ID
uniquely identifies the archive.
Output is customizable via --format / BORG_FIND_FORMAT (ItemFormatter
keys) and --json-lines.
There is no extra index for file paths (keeping one up to date would
cost considerable runtime and space), so this reads the metadata of
all selected archives.
PR #10063 added the "borg help environment" topic and build_man
generates borg-environment.1 since then, but the generated man page
was never committed. Add it (built via scripts/make.py build_man).
Reading from a fifo (or char device) with --read-special could block
borg forever: opening a fifo read-only blocks until a writer connects,
and reads block until data arrives - a broken producer thus hangs an
unattended backup indefinitely.
Now, when no data arrives for --read-special-timeout SECONDS (default:
1800, i.e. 30 minutes; this includes waiting for a fifo's writer to
connect), borg gives up on that file: fifos and char devices are opened
with O_NONBLOCK and all content reads go through the new
SpecialFileReader, which waits for data via select() and raises
OSError(ETIMEDOUT) when the timeout expires. That maps to the new
BackupTimeoutError (rc 111), so the file is skipped with an error (and
not retried) while the backup continues with the remaining files.
--read-special-timeout=0 means: wait forever (previous behavior).
Note: with a timeout active (the default), a writer connecting but
closing without sending anything is reported as a timeout instead of
being stored as a file with empty content.
Co-Authored-By: Claude Fable 5 <[email protected]>
borg2 uses blake3 (as keyed hash) for chunk ids, but the --format
hash keys only offered the python stdlib hashes, so {blake3} raised
a KeyError.
The blake3 package is a hard dependency of borg2 anyway, so just add
blake3 to the set of supported hash algorithms. Output is the default
256bit hexdigest, like for the other hash keys.
Co-Authored-By: Claude Opus 5 <[email protected]>
Set BORG_DEBUG_PROFILE to a filename to get an execution profile written
there; a ".pyprof" suffix still selects the Python-compatible format.
Note that this now applies to EVERY borg invocation while the variable is
set, not just to a single command like the option did - the docs say so and
the test switches it off around the "borg debug convert-profile" call, which
would otherwise profile itself over the profile it is reading.
Unlike the other options removed recently, --iec did work for borg2 repos -
but the same setting is better placed in the environment: it is a display
preference one usually wants for all borg invocations, not per command.
BORG_IEC=yes now selects IEC units (1KiB = 1024B); "true" and "1" are
accepted, too, because jsonargparse's automatic BORG_<OPTION> environment
variables accepted these for --iec. Note that removing the option also
removes that automatic environment variable, so the new use_iec_units()
helper reads BORG_IEC explicitly.
The iec=... plumbing through Archive, Statistics, ArchiveFormatter and Cache
is unchanged, only the places reading args.iec now call use_iec_units().
Both were only ever read by the legacy (borg 1.x, ssh://) remote repository
code, so they had no effect on borg2 repositories - borgstore does not do
rate limiting or upload buffering at all.
For the legacy code path they are not worth keeping either: we only support
transferring FROM borg1 repos, so there is not much traffic in the upload
direction of a borg1 repo anyway.
Removes the SleepingBandwidthLimiter class with them; the only thing left
from it is write_to_fd(), which turns a BrokenPipeError into the nicer
ConnectionBrokenWithHint. Without an upload buffer size limit, the "queue
more calls" condition simplifies to "the send buffer is empty".
Like --remote-path, --rsh was only honoured on the legacy (borg 1.x, ssh://)
code path. For borg2 repositories the remote shell command comes from
borgstore, which reads its own BORGSTORE_RSH environment variable, so --rsh
silently had no effect there.
Remove the option; BORG_RSH is now the only way to set the remote shell
command. For borgstore-based repositories, borg gives the BORG_RSH value to
borgstore as BORGSTORE_RSH, except if that was set explicitly (then the user
wants a different command for borgstore).
borg benchmark crud does not need to inherit --rsh / --remote-path into its
sub-invocations any more (environment variables are inherited anyway), so the
parse_args wrapper is gone.
The --remote-path option was only honoured on the legacy (borg 1.x, ssh://)
code path in legacy/remote.py. For borg2 rest:// repositories, the remote
borg is determined by rest_serve_command(), which only ever looked at the
BORG_REMOTE_PATH environment variable - so giving --remote-path silently had
no effect and the far end failed with "command not found: borg".
Rather than plumbing args into rest_serve_command(), drop the option
completely and use BORG_REMOTE_PATH consistently for both repo types. Also
apply replace_placeholders() to it for rest:// repos, as the legacy code
path already did.
Add a per-directory tar download: append ?tar (or ?tar=1) to a directory URL,
or click the download icon shown next to the heading in the browser listings.
Unlike a plain file download, the tar preserves POSIX metadata (owner, group,
mode, sub-second timestamps, symlinks, special files, xattrs, ACLs), so it is
the metadata-lossless way to restore a whole directory tree over the server.
The tar is rooted at the requested directory (paths above it are stripped);
?tar on the archive root exports the whole archive.
The tar size is not known in advance (PAX header sizes vary), so it is streamed
with chunked transfer encoding. Repository access (item iteration and chunk
fetching) is serialized under repo_lock, but each chunk is written to the client
outside the lock: a slow client cannot block other requests, and the
LockRefresher can keep the repository lock alive during a long download (holding
the lock for the whole stream would starve it and let the lock go stale). A
missing chunk leaves the chunked stream unterminated and closes the connection,
so the client detects the truncation instead of receiving a corrupt archive.
To avoid duplicating the borg-item -> tar mapping, item_to_tarinfo() and
item_to_paxheaders() are factored out of TarMixIn._export_tar to module level in
tar_cmds.py (behaviour-preserving; item_to_tarinfo now returns needs_content and
the caller builds the content stream). webdav writes the tar blocks itself
(TarInfo.tobuf for the header, chunk data, 512-byte padding, end-of-archive
marker) so it can control the per-chunk locking.
Verified end-to-end with the system tar tool: modes, symlink targets and a
multi-chunk file are preserved byte-identically.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add a -f/--foreground flag; without it, the command daemonizes and runs
in the background, matching borg mount. Bind the socket in the foreground
first (so bind errors like "port in use" are reported before forking, and
the listening fd survives the fork), then migrate the repository lock to
the forked process. The LockRefresher and server threads are started after
the fork, since threads do not survive fork().
The daemon stops on SIGTERM (the usual way to stop a daemon) or SIGINT,
shutting down cleanly and releasing the repository lock.
Daemonizing needs os.fork(), which does not exist on Windows; there the
command stays in the foreground (borg webdav is meant to be usable on
Windows, unlike borg mount).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A mounted WebDAV file system reads a big file as many small, sequential
range requests, which tend to hit the same content chunk over and over.
Without caching, each request re-fetched and re-decrypted the chunk.
Add a shared LRU cache of decrypted chunks (keyed by chunk id, so it is
valid across archives and requests), the same approach borg mount uses.
Its size is set by BORG_MOUNT_DATA_CACHE_ENTRIES (default: number of
CPUs), just like for borg mount. The cache is guarded by repo_lock (the
server is multi-threaded and LRUCache is not thread-safe), while writing
to the client still happens outside the lock.
Co-Authored-By: Claude Fable 5 <[email protected]>
The served archives can now be mounted as a read-only network file system
using the WebDAV client built into most operating systems and file managers:
- Windows Explorer: "Map network drive" -> http://localhost:8000/
- macOS Finder: "Go > Connect to Server" -> http://localhost:8000
- GNOME Files / KDE Dolphin: dav://localhost:8000/
- Linux kernel mount: mount -t davfs (davfs2 package)
Implemented (still python stdlib only):
- OPTIONS advertising class 1 DAV support.
- PROPFIND with Depth 0/1 (infinity is refused, as RFC 4918 permits),
supporting allprop / propname / named-prop request bodies, answering
with 207 multistatus XML. Request body size is limited.
- live properties: resourcetype, displayname, getcontentlength,
getcontenttype, getlastmodified, creationdate, getetag, and empty
supportedlock/lockdiscovery (no locking - read-only server).
- Range requests (single range): only the chunks overlapping the
requested range are fetched and decrypted (chunk sizes are known in
advance), so the many small ranged reads DAV file systems do for big
files stay efficient. 206/416 handling, If-Range.
- conditional requests: ETag (archive contents are immutable, so
mtime+size is a solid validator), If-None-Match -> 304.
- all writing/locking methods (PUT, DELETE, PROPPATCH, MKCOL, COPY,
MOVE, LOCK, UNLOCK, ...) are rejected with 405 + Allow, and clients
see a read-only file system.
- symbolic links and special files are not exposed via WebDAV (the
protocol has no concept of them); they remain visible in the web
browser listings.
Verified against real clients: rclone (recursive PROPFIND listing and
reads) and the macOS WebDAV file system (mount_webdav): mount, ls, byte-
identical reads of multi-chunk files, write attempts correctly rejected
with EROFS.
Co-Authored-By: Claude Fable 5 <[email protected]>
New "borg webdav" command: it starts a read-only HTTP server on localhost,
so archive contents can be browsed and files downloaded with a web browser.
This first step implements plain HTTP only (GET/HEAD, HTML directory
listings, file downloads). The WebDAV methods (OPTIONS/PROPFIND), which
enable native mounting by Windows Explorer, macOS Finder and the Linux
desktop file managers, are planned to be added on top of this later.
Implementation notes:
- only uses the python standard library, no new dependencies.
- the directory tree of an archive is built in memory when it is first
accessed, so a big archive only costs time/memory if it is browsed.
- all repository access is serialized with a lock, because borgstore
connections are not thread-safe.
- a LockRefresher keeps the repository lock of an idle server alive,
so it is not killed as stale, see #9872.
- missing chunks abort the download connection, so damaged files are
never served as silently corrupted content.
- symlinks and special files are listed, but not downloadable.
Note: downloads via HTTP do not preserve POSIX metadata (owner, group,
mode, timestamps, xattrs, ACLs) - use borg extract or borg export-tar
for full-fidelity restores.
Co-Authored-By: Claude Fable 5 <[email protected]>
The multi-key feature (#9743) added the "borg key add", "borg key list"
and "borg key remove" subcommands but never wired up their docs:
- scripts/make.py: map key_add/key_list/key_remove to the "key" usage
group, so build_man can locate their examples (it previously aborted
with FileNotFoundError: docs/usage/key_add.rst).
- docs/usage/key.rst: include the three new generated snippets so they
show up on the HTML key page.
- regenerate the affected docs: new key_{add,list,remove}.rst.inc usage
snippets, new borg-key-{add,list,remove}.1 man pages, and borg-key.1
(SYNOPSIS + SEE ALSO now reference the new subcommands).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The authenticated and authenticated-blake3 modes do not encrypt data, but
they still have a real key (id/auth key material) stored as a key blob.
That blob can live as a keyfile or as a repokey just like the encrypted
modes, so make it configurable instead of always forcing repokey storage.
- AuthenticatedKeyBase: set LOCATION_CONFIGURABLE = True so --key-location
(at repo-create) and "borg key change-location" apply.
- key change-location: only copy sessionid/cipher when present (those are
AEAD-only; authenticated keys do not have them).
- repo-info: report the key storage location for authenticated keys too,
and handle the authenticated-blake3 variant (was only "authenticated").
- repo-create help: stop claiming authenticated* has no keyfile/repokey
storage; only "none" truly has no key.
- add change-location round-trip tests for authenticated mode.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
`setup.py` hardcoded crypto library paths for OpenBSD, causing build
issue when OpenBSD drops specific OpenSSL version. Solution is to make
paths configurable.
Addresses #8553.
Thomas Waldmann