Opening a repository loaded four store objects: config/readme (sanity check),
config/version, config/id and config/manifest (only read for its key type
byte since the manifest lost all other content). Now there is one plain text
INI object, config/config, e.g.:
# This is a Borg Backup repository.
# See https://borgbackup.readthedocs.io/
[repository]
version = 5
id = <64 hex digits>
encryption = aes256-ocb
id_hash = sha256
The crypto suite of the key is recorded by the --encryption / --id-hash
names, so key_factory() selects the key class from the config without
reading any repository object. Both names are recorded or none; a config
with one but not the other is invalid, an unsupported suite raises
InvalidRepositoryConfig. The repository version is 5; only version 5 is
accepted, there is no code to read the old layout and no migration (betas
are for new repositories only).
The manifest object is gone for borg 2 repositories: Manifest is now only
the in-memory container for key, repo_objs, repository and archives; its
write() and all callers, get_manifest()/put_manifest(), the manifest checks
and rebuild in "borg check", "borg debug dump-manifest", the key type
detection from stored objects (key_from_repository) and the ROBJ_MANIFEST
type are removed. borg 1.x repositories are still read via their manifest
(legacy_key_factory), as it holds their archives list. NoManifestError is
kept (never raised) so that rc 26 stays reserved.
Repository.create() creates all namespace directories in advance
(store.create_levels()) and writes the config (version, id, no key info)
right away, so a repository created via the Python API is an openable
key/value store. "borg repo-create" alone defers the config write
(create_config=False) until after the key was created, so nothing that
looks like a repository exists until repo-create succeeds: any failure or
interruption before that destroys the store (and the keyfile, in keyfile
mode), and a failure inside create() destroys the store as well.
A store without repository config is reported as not a valid repository
(InvalidRepository; DoesNotExist is only for a missing store backend, the
rest:// case from #10365). repo-create on it raises IncompleteRepository
(rc 11, "has no repository config"), as the store backend refuses any
non-empty directory. "borg repo-delete --force" destroys such a store, so
that ssh/rest/s3 users can remove it without other access to the storage,
but only if it has the packs, archives, index and config namespaces every
store borg created has: a data directory, a home directory or a borg 1.x
repository is never destroyed by borg. key_factory() on a config without
key info raises RepositoryKeyInfoMissing (rc 54).
The cache config's [cache]/[integrity] cross-check (manifest id) could
never detect anything and is dropped.
Docs: new "Repository config" section replaces the manifest section
(including how to recreate a lost config by hand, as check --repair can
not), the layout list, packs and security internals (where the protection
against a swapped crypto suite lives), repo-info example, error list and the
remaining manifest mentions are updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The document still described borg 1.x in many places. All claims were
re-verified against the code (and a live scratch repository):
- repository layout: borgstore namespaces (archives/ cache/ config/
index/ keys/ locks/ packs/) instead of the borg1 nested data/ tree;
document the index/ namespace and the cache/ objects.
- repo objects: the 49-byte header (magic, version, chunk id, meta and
data sizes) and storage inside packs; container details deferred to
the packs document.
- manifest: version 2, archives always empty, item_keys inside config.
- archives directory: entries keyed by archive ID with empty content;
no duplicate-name counters; soft-delete via the .del suffix.
- archive timestamps: time/start/end (time_end is borg1-only).
- item fields synced with item.pyx (target, bsdflags, size, inode,
ACL keys; legacy read-only keys noted).
- files cache: document the digests field and the actual location.
- argon2 salt is 128 bits, not 256.
- key envelope: repository_id refers to the config/id store object;
optional label field added.
- locking: drop the borg1 fslocking cache-lock section; storelocking
is the only locking borg2 does.
- cache config: actual keys + [integrity] section; a corrupt files
cache is discarded and rebuilt, not fatal.
- compaction: describe the current index-driven algorithm; drop the
"reappeared IDs" claim.
- id-hash modes: --id-hash for encrypted modes vs the fixed
authenticated-*/none-* modes.
- default chunker examples use fastcdc; fix the broken "cache" link.
- internals.rst: mention the rabin-aes/toeplitz-aes/goldilocks-aes
chunkers; compression is done by RepoObj, not the key object.
Co-Authored-By: Claude Fable 5 <[email protected]>
New docs/internals/chunker.rst documenting all supported chunkers twice:
- a plain-language section for users: what the chunker choice affects,
the chunk-size fingerprinting threat model, an at-a-glance comparison
table, and concrete recommendations per use case (borg 1.x dedup
compat, speed, untrusted repository storage, disk images);
- a technical section for readers with a crypto background: the FSWC
model, why direct-decision keyed rolling hashes are broken (key
recovery from observed boundaries), the UHF-then-PRF construction and
its epsilon-AU requirement, the three universal hashes with their
collision bounds (rabin-aes ~2^-55 probabilistic, goldilocks-aes
~2^-58, toeplitz-aes exactly 2^-64 unconditional), why the fixed
64-byte window is essential (including the rotation/cancellation
algebra that also explains the classic 4095-vs-4096 buzhash window),
rejected alternatives, measured performance, proof-bound limits and
the wider-digest upgrade path, plus references.
Co-Authored-By: Claude Fable 5 <[email protected]>
fastcdc is ~1.3x faster than buzhash/buzhash64 at the same deduplication
and (with normalized chunking) a tighter chunk size distribution.
It is also the better choice security-wise: its Gear table is derived
from secret key material, while the "buzhash" chunker only XORs a 32bit
seed into its table - so chunk cut points are much harder to predict
without the key (resistance against chunk-size fingerprinting attacks).
Both places using a chunker are switched:
- CHUNKER_PARAMS (file content data): fastcdc,19,23,21,2
- ITEMS_CHUNKER_PARAMS (item metadata stream): fastcdc,15,19,17,2
The metadata stream chunker was still using the 32bit seeded buzhash, so
this also closes that gap.
Also rename CHUNKER64_PARAMS -> BUZHASH64_PARAMS and add BUZHASH_PARAMS,
so the per-algorithm defaults are named consistently and CHUNKER_PARAMS
unambiguously means "the default chunker".
This is intentionally done during the beta phase so it gets plenty of
practical testing before borg2 is released for production.
Rename chunks cache / cached chunks index to chunks index or index, renaming parameters/variables where appropriate to avoid 'cache' terminology for index.
The former section is a bit older (Nov 2016) and has been the piece
responsible for finding CVE-2016-10099, since while writing it I
wondered how the manifest was authenticated to actually
*be* the manifest. Well. There it is ;)
It has been edited to final form only recently and should now be ready
for review.
The latter section is new.
one of the biggest issues with borg < 1.0 was that it had a default target chunk
size of 64kiB, thus it created a lot of chunks, a huge chunk management overhead
(high RAM and disk usage).
the items metadata stream is usually not that big (compared to the file content data) -
it is just file and dir names and other metadata.
if we use too rough granularity there (and big minimum chunk size), we usually will get no deduplication.
use borg instead attic except at the places where it was used:
- as toplevel package name, directory name, file name
- to refer to original attic
remove sphinx upload make command, will be replaced by github.io site later
remove references to binary downloads and linux packages for now
remove some software name references, fix grammar
use borgbackup rather than borg-backup (or borg) in URLs,
less name collision issues, better search results, no validity issues with "-"
