mirror of
https://github.com/borgbackup/borg.git
synced 2026-10-12 02:11:36 +00:00
Thomas WaldmannandClaude Fable 5.1
1110bcddba
Opening a repository loaded four store objects: config/readme (sanity check),
config/version, config/id and config/manifest (only read for its key type
byte since the manifest lost all other content). Now there is one plain text
INI object, config/config, e.g.:
# This is a Borg Backup repository.
# See https://borgbackup.readthedocs.io/
[repository]
version = 5
id = <64 hex digits>
encryption = aes256-ocb
id_hash = sha256
The crypto suite of the key is recorded by the --encryption / --id-hash
names, so key_factory() selects the key class from the config without
reading any repository object. Both names are recorded or none; a config
with one but not the other is invalid, an unsupported suite raises
InvalidRepositoryConfig. The repository version is 5; only version 5 is
accepted, there is no code to read the old layout and no migration (betas
are for new repositories only).
The manifest object is gone for borg 2 repositories: Manifest is now only
the in-memory container for key, repo_objs, repository and archives; its
write() and all callers, get_manifest()/put_manifest(), the manifest checks
and rebuild in "borg check", "borg debug dump-manifest", the key type
detection from stored objects (key_from_repository) and the ROBJ_MANIFEST
type are removed. borg 1.x repositories are still read via their manifest
(legacy_key_factory), as it holds their archives list. NoManifestError is
kept (never raised) so that rc 26 stays reserved.
Repository.create() creates all namespace directories in advance
(store.create_levels()) and writes the config (version, id, no key info)
right away, so a repository created via the Python API is an openable
key/value store. "borg repo-create" alone defers the config write
(create_config=False) until after the key was created, so nothing that
looks like a repository exists until repo-create succeeds: any failure or
interruption before that destroys the store (and the keyfile, in keyfile
mode), and a failure inside create() destroys the store as well.
A store without repository config is reported as not a valid repository
(InvalidRepository; DoesNotExist is only for a missing store backend, the
rest:// case from #10365). repo-create on it raises IncompleteRepository
(rc 11, "has no repository config"), as the store backend refuses any
non-empty directory. "borg repo-delete --force" destroys such a store, so
that ssh/rest/s3 users can remove it without other access to the storage,
but only if it has the packs, archives, index and config namespaces every
store borg created has: a data directory, a home directory or a borg 1.x
repository is never destroyed by borg. key_factory() on a config without
key info raises RepositoryKeyInfoMissing (rc 54).
The cache config's [cache]/[integrity] cross-check (manifest id) could
never detect anything and is dropped.
Docs: new "Repository config" section replaces the manifest section
(including how to recreate a lost config by hand, as check --repair can
not), the layout list, packs and security internals (where the protection
against a swapped crypto suite lives), repo-info example, error list and the
remaining manifest mentions are updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
53 lines
2.0 KiB
ReStructuredText
53 lines
2.0 KiB
ReStructuredText
.. include:: global.rst.inc
|
|
.. _internals:
|
|
|
|
Internals
|
|
=========
|
|
|
|
The internals chapter describes and analyzes most of the inner workings
|
|
of Borg.
|
|
|
|
Borg uses a low-level, key-value store, the :ref:`repository`, and
|
|
implements a more complex data structure on top of it, which is made
|
|
up of the :ref:`archives <archive>`, :ref:`items <item>` and data
|
|
:ref:`chunks`.
|
|
|
|
Each repository can hold multiple :ref:`archives <archive>`, which
|
|
represent individual backups that contain a full archive of the files
|
|
specified when the backup was performed.
|
|
|
|
Deduplication is performed globally across all data in the repository
|
|
(multiple backups and even multiple hosts), both on data and file
|
|
metadata, using :ref:`chunks` created by the chunker using a
|
|
content-defined chunking algorithm - the Gear rolling hash of FastCDC_
|
|
("fastcdc" chunker, the default), Buzhash_ ("buzhash" and "buzhash64"
|
|
chunker) or a universal hash followed by an AES pseudo-random function
|
|
("rabin-aes", "toeplitz-aes" and "goldilocks-aes" chunker) - or a simpler
|
|
fixed block size algorithm ("fixed" chunker).
|
|
|
|
To perform the repository-wide deduplication, a hash of each
|
|
chunk is checked against the :ref:`chunks index <index>`, which is a
|
|
hash table of all chunks that already exist.
|
|
|
|
.. figure:: internals/structure.png
|
|
:figwidth: 100%
|
|
:width: 100%
|
|
|
|
Layers in Borg. At the very top, commands are implemented, using
|
|
a data access layer provided by the Archive and Item classes.
|
|
Below that, the RepoObj class compresses the data (using a Compressor,
|
|
see class RepoObj in ``repoobj.py``) and then hands it to the "key"
|
|
object, which provides the authenticated encryption. The "key" object
|
|
represents the sole trust boundary in Borg.
|
|
The lowest layer is the repository accessed via class Repository.
|
|
Repository uses ``borgstore`` internally.
|
|
|
|
.. toctree::
|
|
:caption: Internals contents
|
|
|
|
internals/security
|
|
internals/data-structures
|
|
internals/chunker
|
|
internals/packs
|
|
internals/frontends
|