Compare commits

..
Author SHA1 Message Date
beardthelion 49a9ade2cd docs(node): narrow the storage doc to its pinning charter, fix the Durability bullet
- opening now says what the page is: optional post-push pinning tiers in
  ipfs_pin.rs and pinata.rs, with one pointer to bare-repo storage so the
  page does not read as the answer to "where is my data?"
- the "Running both" Durability bullet now carries the warm-first caveat
  the dedup paragraph already documents: an object warm-pinned before
  Kubo was configured keeps only the warm copy
- README links the doc from the pin configuration row, where operators
  configuring pins actually look; the original staking-list link stays
2026-11-10 15:41:06 -06:00
beardthelion f2660e4413 Merge remote-tracking branch 'origin/main' into pr-363 2026-11-10 15:40:46 -06:00
Kevin Codex bc69549ff8 docs: name the hot tier as the unbounded side of the parked-list memory
Also record the two wording residuals from the same round: dedup isolation
is one-directional (warm row presence suppresses a later hot pin), and an
unset Pinata JWT disables uploads but not the tail's permit + walk cost.
2026-08-31 14:20:01 +08:00
Kevin Codex 7b640976c8 Merge remote-tracking branch 'origin/main' into HEAD 2026-08-31 14:19:32 +08:00
andreolf 8a577d3aa3 docs: address review — fix table/no-op/peer/budget claims, link from README 2026-08-18 12:39:11 +02:00
andreolf 706db57c65 docs: clarify pin de-duplication is best-effort, not atomic 2026-08-17 22:26:22 +02:00
andreolf b81f00e673 docs: document the two-tier storage and pinning model 2026-08-17 21:12:17 +02:00
2 changed files with 101 additions and 0 deletions

No files matched your search

+6
View File
@@ -412,6 +412,11 @@ Important node settings:
| `GITLAWB_MAX_CONCURRENT_GIT_PUSHES` | Max concurrent `git-receive-pack` POST operations, in a pool separate from the read pool. The anon receive-pack `info/refs` advertisement runs in a third pool of the same size, disjoint from both, so an advertisement flood cannot shed a push either. Two per-source push caps are derived from this value (`/8`, floor 1) and have no env var of their own. Over-cap sheds a 503 + Retry-After. Default 32. |
| `GITLAWB_MAX_CONCURRENT_READS_PER_CALLER` | Max concurrent read ops a single caller may hold, so one caller cannot monopolize the read pool. Keyed on the resolved source IP, never the DID, and only as granular as `GITLAWB_TRUSTED_PROXY`: left unset, a node behind an edge or NAT keys every caller on the edge IP and this collapses to one global cap. Default 16. |
| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | Max post-push pin loops (IPFS + Pinata) running concurrently across all repos. This caps how many loops RUN at once, not how much object-id list memory the node retains: on the local IPFS path a loop parked waiting for a permit still holds its full list. Do not size memory from this knob alone. A loop over cap waits, never drops a pin. Default 8. |
| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | Max post-push pin loops (IPFS + Pinata) running concurrently across all repos. This caps how many loops RUN at once, not how much object-id list memory the node retains: on the local IPFS path a loop parked waiting for a permit still holds its full list. Do not size memory from this knob alone. A loop over cap waits, never drops a pin. Default 8. |
For the two-tier pinning model (hot Kubo + warm Pinata), de-duplication
semantics, and parked-list memory behavior see
[`docs/STORAGE-AND-PINNING.md`](docs/STORAGE-AND-PINNING.md).
| `GITLAWB_REPO_LEASE_MAX_WAITERS` | Max pushes parked at once waiting for the same repo's write lease. Each waiter pins its buffered pack body, so this bounds that memory for a hot repo; past the cap the newest push sheds a 503 + Retry-After instead of queueing. Pushes to other repos are unaffected, and the lease holder is not counted. Default 8. |
| `GITLAWB_MAX_CONCURRENT_IPFS_WALKS` | Max concurrent `GET /ipfs/{cid}` visibility walks across all callers (own pool, disjoint from the served-git pools); over-cap sheds 503. Default 32. |
| `GITLAWB_IPFS_WALK_PER_SOURCE` | Max concurrent `/ipfs` walks a single source IP may hold. Default 4. |
@@ -478,6 +483,7 @@ Read:
- [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md)
- [`docs/ECONOMICS.md`](docs/ECONOMICS.md)
- [`docs/STORAGE-AND-PINNING.md`](docs/STORAGE-AND-PINNING.md)
Use a dedicated low-balance operator wallet. Do not use a treasury wallet as the heartbeat key.
+95
View File
@@ -0,0 +1,95 @@
# Storage and pinning
How a gitlawb node's **optional post-push pinning tiers** work. This documents
the behavior implemented in `crates/gitlawb-node/src/ipfs_pin.rs` and
`crates/gitlawb-node/src/pinata.rs`; it does not change any behavior. Primary
git storage for clone/fetch lives in bare repos under `GITLAWB_REPOS_DIR`
(with optional `GITLAWB_TIGRIS_BUCKET` backing), covered in the README
Configuration table.
## Two-tier model
After a push lands, new git objects are pinned to up to two independent sinks.
Both are **opt-in and independent** — a node runs fine with neither, either, or
both configured.
| Tier | Sink | Module | Enabled by | Purpose |
|------|------|--------|-----------|---------|
| Hot | Local Kubo (IPFS) | `ipfs_pin.rs` | `GITLAWB_IPFS_API` set | Node-local availability; the node is an HTTP client of a co-located Kubo daemon |
| Warm | Pinata (Filecoin-backed) | `pinata.rs` | `GITLAWB_PINATA_JWT` set | Off-node durability + public IPFS gateway reachability |
If a sink's config value is empty, its **pin** paths are no-ops — so leaving
`GITLAWB_PINATA_JWT` unset disables warm-tier *uploads*, though not the tail's
cost: the post-push Pinata tail still takes a global pin permit and runs the full
object-list re-derivation walk before the upload step returns empty, since its
gate is the announce/walk state, not the JWT. Note the no-op applies to pinning
only, not reads: the hot-tier read path (`ipfs_pin::cat`) returns an error rather
than a no-op when `GITLAWB_IPFS_API` is unset, so a node that serves the
encrypted-blob read endpoint needs Kubo configured.
## Configuration
| Env var | Default | Meaning |
|---------|---------|---------|
| `GITLAWB_IPFS_API` | `""` (disabled) | Base URL of the local Kubo HTTP API, e.g. `http://127.0.0.1:5001` |
| `GITLAWB_PINATA_JWT` | `""` (disabled) | Pinata bearer JWT enabling the warm tier |
| `GITLAWB_PINATA_UPLOAD_URL` | `https://uploads.pinata.cloud/v3/files` | Pinata v3 upload endpoint |
| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | `8` | Cap on concurrent post-push pin loops across all repos |
## How pinning runs
Pinning happens **after** a push is accepted, not on the push's critical path:
- The **hot** tier pins inline in the post-push encrypt/pin task.
- The **warm** (Pinata) tier runs in a spawned replication tail, so a slow or
unreachable Pinata never blocks the pusher.
Both tiers share a single global **pin admission semaphore**
(`max_concurrent_pin_tasks`). The pool **defers rather than sheds**: when it is
saturated, a pin loop waits for a slot instead of dropping the pin. Each batch is
bounded by `PIN_BATCH_BUDGET` (120s) so the pin batch itself cannot hold a slot
indefinitely. The Pinata tail re-derives its object list only *after* acquiring a
slot, which bounds *its* outstanding memory to O(refs) rather than
O(pushes × objects) — but that bound is the warm tail's alone. A **hot-tier** pin
loop takes an owned object list materialized by its caller *before* it waits for
a slot, so every hot loop parked on a saturated pool is holding its full list:
the hot side of the parked-list memory is O(pushes × objects), capped per repo by
`EncryptInflight` but not across repos. Do not size a node's memory from the
semaphore knob (README says the same).
Note the budget covers the pin batch, not the preceding object-list re-derivation
walk: that walk holds the slot too, and bounds only each child git process
individually (no aggregate deadline).
De-duplication is per sink and best-effort, backed by the single `pinned_cids`
table: the hot tier keys on its `cid`/`sha256_hex` rows and the warm tier on the
nullable `pinata_cid` column, so later pushes normally skip objects whose
successful pin is already recorded. The isolation is one-directional: a hot-only
pin does not suppress the warm tier (`pinata_cid` stays null), but the hot tier's
skip check is row *presence*, and a warm-tier pin inserts the row — so an object
warm-pinned while `GITLAWB_IPFS_API` was unset is skipped by the hot tier after
Kubo is configured, until something re-pins it. The check-upload-record sequence is not atomic,
so concurrent post-push tasks for the same object, or a failure to record after a
successful upload, can still cause a repeat upload attempt.
## Durability notes
- With only the **hot** tier, availability depends on the node (and any IPFS
peers that have fetched the CIDs). If the node is down and no peer holds the
objects, they are unreachable until it returns.
- The **warm** tier adds off-node durability via Pinata's Filecoin-backed
storage and makes objects reachable through the public IPFS gateway.
- Running **both** gives a node-local hot copy plus an off-node warm copy for
objects pinned while both sinks are enabled. An object warm-pinned before
Kubo was configured keeps only the warm copy (the dedup paragraph above)
unless something re-pins it.
> The two sinks are currently invoked as separate call paths. Unifying them
> behind a single pluggable backend interface (to add providers such as direct
> Filecoin deals or self-hosted clusters without touching push logic) is tracked
> separately.
## See also
- [RUN-A-NODE.md](RUN-A-NODE.md) — provisioning and running a node
- `crates/gitlawb-node/src/ipfs_pin.rs` — hot-tier implementation
- `crates/gitlawb-node/src/pinata.rs` — warm-tier implementation