Merge remote-tracking branch 'origin/main' into pr-363

- README: the duplicate GITLAWB_MAX_CONCURRENT_PIN_TASKS row and the
  free-standing paragraph it introduced broke the table; the single row
  now carries the STORAGE-AND-PINNING cross-link inline
- dedup: the hot tier skips on has_ipfs_cid (local_ipfs_provenance =
  TRUE), not row presence; a Pinata-only row is local_ipfs_provenance =
  FALSE, so it enters the hot path on later pushes once Kubo is
  configured instead of being permanently suppressed
- the "Running both" durability bullet now agrees: enabling Kubo
  backfills local copies for warm-only objects

PROTOCOL.md

@@ -1,106 +0,0 @@
-# Gitlawb Wire Protocol Specification (v1, alpha)
-
-This document describes the wire behavior a second implementation can interop
-with **today**. Anything the node does not ship yet is explicitly marked
-*planned*; nothing below is aspirational unless it says so.
-
-## 1. Identity & DIDs
-
-- **Key type:** Ed25519.
-- **Method (alpha):** `did:key` only. Resolution is deterministic public-key
-  extraction from the multicodec prefix (`z6Mk…`, multicodec `0xed01`).
-- Signed requests presenting any other DID method are rejected; the node's
-  auth layer answers with a hint that only `did:key` is supported in alpha.
-- *Planned:* `did:web` and `did:gitlawb` resolution. Clients MUST NOT assume
-  either is accepted anywhere yet.
-
-## 2. Authentication (RFC 9421)
-
-- **Transport:** signed requests are expected over TLS; the node does not
-  terminate HTTP signatures over cleartext.
-- **HTTP signatures:** `Signature-Input` covers `@method`, `@path`, and
-  `content-digest`:
-  `sig1=("@method" "@path" "content-digest");keyid="did:key:…";alg="ed25519";created=<unix>`
-  All three components are mandatory: a `Signature-Input` missing one is
-  rejected as `incomplete_signature`.
-- **`@path` value:** the node binds `@path` to the path *and* query string of
-  the request, not to the path-only value RFC 9421 defines for `@path`. A
-  second implementation must sign `GET /r/info/refs?service=git-upload-pack`
-  including the `?service=…` part.
-- **Signer identity:** `keyid` is the actor DID (`did:key` only, see §1).
-- **Integrity:** `Content-Digest` is SHA-256 over the request body and is
-  verified when the header is present. Because `content-digest` is a covered
-  component, every signed request signs a digest value: on a body-less
-  request the signed value is the SHA-256 of the empty body, which the node
-  also accepts when the header is absent entirely.
-
-## 3. Proof of Intelligence (iCaptcha)
-
-- **Mode:** controlled by `ICAPTCHA_MODE` = `off` | `shadow` | `enforce`,
-  **default `off`**. A node with the gate off never demands proof.
-- **Scope:** the gate applies to a small set of write endpoints (agent
-  registration and repo create/fork), not to all writes.
-- **Wire shape:** the client-supplied header is `x-icaptcha-proof` — that is
-  the only iCaptcha header a node reads from a request. When proof is
-  required and missing/invalid, the node answers
-  `403 icaptcha_proof_required` and sets `x-icaptcha-url` and
-  `x-icaptcha-level` **on the response** (also mirrored as JSON fields) so
-  the client can go solve a challenge and retry.
-
-## 4. Ref-Update Certificate
-
-*Planned:* the `gitlawb/ref-update/v1` schema below is defined in code, but no
-shipped endpoint serves or verifies these certificates today; nothing in
-sections 2-3 produces one on the wire. It is documented here so a second
-implementation shares the type vocabulary ahead of the wire surface.
-
-The hash fields reserve 64 hex characters for the SHA-256 object format; the
-repositories the node serves are sha1 with 40-hex OIDs (see §5), so the field
-width is a forward reservation, not what a current push produces.
-
-Two distinct certificate-shaped payloads exist; this section documents the
-**core signing schema**, which is what `gitlawb/ref-update/v1` names:
-
-- **Type string:** `gitlawb/ref-update/v1`.
-- **Body (canonical JSON, the signed bytes):** `type`, `repo` (the
-  repository DID), `ref_name`, `from` (previous hash, 64 hex chars,
-  all-zeros for a new ref), `to` (target hash, 64 hex chars), `seq`
-  (monotonically increasing, replay prevention), `timestamp` (RFC 3339),
-  `nonce`.
-- **Signatures:** a list of `{signer: did:key, sig: base64url-unpadded
-  Ed25519}` entries appended outside the signed body; thresholds are counted
-  over distinct signers that verify.
-
-The node's REST API separately serves **push receipts** — a different,
-node-issued JSON payload attesting that a push was processed. Receipts are
-not `gitlawb/ref-update/v1` documents and are not interchangeable with them;
-they are documented with the node's HTTP API, not here.
-
-## 5. Storage & Git Transport
-
-- **Object format:** bare repositories are created with
-  `--object-format=sha1`; production OIDs are 40 hex chars. *Planned:*
-  SHA-256 object format.
-- **Smart-HTTP endpoints:**
-  - `GET /{owner}/{repo}/info/refs?service=git-upload-pack` — fetch
-    advertisement.
-  - `GET /{owner}/{repo}/info/refs?service=git-receive-pack` — push
-    advertisement.
-    The `?service=` parameter is mandatory and takes exactly those two
-    values; a missing parameter is a `400`, not an omitted-service default.
-  - `POST /{owner}/{repo}/git-upload-pack` — clone/fetch. Request and
-    response bodies are pkt-line framed; the result content type is
-    `application/x-git-upload-pack-result`.
-  - `POST /{owner}/{repo}/git-receive-pack` — push. **Authentication is
-    mandatory**: an unsigned push is refused with `401`. Same pkt-line
-    framing; the result content type is
-    `application/x-git-receive-pack-result`.
-    Signed requests on this path follow §2, including TLS transport.
-- **Fetch vs push auth split:** fetch-side requests against a withheld or
-  private repo answer `404` (existence is not confirmed to unauthorized
-  callers), while the push path answers `401` for missing/invalid
-  signatures. A second implementation must not treat the two symmetrically.
-- **Content addressing:** the node records IPFS CID metadata per branch and
-  serves it over REST (the repo `refs` endpoint maps branch heads to CIDs).
-  *Planned:* IPNS-backed branch pointers; IPNS is not part of the shipped
-  wire contract.

README.md

@@ -411,7 +411,7 @@ Important node settings:
 | `GITLAWB_MAX_CONCURRENT_GIT_OPS` | Max concurrent served git READ ops (upload-pack and its `info/refs` advertisement) across all callers; over-cap sheds a 503 + Retry-After. Anonymous reads draw from this pool, so pair it with `GITLAWB_MAX_CONCURRENT_READS_PER_CALLER`. Pushes and the receive-pack advertisement have their own pools, so a read flood cannot shed an authenticated push. Default 128. |
 | `GITLAWB_MAX_CONCURRENT_GIT_PUSHES` | Max concurrent `git-receive-pack` POST operations, in a pool separate from the read pool. The anon receive-pack `info/refs` advertisement runs in a third pool of the same size, disjoint from both, so an advertisement flood cannot shed a push either. Two per-source push caps are derived from this value (`/8`, floor 1) and have no env var of their own. Over-cap sheds a 503 + Retry-After. Default 32. |
 | `GITLAWB_MAX_CONCURRENT_READS_PER_CALLER` | Max concurrent read ops a single caller may hold, so one caller cannot monopolize the read pool. Keyed on the resolved source IP, never the DID, and only as granular as `GITLAWB_TRUSTED_PROXY`: left unset, a node behind an edge or NAT keys every caller on the edge IP and this collapses to one global cap. Default 16. |
-| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | Max post-push pin loops (IPFS + Pinata) running concurrently across all repos. This caps how many loops RUN at once, not how much object-id list memory the node retains: on the local IPFS path a loop parked waiting for a permit still holds its full list. Do not size memory from this knob alone. A loop over cap waits, never drops a pin. Default 8. |
+| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | Max post-push pin loops (IPFS + Pinata) running concurrently across all repos. This caps how many loops RUN at once, not how much object-id list memory the node retains: on the local IPFS path a loop parked waiting for a permit still holds its full list. Do not size memory from this knob alone. A loop over cap waits, never drops a pin. See [docs/STORAGE-AND-PINNING.md](docs/STORAGE-AND-PINNING.md) for the two-tier model, dedup, and parked-list memory. Default 8. |
 | `GITLAWB_REPO_LEASE_MAX_WAITERS` | Max pushes parked at once waiting for the same repo's write lease. Each waiter pins its buffered pack body, so this bounds that memory for a hot repo; past the cap the newest push sheds a 503 + Retry-After instead of queueing. Pushes to other repos are unaffected, and the lease holder is not counted. Default 8. |
 | `GITLAWB_MAX_CONCURRENT_IPFS_WALKS` | Max concurrent `GET /ipfs/{cid}` visibility walks across all callers (own pool, disjoint from the served-git pools); over-cap sheds 503. Default 32. |
 | `GITLAWB_IPFS_WALK_PER_SOURCE` | Max concurrent `/ipfs` walks a single source IP may hold. Default 4. |
@@ -478,6 +478,7 @@ Read:
 
 - [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md)
 - [`docs/ECONOMICS.md`](docs/ECONOMICS.md)
+- [`docs/STORAGE-AND-PINNING.md`](docs/STORAGE-AND-PINNING.md)
 
 Use a dedicated low-balance operator wallet. Do not use a treasury wallet as the heartbeat key.
 

docs/STORAGE-AND-PINNING.md

@@ -0,0 +1,100 @@
+# Storage and pinning
+
+How a gitlawb node's **optional post-push pinning tiers** work. This documents
+the behavior implemented in `crates/gitlawb-node/src/ipfs_pin.rs` and
+`crates/gitlawb-node/src/pinata.rs`; it does not change any behavior. Primary
+git storage for clone/fetch lives in bare repos under `GITLAWB_REPOS_DIR`
+(with optional `GITLAWB_TIGRIS_BUCKET` backing), covered in the README
+Configuration table.
+
+## Two-tier model
+
+After a push lands, new git objects are pinned to up to two independent sinks.
+Both are **opt-in and independent** — a node runs fine with neither, either, or
+both configured.
+
+| Tier | Sink | Module | Enabled by | Purpose |
+|------|------|--------|-----------|---------|
+| Hot  | Local Kubo (IPFS) | `ipfs_pin.rs` | `GITLAWB_IPFS_API` set | Node-local availability; the node is an HTTP client of a co-located Kubo daemon |
+| Warm | Pinata (Filecoin-backed) | `pinata.rs` | `GITLAWB_PINATA_JWT` set | Off-node durability + public IPFS gateway reachability |
+
+If a sink's config value is empty, its **pin** paths are no-ops — so leaving
+`GITLAWB_PINATA_JWT` unset disables warm-tier *uploads*, though not the tail's
+cost: the post-push Pinata tail still takes a global pin permit and runs the full
+object-list re-derivation walk before the upload step returns empty, since its
+gate is the announce/walk state, not the JWT. Note the no-op applies to pinning
+only, not reads: the hot-tier read path (`ipfs_pin::cat`) returns an error rather
+than a no-op when `GITLAWB_IPFS_API` is unset, so a node that serves the
+encrypted-blob read endpoint needs Kubo configured.
+
+## Configuration
+
+| Env var | Default | Meaning |
+|---------|---------|---------|
+| `GITLAWB_IPFS_API` | `""` (disabled) | Base URL of the local Kubo HTTP API, e.g. `http://127.0.0.1:5001` |
+| `GITLAWB_PINATA_JWT` | `""` (disabled) | Pinata bearer JWT enabling the warm tier |
+| `GITLAWB_PINATA_UPLOAD_URL` | `https://uploads.pinata.cloud/v3/files` | Pinata v3 upload endpoint |
+| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | `8` | Cap on concurrent post-push pin loops across all repos |
+
+## How pinning runs
+
+Pinning happens **after** a push is accepted, not on the push's critical path:
+
+- The **hot** tier pins inline in the post-push encrypt/pin task.
+- The **warm** (Pinata) tier runs in a spawned replication tail, so a slow or
+  unreachable Pinata never blocks the pusher.
+
+Both tiers share a single global **pin admission semaphore**
+(`max_concurrent_pin_tasks`). The pool **defers rather than sheds**: when it is
+saturated, a pin loop waits for a slot instead of dropping the pin. Each batch is
+bounded by `PIN_BATCH_BUDGET` (120s) so the pin batch itself cannot hold a slot
+indefinitely. The Pinata tail re-derives its object list only *after* acquiring a
+slot, which bounds *its* outstanding memory to O(refs) rather than
+O(pushes × objects) — but that bound is the warm tail's alone. A **hot-tier** pin
+loop takes an owned object list materialized by its caller *before* it waits for
+a slot, so every hot loop parked on a saturated pool is holding its full list:
+the hot side of the parked-list memory is O(pushes × objects), capped per repo by
+`EncryptInflight` but not across repos. Do not size a node's memory from the
+semaphore knob (README says the same).
+Note the budget covers the pin batch, not the preceding object-list re-derivation
+walk: that walk holds the slot too, and bounds only each child git process
+individually (no aggregate deadline).
+
+De-duplication is per sink and best-effort, backed by the single `pinned_cids`
+table: the hot tier skips an object when `has_ipfs_cid` reports true, which is
+`local_ipfs_provenance = TRUE` — a row that records a real local Kubo CID — and
+the warm tier on the nullable `pinata_cid` column, so later pushes normally skip
+objects whose successful pin is already recorded for that sink. The isolation is
+one-directional: a hot-only pin does not suppress the warm tier (`pinata_cid`
+stays null), and a warm-tier pin does not suppress the hot tier either: a row
+with only `pinata_cid` set has `local_ipfs_provenance = FALSE`, so an object
+warm-pinned while `GITLAWB_IPFS_API` was unset still enters the hot upload path
+on later pushes once Kubo is configured, subject to the normal budgets and
+failure handling (enabling Kubo can backfill local copies rather than leaving
+the object warm-only). The check-upload-record sequence is not atomic,
+so concurrent post-push tasks for the same object, or a failure to record after a
+successful upload, can still cause a repeat upload attempt.
+
+## Durability notes
+
+- With only the **hot** tier, availability depends on the node (and any IPFS
+  peers that have fetched the CIDs). If the node is down and no peer holds the
+  objects, they are unreachable until it returns.
+- The **warm** tier adds off-node durability via Pinata's Filecoin-backed
+  storage and makes objects reachable through the public IPFS gateway.
+- Running **both** gives a node-local hot copy plus an off-node warm copy for
+  objects pinned while both sinks are enabled. An object warm-pinned before
+  Kubo was configured still enters the hot path on later pushes (the dedup
+  paragraph above), so enabling Kubo can backfill local copies instead of
+  leaving it warm-only.
+
+> The two sinks are currently invoked as separate call paths. Unifying them
+> behind a single pluggable backend interface (to add providers such as direct
+> Filecoin deals or self-hosted clusters without touching push logic) is tracked
+> separately.
+
+## See also
+
+- [RUN-A-NODE.md](RUN-A-NODE.md) — provisioning and running a node
+- `crates/gitlawb-node/src/ipfs_pin.rs` — hot-tier implementation
+- `crates/gitlawb-node/src/pinata.rs` — warm-tier implementation