Compare commits

..
Author SHA1 Message Date
beardthelion 3a82ccb8a1 fix(node): satisfy clippy on stale-local download test path join 2026-08-30 16:31:25 -05:00
beardthelion c306933e07 style(node): rustfmt advisory-lock gap-closure commits 2026-08-30 16:26:21 -05:00
beardthelion 22da2d6b96 fix(node): refuse stale under-lock refresh and gate tail durability on release
Under-lock GET failures now always shed as RepoUnavailable, even when a local
copy exists, so a transient download error cannot publish over a newer stored
generation. PublishDurabilitySlot only synthesizes UploadUnknowable after
release starts, and receive-pack marks that boundary before spawning the tail.

Adds regression tests for both paths; inv22 F4 gate accepts the release wrapper.
2026-08-30 16:14:25 -05:00
beardthelion 5cd6703aad fix(node): compensate under the write lock and harden tail durability handoff
Run create_issue rollback through release_compensating so issue refs are
deleted only while the advisory lock is still held and only on definite
publish refusals, not UploadUnknowable. Replace the tail durability slot's
async try_lock drop with a std mutex so cancellation during polling cannot
skip installing UploadUnknowable.
2026-08-30 15:30:34 -05:00
beardthelion 85c6f48bc4 fix(node): verify adversarial findings on advisory-lock PR
Fail closed when publish durability never records, bound pg_advisory_unlock
with the same transfer budget as upload, and cap close_issue pre-lock snapshots
at git_acquire_timeout_secs instead of the write-lock transfer bound.
2026-08-30 13:25:06 -05:00
beardthelion 9d02139dad fix(node): close three jatmn findings on advisory-lock PR
Retry fork create confirmation before compensating or returning, schedule
background recovery when the lookup stays unavailable, install publish
durability outcomes before the recorded flag, and size the lock pool for
non-push mutations (default 40).
2026-08-30 13:04:54 -05:00
beardthelion 487687c5fd fix(node): add ValidatedRepoDiskPath for CodeQL and reconcile tail durability
Introduce a validated-path newtype with the same inline join and component
walk main uses, so path-injection sinks only accept sanitised repo paths.
Reconcile the publish-durability gate with the disconnect-safe tail spawn via
PublishDurabilitySlot and explicit UploadUnknowable handling.
2026-08-29 15:29:01 -05:00
beardthelion 2ad2f12860 fix(node): gate replication tail on publish durability and split close_issue limits
Wait for a confirmed release outcome before post_receive_replication_tail runs
any walks or coalescing work. Give close_issue its own per-IP rate bucket so
it cannot drain receive-pack quota. Add regression tests for both paths.
2026-08-29 12:29:36 -05:00
beardthelion 5d3a4ea462 fix(node): close write-attempt ownership gaps on advisory-lock release
Make publish swaps claim-exclusive via CAS, invalidate the local read cache
on definite publish refusals, harden fork create compensation and clone
cleanup, and keep snapshot temp dirs owned through the async handoff.
2026-08-29 12:01:06 -05:00
beardthelion 1911291533 fix(node): harden issue close/create and fork publish paths on #285
Move close_issue rate limiting after the read gate so denied readers see 404
not 429, roll back local issue refs when publish refuses, re-check swap
authority under publish_lock, propagate fork upload failures before DB insert,
and update lock-pool config help text.
2026-08-28 22:53:17 -05:00
beardthelion 8e66e77c57 fix(node): close jatmn round on publish authority and durability gating
Revoke under-lock refresh swaps when the advisory lock ends, fail closed on
unrecorded release outcomes before Pinata/gossip, compensate fork archives when
create_repo fails, recompute acquire backoff from the live deadline, and route
publish swaps through validated_repo_disk_path for CodeQL.
2026-08-28 22:10:38 -05:00
beardthelion d1ce2e3291 fix(node): harden release-upload outcomes and replication tail gating
Treat timed-out or failed archive uploads as non-durable in into_result,
gate pinata/gossip on confirmed publish durability while keeping inv22 spawn
order, split lock-pool vs main DB pool validation, map LockPoolBusy through
From<anyhow::Error>, and attach Retry-After on repo write contention errors.
2026-08-28 16:02:06 -05:00
beardthelion 4eb0c4479c fix(node): route fork disk paths through validated_repo_disk_path
CodeQL flagged remove_dir_all on a user-derived path in fork_repo. Use the
same three-layer barrier as RepoStore::local_path and reject names the
validator refuses before clone.
2026-08-28 15:17:52 -05:00
beardthelion 4f1c1bbd09 fix(node): drop unused logcap helper after drain_faults seam 2026-08-28 14:45:25 -05:00
beardthelion 98fe670c11 fix(node): carry transient acquire failures through every write handler
Extend acquire_write_app_error so RepoBusy and RepoUnavailable map to
fixed-body 503s on issue and pull paths, matching receive-pack. Fix the
flaky u2 drain exhaustion test via a drain_faults seam and add handler
regressions for contention and storage unavailability on create_issue.
2026-08-28 13:56:42 -05:00
beardthelion 3df71c9aef fix: unblock pre-push after main merge
Read CappedBody.text in gl whoami (#381 shape), restore legacy CID sweep
wiring dropped in the merge, route rate-limiter cleanup through AppState,
and allow build_lock_pool for test-only callers.
2026-08-25 23:29:00 -05:00
beardthelion 7ba0fefd75 fix(node): mark close_issue mirror visibility gate for push hook
Annotate list_visibility_rules with mirror-rows-handled so the pre-push
surface detector accepts the synced-row read gate.
2026-08-25 23:09:33 -05:00
beardthelion 3c201486cc fix(node): cap close_issue snapshot work on the read pool
Take a git_read_semaphore permit before the non-owner author snapshot so
parallel close attempts cannot each drive a full archive extraction while still
under the hourly rate bucket. Format owner-DID test helpers.
2026-08-25 23:03:07 -05:00
beardthelion b35cb31944 fix(node): address jatmn review on merged advisory-lock head
Align .env.example db pool with validate floor, move snapshot cleanup into
the blocking extract task, remove fork clones on PreconditionLost, restore
upload_site_reached test plumbing, and use owner DIDs in receive-pack tests
after #330 owner-push default.
2026-08-25 22:30:07 -05:00
beardthelion 4cee7e0935 merge origin/main into fix/279-advisory-lock-session-affinity
Reconcile the advisory-lock series with main after #173, #330, and related
landings. Conflict resolution in repo_store, repos, issues, pulls, ipfs, and
main.
2026-08-25 22:30:00 -05:00
Kevin Codex 73fd7473af Merge pull request #330 from Gitlawb/fix/default-enforce-owner-push
fix(node)!: enforce owner-only push by default
2026-08-25 22:06:12 +08:00
Kevin Codex 21c74c65d9 Merge pull request #223 from Gravirei/fix/issue-220-whoami-registered-false-trust-lookup
fix(gl): only report registered: false on 404 in whoami (#220)
2026-08-25 22:04:56 +08:00
Kevin Codex 21f8cb6135 Merge pull request #173 from Gitlawb/fix/issue-135-ipfs-cid-tree-gate
fix(node): gate GET /ipfs/{cid} tree objects so a withheld subtree's structure can't leak (#135)
2026-08-25 22:04:14 +08:00
Gravirei 861cced8f8 chore: update h2 0.4.13 → 0.4.18 to resolve RUSTSEC-2026-0258 2026-08-24 15:10:32 +06:00
Gravirei 39ce655904 fix(gl): sanitize full transport error chain in whoami, remove raw error attachment (#220) 2026-08-23 22:12:19 +06:00
Gravirei d25cd0c6e6 merge: resolve sync.rs conflict — move read_body_capped/sanitize_node_msg to http.rs (#220) 2026-08-23 22:07:46 +06:00
beardthelion 9dd71e999c fix(node): record why the close-issue read gate cannot carry a synced row
The gate added for the pre-lock snapshot reads a repo row's own visibility
rules and public flag. A row synced from a peer is stored public and carries
none of the owner's rules, so for that class the gate can only return allow.

Refusing instead would deny the repo's real owner and the issue's real author
on any node whose only copy is a synced one, and it would not buy the
protection it appears to, because a synced row's recorded owner comes from the
peer that sent it. Every other read gate in the API reaches the same verdict
for such a row, so this is left as is and stated at the call site rather than
special-cased here.

The test that covered the gate seeds a locally created repo and cannot observe
this. Adds one that pins the property directly: the gate allows an arbitrary
caller on a synced row, and the handler refuses that caller anyway, because the
owner-or-author check is what decides this route.
2026-08-18 00:43:06 -05:00
beardthelion 9f35dca002 fix(node): address the 2026-08-10 review round on the advisory-lock series
- P1: serve the receive-pack advertisement from a non-mutating snapshot
  instead of acquire_fresh, so the unlocked advertisement cannot delete or
  swap the live repo directory under a concurrent guarded write (the stale-ETag
  success ordering). acquire_fresh loses its only production caller and is
  removed.
- P1: read-gate, rate-limit, bound, and cancellation-clean the close_issue
  pre-lock snapshot, so a signed non-owner cannot drive unbounded Tigris
  downloads and blocking extractions, and an abandoned extraction no longer
  leaks its temp dir.
- P1: classify raw 409/412 responses as a lost conditional write via
  SdkError::raw_response() in both upload and the publish supersede-retry, so
  an unparsable error body cannot acknowledge a write that was never published.
- P2: wrap the cold-cache under-lock download failure in RepoUnavailable so it
  sheds as a retryable 503 like the HEAD arm, not a permanent 500.
- P2: surface a refused create-only fork upload as PreconditionLost and refuse
  the fork, so an orphan archive cannot shadow a fork's DB record.

Refs #279
2026-08-18 00:04:50 -05:00
beardthelion 2a92a1f73a fix(node): reconcile F-series tests and test seams with the rebased guard shape
The rebase onto main carried main's #174 F-series guard tests forward. The
branch's guard replaces main's locked/released/test_pre_unlock_gate Drop
mechanics with close-on-drop (runtime) and leak (off-runtime), so two tests
that asserted the replaced mechanics are reconciled: the off-runtime disposal
test now asserts the leak observable (the slot never returns to idle) and the
detached-unlock-returns-connection test is dropped, its invariant covered by
the U-series drop-frees-the-lock gates. The pre-unlock gate seam is restored
(test-only) so the mid-unlock cancellation tests keep their deterministic
park. ipfs.rs tests are updated to the branch's sync test-client constructor
and 4-arg RepoStore::new.

Refs #279
2026-08-17 23:23:32 -05:00
beardthelion 3d803ff217 test(node): drive the abandoned-upload race end to end, and probe the real backend
The headline test is the one that had to exist. Writer A's release is parked past
its transfer bound and returns with the outcome unknowable, B acquires and
publishes, and A's captured PUT is then replayed: the store answers 412 and B's
archive survives. The create-only variant covers the arm whose real-world failure
is silent rather than loud.

The control is what makes those attributable. With no interleaved B, an
abandoned-then-replayed PUT whose generation still matches lands. Without it the
headline would only show that replays get rejected, not that staleness is what
rejects them.

Header assertions come last in all three, so a lost fence reds on the outcome it
is about rather than on a wire-format check.

A mock cannot prove Tigris honors any of this, and the vendor requires a
Single-region or Multi-region bucket for conditional operations, so against a
Global or Dual-region bucket the fence is a silent no-op. The credentials-gated
probe checks both arms against the real endpoint and cleans up unconditionally,
including when an assertion fails, which is the case it exists to catch. It
accepts 412 or 409 on the create-only arm because both mean the precondition was
enforced and both are already classified as a loss.
2026-08-17 23:00:06 -05:00
beardthelion 317841ee24 fix(node): fence the release publish so an abandoned upload cannot win
acquire_write now reads the archive's ETag under the advisory lock and the guard
carries it, so release publishes conditionally on the generation it actually
refreshed from. An upload abandoned by an earlier writer no longer overwrites a
successor's archive: the store rejects it, because the ETag it was written
against is gone.

A refused precondition gets exactly one supersede-retry, never a loop. The
distinction that makes this sound is that a 412 is a definite answer, unlike the
timeout arm where nothing is knowable, and the retrying writer still holds the
lock, so whatever landed underneath was written without one and its tree is not
the authority. Two losses in a row refuse instead of escalating.

That retry is what keeps ordinary pushes working now that init publishes
create-only: a first push racing init's upload of the empty repo loses once and
then wins, rather than surfacing a 503 on the most common operation there is.

release returns a must-use outcome and the four publishing handlers propagate it
before any trust bump, webhook, or success body, so a publish the store refused
reads as a retryable 503 instead of a 201. The three release(false) sites
deliberately do not map it: they publish nothing, and a 503 there would shadow
the 403 or 404 the route means to return.

The download-failure fallback also publishes fenced now. That arm knows the
stored generation (its HEAD succeeded, only the GET failed), so publishing
unconditionally from it would reintroduce the same overwrite.
2026-08-17 23:00:06 -05:00
beardthelion 3eabf794a5 feat(node): let the storage client express and detect a conditional write
upload takes an UploadPrecondition (IfMatch, IfAbsent, Unconditional) rather
than an Option<String>, so the absent case and the deliberate no-fence case are
distinguishable at the type level and a caller cannot lose the fence by passing
None. head_etag reads the current ETag alongside exists, which keeps exists and
its other callers untouched.

A failed precondition has to be classified off the raw HTTP status: PutObjectError
models no PreconditionFailed variant, so a 412 arrives as Unhandled with nothing
useful on it. 412 is always a lost precondition and 409 is one under IfAbsent.
404 deliberately is not: no archive delete exists on this line, so a 404 on a
conditional PUT means a wrong bucket or endpoint, and reporting that as retryable
would send clients into a loop against a permanent fault.

The three background uploads outside the write guard now publish with IfAbsent.
They fire only where the archive is expected absent, and leaving them
unconditional would defeat the fence from the side: init uploads an empty bare
repo, so a push landing just before it could have its archive replaced by that
empty one. A refusal there means someone else already published the key, which is
logged as the correct outcome rather than a failure.
2026-08-17 22:59:52 -05:00
beardthelion 31f90c4e05 fix(node): stop pretending a held lock fences a late upload
The timeout arm claimed that keeping the advisory lock protected a successor
from an abandoned PUT. It does not. release takes mut self, so the guard drops
the moment it returns and Drop closes the session; measured on this branch, a
successor took the same repo's lock 5ms after release returned while the PUT
was still in flight.

The comment and warn now say what is actually true: the outcome is unknowable,
the PUT may still land, the lock releases normally, and a conditional upload is
what keeps a late publish from overwriting a successor's archive.

Two tests replace the claim. Session disposition is the observable that
separates the two shapes, so the unlock is pinned to run and be confirmed on the
guard's own session with the connection returned to the pool, checked by backend
pid. Successor admission is pinned too, but noted as not what proves the point,
since the lock frees within milliseconds either way.
2026-08-17 22:59:52 -05:00
beardthelion 60d70c7fc7 test(node): add an S3 mock that really enforces conditional writes
The fence work landing next is only as good as what the tests can observe, and
a mock that answers 200 to every PUT would make the whole suite vacuous. This
one holds the object and its ETag, refuses a mismatched If-Match and an
If-None-Match "*" over an existing object with 412, and mints a fresh ETag per
successful PUT so two byte-identical archives never share a token.

Capture-then-replay is deliberate rather than parking a handler and hoping it
resumes: when tokio drops an SDK future the client can tear the connection down
and cancel the server task with it. Replaying what arrived models the arm that
matters (body fully transmitted, commit decided later) with no timing in it.

Six tests pin the semantics in both directions so a hollowed mock cannot hide.
2026-08-17 22:59:52 -05:00
beardthelion 9f313b9bf7 fix(node): bound every await in the lock-acquire loop by the deadline
The remaining budget was checked before the pool checkout but bounded neither
the checkout nor the pg_try_advisory_lock query that follows it. A checkout
starting just under the deadline could wait out the pool's own acquire timeout,
and a slow query could be accepted after the budget was spent, so the advertised
wall-clock cap held only on the fast path.

Both awaits now run under the remaining budget and shed as RepoBusy when it runs
out. The probe's Drop closes its session, which cannot hold a lock it never
confirmed taking, so the query-timeout arm is a plain shed.
2026-08-17 22:59:52 -05:00
beardthelion c4dbd8a797 fix(node): read the close_issue author pre-check from a non-mutating snapshot
The non-owner author fallback refreshed through acquire_fresh, which publishes
into the live repo directory: its extract step removes the existing directory
and renames the new one into place. That runs with no write lock held, so any
signed non-owner could trigger a directory swap underneath an in-flight
guarded write on the same path.

read_snapshot downloads to a throwaway temp dir and hands back a RepoSnapshot
that removes it on drop, so the pre-check still sees fresh data and the live
path is never touched. download_to grows a publish flag to serve both shapes
from one path.

The wedge invariant still holds: a stranger is refused without waiting on the
write lock.
2026-08-17 22:59:52 -05:00
beardthelion f81ebbe614 test(node): assert the second writer sheds as RepoBusy instead of timing out
two_writers_on_the_same_repo_are_not_both_admitted wrapped the second
acquire_write in an 8-second outer timeout and only checked the future had not
finished. That passes for any stall, including lock-pool saturation or a slow
CI box, so it could not tell a working shed from an unrelated hang, and it cost
8 seconds on every suite run.

Use with_lock_acquire_deadline and assert the typed RepoBusy downcast while the
first guard is still held, matching what contended_acquire_sheds_as_repo_busy_
not_internal_error already does. It now also fails loudly if a second writer is
admitted, which the timeout version could not distinguish.
2026-08-17 22:59:52 -05:00
beardthelion 5fdb902096 fix(node): raise a failed fresh download with no local copy as RepoUnavailable
acquire_fresh already refused a failed Tigris HEAD as RepoUnavailable, so the
handler layer mapped it to a retryable 503. A failed GET on the same path still
returned a bare anyhow error, which the info/refs map_err closure stringified to
AppError::Git and answered 500. One endpoint therefore told the client a
transient object-storage blip was permanent or retryable depending on which call
failed, and close_issue's pre-check inherited the same split through its bare ?.

Raise it at the source instead of at each consumer: From<anyhow::Error> for
AppError already downcasts RepoUnavailable out of the context chain, so both
callers pick up the retryable mapping without touching either.

The new test drives HEAD 200 with GET 500, which is the exact state the refusal
is for: archive present per HEAD, GET failed, no local copy to fall back on.
2026-08-17 22:59:52 -05:00
beardthelion 999ee72f3c test(node): poll for the closed session instead of sleeping a fixed 300ms
The release-invariant test slept 300ms for a Drop-spawned close before
comparing backend pids, which is a coin flip on a loaded CI runner. It now
polls pg_stat_activity for the captured pid on a standalone connection, the
same discipline poll_until_free documents: a pooled observer would be handed
the session under measurement and hide the effect.

The conversion was checked against the failure it exists to catch rather than
assumed. Pooling the session on an unlock that returned false makes the test
fail after the poll deadline, not hang, and no_reap_pool disables idle timeout
and max lifetime so nothing but the close under test can retire that backend.
A generous deadline would have been the same defect as the sleep.
2026-08-17 22:59:52 -05:00
beardthelion 259d5873ec fix(node): log the read failure the close_issue pre-check folds into a denial
The authorship pre-check treated a get_issue I/O error and a genuinely absent
issue as the same None, with no log line. The client answer is deliberately
identical, since a caller who cannot write must not learn whether the issue
exists, but the two are not the same event and an operator had no way to tell
a real authorization denial from a filesystem or parse failure behind it.

Splitting the arm leaves the 403 exactly where it was and makes the read
failure visible in the log.
2026-08-17 22:59:52 -05:00
beardthelion 10230ac50e fix(node): log expected transient acquire failures at warn, not error
Both acquire call sites logged every failure at error, including RepoBusy,
which the raise site already logs at warn. Ordinary write contention paged.
The previous commit widened the problem: info_refs now raises RepoUnavailable
on a storage blip, so that site would have started paging on the condition
this series just classified as transient and retryable.

A classifier over the two typed refusals picks the level at both sites,
mirroring the startup path's permanent-vs-transient split. Anything it cannot
classify still logs at error, so an unknown failure keeps paging.
2026-08-17 22:59:52 -05:00
beardthelion 63a9ef8b1b fix(node): refuse a fresh-copy write when the archive HEAD cannot be read
acquire_fresh collapsed a failed HEAD into "no archive exists" via
unwrap_or(false) and served the local copy, while the sibling path under the
lock already refused on the same condition. A push that hit a storage blip got
an advertisement built from a possibly-stale tree, uploaded a whole pack, and
was then refused by acquire_write for the reason the advertisement had already
swallowed. The authorship pre-check on close_issue read the same way, which is
an infrastructure failure resolving toward a denial.

A failed HEAD now raises RepoUnavailable, so both callers surface it as the
retryable 503 rather than a stale success. The download-failure fallback is
unchanged: a present-but-unreadable archive is still self-healed from local.

info_refs keeps its map_err for every other failure so the read path's error
vocabulary does not move; only the typed error takes the From chain.

A test-only TigrisClient constructor pointed at a closed port makes both
refusals executable, so this is no longer verified by reading. It also proves
the under-lock arm end to end, which the earlier draft had recorded as an
untestable gap.
2026-08-17 22:58:55 -05:00
beardthelion e5558fbb27 fix(node): shed an unrefreshable write as a retryable 503 with a fixed body
The two refusal arms added for the under-lock transfer bound returned bare
anyhow errors, so AppError's From impl (which downcasts only sqlx::Error and
RepoBusy) landed them in Internal: a 500 internal_error whose body was the
error string. Two defects in one. A transient object-storage failure told the
client not to retry, and the timeout arm's message interpolated the owner slug
and repo name straight into the response body, contradicting the fixed-body
policy the RepoBusy arm sets six lines above it.

RepoUnavailable follows RepoBusy exactly: a fieldless type raised with the
operator detail in a context string, downcast to its own rung, mapped to a 503
whose body interpolates nothing. The detail stays in the log at the raise site.

The timeout arm logs at error! where its sibling logs at warn!, deliberately: a
300s stall pinned a lock-pool slot for five minutes and is the condition the
transfer bound exists to surface, so it must keep paging when the handler
classifier demotes the ordinary blip case.
2026-08-17 22:58:26 -05:00
beardthelion af5948b204 style(node): rustfmt the new contention test
Whitespace only; cargo fmt --check gates the push.
2026-08-17 22:58:26 -05:00
beardthelion 8cf6b7ae46 fix(node): re-authorize close_issue under the guard instead of only re-checking existence
The guarded re-read matched `Ok(Some(_))` and discarded the blob, so the
authorization decision rested entirely on the pre-lock read. A comment above
claimed the opposite. That is not a narrow window: `acquire_write` re-downloads
the archive after locking, so the tree that gets mutated is routinely not the one
the author was read from. With owner-push enforcement defaulting to false and
branch protection covering only `refs/heads/*`, `refs/gitlawb/issues/*` is
pushable, so a forged author blob landing between the two reads was honored.

The blob is already in hand under the guard, so re-asserting owner-or-author
costs a deserialize. A non-owner whose issue has vanished gets 403 rather than
404, matching the pre-check's existing refusal to reveal existence.

`owner_can_still_close_after_the_reorder` seeded the owner as their own issue's
author, which made it unable to fail: with the owner check disabled the author
fallback granted the close and the test stayed green. It now seeds a third party,
so only the owner arm can grant. RED with `is_owner = false`.

Also drops the claim that the author twin covers the acquire-vs-acquire_fresh
distinction. `RepoStore::for_testing` hardcodes `tigris: None`, so the two calls
are identical in every test here and reverting that line leaves the twin green.
Separating them needs an object-storage seam. Stating the gap beats asserting
coverage that does not exist.
2026-08-17 22:58:26 -05:00
beardthelion 82143ad692 fix(node): refuse a write when object storage is unknowable, and shed contention as 503
Four defects a seven-reviewer pass found in the previous two commits, three of
them cases where one arm of a match contradicted its neighbour.

The under-lock refresh read a failed Tigris HEAD as "no archive"
(`exists().await.unwrap_or(false)`), skipped the refresh, and wrote against a
possibly-stale tree, then re-uploaded over it. That is the outcome the timeout
arm twenty lines below explicitly refuses. A failed HEAD and a failed download
leave us knowing different things, so they no longer share a branch: only a
download failure after a successful HEAD establishes that the local copy is a
sound thing to fall back to and re-upload. A HEAD failure now refuses the write.

Lock contention that ran out the acquire deadline surfaced as a 500 carrying the
owner slug and repo name in the client-visible body, while pool exhaustion
fifteen lines up returned a deliberate 503. Contention is transient and ordinary,
so it now maps through a typed `RepoBusy` to a retryable 503 with a fixed body;
the detail stays in the log at the raise site.

`lock_not_taken` was assigned after the try-lock answered, so an error left a
previous `true`-derived value standing and could return a lock-holding session to
the pool. It is now cleared before the statement is sent.

`leak()`'s no-panic property in the guard's no-runtime Drop branch depended on
`min_connections == 0` without saying so; a future tuning change would have
silently re-armed a panic inside Drop. Made explicit with the reason.

Two tests that did not bind:

  - `waiter_on_one_repo_does_not_block_another` proved only that two lock keys do
    not collide. It now samples the pool counters across more than two backoff
    cycles. RED at 0/50 samples with `drop(probe)` moved after the sleep, the
    exact defect it names, which the previous version survived.
  - the exhaustion test's `Ok(Err(_)) | Err(_)` was satisfied by its own outer
    timeout. It now requires `PoolTimedOut` and asserts the slots are accounted
    for. RED with the pool sized N+1.

`contended_acquire_sheds_as_repo_busy_not_internal_error` is new and drives the
deadline path for real; the deadline became a field so it does not wait 90s. RED
at 500-vs-503 with the downcast removed.

The acquire deadline's docstring claimed a total under the 120s proxy idle
timeout, which the 300s under-lock transfer bound in the same function
contradicts. It bounds the wait only, and now says so. The backoff is clamped to
the remaining budget.
2026-08-17 22:58:26 -05:00
beardthelion d346bde532 fix(node): log lock-pool saturation in the request path, not via readiness
Completes the shed-in-path half of the availability story. F7 already made a
pool timeout a retryable 503 by letting the sqlx downcast through; this adds
the operator-facing half, with the pool's own size/idle counters so an
incident can distinguish "the pool is full" from "the database is gone"
without reproducing it.

Deliberately NOT a readiness probe. /ready gates Fly routing with no
fail-open, so failing it on a saturated pool would pull this node's READS
out of service too and push its write load onto peers carrying the same
load. That is the downward spiral AWS's health-check guidance names and the
SRE Book documents independently. A lock-pool readiness probe would also add
nothing on the reachability axis, because both pools are built from the same
database_url, so the existing app-pool ping already answers it.

The error is still returned via .context() rather than replaced, because
anyhow preserves downcastability through context layers and the 503 mapping
depends on it.

Cost is bounded by construction: one line per failed acquire, and a failed
acquire has already paid a multi-second pool timeout.

Refs #279
2026-08-17 22:58:14 -05:00
beardthelion 5d254fb336 fix(node): address the review findings on the advisory-lock series
Seven fixes from a seven-reviewer pass. Eight of the nine substantive
findings were defects introduced by this branch, and two contradicted
claims made in its own earlier commit messages.

P0, the worst of them. The under-lock download's timeout was folded into
the same Err as a corrupt archive, so it fell through to the
use-the-local-copy fallback and the write proceeded. Two ways that
destroys data: we do not know whether we hold the latest tree, so
re-uploading can overwrite another node's newer archive; and the abandoned
download's extraction runs in an uncancellable spawn_blocking that ends in
remove_dir_all + rename over the same path, so git would have been running
against a directory a background task was about to delete. Now the
Option is branched rather than collapsed: a genuine fetch error keeps the
local-copy fallback, a timeout refuses the acquire and lets the guard's
Drop free the lock.

LockProbe closed its connection on ordinary contention, so a 60-attempt
spinner tore down 60 backends -- the exact behavior an earlier commit
claimed it avoided. The test written to guarantee that asserted the
HOLDER's lock count, which cannot observe the probe's own connection, so it
passed throughout. The correct predicate turned out to be narrower than the
first attempt at this fix: not "we saw an answer" but "we positively know
nothing was acquired," because a true answer means the lock IS held and
dropping without handoff leaks it exactly as a cancellation would. The
first attempt reopened U1's leak and U1's gate test caught it.

tigris.exists() ran unbounded inside the lock-held span, so the knob's
promise was not kept and the docs were wrong about the shape. The HEAD and
the download now share one budget, so worst-case occupancy is one budget
per span rather than two.

close_issue's pre-lock author read used acquire(), whose fast path returns
on a cached directory and never contacts object storage, so on a multi-node
deploy an issue author would be refused their own issue. Now acquire_fresh,
which refreshes without taking the lock -- the property the comment
already claimed. That comment also asserted authorship is immutable, which
a reviewer disproved by force-pushing a forged blob at refs/gitlawb/**;
reworded to the real justification (a pre-check whose mutation re-reads
under the guard). The underlying pushability is pre-existing and larger
than this branch.

The retry loop had no wall-clock cap, reaching ~360s against a 120s proxy
idle timeout that was itself lowered from 600s after held connection slots
caused a production outage. Capped at 90s, with the bail message naming the
real bound.

All four acquire_write call sites stringified their error, destroying the
anyhow downcast that error.rs documents explicitly ("without this, every
database outage surfaces as a 500 instead of a 503"). PoolTimedOut is in
that 503 set, so a saturated lock pool told clients not to retry something
transient. Now propagated.

Tests: six that the plan required and the first pass never wrote. The R4
pool-isolation test (named in the plan as proving the pool split, entirely
absent), close_issue's owner and non-owner-author twins (INV-21c, a
two-principal gate with only its deny arm covered), a waiter-does-not-block-
an-unrelated-repo case, and a config test for the transfer knob. The
runtime-teardown test no longer returns green when DATABASE_URL is unset.

Drop's no-runtime branch now detaches via leak() instead of mem::forget:
PgConnection has no Drop impl, so the socket closes synchronously and the
lock frees immediately rather than at process exit.

Evidence. The probe predicate is proven load-bearing in BOTH directions:
forcing always-close reddens the churn regression, forcing never-close
reddens the cancellation gate, and only the correct predicate satisfies
both. The author twin reddens when the fallback is removed.

One honest limit: the author twin does NOT redden when acquire_fresh is
reverted to acquire. With tigris disabled in tests the two calls are
identical, so that fix is correct by reading, not by execution -- the same
seam that leaves the transfer bound's wiring untested.

Full crate suite 529 passed, clippy -D warnings clean.

Refs #279
2026-08-17 22:58:14 -05:00
beardthelion 9da0939312 fix(node): authorize before taking the write lock in close_issue
close_issue acquired the per-repo advisory lock, then ran the
owner-or-author check, then returned 403 with the lock still held. That was
harmless only because the lock excluded nothing. Making it work, as the
preceding commits do, turns this ordering into a denial primitive: any
caller with repo read access could take the write lock on demand and be
refused the write, while a legitimate writer burned its 60-attempt retry
budget against a lock held by someone with no write authorization. On a
public repo that is every permissionless identity. This series creates the
exposure, so it closes it in the same series.

The owner check is cheap and moves above the lock outright. The author
fallback needs the issue's git-JSON blob, since there is no author column,
so it now reads through acquire() rather than acquire_write(): an issue's
author is set at creation and never changes, so reading it outside the lock
races nothing, and only the mutation needs exclusion.

Two deliberate behavior choices. A non-owner whose authorship cannot be
established gets 403 rather than 404, so this route does not tell an
unauthorized caller whether an issue exists. And the owner's existing
404-for-a-missing-issue path is preserved by re-reading under the guard,
which the mutation wants anyway.

Swept the other three acquire_write sites: merge_pr owner-gates at :200
before acquiring at :214, the repo write path checks did_matches at :916
before :933, and create_issue's read gate is legitimate because that caller
IS authorized for the action it performs. close_issue was the only one with
the wrong order.

RED before: with an independent session holding the repo's lock, a
stranger's request sat in the retry loop until the 3s deadline fired
(Elapsed), which is the wedge itself. GREEN after, refused immediately.
Proven load-bearing: disabling the pre-lock refusal restores the Elapsed.

Full crate suite 523 passed.

Refs #279
2026-08-17 22:57:57 -05:00
beardthelion 9321d3b993 fix(node): bound the object-storage transfers that run under the write lock
Two transfers happen while the per-repo advisory lock is held: the archive
download inside acquire_write, which runs after the lock is taken and
before the guard exists, and the upload inside release. Both were free
before the guard pinned a lock-pool connection, because the lock's
connection went back to the pool immediately. Now an unbounded stall holds
a lock-pool slot for its whole duration, so enough concurrent stalls deny
every write on the node with no reaping path. Add
GITLAWB_LOCK_HELD_TRANSFER_TIMEOUT_SECS (default 300) and route both
through it.

A timed-out upload is UNKNOWABLE rather than failed, so the timeout arm
takes no compensating action: the PUT may well have landed. The lock is
released either way, which trades a narrow last-writer-wins window for not
wedging the repo behind a stalled transfer. Deliberate, and stated here
rather than discovered later.

Note what is NOT bounded by this knob: acquire_fresh's download. It runs
before any lock is taken, so it holds nothing. An earlier revision of this
change put the bound there by mistake, because both functions contain an
identical download call and the first match won.

Two disclosures.

Coverage: the committed tests cover the bound mechanism, not the wiring.
Driving a genuinely stalled transfer through acquire_write needs either the
object-store abstraction (out of scope) or a process-global
AWS_ENDPOINT_URL_S3 mutation, which would make the suite order-dependent
under the concurrent runner. That a stalled transfer is bounded is
therefore verified by reading, not by execution.

Behavior: on timeout with a local copy present, the download falls into the
pre-existing self-healing fallback and the write proceeds against that
local copy. This widens the conditions reaching that path from
corrupt-or-unreachable to include merely-slow, so a stale tree could now be
written and re-uploaded on a slow link. Kept consistent with the existing
failed-download behavior rather than inventing new semantics here; changing
it is its own decision.

Full crate suite 522 passed.

Refs #279
2026-08-17 22:57:57 -05:00
beardthelion 98f2daf804 fix(node): read the advisory unlock's result instead of discarding it
pg_advisory_unlock reports "you did not hold this lock" as a false RETURN
VALUE plus a server WARNING, never an error, so let _ = execute(...) could
not distinguish a real release from a no-op. Three blindnesses stacked in
those four lines: execute discards the row, the boolean lives in the row,
and let _ discarded the Result too.

Read it through fetch_one into (bool,). A false means this session's lock
state is not what we believe it is, so the connection stays in the guard
for Drop to close rather than being handed back to the pool as clean. Only
a confirmed unlock returns it. A query error gets the same treatment, since
the lock must not outlive a session we can no longer reason about.

Note this is the only unlock site: the pre-write download's error path
returns through the guard, so Drop covers it and there is no second place
to keep in sync.

RED before: the connection came back on the same backend pid after an
unlock that returned false. GREEN after, and proven load-bearing by
treating false as success, which turns it RED again. The must-not case (a
normal release still reuses its backend) passes in both states, so this
does not over-close the happy path.

Full crate suite 520 passed.

Refs #279
2026-08-17 22:57:43 -05:00
beardthelion c49d3cdb57 fix(node): free the advisory lock when a write guard dies without releasing
A guard can exit without reaching release(): an early ? on the pre-write
download, a panic, or an axum handler future cancelled when the client
disconnects. Its session still holds the lock, so returning that
connection to the pool would block every future write to the repo until
sqlx recycles it, which on the 0.8.6 defaults is ten minutes idle or
thirty minutes lifetime. Close the session instead and let Postgres free
the lock at session end.

One hazard found by writing the teardown test rather than by reasoning:
PoolConnection::drop spawns onto the runtime for both closing and
returning, and panics outright when no runtime handle exists. That panic
would fire inside a Drop and abort the process during unwind. It is not
introduced by this commit, it comes with owning a PoolConnection at all,
but this is where it becomes reachable. So Drop checks for a runtime
first and, with none, leaks the handle deliberately rather than panicking:
the process is already exiting and socket teardown ends the session, which
is what frees the lock at exit anyway.

Observed RED before the fix, with the lock still held for the full 10s
poll window against a standalone observer on a no-reap pool, and the
teardown case panicking in sqlx-core connection.rs:208. Proven
load-bearing after: neutering the close_on_drop call turns the drop test
RED again. The must-not case (a released guard reuses its backend pid
across four writes on a pool sized 1) passes in both states, so the
signal is specific to the abandoned-guard path.

Full crate suite 519 passed.

Refs #279
2026-08-17 22:57:43 -05:00
beardthelion 5ce3c7f413 fix(node): hold the lock-owning connection in RepoWriteGuard (#279)
Postgres advisory locks are session-scoped: only the backend that took one
can release it. acquire_write took the lock through fetch_one(&pool) and
release unlocked through execute(&pool), two independent checkouts, so the
unlock usually landed on a session that held nothing and returned false.
Measured on main: two writers on one node and the same repo BOTH acquired,
50 of 50 sequential cycles leaked, and 100 writes left 100 orphaned
advisory locks on the server.

The guard now owns the PoolConnection that took the lock, drawn from the
dedicated lock pool, and releases on that same session. The retry loop
probes through LockProbe so a cancellation mid-acquire cannot strand the
lock, and hands the connection back before each backoff so a spinner on a
contended repo does not pin a slot while idle.

Pool exhaustion is deliberately not retried. It is a different condition
from lock contention, and retrying it would spend all 60 attempts on a
capacity problem unrelated to this repo while reporting it as someone else
holding the lock.

Both #279 acceptance tests were observed RED first: the exclusion test
admitted the second writer, and the leak test reported 1 lock held where 0
was required. Both GREEN after. Full crate suite 516 passed.

Db::pool() is removed because this change was its only caller.

Refs #279
2026-08-17 22:57:43 -05:00
beardthelion edcf0f2ccf fix(node): add a dedicated, lazily-connected advisory-lock pool
Pinning a connection for the lock's lifetime is only safe if those
connections come from somewhere other than the pool serving ordinary
request handlers, otherwise a push burst starves every other query. Add
GITLAWB_DB_LOCK_POOL_MAX_CONNECTIONS (default 32) and a Db::lock_pool
builder, with the sizing tradeoff documented on the field and in
.env.example: every in-flight write pins one connection here, so the
value is a hard ceiling on simultaneous writes node-wide.

The pool connects lazily on purpose. The main pool must connect eagerly
because it runs migrations, which is why it needs connect_db_with_retry's
backoff and degraded-server handoff; that function is not a generic retry
helper and the lock pool is built well after the db-ready handoff has
already resolved. A lazy pool has no startup work, so it adds no new way
for the process to fail to boot and needs no second copy of that
machinery. If Postgres is unreachable when the first write arrives, that
write fails on the pool's own acquire timeout, like any other
database-backed request.

Pure configuration, so no proof-first cycle: the knob is covered by a
parse/default/reject-zero test. Db::lock_pool has no caller until the
guard wiring lands, hence the temporary dead_code attribute.

Refs #279
2026-08-17 22:57:07 -05:00
beardthelion 7af306701d fix(node): close the advisory-lock probe's session if it is dropped mid-acquire
A cancelled .await does not cancel an already-sent SQL statement, so a
pg_try_advisory_lock whose future is dropped still takes the lock
server-side while the caller abandons the result, leaving nothing to
release it. The connection then returns to the pool holding the lock and
wedges that repo until sqlx recycles the session.

Introduce LockProbe, which owns the connection across the in-flight
try-lock and closes it in its own Drop if it is still held. close_on_drop
is a one-way setter, so the arming lives in Drop rather than being set up
front and cleared on success; disarming is Option::take, which is what
into_conn does once an acquire is actually observed. This is now the only
place that issues pg_try_advisory_lock.

The committed gate drops a probe without taking its connection, which is
the state a cancellation leaves behind, and polls a standalone observer
until the lock frees. Deterministic on purpose: the timing sweep that
found this window leaks roughly 1 in 600, which is not something a CI
gate can rest on. Observed RED before this change with the lock still
held for the full 10s window.

Refs #279
2026-08-17 22:56:50 -05:00
Kevin Codex e4c7458fb3 Merge pull request #332 from anandh8x/agent/docs-correct-security-status
docs: correct security and workspace status
2026-08-16 21:40:30 +08:00
beardthelion 76f630008f fix(node): never hand back a continuation that made no progress
On a resumed request whose visit budget was already spent by the provenance
phase, the scan's top-of-loop visit arm sealed pager.cursor, which at that
moment is the position the caller just sent. The node returned the caller's own
token, verbatim, rung after rung. Three rungs were observed returning an
identical position.

A token looks like progress, so the client keeps going: gl retries to its
resume cap, and every one of those requests re-runs the full provenance phase,
up to seventeen repo acquires and cat-file subprocesses, advancing nothing
before it errors. That is roughly nine anonymous requests worth of work for
none, and it is worse than shedding nothing, because a caller who is told the
ladder is over stops immediately.

A seal now has to be strictly ahead of where the request itself started: a
different candidate is ahead by construction, since only the gated advance can
name one, and the same candidate needs a row past the start row. A request that
started at the front is before everything, so its seals pass untouched.

When the proposer settled at least one row this rung, the existing ceiling arm
already seals that row, and it is strictly ahead because a resumed scan only
walks rows past its cursor. Only a rung that settled nothing sheds without a
token, and that is honest: the spender is the provenance phase, which runs the
same way every rung, so no retry can do better.

The filter sits at the single mint site, where a future call site cannot bypass
it, and it logs the drop as a boolean. record_scan_truncation has already
logged that a position was sealed by then, and a 503 carrying no token next to
that line is the confusion that log exists to prevent.
2026-08-16 02:20:05 -05:00
Vasanthdev2004 888fab6028 fix(infra): serialize the upgrade and write compose through a unique temp path
The compose reinstall added last round is not safe under overlap, and `main`
never wrote compose from this document, so the race arrives with that commit.

Two executions — a double run, a retry while the first is still pulling, two
operators — shared the fixed `/opt/gitlawb/compose.yaml.new`. The second
truncates the file the first is about to install; and once the first renames,
the second's open descriptor keeps writing into the live `compose.yaml` inode.
Both outcomes stay valid YAML, so the `up -d --remove-orphans` on the next line
accepts a blended or truncated service set and deletes whatever fell off — on a
live node, the local postgres or the TLS terminator. Both reviewers reproduced
it independently; one measured a short file in 12 of 12 trials.

The whole step now runs under `flock -n` on /opt/gitlawb/.upgrade.lock and
exits non-zero if the lock is held, so a second run cannot restart against a
file the first is mid-install. The temp file comes from `mktemp`, so concurrent
writers cannot truncate each other, with a trap to clean it up on failure.

The compose body is base64-encoded rather than written through a nested
heredoc. The rendering carries `$${VAR}` passthroughs that must reach the file
unexpanded, which needs a quoted delimiter, and a shell heredoc nested inside an
indented Terraform heredoc depends on the dedent landing that delimiter at
column 0. Decoding one line has neither failure mode and no delimiter can
collide with the content.

Not validated locally: terraform is not installed on this machine, so this
change has had no `fmt`, `validate`, or `plan` run against it.
2026-08-16 11:00:21 +05:30
beardthelion d0635d95b7 fix(node): resume the /ipfs scan per candidate so a starved one is not skipped
A CID that maps to several oids shared one resume slot across every candidate,
and the slot kept the maximum position. An earlier candidate could spend the
probe budget walking past a repo that holds the object for a later one, seal a
position beyond it, and the next rung would resume past a row that candidate
never examined, wrap, and shed tokenless. The client reads an absent token as
the ladder being over, so the object became permanently unretrievable, at stock
config, deterministically on every retry.

The token now names which candidate it is resuming, and the rules that keep
that sound are narrower than they first look:

Only one candidate per request may seal, and which one depends on where the
REQUEST started. On a resumed request it is the resumed candidate alone, since
the shared pager holds only the table suffix from the caller's cursor, so a
later candidate walked a suffix and never saw the front. On a front-started
request it is the first unfinished candidate, since there every candidate walks
from the front and a later candidate's stop is honest coverage. Silencing later
candidates unconditionally would remove the only thing that mints rung 1 when
the first candidate wraps untruncated.

A candidate is finished when its row loop walked every fetched row, or when it
owed no scan at all. Both matter: a properly provenanced candidate never wraps,
so without the second arm the ladder dies every rung. The wrap is witnessed per
candidate at the row loop's own two exits, never by reading the shared pager
flag at the tail, which any short page sets and which would let a candidate that
truncated mid-page look finished and strand the rows it refused.

Finishing a non-final candidate advances the seal to the next one at a front
sentinel and taints, because the tail emits a continuation only when something
tainted; sealing without tainting would suppress the taint and return a
definitive 404 while discarding the token it had just minted.

The keep-the-maximum comparison is gone. With one proposer per request the slot
is written at most once, so an assertion states that directly instead.

The pager stays shared per request. A per-candidate pager would restore the
fan-out the paging exists to remove.
2026-08-15 23:09:36 -05:00
beardthelion 2656718916 feat(core): carry the resumed candidate's identity in the scan token
A CID can map to several git oids, and the ladder needs to name which one it is
resuming. The sealed position gains the candidate's oid hex so a rung resumes
that candidate rather than a position in a list: oids_for_cid is a sorted set,
so an ordinal silently repoints at a different candidate when a pin that sorts
earlier arrives between rungs, while an identity degrades safely to "not found,
restart at the front".

The field is length-prefixed and padded to 64, matching the framing the row
fields already use, because production oids are 40 hex, not 64: repos are
created with --object-format=sha1 and only the test fixtures are sha256. A
fixed 64-byte field would fail every seal on a real deployment and shed a
tokenless 503, which the client reads as the ladder being over. Both widths are
exercised, and a zero-length candidate is rejected at decode so it cannot be
confused with the front-of-table sentinel, which is empty row fields with a
real candidate.

VERSION goes to 3 and the plaintext to 527 bytes, so a token minted under the
old layout opens to None and the caller restarts at the front. Nothing has
minted one outside tests.

The slot carrying the position is a struct rather than a widened tuple on
purpose: a 3-tuple would have pulled the hex into the existing keep-the-maximum
comparison, changing behavior this commit is meant to leave alone.

Token length stays invariant across both oid widths, since length would
otherwise be a side channel for the withheld row it names. That is asserted on
a real seal in gitlawb-core, not on the gl fixtures: nothing in gl seals or
opens a token, so its width constant cannot detect a wrong layout.
2026-08-15 22:20:34 -05:00
beardthelion 366b599331 fix(node): order the CID candidate list so it cannot depend on heap order
oids_for_cid ran a bare SELECT with no ORDER BY, so Postgres was free to return
the candidates in physical heap order. get_by_cid walks those candidates under
one shared probe budget, visit budget and pager, so whichever comes back first
is the one that spends the request's budget: two nodes holding identical data,
or one node before and after an unrelated write, could resolve the same CID
differently and one could shed a 503 where the other serves.

The instability is not hypothetical. An unpin and re-pin of a single object,
which is an ordinary production sequence, moves that row to the end of the heap
and rotates the list.

The sibling pin_sources_for_oid already orders its union for exactly this
reason, and the handler comment next to it leans on that determinism.
2026-08-15 21:56:07 -05:00
beardthelion 1bdefb18c2 chore(node): log every /ipfs scan truncation and correct the helper's docs
Code review of the previous commit turned up three things worth fixing in place.

record_scan_truncation centralized the taint and the seal but logged nothing, so
centralizing actually made a truncation less visible than the scattered inline
taints it replaced: only the visit ceiling logged, and it logs from inside the
gate, before the caller decides whether a position gets sealed. Two identical log
lines could therefore mean "the ladder continues" or "the caller is stranded".
One debug line now carries the reason and whether a position was sealed. It logs
only whether one exists, never its value, since the position names a withheld
row's created_at and id.

The doc comment claimed to be "the one site that records a scan truncation" while
eight other taint sites bypass it. The distinction is real but it is not the one
the comment drew: a ceiling stops the scan and owes the caller a position, while
a transient skip refuses one row and the rows behind it are still walked. Says
that now.

The forward-only rule's stated justification was wrong. A later candidate's
position is still ahead of the token the caller echoed, so letting it win would
not move the ladder backwards; it would shrink each rung toward a single row.
Also records that the comparison is Rust byte order while the pager's keyset runs
under the database collation, which can disagree on a non-C collation, and why
that costs a replay rather than a skipped row.
2026-08-15 19:45:08 -05:00
beardthelion 42e6aabc10 Merge origin/main into fix/issue-135-ipfs-cid-tree-gate
Carries #326 (distinct-signer certificate threshold) onto this head.
2026-08-15 18:38:15 -05:00
beardthelion 69cfbd4b3c fix(node): mint a continuation when a ceiling stops the /ipfs scan mid-page
The probe and visit ceilings taint inside gate_and_serve and returned Skip, so
the row they refused and every row behind it were walked past without a verdict
while the resume position was sealed from pager.cursor, the end of the fetched
page. Two ways that stranded content:

- On the final page, `pager.exhausted` breaks ahead of every mint arm, so the
  shed carried no continuation at all. A tokenless search_incomplete is the
  wrapped-scan answer, which tells `gl ipfs get` its ladder is over, so a holder
  on that page was unreachable on every retry.
- Mid-page, the sealed cursor sat past the refused rows, so the resume skipped
  them. The shipped ladder tests all set page_rows == ceiling, which puts the
  break exactly on a page boundary and hides both.

The ceiling now returns GateOutcome::CeilingStop rather than tainting on its way
out, and the scan loop stops there and seals the row in front of the one that was
refused. record_scan_truncation is the single site that records a truncation:
it taints and seals together, and only ever moves the position forward, so a
later oid candidate re-walking the same rows on a spent budget cannot hand back a
token the caller already echoed. The wrapped-scan tail no longer clears a seal,
since a ceiling can stop a resumed scan part way through the last page.
2026-08-15 18:38:09 -05:00
Vasanthdev2004 579eee07bc docs(infra): correct the ignore_changes comment for the new upgrade path
The lifecycle comment still listed image_tag among the values needing a deliberate instance replacement. It is rendered into compose.yaml, which the upgrade command now reinstalls, so apply plus upgrade rolls it out. Only the values user-data writes into .env at first boot stay replacement-only.
2026-08-15 14:12:14 +05:30
Vasanthdev2004 de4d932609 docs(infra,macos): make the opt-out reachable and stop the AWS guide contradicting itself
All four are in the deployment half added last round; none is in the Rust.

The AWS guide told operators two different things
--------------------------------------------------
"Changing configuration" still said editing `image_tag` does not affect a running
instance and routed to `-replace`, while the upgrade section thirty lines above
now reinstalls the rendered compose, which carries `image_tag`. The same
paragraph told operators to edit `.env` and restart — which the new "Why the
upgrade rewrites the compose file" section explains is silently ineffective for
any key the installed compose does not already name.

Split by which file carries the change rather than deleted, because the
distinction is real: values rendered into `compose.yaml` (`image_tag`, ports,
`domain_name`, `db_host`, `icaptcha_*`) roll out with `apply` + `upgrade_command`;
values user-data writes into `.env` at first boot (`public_url`,
`bootstrap_peers`, integrations) do not propagate from tfvars at all and need an
edit on the instance or a replacement. The third case is the one that bites: a
setting the installed compose does not yet name is read from `.env` and dropped,
so it needs the upgrade first.

`--remove-orphans` stopped being inert
---------------------------------------
The flag is not new, but until the upgrade rewrote `compose.yaml` it reconciled
against the instance's own file, so the only orphans were hand-started
containers. It now reconciles against a fresh rendering whose service set is
conditional — `postgres` only when `db_host` is empty, `caddy` only when
`domain_name` is set. Setting `use_rds` or clearing `domain_name` and then
running what the README calls a pull-and-restart stops and removes the local
postgres or the TLS terminator. That is what those variables mean, but it happens
on the upgrade rather than on apply, so it is now stated next to the
Terraform-owned warning. Data survives either way; both bind-mount under
/mnt/data.

The macOS setting had no writer
--------------------------------
`enforceOwnerPush` was persisted, loaded, emitted into `.env`, and forwarded by
the generated compose — and nothing set it. `SettingsWindow` had no state and no
assignment, so the only way to reach the opt-out on that launch path was hand
editing config.json in Application Support, which no document mentions. Editing
`.env` does not work there either: `writeEnvFile()` rebuilds it from `Config` on
every start. It now has a Toggle beside "Sync repos from peers", following the
pattern already in that view.

Three lines the change made untrue
-----------------------------------
The handler test asserted "with no configuration at all" while setting
`enforce_owner_push` explicitly — the right thing for the test, the wrong claim
for the message. The cross-tenant docstring still described three distinct pusher
DIDs after A and B were given `n1`'s owner. And OSS-READINESS-AUDIT listed
unprotected branches not enforcing owner checks as a live blocker; owner checks
now cover every branch, and only the UCAN half of that sentence still stands.

Also dropped the `#[allow(dead_code)]` on `test_state_with`, redundant since the
same commit added its call site.
2026-08-15 11:47:36 +05:30
anandh8x 0b47e1fc08 docs: clarify visibility limits 2026-08-15 10:32:24 +05:30
anandh8x 505c452378 docs: correct security and workspace status 2026-08-15 10:30:51 +05:30
Kevin Codex 50d3cbbe97 Merge pull request #326 from euxaristia/fix/cert-threshold-duplicate-signatures
fix(core): Count distinct signer DIDs in certificate threshold check
2026-08-15 10:49:11 +08:00
beardthelion a7e8e35030 fix(review): make the coupled sites findable and pin two more properties
Code review found the doc comments carrying claims the code does not
support and two coupling points documented from one side only.

The floor and the resolver's walk_cap are not textual twins: the floor
reads the constant, the resolver reads the AppState seam, and they agree
only because every construction seeds one from the other. Say that, and
give walk_cap the back-reference it lacked. The lifted node_verifies
helper no longer named the middleware it mirrors, so an edit to
require_signature would not find either copy.

The request-target clause pins @path, not @method or content-digest: a
301, 302 or 303 still rewrites a signed POST to a bodyless GET while the
signature headers ride along. Record that rather than implying the hop is
safe. Add the query-removed and fragment-only matrix rows, and a mutant
pinning that scan-phase walks are not charged to the work bucket, which
nothing covered.
2026-08-14 19:21:44 -05:00
beardthelion dc5daa14b4 fix(node): size the ipfs work floor for provenance walks plus one full search
The floor reserved one complete legacy search per window, probes plus page
tolls, but the provenance visibility walk debits the same bucket before the
fallback runs and its cap is charged per phase. With a route limit set below
the floor the provenance phase spent from the budget the search was promised,
the fallback 429d short of its configured reach, and the retry re-paid the
same walk charges, so a readable holder past that point stayed unreachable.

Add the walk term, min(MAX_HISTORY_WALKS_PER_REQUEST, ipfs_max_repos_walked),
which is textually the resolver's own walk_cap so the two move together. The
ladder fixture pins repos-walked to 1 to keep the page toll, not the new term,
as the thing binding it.
2026-08-14 17:43:48 -05:00
beardthelion 1c86ee22d6 test(node): pin require_signature to the request-target it received
The verifier already rebuilds @path from the URI it was sent, which is why
the client-side redirect bug surfaced as a 401 rather than as a bypass.
Nothing pinned that, so a change to the reconstruction could drop the query
half or collapse it to a constant and only the clients would notice.

Drive the production router fixture with a signature made over one path and
a request on another, and again with a query mismatch, plus an identically
signed control. No pre-fix RED is obtainable here by construction, so each
case is proven load-bearing by injecting the defect it names.
2026-08-14 15:48:46 -05:00
beardthelion 92bb55441d test(clients): prove the refused redirect never reaches the node verifier
The two same-origin follow tests drove exactly the path-changing hop the
predicate now refuses, and their mocks only proved a target was reached,
never that the signature verified there.

Rewrite both into refusal tests and run the real gitlawb-core verification
over the request the target actually received, so the recorded verdict is
what the assertion speaks about. Each refusal is paired with a positive
control that verifies, so an empty verdict slot is attributable to the
refusal rather than to a dead harness.
2026-08-14 15:10:52 -05:00
beardthelion 5c0d6f3252 fix(core): refuse a redirect that rewrites the signed request-target
may_follow compared host, port and scheme only, so a same-origin hop that
normalized a trailing slash or a query was followed. The signature binds
@path as the client sent it and the node rebuilds it from the URI it
received, so that hop left the signature covering a target nobody asked
for and the read 401d.

Add the request-target clause and repoint the seven matrix rows that rode
on a path change, so each keeps pinning the host or port property it
exists for rather than going false on the path alone.
2026-08-14 15:10:44 -05:00
Vasanthdev2004 1f0d9ff368 fix(node,infra,macos): make the owner-push opt-out reach every launch path
The previous round forwarded GITLAWB_ENFORCE_OWNER_PUSH in three compose
templates. Two of those are not the file their runtime actually reads, so the
documented rolling-upgrade opt-out did not work where it mattered.

Existing AWS instances could not opt out
-----------------------------------------
`aws_instance` ignores user_data drift, so an instance keeps the compose file it
was created with, and the published SSM upgrade only ran `docker compose pull &&
up -d`. Compose passes only the variables named in a service's `environment:`
block, so a key added to the template never reached a running instance: an
operator following our own instructions wrote GITLAWB_ENFORCE_OWNER_PUSH=false
to /opt/gitlawb/.env, upgraded, and got the new default anyway — silently, with
their CI pushers refused.

The upgrade document now reinstalls the rendered compose file before restarting.
It is idempotent (identical bytes on an already-current instance) and it makes
the upgrade a migration rather than a restart. compose.yaml is Terraform-owned
and overwritten; per-instance settings belong in .env, which is untouched.
infra/aws/README.md documents that `terraform apply` must precede the upgrade,
since the SSM document embeds the rendering.

The packaged macOS app read a different compose file
-----------------------------------------------------
The bundled Resources/docker-compose.yml is not what a packaged app runs: with
no source checkout selected, `generateComposeFile()` writes its own template,
whose environment block did not carry the key. `Config.writeEnvFile()` also
regenerates .env on every start, so a hand-added value was discarded. Between
them, a packaged-app user had no way to opt out at all.

`enforceOwnerPush` is now a persisted app setting, emitted into .env on every
start, and `generateComposeFile()` forwards it with the same interpolation as
the bundled resource — one definition, three consistent sites.

A test inherited the environment it was supposed to assert about
-----------------------------------------------------------------
`default_config_refuses_a_non_owner_push` built state through `test_state`, which
parses `Config` from the process environment — 47 fields come from GITLAWB_*
variables. On a host exporting GITLAWB_ENFORCE_OWNER_PUSH=false, which is exactly
what the rolling-upgrade guidance tells operators to set, it constructed a
disabled state, let the push through to git, and failed for a reason unrelated to
the code under test.

Renamed to `enforced_owner_push_refuses_a_signed_non_owner` and given an explicit
config through a new `test_state_with`, so it states the configuration it is
about. Verified by running it both with and without the variable exported. The
shipped default keeps its own env-independent proof off the parser declaration,
in config::tests — a parser question and an authorization question should not
share a failure mode. `test_state` now carries a warning about the hazard, since
any of those 47 fields can skew a test the same way.
2026-08-14 23:35:50 +05:30
Vasanthdev2004 2f0174cc04 fix(node): close review findings on the owner-push default
Every item below was verified against the code before changing it; two of them
were confirmed by running the probe the reviewer described.

A test the fixture sweep missed, and that could not fail
--------------------------------------------------------
`f1_write_cap_is_inert_without_a_resolvable_source_key` seeded `z6f1none` and
pushed as `z6MkF1NoKeyPusher…`, so the owner gate rejected it before it reached
the per-caller cap the test exists to measure. Its assertion was
`!matches!(r, Err(Overloaded(_)))`, which a `Forbidden` satisfies, so it reported
ok while measuring nothing — and it is not `#[cfg(unix)]`, so it did that inside
every green run. Confirmed by inverting the assertion to expect `Forbidden`: it
passed, proving the request stopped at the gate.

The pusher is now the owner, and the assertion is positive: reaching git on the
missing on-disk repo (`Err(Git(_))`) is proof the request cleared both the gate
and the cap. The sibling `f1_write_cap_key_is_per_source_not_per_repo` carried
the same weak negative in its second assertion and is tightened the same way.

The default was barely tested, and what tested it read the environment
---------------------------------------------------------------------
`Config::parse_from` consults the process environment, so on a host exporting
`GITLAWB_ENFORCE_OWNER_PUSH=false` the old `enforce_owner_push_defaults_to_true`
reported the operator's setting rather than what this crate declares — it failed
outright, which would also break any contributor who sets the variable. It is
replaced by two tests that are env-independent and behavioural:

  * `enforce_owner_push_is_declared_true_independent_of_the_environment` reads
    the declared default off `Config::command()`.
  * `default_config_refuses_a_non_owner_push` drives the handler on an untouched
    `test_state` config and asserts 403 — authenticated by a real RFC 9421
    signature, so 403 rather than 401 is what makes it a statement about
    authorization.

Both were checked by reverting `default_value_t` to `false`: the handler test
reports 500 where it wants 403, and the declared test reports `["false"]`.

The cross-tenant test lost its second tenant
--------------------------------------------
The sweep collapsed `u1_parked_push_does_not_shed_another_pusher_behind_the_same
_ip` onto one identity, leaving the test named for cross-tenant behaviour with a
single tenant and two docstrings describing a scenario it no longer ran. `n2` is
now owned by `z6u1nat2` and its push signs as that owner, so two distinct tenants
share one edge IP again — which is the property under test, since the per-source
key is the resolved peer address and never the DID.

Four comments outlived their premise
-------------------------------------
`repos.rs` (3 sites) and `.env.example` still said owner enforcement defaults
off. The IP-keying rationale they support is unchanged and still correct — a DID
farm defeats a DID key — so only the dead premise is rewritten: the cap is
load-bearing because the rate limiter bounds arrival rate rather than occupancy,
because the gate can be turned off for a rolling upgrade, and because one owner
can open many concurrent pushes to their own repo.

Operators on the container path got none of this
-------------------------------------------------
`GITLAWB_ENFORCE_OWNER_PUSH` is now forwarded in `docker-compose.yml`, the macOS
app's bundled compose, and `infra/aws/compose.yaml.tftpl`, defaulting to `true`
to match the node. Without it the rolling-upgrade opt-out this PR documents in
three places was unreachable for anyone running compose.

A cutover this project never promised
--------------------------------------
The README paragraph announcing a 15 September 2026 flip of
`GITLAWB_REQUIRE_SIGNED_PEER_WRITES` is removed. That flag is not changed here,
the date appears nowhere on main, and a PR about a different flag is not where
that commitment would be made. The table row for `GITLAWB_ENFORCE_OWNER_PUSH`
carries the part that belongs.

A docstring that credited the wrong thing
------------------------------------------
The field and test docs claimed the value-taking action is what makes
`GITLAWB_ENFORCE_OWNER_PUSH=false` resolve to `false`. It is not: the env form
resolved under the presence-only declaration too. What the change buys is
`--enforce-owner-push false` parsing at all, rather than erroring as an
unexpected argument. Both docstrings now say so.
2026-08-14 20:43:43 +05:30
Vasanthdev2004 7de6f7cf5d test(node): derive the P2 pusher from the repo it targets
`p2_push` signed every push as one shared `P2_PUSHER` constant, but its two
callers own different repos — `z6p2tail` and `z6p2fail`. Pinning the constant to
either owner leaves the other rejected by the owner-push gate, so the identity
now follows the `owner` argument the helper already takes, the same way the
burst test derives it.

The rejection surfaced as `async fn resumed after completion` rather than a 403:
a denied push completes on the first poll, `p2_logged` never observes
`receive-pack`, and the polling loop then polls the finished future again.
2026-08-14 18:31:00 +05:30
Vasanthdev2004 586896d8b2 test(node): push as the repo owner in the remaining lease and scan fixtures
Fourteen concurrency tests seed a repo through `f4_state_with_repo` or
`p2_parked_release_state` and then push under an unrelated DID, which the
owner-push gate now rejects before the machinery each test asserts on is ever
reached. They are transport and lease tests, not authorization tests, so each
now signs as the owner of the repo it targets.

Two needed more than a swapped constant:

- `receive_pack_burst_scans_serialized_and_both_pushes_succeed` pushes to two
  repos with different owners, so the identity is derived per push from the
  owner already threaded through the closure.
- `u1_parked_push_does_not_shed_another_pusher_behind_the_same_ip` used three
  distinct pusher DIDs, which owner-only push makes impossible. The distinction
  was narrative: the per-source key is the resolved peer IP and never the DID,
  so three concurrent pushes from one edge IP contend exactly as before. The
  comment now says so rather than leaving the reader to infer it.

The gate keeps its own coverage in `enforced_rejects_non_owner_with_forbidden`
and `enforced_rejects_missing_did_with_forbidden`, so nothing here removes the
assertion that a non-owner push is refused.

These tests are `#[cfg(unix)]`, so a Windows checkout cannot run them; they were
missed in the previous round because the Windows suite runs 717 of the 825 tests
Linux runs.
2026-08-14 18:00:36 +05:30
Vasanthdev2004 cdbda19c29 fix(node)!: enforce owner-only push by default
A `did:key` signature is authentication, not authorization: the method is
self-certifying, so any party can generate a keypair, derive its DID and sign.
With `GITLAWB_ENFORCE_OWNER_PUSH` defaulting to `false`, `owner_push_rejection`
short-circuited and every `git-receive-pack` carrying any valid signature was
accepted — including pushes to a repository the signer does not own, and
including private ones, on every branch not explicitly protected.

The gate itself already worked when switched on. What was missing is that no
reachable default configuration switched it on, so a node started with nothing
configured accepted a push from anyone. That default is what this changes.

The argument now takes a value so there is a way back: `--enforce-owner-push
false` and `GITLAWB_ENFORCE_OWNER_PUSH=false` both disable it, while the bare
`--enforce-owner-push` form still means `true`. Without this the flag would be
presence-only (`ArgAction::SetTrue`), which with a `true` default would leave
operators no way to opt out during a rolling upgrade.

Three concurrency tests pushed as non-owners in order to reach the shedding
logic they actually assert on. They now push as the repo owner, so they keep
exercising the shipped default rather than opting out of the gate.

Docs updated in README.md, .env.example and docs/RUN-A-NODE.md, including the
delegated-key caveat below. README also carries a dated cutover for
`GITLAWB_REQUIRE_SIGNED_PEER_WRITES`, which stays `false` for now so live peers
can finish upgrading.

BREAKING CHANGE: `git-receive-pack` now rejects a push whose authenticated DID
is not the repo owner, returning 403 before any ref update is applied.
Delegated and CI keys count as non-owners: a UCAN `git/push` capability is
verified but not yet honored for authorization, so an agent pushing under its
own DID cannot push while this is on. Set `GITLAWB_ENFORCE_OWNER_PUSH=false`
during a rolling upgrade, or have automation push as the repo owner, until
scoped collaborator / UCAN-delegated push rights land.
2026-08-14 17:22:23 +05:30
Vasanthdev2004 5a7a77b8dc fix(node): gate two Unix-only tests so the crate compiles on Windows
Two tests build a fake git as a `/bin/sh` script, mark it executable through
`std::os::unix::fs::PermissionsExt`, and reap the hung `rev-list` with
`libc::kill(SIGKILL)`. None of that exists on Windows, and neither test was
cfg-gated, so `cargo test -p gitlawb-node` failed to compile there with six
errors before running anything. A Windows checkout could not run a single test
in the crate, including the ones that are platform-independent.

Gate both with `#[cfg(unix)]`. The attribute is a no-op on Linux, so CI keeps
running them exactly as before; only the Windows build changes, from "does not
compile" to "runs the platform-independent tests".

Refs #228
2026-08-14 17:22:23 +05:30
beardthelion c82a70ab21 fix(core): put the redirect predicate behind a feature so core stays lean
The dependency allowlist caught this: sharing the predicate pulled url, and
behind it idna and the icu crates, into the closure of the one crate that is
supposed to stay embeddable. Both clients that need the predicate already parse
URLs, so they opt in and nothing else pays for it.

The test dependency is not optional, so the origin matrix still runs under a
bare cargo test -p gitlawb-core. Gating the module on the feature alone would
have left those tests silently unbuilt, which is the failure the module is
there to prevent.
2026-08-13 18:55:58 -05:00
beardthelion ba44a018ec fix(core): share the origin-scoped redirect rule with the git helper
The previous commit stopped the CLI carrying request signatures to a host the
node redirected it to, and left the other signing client alone. That one is the
git remote helper, which is what actually runs on clone, fetch and push, and it
had no redirect policy at all. A push signs from its first request, so a node
answering 302 handed out a working credential, and on a 307 or 308 the pack
body went with it. The rule now lives in core and both clients use it, so the
two cannot drift apart again.

Three guards were not holding what they claimed. Bounding the retry wait by the
remaining deadline could be deleted with every test still green, which the
timing shows was four real seconds of overshoot past the bound the docs
advertise; the assertion was loose enough to pass either way and is now tight
enough to fail. The refusing-gateway warning was asserted through a test-only
mirror rather than the write the user sees, so deleting the write kept the test
green. And both no-hint exceptions, the definitive not-found and the wrapped
scan, were only ever reached without a token in hand, which is the one state
where they do nothing.

A response whose body failed partway through the read still replayed the held
token for every remaining attempt, spending the whole ladder on one position and
then reporting that the object might lie further on. A body cut by the read cap
was already terminal for that reason; a body cut by a broken connection is the
same unreadable state and is terminal now too.

A successful fetch no longer buffers the whole object before writing a byte of
it, so a node cannot make the client hold an arbitrary amount of memory inside a
time limit that was never a memory limit. The remaining silent skips in the
recovery loops now say what went wrong, including the manifest fetch and the
body reads, so a failure there is reported rather than looking like an absent
blob.

Also: the redirect chain bound is off by one against reqwest's own, has a test,
and its comment no longer claims to prevent something the request timeout
already prevents.
2026-08-13 17:34:14 -05:00
beardthelion de3f230940 fix(gl): stop carrying request signatures to a host the node redirected us to
The node client followed redirects with no policy, and reqwest strips only the
classic credential headers when a redirect crosses origins. RFC 9421 signature
headers are not on that list, so a node answering 302 could hand a caller's
signature to another host, which then served whatever bytes it liked for a
content address that never hashed to them. Following continuations made it
worse by turning one signed request per invocation into as many as nine. The
signature covers the method, the path and the body digest but not the
authority, and the node accepts any signature inside a five minute skew with no
nonce ledger, so one harvested header reads a caller's private paths at any
node in the fleet.

Redirects are now scoped to the origin that issued them. Same-origin ones still
follow, so an http to https upgrade or a trailing-slash normalization keeps
working, and a cross-origin one stops rather than errors, so the caller sees
the 3xx through the status path it already has. A downgrade back to http on the
same host is refused too: the origin matches, but the signature would leave in
cleartext, which is the same leak by a slower route. A custom policy replaces
reqwest's built-in chain limit, so the limit is restated rather than lost.

The recovery path in clone had the same silence the CLI just lost: it skipped a
blob on any non-success without saying why, so a truncated scan, a permission
denial and a dead gateway were one indistinguishable outcome. It now names the
status or the transport error, capped and stripped of control characters since
the text comes from a caller-chosen gateway, and continues as before.

Reading a response body no longer conflates three endings. The cap cutting the
body, the body finishing, and the read failing partway were all one silent
break, so a failed read surfaced as an error message that stopped after the
colon. Each is now reported, and a body that could not be read says so.

Coverage for three paths that were previously argued rather than run: a
transport failure mid-ladder still hands back the resume token, a scan that
ends before any ceiling still answers a definitive not-found, and a response
whose body stalls after its headers is still ended by the client's own
deadline.
2026-08-13 16:10:04 -05:00
beardthelion 8c39cfdcb3 fix(review): keep the resume token on every terminal that still holds one
Review of the scan-continuation work found the ladder surfacing its token on
three terminals and dropping it on three others: a mid-ladder 5xx that is not
an overload, a truncation whose offered continuation fails validation while a
good token is still held, and a transport error. In each case the user is told
to start over, and a bare re-run restarts at the first row and re-spends the
node's per-IP budget, which is the failure the token exists to prevent. It is
surfaced on those paths now, still never on a definitive 404 or on the node's
deliberate scan-wrapped signal, and a rejected token is still never echoed
back.

A body cut at the read cap used to change the answer. Truncated mid-JSON it
failed to parse, lost its error code, and fell to the transient-overload arm,
which retried the old position for every remaining attempt and then reported
that the object might lie beyond the rows scanned when nothing past the first
page had been. The cap now reports that it cut, and a cut body is terminal.

Three guards were doing nothing. Mutation testing showed the Retry-After clamp
could be deleted with every test still green, so a node advertising a day-long
wait would have parked the client inside a single sleep that neither the
attempt cap nor the deadline can interrupt. The rejection of a malformed
caller-supplied token was equally untested, though the same property is proven
on the node-offered side. The withheld-blob ladder asserted only that each rung
was not a success, which any early exit satisfies without the rule ever being
evaluated; it now witnesses that the scan reached the end of the table.

The inter-attempt wait is bounded by the time left on the deadline, so a run
can no longer overshoot the bound it advertises. That drops the wait out of the
worst case entirely rather than shrinking it: the last attempt starts before
the deadline and the client's total timeout covers that request through the end
of its body, so a run is over by the deadline plus that timeout. The docs said
about 95 seconds and now say about 90.

They also said a large download already streaming is never cut off part way.
That was wrong. The scan deadline does not cover the download, but the client's
timeout is a total request deadline running until the body finishes, so a slow
transfer is cut off. Both prose sites now say what the code does.

Also: stdout is flushed before the command returns, so a closed pipe fails
loudly instead of yielding a truncated object with a success status.
2026-08-13 15:09:45 -05:00
beardthelion eac4d43c67 docs(gl): describe how gl ipfs get resumes a truncated scan, and its limits
Document the behaviour rather than the mechanism: that an object pinned before
the node recorded its repo is found by scanning the inventory, that the scan
stops at the node's ceilings, and that the command follows the resume token the
node hands back instead of reporting a false not-found.

State the bounds in values, not constant names, and state honestly what each
one covers. The deadline bounds the search: every attempt gets the time left on
it to produce response headers, and it deliberately does not cover downloading
an object once found, so a large blob already streaming is never cut off. The
error-body read runs outside it under the client's own timeout, which is why
the longest a single run can take is about 95 seconds rather than the flat
deadline, and the in-code comment now derives that instead of asserting it.

Say what is not promised, too. A rate limit ends the ladder because its window
is an hour and its Retry-After cannot be honored inside one invocation, and the
node's per-IP fanout brake can end a ladder well short of the cap, so automatic
resumption is not a guarantee that the object will be reached.

Where a bound stops the ladder, the printed invocation is the only thing that
makes progress: a bare re-run restarts at the first row, reproduces the same
truncation and spends the per-IP budget again.
2026-08-13 14:11:59 -05:00
beardthelion 181f6178a0 fix(gl): accept the resume token the incomplete-scan message tells you to pass
The bounded terminals printed "gl ipfs get <cid> --scan <token>", but the
command took no --scan argument, so following that advice failed on an
unexpected argument and the resume input existed only behind a private seam.
The flag now exists and forwards to the same validation a node-supplied token
gets, which is what makes surfacing the token worth anything: without it a
stopped ladder restarts from the front on every rerun.

Three test fixes ride along, all found by running the mutation matrix rather
than by reading.

The deadline scenario asserted its message "must name the deadline" while its
fixture CID was bafkreideadline, so any message echoing the CID satisfied it.
Under a mutant that removed the deadline the ladder ran to the cap and that
assertion still passed. The CID is renamed so the assertion can only pass when
the message genuinely names the deadline, and the mutant now reddens on it
instead of on the call count.

The default classification arm was certified only for the token-less case: no
fixture reached it with a token already held, so degrading it to retry was an
equivalent mutation against the whole suite. A scenario now drives an
unrecognized code mid-ladder with a token in hand and asserts it is terminal.

The wired path from the argument to the resume input had no coverage; only the
seam was tested. A scenario now goes through the function clap dispatches to,
so dropping the argument turns it red.
2026-08-13 14:02:34 -05:00
beardthelion 9a8d190492 fix(gl): follow scan continuations in gl ipfs get, bounded three ways
A truncated legacy CID scan answers 503 with a sealed continuation token, and
the command turned that into a hard error, so content past a scan ceiling was
unreachable from the first-party client. It now follows the continuation
automatically.

Three bounds, each protecting a different resource. The attempt cap bounds the
work asked of the node. A wall-clock deadline bounds the caller's own time and
is passed into each request as that request's timeout, so a slow-but-alive node
cannot outlast it. The per-wait clamp bounds a single Retry-After, so a hostile
value cannot stall the client. There is deliberately no non-advancing-token
check: tokens are nonce-randomized ciphertext, so two seals of one position are
never byte-equal and a client cannot detect a wedged ladder by comparing them.

Classification happens at one site with no wildcard to retry. Status is checked
before the body is read. A rate limit is terminal on status alone, since its
window is an hour and its Retry-After cannot be honored inside one invocation.
An overload mid-ladder is retried on the still-held token, because that shed
releases its permit when the request ends and the node advertises a retry. Any
other status or code is terminal through the default arm, including an overload
on the first request, where no token is held and there is nothing to resume.

Every terminal message is capped and sanitized before it reaches the terminal,
and a rejected token is never echoed back; the bound is named instead. Tokens
are accepted only as non-empty base64url within a generous ceiling, so a future
change to the sealed layout cannot silently disable resume.

Where a bound stops the ladder, the still-held token is printed with the
invocation that resumes from it, rather than wording that implies a bare re-run
makes progress.

gl suite 332 green, fmt and clippy clean.
2026-08-13 13:47:41 -05:00
beardthelion 1d37cb1ce9 test(gl): cover the scan-continuation resume ladder in gl ipfs get
The node answers a truncated legacy CID scan with a 503 carrying a sealed
continuation token, and gl ipfs get turns every non-success into a hard error,
so content reachable only past a scan ceiling cannot be fetched with the
first-party client at all. The old test asserted that terminal 503 as the
desired behaviour, and its fixture body predates the token, so it is replaced
rather than adjusted.

Thirteen scenarios pin the contract: a resume that completes, the attempt cap,
a wedged server that keeps minting fresh tokens for the same position, a
missing or malformed token, a first-request overload, the signed resume, the
wall-clock deadline, a mid-ladder rate limit that is terminal, a mid-ladder
overload that is retried, the token length boundary pair, the token surfaced at
every bounded terminal, and a caller-supplied continuation that starts from the
token rather than the front.

Hostile-input scenarios assert the surfaced message is capped and stripped of
control and bidi characters and that a rejected token is never echoed back.
Ladder fixtures answer Retry-After: 0 so the clamped waits are zero and the
whole gl suite still runs in under a second in real time.

The seam signatures land here so the tests compile and fail on their
assertions: cmd_get delegates to cmd_get_inner, which threads a deadline, an
attempt cap, and an optional starting continuation. Its body is unchanged and
still bails on any non-success, so all fourteen new tests fail on the property
each one names.
2026-08-13 13:13:29 -05:00
beardthelion 21e6873683 fix(node): bound the legacy scan's query by the remaining row budget
GITLAWB_IPFS_MAX_LEGACY_SCAN_ROWS accepts any value from one upward, but the
pager always asked the database for a full page and the ceiling was only
checked afterwards, in the scan loop. With the ceiling below the page size an
anonymous request still selected and rule-loaded a whole page while holding
walk admission, and any ceiling that is not a page multiple overshot by up to
a page. Compute the remaining budget first and ask for min(page, remaining),
so the page size stays a batching detail rather than an implicit floor under an
operator ceiling.

The exhaustion inference moves with it. A short page used to prove the table
was exhausted; once the limit can be shortened by the budget it proves nothing,
and marking the pager exhausted breaks at the top-of-loop arm that sits ahead
of every ceiling arm, taints nothing, and mints no continuation. That path ends
in a definitive 404 for content that exists, so the predicate now compares the
returned rows against the limit actually sent.

The budget computation sits above the timeout opener so the deadline-wrapping
guard's lookback window is unaffected, and the limit is read from the AppState
ceiling the row-ceiling arm reads, not the config field, so the arm and the
query cannot disagree.

Full suite 1059 green, fmt, clippy, and --locked clean.
2026-08-13 12:39:05 -05:00
beardthelion a60026bbfa test(node): pin the legacy scan's DB-facing row selection to the configured ceiling
The row ceiling is enforced in the scan loop, after a full page has already
been fetched and rule-loaded, so a cap below the page size is ignored for the
first fetch and any cap that is not a page multiple overshoots by up to a page.

The existing coverage could not catch that. Both row-ceiling tests asserted
rows <= ceiling + one page, which passes whether or not the query is bounded,
and rows-returned cannot distinguish a bounded query from a full fetch that is
trimmed afterwards. Add a scan_limit seam recording the limit actually sent to
the database, tighten both tolerant assertions to exact counts, and add two
tests that pin the limit: a ceiling below the page size and a ceiling that is
not a page multiple.

Also add the fail-open fixture. A repo evaluated with a partial rule set is
served as if it had no rules, so the boundary repo is seeded public at root
with a path-scoped rule and a real blob on disk, pinned with NULL provenance so
the scan path's rule map is the one consulted, and asserted never served across
its own ladder. Its positive control serves the same blob with the rule absent,
so a failure there is attributable to the rules rather than to a probe or a
budget skip.

Both new tests fail against current code on the scan_limit assertion.
2026-08-13 11:54:41 -05:00
beardthelion 75c0e282af test(node): probe the production advisory-lock key, not a stale copy
Both write-lock tests hand-copied advisory_lock_key's derivation. The copy
still used DefaultHasher, so once the key moved to SHA-256 (#210) the probe
checked a key nobody held. The disconnect test then failed asserting the
lock was held, and the success test passed asserting a release it never
actually observed.

Import the production function, as the acquire-deadline test already does,
so the two cannot diverge again.
2026-08-13 09:55:24 -05:00
beardthelion f4f0f4bff0 Merge remote-tracking branch 'origin/main' into fix/issue-135-ipfs-cid-tree-gate 2026-08-13 09:44:35 -05:00
beardthelion e6cfb588c2 fix(review): bound token key, rule bytes, and the discovery load
An external review found three costs that outlive the request or the boot
that paid for them.

The scan token was sealed with a key minted per boot, so a restart or a
rolling deploy made every outstanding continuation undecryptable and the
handler reads an unopenable token as absent. A caller walking a large
inventory could be sent back to the front indefinitely. The key is now
derived from the node's Ed25519 seed with HKDF, so it is the same after a
reload, under a distinct salt and info string so it is not the signing
key and cannot be turned against signatures. Rotation is a version bump in
the info string. An earlier comment rejected derivation on the grounds
that surviving restarts bought nothing; that is exactly what it buys, and
the comment now says so.

The rule byte ceiling was summed after the page had already been fetched,
so it truncated the request without bounding the work: the allocation and
the transfer had happened. The budget now bounds the query itself. The cut
lands on a repo boundary, which is not cosmetic: a mid-repo cut leaves a
partial rule set, and a partial rule set at the gate is indistinguishable
from a repo with no rules, so it would fail open and serve an object those
rules withheld. Only fully loaded repos are gated. The query always admits
the first rule-carrying repo of a page, so the bound is the budget plus
one repo rather than the budget; without that a single oversized repo
would put the cut at the cursor and every retry would reproduce the same
page.

The old check carried a not-exhausted condition so a short final page
could not turn a complete scan into a permanent shed. That condition is
gone because the taint no longer keys on page length; it keys on the query
having left repos unloaded. A short final page that fits produces no cut
and still 404s, and one that is cut is honestly incomplete and resumable.

Discovery paged the whole repo table and stat'd every warm repo to pick a
sixteen candidate window. Its comment called that acceptable because the
cost was paid once, which was true when the sweep ran once per boot. This
round made the sweep periodic, so an upgraded node with one unrepairable
row paid a full table scan on every run, forever. The load now stops once
the window is full. It collects one candidate past the cap so that running
out still distinguishes a warm set that fits from one that does not, which
is what the reset arm needs now that the total count is no longer known.
Ordering is unchanged, so an attacker who mints repos still cannot move
the boundary.

Verified by mutation: a per-boot key strands a continuation across a
restart, using the signing seed directly defeats the domain separation,
summing bytes after the fetch materializes the whole page, paging to
exhaustion buys pages the window cannot use, and reading the fit under cap
from the window length loses the reset.
2026-08-12 23:38:15 -05:00
beardthelion 342dd48d24 fix(review): mint a continuation at every ceiling and widen the token
Code review found two ways the scan ladder still dead-ends, both the same
shape the round has been fixing: a truncation that sheds without a token,
which this design reads as "the ladder is over", so the caller restarts at
the front forever and everything past that point is unreachable.

The continuation was wired into the row and rules ceilings but not the
probe and visit ceilings, and those two bind first on any inventory with
root-readable repos, long before the row ceiling. Every fixture seeded
root-denying repos, so no probe was ever charged and no test could reach
the arms that were missing it. All four breaks stop at the same cursor, so
all four now carry it.

The token's field width was 128 bytes against a repo id the write path
admits at 356 (255 owner, a slash, 100 name); a plain full DID with a long
name already passes 128. A repo over the width at a truncation boundary
failed the seal and shed tokenless, and repo names are peer-controlled, so
the boundary was steerable rather than unlucky. Fields are now sized
separately, 64 for the timestamp key and 384 for the id, and the version
byte moves so older tokens fail open to absent instead of mis-parsing.
Tokens stay constant length at 668 characters.

The sweep no longer exits for good on a failed pass. Making it periodic
was the point of the round, and returning on the first error meant one
deadlock disabled repair until the next boot, silently. It re-arms on a
longer interval, escalates the log once failures are consecutive, and
backs off again when a run repairs nothing so a table that will never
repair stops costing 64 object reads every five minutes.

The rule ceiling counts retained bytes instead of rule rows and is checked
where the rules are actually fetched, since a page can carry unboundedly
many rules and rule count is the wrong unit for a memory bound. It skips a
short final page: without that, a complete scan of an absent object turns
into a permanent 503.

Two docs contradicted the code and are corrected: the walk cap is charged
per phase now, so a request can spend twice the knob, and the README's
retry bound holds when the row ceiling binds.

Verified by mutation: reverting either ceiling's continuation strands the
holder, narrowing the token width fails the seal at the boundary,
restoring the terminal return stops the sweep, counting rules instead of
bytes lets a page overshoot, and removing the idle backoff keeps a
fruitless table under the base interval.
2026-08-12 19:12:24 -05:00
Kevin Codex 96d8123e0f Merge pull request #320 from euxaristia/fix/like-prefix-escaping
fix(node): Escape LIKE wildcards in certificate prefix search
2026-08-13 06:22:26 +08:00
Kevin Codex c380f9cee5 Merge pull request #215 from Gravirei/fix/issue-210-stable-advisory-lock-key
fix(repo_store): use SHA-256 for stable advisory-lock key (#210)
2026-08-13 06:21:50 +08:00
beardthelion 0d7711c089 fix(node): bound the pre-walk CID resolve on its own short budget
get_by_cid takes the global walk permit and the per-source permit before
resolving the CID, and that lookup was clamped only by the 600s request
budget. A well-formed CID with no pinned row does no probe and no walk,
but against a stalled pool it held a scarce walk slot for the full
window, so requests from enough distinct sources could reject every real
retrieval at admission while nothing was walking.

The lookup now runs under its own budget, defaulting to ten seconds and
clamped by the request budget, so a larger value degrades rather than
extends. Admission still comes first. Resolving before admission was the
other way to fix this and it trades one amplification for another: taking
the read out from behind the gate lets unadmitted anonymous callers stack
concurrent queries, and shedding a flood cheaply before it reaches the
database is what admission-first is for.

Only that one await moves. Everything after it stays on the request
budget because it runs interleaved with work the caller has already been
admitted for: the per-oid source lookup runs after real probing from the
second candidate on, the marker pair only on a provenance miss, and the
pager fetches between pages that cost probes. A short clock anchored at
admission would be long spent by then, so widening it would shed
requests that are making progress. The region comment now carries that
assignment per await.

The shed names the knob an operator can actually turn, separately from
the request-budget shed, so the two are distinguishable in the body and
the log.

Verified by mutation: reverting the clamp overruns the short budget, and
anchoring one short clock across the whole permit-held region sheds a
slow but progressing walk. The second leaves the first test green, which
is what makes it a boundary case rather than a second removal.
2026-08-12 16:59:02 -05:00
beardthelion ae96ffa467 fix(node): budget provenance and fallback walks separately
MAX_HISTORY_WALKS_PER_REQUEST is MAX_PIN_SOURCES + 1, which is exactly
enough for the first pinner plus every recorded additional source. Those
seventeen sources each spend a walk reaching their deny when they are
root-readable but path-scoped, so the walk budget is gone by the time the
at-cap fallback reaches a public source that record_pin_source dropped at
its sixteen-row cap. If that source needs a walk too it is skipped at the
ceiling and the request returns 503 search_incomplete on every retry: a
public object nobody can ever fetch.

The counter is now per phase. gate_and_serve already knows which phase it
is in, so provenance and fallback each get their own budget against the
same cap, and the fallback's capacity no longer depends on what the
provenance phase spent. Total walk work per request stays bounded at
twice the cap, a constant, so the amplification story holds.

Raising the shared ceiling was the obvious alternative and does not work:
the adversary picks how many provenance slots exist, since they are
grindable repo ids filling pin_repo_sources, so cap + N denials recreate
the exhaustion for any constant N.

The test uses real repos with real path-scoped rules that each spend a
real walk, proven by the walk log. The existing buried-source test seeds
repos that do not exist, which never enter the walk block at all, and
that is why it could not see this.

The must-not assertion needed changing to mean anything. A CID is
content-addressed, so every holder of it has byte-identical content and
comparing the served bytes against the public repo's object cannot show
which repo served. The gate is pinned instead by a second object held
only by a path-denying repo, with the fallback armed for it, asserted to
404 for an anonymous caller.

Verified by mutation: re-merging the counters restores the deterministic
503; capping the fallback at what the provenance phase left over is a
reserve that still couples them and fails the same way; and dropping the
fallback cap entirely unbounds the total. Each reddens exactly one of the
two tests, so neither is standing in for the other.
2026-08-12 16:27:23 -05:00
beardthelion 22f4db94b2 fix(node): bound the legacy CID scan and let the caller carry its position
A denial-only inventory paged through the whole repo table. gate_and_serve
returns Skip on quarantine and on the root visibility deny before either
walk.probes or walk.visits increments, and those two counters were the
pager's only stop conditions, so an anonymous caller holding a CID from
the public pins index could read every row and its rules while the global
IPFS walk admission stayed held for the request budget.

Three parts, and the shape of each is set by a way an earlier attempt was
wrong.

A row ceiling, with a rules companion. Rows scanned bounds the DB reads
and the pager's row retention; it does not bound rules, since the rules
query has no LIMIT and an owner picks how many rules their repos carry, so
that gets its own ceiling. Both taint into the existing retryable 503 and
never a 404.

A ceiling alone would stop there and strand content. The scan starts from
a NULL cursor over (created_at, id) ASC, so a bare ceiling cuts the same
oldest prefix on every request and a holder past it is unservable for
good, which is the defect this PR already carries a finding about. So the
503 hands back the position. Server-side alternatives were tried and
dropped: a node-global cursor lets concurrent callers skip each other's
windows, and a per-caller map lets a key farm evict an honest caller
mid-ladder. State the caller carries has neither problem.

The token is sealed, not encoded. It names a row the caller may not be
allowed to see, and id derives from the owner DID, so plaintext would leak
a withheld repo's existence, creation time and owner. XChaCha20-Poly1305
with a fresh nonce per seal, the canonical CID as associated data, an
expiry inside the sealed bytes, and a fixed-width plaintext so token
length cannot vary with the row it names. Undecryptable, expired,
malformed and foreign-CID tokens are one indistinguishable case: absent,
start at the front.

The wrap taint keys on the pager reaching exhaustion while resumed, not
on any break site, so a token pointing at the last row still sheds a
retryable 503 instead of falling through to the 404 tail.

Pages are charged to the per-IP work bucket. Without that the ceiling
bounds one request and nothing across requests, and the denial path pays
nothing, which is the per-request-free-allowance shape already rejected
once on this branch. The derived work-budget floor grows by the same page
term so a deep scan cannot self-throttle mid-ladder.

A toll shed renders 429 rather than 503, which reverses the tail's
documented precedence for this one case. The tail comment names the
exception: the caller's own bucket stopped the scan, not the node's
search, and a 503 would invite an immediate retry into an empty bucket.

Four existing tests hand-size the work limiter to their exact probe count
and now also pay for a page; each gained one token and a comment showing
the arithmetic. Production is unaffected because the floor grew by the
same term.

Verified by mutation: removing the ceiling unbounds the scan; truncating
without tainting turns it into a 404; emitting a token but never resuming
never reaches the holder; dropping the toll makes pages free; encoding
the token as base64 plaintext leaks the withheld fields; dropping the CID
from the associated data honors a cross-CID resume; a variable-width
encoding makes token length vary with the row; and a constant nonce makes
two seals identical.
2026-08-12 15:49:51 -05:00
euxaristia 3993fd1f5d fix(core): count distinct signer DIDs in certificate threshold check
satisfies_threshold counted raw signature entries against the maintainer
list, so a certificate carrying the same maintainer's signature duplicated
across two entries satisfied a 2-of-N threshold with only one real
signer. Dedupe by collecting valid signers into a set before comparing
against the threshold, so each maintainer counts once regardless of how
many times their signature appears in the certificate.
2026-08-12 15:44:06 -04:00
beardthelion 9babb5ece0 fix(node): record a discovered source and its marker in one commit
Discovery recorded the source and armed the fallback in two independent
best-effort writes. If the insert landed and the marker did not, the row
was left raw CIDv1 with a nonempty, below-cap, unmarked source set. Later
sweeps skip such a row, and get_by_cid reads that set as complete and
suppresses the fallback scan, so once the transient error cleared, a
public duplicate of an object whose recorded holder is private stayed
unreachable for good.

record_discovered_pin_source does both writes in one transaction: the
capped source insert, the gated failure-row delete, and the sentinel that
arms the fallback. The sentinel is unconditional because one holder found
by a bounded warm-only probe never proves the set complete.

The method ends in an explicit commit, so an elapsed deadline is a
definite non-write rather than an unknown. A failed record still leaves
an empty source set, which the resolver's own empty-set signal routes to
the fallback, so the degradation posture is unchanged and now has a test.

The fault injection is a BEFORE INSERT trigger on pin_source_failures,
not a dropped table. Dropping it would fail the delete inside
record_pin_source's own transaction, roll that back, and produce the same
empty set as the fix, so the test would pass without ever reaching the
half-state.

Verified by mutation: splitting the transaction back into two writes
404s the public copy, and dropping the sentinel leaves the fallback
unarmed.
2026-08-12 14:13:17 -05:00
beardthelion 4fdfa1cedb fix(node): rotate discovery candidates across traversals
Discovery probed the first sixteen warm candidates of a stable
(created_at, id) ordering on every pass, so an object held only by the
seventeenth repo was never repaired and every boot repeated the same
work. The probe cap bounds expensive reads, which is right; it was also
silently bounding which candidates are ever considered.

A persisted continuation now rotates the window. Four choices carry it,
each forced by a way an earlier draft was wrong:

Keyset, not offset. An offset indexes into a list whose length anyone can
change, so candidates entering below the boundary shift the window off the
holder. A key names a position that survives insertions. created_at is
server-stamped, so minted repos only ever append.

Per traversal, not per pass. The row cursor also advances per pass, so
per-pass rotation gives row R the window index (t*m + j) mod W and a row
sees only W/gcd(m, W) of the W windows. Sharing one window per traversal
makes coverage a per-row property.

Advanced only over probes that started with a live deadline. A probe
against a spent deadline is charged a read but never opens the repo, so
counting it would step the window over candidates nobody examined.

Persisted from inside sweep_pass, next to the row cursor. The run-end site
never executes: spawn_legacy_cid_sweep drops the future on SIGTERM.
Traversal state is owned by the wrapper because the dead-read cap can pause
a run mid-traversal and a later run ends it.

The sweep re-arms on a timer instead of running once per boot, so coverage
is wall-clock rather than a reboot count, and returns only on a failing
pass query. The unconditional sleep is what keeps a repaired table from
spinning.

Verified by mutation: no rotation strands the holder; an offset
continuation shifts the window; advancing to the window end both skips an
unread holder and burns an unprobed window; per-pass advancement drifts
one row's window from another's; no re-arm leaves the holder unreached; a
run-scoped accumulator applies the wrong arm after a cap pause; and
deleting the v26 entry fails the upgrade path.
2026-08-12 13:46:03 -05:00
beardthelion 22560b0872 fix(node): give each discovery row its own slice of the pass budget
DiscoveryCtx held one deadline for a whole sweep pass, so a single hung
candidate on the first source-less row spent it for every row behind that
one. Those rows then called repair_legacy_provider_cid with an expired
deadline, came back retryable without a meaningful probe, and were charged
reads for work they could not do. CID order is stable, so the same first
row consumed the budget on every boot and the rows behind it starved
permanently.

Each row now probes under min(now + git_timeout / 4, pass_deadline), so at
least four rows per pass get a live deadline and no row can take more than
a quarter. A row that starts with the pass deadline already spent returns
PassBudgetSpent: no probe, zero reads charged, retryable, and surfaced on
SweepStats so a run can say it happened. Charging those reads would burn
MAX_DEAD_ROW_READS_PER_RUN on probes that cannot succeed.

The field is renamed to pass_deadline because that is what it is:
load_discovery_ctx runs once per pass and a run loops passes. Naming it
for the run would have stated a bound the code does not hold.

The divisor is not a bound on a single probe. A row's slice is still
shared by up to MAX_LEGACY_DISCOVERY_PROBES candidates, and the doc says
so rather than implying a guarantee.

Verified by mutation: collapsing the row deadline back to the pass
deadline starves the second row, and folding the spent-budget arm into
ordinary retryable accounting charges reads that must stay free.
2026-08-12 12:20:19 -05:00
beardthelion 650460acc0 test(node): cover the pre-provenance upgrade row end to end
The sweep's source-less discovery had no fixture built from a real
pre-migration row. Every existing test seeded its pin through the modern
path, which cannot reproduce the shape an upgraded node actually carries:
a row written when pinned_cids.repo_id and pin_repo_sources did not exist
yet, so its source set is empty with no other signal.

The fixture un-applies migrations 19 and 20, inserts the provider-keyed
pin naming only (sha256_hex, cid, pinned_at), then migrates forward. That
ordering is what makes it real; seed_legacy_pin binds repo_id, the column
being proven absent, so the insert is inline.

It brackets the repair rather than asserting an end state: the anonymous
GET is driven before the sweep and asserted not to serve, then after the
sweep asserted to return the object's bytes. The marker assertion queries
pin_source_failures directly for the "" sentinel instead of reading the
pin_sources_incomplete bool, which cannot distinguish the sentinel from a
repo id that record_pin_source would later clear.

Verified by mutation: short-circuiting the sources.is_empty() discovery
arm restores the pre-fold fall-through and the test goes red on the
repair assertion.
2026-08-12 11:56:56 -05:00
Gravirei b01d17b7f3 docs(repo_store): pin colon-free owner_slug contract on advisory_lock_key
The domain separation owner_slug + ':' + repo_name has no length prefix, so
it is injective only while owner_slug cannot contain a colon. State that on
the pub(crate) function and guard it with a debug_assert, since a raw DID
would collide (('did:key:abc','x') and ('did','key:abc:x') hash the same).
2026-08-12 13:07:50 +06:00
beardthelion 4c665fbca8 fix(node): port discovery's candidate load onto the paged repo query
#173 removed Db::list_all_repos in favour of a keyset-paged
list_repos_page_for_scan, so this branch stopped compiling on rebase. One call
site, load_discovery_ctx, ported with its reviewed semantics intact.

The quarantine drop gets simpler rather than different: each page row carries its
own quarantined flag, so the separate list_quarantined_repos query is gone and
the filter runs over rows the pass already read. Private non-quarantined repos
still stay in the list. Warm-only via validated_repo_disk_path inside
spawn_blocking is untouched, so nothing here reaches repo_store.acquire and a
repair pass still cannot become a bulk restore, and an unsafe path is still
dropped with the same warn and stays terminal.

Oldest-first by (created_at, id) now comes from the query's ORDER BY (index
backed by migration v25) instead of a client-side sort, and pages concatenate in
that order, so the explicit sort_by is redundant and removed. The anti-grinding
reason for that order is unchanged and the doc comment keeps it: repo_id derives
from a grindable owner DID, so an id sort would let someone bury the true holder
past the probe cap. The probe cap still bounds expensive reads rather than
candidates considered, and no cap on candidates was added.

This pager runs to exhaustion, unlike the resolver's, which drives the same query
and stops when its probe or visit budget is spent. The comment says why: the
resolver is an anonymously reachable route holding scarce walk admission, while
this is background maintenance on a timer with no caller and no permit
contention that needs the whole warm candidate set to settle a row. Paging is per
pass, matching load_discovery_ctx's existing once-per-pass lifetime.

Refs Gitlawb/node#321.
2026-08-11 22:46:29 -05:00
beardthelion bc202368e1 fix(node): charge discovery's probes against the per-run read budget
The two halves of round 12 left a hole between them. The cursor rewind became
unconditional on reaching the end of the table, so every completed run re-walks
every row, and MAX_DEAD_ROW_READS_PER_RUN is what stops that from costing
O(rows) object reads on every boot. But row_read_attempted was only ever set in
the provenance loop, so a source-less row, the one shape discovery exists for,
spent up to MAX_LEGACY_DISCOVERY_PROBES reads and contributed nothing. The
per-row cap bounded one row; nothing bounded the run.

discover_legacy_row now reports the probes it spent and the caller charges them
whenever the row was not repaired.

Per probe rather than per row, because at 16 probes a row the budget would
otherwise admit 16 times the reads it names.

Including the retryable arm, which is where discovery differs from the
provenance loop. There a retryable read is against a repo the row names as a
holder and is expected to succeed once that repo warms. Discovery probes repos
the row does not name and re-derives its candidate list from scratch each run,
so an unrepaired row costs the same reads again on the next boot whatever its
outcome was. The cap-reached arm is retryable by design, to stop a grindable
repo id burying the true holder, and leaving it uncharged would leave exactly
the steerable path uncharged.
2026-08-11 22:08:10 -05:00
beardthelion bd0bef1476 fix(node): carry discovery onto the per-(oid, repo) incompleteness marker
Rebasing onto the #173 head that made the marker per (object, repo) broke
three assumptions this branch was written against.

Discovery re-marked the set incomplete after recording a discovered source,
because a successful record_pin_source used to clear the whole per-object
boolean. It no longer does, so that call went from compensating for a clear to
being the only thing arming the resolver's fallback. It now marks against the
unknown-repo sentinel rather than the repo it just recorded, which would claim
that repo failed when it is the one that succeeded. The sentinel is what the
v24 migration carries pre-upgrade markers under and means the same thing here:
the bounded warm-only probe may have missed a holder and nobody knows which.

Two tests asserted the superseded behavior and are inverted rather than
deleted, which is what their own comments asked for. GAP 2 documented the
per-object marker being cleared by any later record; that residual is closed,
so the test now proves discovery's marker survives one. The cold-candidate test
proved the cursor was not rewound; the rewind is unconditional on completion
now, so it asserts the cost that still matters, that re-walking the row reads
no object bytes and restores nothing.
2026-08-11 22:08:10 -05:00
beardthelion 25cab14024 test(node): pin the per-object marker's third-repo residual
record_pin_source's doc already names this residual and accepts it: the
pin_sources_incomplete marker is one boolean per object rather than per
(object, repo), so a genuine record from a third repo clears a marker that a
different repo's failed record set. Its two neighbouring cases are tested,
the same repo re-recording and a no-op insert that must not clear, and this
one sat between them described in prose and guarded by nothing.

The test drives the real path rather than writing rows. Repo A first-pins the
object. Repo B genuinely holds it and pushes, its skip-branch record fails on
the definite-error arm, and the code marks the set incomplete. An anonymous
read is then SERVED from B, which is the assertion that matters: it proves the
unrecorded holder is reachable while the marker stands. Repo C pushes, its
record succeeds, and the clear fires inside C's transaction. The identical
read now 404s, with the preload counter proving the fallback scan never ran.

Nothing here changes behavior. It converts an accepted tradeoff from a comment
into a guard, so widening it takes a deliberate edit rather than going
unnoticed. The doc comment says the residual fails closed, and that holds:
the provenance loop and the fallback scan both serve through the same
gate_and_serve, so clearing the marker only ever removes a gated search and
can never serve an object the caller could not already read. The window is
exactly one to fifteen sources, because at cap the count-guarded insert is a
no-op and the marker survives.

If someone implements the per-(oid, repo) marker the doc proposes, this test
is expected to go red. It should be updated to the new behavior, not deleted.

Proven load-bearing: removing the clear from record_pin_source leaves the
second read at 200 and turns the test red on the marker assertion.
2026-08-11 22:08:10 -05:00
beardthelion d88c0c726c fix(node): repair source-less legacy pins by bounded additive discovery
The boot sweep resolved each legacy row's bytes only through
pin_sources_for_oid. A row predating provenance has repo_id NULL and no
pin_repo_sources entry, so the source set is empty, the `for repo_id in
sources` body never runs, and the cursor advances past it. list_pinned_cids
also filters those rows out on is_raw_cidv1, so they were both unadvertised
and unrepairable: nothing on the node would ever fix them. The sweep exists
precisely for pre-provenance rows, and that was the shape it could not
handle.

For a source-less row, discovery now probes a bounded set of warm local repos
for the object and, on a hit, repairs the key through the existing
repair_legacy_provider_cid.

What it records is the part worth reading closely. It calls record_pin_source
additively and then mark_pin_sources_incomplete. It does NOT call
backfill_pin_provenance, and repo_id stays NULL. Reading identical bytes out
of a repo proves that repo HOLDS the object, not that it is the source: forks,
a vendored file, a shared LICENSE blob and the empty tree all collide across
repos. Since backfill_pin_provenance guards on AND repo_id IS NULL, the first
claim written is permanent, so an exclusive claim would let probe order decide
which repo's visibility rules gate a public read surface, with no correction
path. It would also disable the resolver's own fallback: needs_scan is
sources.is_empty() || at_cap || incomplete, so a non-empty below-cap unmarked
set reads as complete and stops the scan that would have found a servable
copy. The additive record plus the incomplete marker keeps that fallback
alive and makes nothing permanent.

Candidates are quarantined-filtered. list_all_repos is a bare SELECT over
repos with no quarantine or visibility filter, while the resolver's own legacy
scan loads list_quarantined_repos and gates on it, so discovery loads the same
set and drops those ids. A quarantined repo is hidden from every reader, so it
must not become a recorded source. Private non-quarantined repos stay in the
list: an additive source record does not widen what the resolver will serve,
since every source is gated independently.

The candidate list and the warm filter load once per pass, lazily, on the
first source-less row, and the is_dir sweep runs under spawn_blocking rather
than inline on a worker. Nothing goes through repo_store.acquire, so a cold
repo is never pulled back from remote storage; a repair pass must not become
a bulk restore.

Two classification rules matter more than they look. A cold or quarantined
candidate does NOT mark the row retryable. The candidate list is every repo on
the node rather than the row's holders, and sweep_legacy_provider_cids rewinds
the cursor whenever any retryable skip occurred, so counting a cold candidate
would make every source-less row retryable on every run on a cold-storage
node: the cursor would rewind forever, the sweep would never drain, and each
pass would pay full discovery cost. Only a read error on a warm candidate is
evidence about the row. Conversely, cap exhaustion IS retryable rather than
terminal, and candidates are ordered oldest-first by (created_at, id) rather
than by id: repo_id derives from the owner DID, which anyone can grind, so a
first-N-wins cap over an id-sorted list would let someone bury the true holder
permanently behind cheap registrations.

MAX_LEGACY_DISCOVERY_PROBES bounds the expensive unit, a bounded object read
from a warm repo. A candidate dropped cold, quarantined, or unsafe costs no
probe. Its doc states the contract it encodes rather than its resemblance to
the resolver's own cap, which happens to share the value under a different
contract.

The source record is best-effort and says so at the site rather than claiming
a later pass heals it. Once the key is rewritten the row is raw CIDv1, so the
cost gate skips it free and nothing revisits it; the healing path is the
resolver's needs_scan fallback, which the incomplete marker keeps available.

The no-remote-fetch property is asserted from both sides. On the effect side,
a cold candidate's disk path still does not exist after two full runs, with a
control proving the sweep declined an available copy rather than finding
nothing to take. On the call side, a source scan of this module (comments and
string literals stripped, four anti-vacuity assertions) rejects the
fetch-capable call shapes, and a mutation planting a real acquire in the probe
loop turns it red. What that pair does not cover, and the test says so: a
fetch reached transitively through git::store or db, and git's own promisor
lazy-fetch on a partial clone. Neither shape exists in the tree today.

Concurrency around the marker is executed rather than assumed. Discovery
records then marks, in that order, because a successful record_pin_source
clears the marker. Under a proven lock-wait interleaving, a concurrent record
committing before discovery's mark leaves the row marked, and two concurrent
passes over one row converge on a single source with the marker set. Both the
swap and the drop of that pair are red-checked.

One residual is pinned by a test rather than papered over, and it is not
introduced here. pin_sources_incomplete is one boolean per object, not per
(object, repo), so a genuine pusher whose record_pin_source commits strictly
AFTER discovery's mark clears it, leaving a non-empty below-cap unmarked set
that tells needs_scan to stop scanning. No ordering inside this sweep can
prevent that, and record_pin_source's own doc already states the same residual
for its existing call sites; closing it needs a per-(oid, repo) marker, which
is a change to the marker's shape. It is bounded in practice: the clearing
writer really holds the bytes, so every source in the set is servable, and for
these rows it cannot be a regression, since before discovery ran they were
keyed on a provider CID and were neither served nor advertised.

Tests: fifteen regressions driving the sweep entry points. The repair, cheap
second pass, probe cap and multi-holder cases were observed RED first. The
must-not cases carry mutations instead, since a guard against something
happening has no honest pre-fix red: quarantined holder never serves, a cold
candidate is neither fetched nor materialized and does not rewind the cursor,
an unsafe candidate name is dropped without aborting the pass while a later
safe candidate still repairs, a raw-CIDv1 row never enters discovery, and a
provenanced row still resolves through the source loop alone. Every guard
carries a red-check mutation with an attributed message, including three on
the design itself: one adds backfill_pin_provenance beside the additive record
and must redden the no-exclusive-claim assertion, one neutralizes the
incomplete marker, and one swaps the record and mark order.
2026-08-11 22:08:10 -05:00
beardthelion 4985e38af2 fix(ipfs): page the legacy CID scan instead of materializing every repo
GITLAWB_IPFS_MAX_LEGACY_PROBES bounded the probe fan-out, but nothing bounded
the load that ran before the first probe. On a legacy (NULL-provenance) or
capped/incomplete provenance set, get_by_cid called list_all_repos, cloned every
id, fetched visibility rules for the whole set, and fetched the node's entire
quarantine list, then retained all three for the request. /ipfs/{cid} is
anon-reachable and CIDs are enumerable from the unsigned pins index, and an
object pinned from MAX_PIN_SOURCES repos is permanently at cap, so every later
anonymous GET for it paid that whole preload with the scarce walk permits held,
even at a probe budget of 1.

The selection is now keyset-paged on (created_at, id) ASC, with the quarantine
flag returned per row so the separate whole-node quarantine query is gone. Both
key columns are immutable, so paging is exact: a repo touched mid-scan cannot
cross a page boundary and go unvisited, and updated_at is attacker-bumpable,
which would have let a caller sort their own repos ahead of the true holder.
The pager lives outside the oid loop, so its cursor and its accounting are per
request rather than per candidate. Before fetching another page the scan checks
that a row on it could still reach a verdict; a spent probe or visit budget
stops the read and TAINTS, so a truncated search still sheds the retryable 503
and never a false 404. A page of pure denials is deliberately not a stop
condition, since a cheap denial costs no probe and a public object buried behind
private repos must still be reached. Both page queries stay clamped to the
remaining request budget, and the rules query still fails closed.

A new v25 migration adds idx_repos_created_at_id. Bounding what the handler
materializes is only half the property: repos carried no index in the new keyset
order, so Postgres seq-scanned the table and top-N sorted it to return every
page while the walk admission was held, which moves the same O(rows) cost onto
the database rather than removing it. Measured on 50k rows: 951 shared buffers
and ~44ms per page without the index, versus an Index Only Scan at 7 buffers and
~0.09ms with it, and the keyset predicate pushed down as an Index Cond instead
of filtering after a scan. Nothing names the index in any query text, so its
entry says why it exists and a test asserts its presence structurally.

Two order-dependent tests were reseeded rather than left green: the buried-row
and acquire-taint cases encoded their ordering through updated_at, which no
longer determines iteration. Both were re-confirmed RED under their documented
mutations after the reseed. Three regressions land with the fix. One drives a
one-probe, one-row-per-page request against three candidate repos and asserts it
materializes one row and runs one query, which a query counter alone could not
have caught. One seeds two distinct source-less oids under a single CID and pins
the per-request property directly: the second candidate re-reads the pages the
first paid for, so the whole request costs one pass rather than one per
candidate. The third covers the index, including the INV-7 upgrade path.

Refs Gitlawb/node#173.
2026-08-11 21:40:19 -05:00
euxaristia 1d3987bba6 fix(review): make the LIKE escape independent of the SQL parser mode
Spelling the escape character as `ESCAPE '\'` puts a backslash inside an
ordinary SQL string literal. Under the legacy `standard_conforming_strings=off`
setting that backslash escapes the closing quote, so the literal runs on and the
statement fails to parse. The failure happens before the bound parameters are
considered, so every prefix lookup breaks, including prefixes that contain no
metacharacter at all, and short certificate IDs stop resolving entirely.
Connection settings come from outside the process, so that mode can arrive from
a database- or role-level setting.

Use `!` as the LIKE escape character instead, escaping `%`, `_`, and `!` in the
bound pattern. The statement then holds no backslash and parses the same either
way, without needing a second layer of backslash escaping. A backslash in the
prefix becomes an ordinary character, which is the literal match the search
wants anyway.

Cover it with a test that runs the production query through a second pool that
requests the legacy mode at connection startup, so a future return to a
parser-mode-dependent literal is caught rather than only wildcard handling under
the default configuration.

Refs #319
2026-08-11 14:09:13 -04:00
beardthelion 9758bc7303 docs(node): name the knob that actually sets the legacy-probe budget
The comment beside ipfs_max_legacy_probes told operators to tune
GITLAWB_IPFS_MAX_REPOS_WALKED, but AppState::ipfs_legacy_probe_budget reads
GITLAWB_IPFS_MAX_LEGACY_PROBES. The knob it named is the separate cap on
expensive visibility walks, so anyone following the guidance next to this line
adjusted the walk cap and left the legacy scan fan-out at its default.
2026-08-11 12:23:08 -05:00
beardthelion 5575da4b65 fix(node): make the pin-source incompleteness marker per (object, repo)
pinned_cids.pin_sources_incomplete was one boolean per object, so any
successful source record cleared it, including a genuine record from a repo
unrelated to the failure. The resolver reads a cleared marker as "every source
is recorded", drops the scan fallback, and 404s an anonymous caller whose only
servable copy is the one whose record failed. The missing source is a property
of an (object, repo) pair, so v24 stores it as one and only a later record from
that same repo clears it.

Pre-upgrade markers carry over with an empty sentinel repo id that no real
record matches, so they keep their fallback instead of being cleared by the
next unrelated record. pinned_cids.pin_sources_incomplete is left in place,
unread, so a rollback to the previous release still finds its markers.

Also adds the handler-level coverage the size-probe commit could not: a test
seam drops the object between the type probe and the size probe, and the
request tails to a retryable 503 rather than a definitive 404. The seam is
keyed on (repo path, oid) because fixture oids are content-derived and shared
across tests, and an oid-only key deleted objects out from under two unrelated
tests running in parallel.
2026-08-11 12:20:39 -05:00
beardthelion ed6355ed2c fix(node): stop the /ipfs size probe reporting a fault as a missing object
object_size_bounded mapped every non-timeout failure to Ok(None), which
gate_and_serve reads as a verified absence and does not taint the search for.
A corrupt object, an unreadable pack, or a failed spawn therefore ended the
search cleanly and handed an authorized caller a definitive 404 instead of the
retryable 503 tail, even though the type probe had already reported the object
present. Confirmed by execution against the old code: a corrupt loose object
returned Ok(None) where the healthy read returned Ok(Some(n)).

The probe now returns Result<u64, ProbeError>, the vocabulary the type and
content stages either side of it already use: a reaped child is Transient, and
everything else is classified by store readability, Transient when the store
cannot be read and Deterministic when it can. There is no absence value left,
so the collapse is unrepresentable rather than fixed at one call site, and the
gc-between-stages case is classified like any other fault.

ServedRead::Gone lost its only producer and is removed with it.

Covered by two store tests, each asserting a healthy read first so the fault
case cannot pass on a broken fixture. Not covered: the handler path that turns
the fault into a 503, because that call site hardcodes "git" rather than
state.git_bin, so no shim can be injected there.
2026-08-11 11:38:59 -05:00
beardthelion e4aaae8c52 fix(node): bound the fruitless reads one sweep run will spend
Rewinding on every completed walk means a later run re-attempts the object
read for every row whose bytes are permanently gone, so a node that collected
dead pins from a deleted repo or a force-pushed history paid O(dead rows) git
invocations on every boot with no decay. A run now stops after 64 such reads
and keeps its cursor, so the next boot resumes past what it already walked and
the table is still covered across boots.

Also corrects two pieces of prose that described the superseded round 11
behavior: the SweepStats::retryable_skips field doc still claimed that field
drives the rewind, and the new below-cursor test's comments still said a clean
run parks its cursor.

The pass-failure test waited on a fixed 100ms sleep for the first pass to
finish before renaming the table out from under the walk. It now waits on the
cursor that pass writes, so a runner slow enough to push the first pass past
the sleep no longer fails the assertion for the wrong reason.
2026-08-11 11:11:34 -05:00
Gravirei aa68d97a8d refactor(repo_store): expose advisory_lock_key pub(crate) and drop test copy
The deadline test hand-copied the advisory-lock key derivation, and that
copy drifted once the key moved to SHA-256, silently breaking the lock
collision the test is built around. Import the production function
instead so the key cannot diverge again.
2026-08-11 17:59:33 +06:00
beardthelion 1b44b12b46 fix(node): rewind the pin repair sweep cursor on every completed walk
A clean sweep run parked its cursor at the table's maximum sha256_hex and
later runs read only rows above it, so a provider-CID row written below that
point by another node during a rolling upgrade was never swept. The resolver
withholds such a row's advertised key, leaving the object unretrievable with
nothing left to repair it.

The round 11 rewind could not cover this: it fired only when a run skipped a
retryable row, and the run that parks the cursor is by definition clean. The
rewind is now unconditional on reaching the end of the table. A run that stops
on a pass error keeps its position instead, so a node whose DB fails part-way
through does not restart the walk from the beginning on every boot.

The cost is one ordered scan per run, plus one repair attempt per run for a
row that is unrepairable in principle. Rows already carrying the canonical raw
key cost a codec decode and no object read, so a node that has finished
repairing pays the scan and nothing more. The terminal-skip test now asserts
that bounded cost rather than the row going unread.
2026-08-11 04:33:47 -05:00
beardthelion 9ac3fd3f6b Merge origin/main into fix/issue-135-ipfs-cid-tree-gate
#254 landed after this branch was last pushed and it moves the same surface.
It converts get_by_cid's sqlx failure arms from AppError::Internal to a bare
Into<AppError> conversion, so a connection-class failure downcasts to
AppError::Db and answers 503 db_unavailable instead of a 500 (#251).

Both textual conflicts were spurious alignments rather than real overlaps.
Git matched this branch's provenance path against main's legacy-scan preload
because both regions end the same way, and oids_for_cid does not exist on main
at all. Resolved to this branch's structure, which already contains main's
preload in its own place with all three of its queries clamped.

The semantic change is the part that mattered and it was applied by hand
afterwards, not inherited from the resolution. All ten clamped DB awaits in
get_by_cid now use the bare conversion. Taking this branch's side wholesale
would have silently reverted #251 across every one of them, including the four
this branch added, since the arm shape they were written to mirror predated
#254.

Suite green on the merged tree with fmt, clippy, and metadata --locked clean.
2026-08-10 22:46:18 -05:00
euxaristiaandcairn-code eeafa1fa53 fix(node): Escape LIKE metacharacters in certificate prefix search
Short-ID resolution passes a caller-provided prefix into a LIKE pattern. Any '%', '_' or '\' in that prefix is interpreted as a LIKE wildcard rather than a literal character, so lookups can match certificates outside the requested prefix.

Escape the three metacharacters in the prefix and add an explicit ESCAPE '\' clause so prefix matching is always literal.

Add a regression test covering literal '%', '_' and '\' prefixes.

Co-Authored-By: cairn-code <[email protected]>
2026-08-10 23:38:05 -04:00
Kevin Codex 0e2328bd80 Merge pull request #254 from Ayush7614/fix/251-db-outage-503-ipfs-arweave
fix(node): map ipfs/arweave DB outages to 503 via bare ? (#251)
2026-08-11 09:54:29 +08:00
Kevin Codex 10e0840dd9 Merge pull request #312 from Gitlawb/fix/shim-nonblocking-accept
fix(git-remote): normalize the shim's accepted socket to blocking
2026-08-11 09:53:00 +08:00
beardthelion 9984e08212 fix(node): bound the pin loops' DB work by the batch deadline, both lanes
api/repos.rs holds the global pin_semaphore across ipfs_pin::pin_new_objects
and across pinata::pin_new_objects. Inside both, batch_budget_gate only gates
between objects and the git read was already clamped, but the DB calls were
bare awaits. One stalled query parked the permit past every budget, and once
all pin permits were so held, post-push IPFS and Pinata replication stopped
for every repository on the node. The pool defers rather than sheds, so the
queue behind it only grows.

The bound set is derived callee-transitively from each budgeted region's call
graph, not from the shape of the fix. A grep for db./retry_db_record over the
loop bodies cannot see a DB await behind a call site, and that is not
hypothetical: it missed repair_legacy_provider_cid, whose own cid_for_oid and
repair_legacy_provider_cid awaits run under the permit from both lanes while
its deadline argument bounds only the spawn_blocking git read. Kubo: six
direct sites plus those two transitive ones. Pinata: eight direct sites, two
more than Kubo because its post-upload path has its own record_pin_source and
incomplete marker. Completeness here is the enumeration, scoped to the call
graph at this head, not a compile-enforced guarantee.

db_bounded takes the ABSOLUTE deadline, so a slow predecessor cannot hand a
later call a fresh budget. What its Elapsed arm MEANS is a property of the
operation, not of the timeout. A multi-statement transaction whose tx.commit()
is never reached definitively did not land: record_pin_source is
pool.begin(), INSERT, conditional UPDATE, tx.commit(), so a future cancelled
while the INSERT waits on a lock cannot have committed. A single autocommit
statement may well have landed server-side after the client future was
dropped, since tokio cancels the future but not the statement Postgres is
running; mark_pin_sources_incomplete and record_pinata_cid are that shape.

Each site maps its elapsed arm on that basis. Reads skip the object.
Provenance and backfill warn and continue, since the backfill is idempotent
under its repo_id IS NULL guard. A timed-out record_pin_source writes the
incomplete marker, exactly as its definite-error arm does, because the source
definitively was not recorded and leaving the set incomplete AND unmarked is
the state this change exists to prevent: /ipfs would read a non-empty
below-cap source set as complete and 404 a copy the repo would serve. The
arms stay separate so the warn still distinguishes a stalled batch from
scattered per-object failures.

The durability writes take a floored remainder. batch_budget_gate only
guarantees PIN_READ_FLOOR before an object starts and the add gets the whole
remainder, so a successful add can finish with ~0 budget left; an unfloored
bound would fail a write that today completes in milliseconds. That leaves
bytes in Kubo with no pinned_cids row, and on the Pinata side it is worse,
because pinned.push is unconditional there, so repos.rs would build cid_map
and gossip publish_ref_update for a CID that /ipfs will 404. The floor is
re-taken from now at each call, so graces chain within one iteration: the
worst case is deadline+4s on the Kubo skip branch and deadline+6s on the
Pinata add path, about 5% of a 120s PIN_BATCH_BUDGET. It does not stack per
object, because batch_budget_gate breaks at the next iteration's first
statement.

pin_new_objects_gated now takes the batch budget instead of hardcoding
PIN_BATCH_BUDGET. The permit-release regression asserts a bound that IS the
budget, so with the constant inlined it cost 120s of wall clock; passing it
drops the test to 3.4s. Production passes the same constant.

Both twin doc comments are updated together. Each previously stated that the
DB round-trips were untimed inside the budgeted region, which is now false.
The Pinata permit's total hold across pinata_object_list_for_refs stays
unbounded and is called out as such: repos.rs acquires the permit before that
pre-loop walk, so this change closes the DB-stall trigger, not the git-walk
one.

Tests: thirteen regressions, the stall cases observed RED first with the bare
await sitting on the lock until the outer wrap fired. Every guard is proven
load-bearing by a mutation carrying an attributed message, so a RED is pinned
to the named property rather than to seam removal.

Three properties needed a fixture built specifically to bind them, because
the obvious test passed either way. The record floor: a zero remainder still
landed the write, since tokio polls the inner future before the timer and a
local UPDATE round-trips inside one tick, so the marker's table is now locked
and released inside the grace window, which fails without the floor and lands
with it. The absolute deadline: every single-object stall test passes under a
per-call duration too, so a helper unit test now runs two calls against one
shared deadline and asserts the second is cut off by the remainder its
predecessor left, with the mutation granting a fresh duration equal to the
budget rather than larger than it. The Pinata post-upload elapsed arm: it had
been changed in lockstep with the two covered sites but nothing executed it,
so a fixture now lets the upload succeed and stalls only the record that
follows.
2026-08-10 20:36:04 -05:00
beardthelion bad82b3cd9 fix(node): clamp every admission-held DB await in the CID resolver
get_by_cid builds its walk admission (the global permit plus the
per-caller permit) and then made several DB awaits with no deadline while
those scarce permits were held. A stalled pool or a blocked query pinned
every walk slot for the whole stall, so later /ipfs requests shed capacity
503s long after GITLAWB_IPFS_REQUEST_BUDGET_SECS had elapsed. Any
unauthenticated caller can reach it.

The clamped set is derived from the admission-held region itself, not from
the two sites the review named. Walking the handler body found four bare
awaits:

  oids_for_cid, pin_sources_for_oid, pin_sources_at_cap,
  pin_sources_incomplete

The last two were not named in review. Already clamped and left alone: the
three per-source lookups and the legacy-scan preload trio. Two negatives
worth recording: gate_and_serve is the only callee reached from inside the
region and holds no DB awaits of its own, so the region does not extend
transitively, and ipfs_work_rate_limiter.is_throttled is in-memory, not a
round trip. Ten DB awaits in the body, zero bare after this change.

Each clamp takes the arm shape the per-source lookups already use, so a
timeout returns the budget-tainted retryable 503 and the early return
RAII-drops the admission. budget_shed and remaining are hoisted above the
oids_for_cid await so the provenance-path sites share one definition; the
legacy-scan preload keeps its own in its nested scope.

Tests: four stall regressions, one per clamped site. The first two lock the
table their query reads and were observed RED first, with the bare await
sitting on the lock until the 10s outer wrap fired.

The marker pair needed a test-only seam rather than a lock. Both queries read
tables that earlier queries in the same request touch first, so a LOCK TABLE
aimed at them stalls an earlier clamp instead, and budget exhaustion has the
same problem because an earlier query elapses before the pair is reached. The
remaining window between pin_sources_for_oid returning and the pair running
is a single get_repo_by_id round trip, measured at 0.14 to 0.17 ms, which is
too small to time a lock into without flaking. So there is a cfg(test) stall
armed per target, placed INSIDE the timeout wrapper so the clamp is what
fires. Every seam item is cfg(test), including the function itself, so a
non-test caller would not compile, and a full release build is clean.

Each of the four asserts the same contract: a 503 inside the budget, the
budget taint in the body, the walk permit returned, and a follow-up request
admitted rather than capacity-shed. All four clamps are proven load-bearing
by reverting the clamp alone with an attributed message, and the mutations
cross-isolate: reverting one turns only its own test red, so each test
stalls at its own query rather than a sibling's.
2026-08-10 20:35:36 -05:00
beardthelion 208fd808b6 Merge remote-tracking branch 'origin/main' into fix/issue-135-ipfs-cid-tree-gate
# Conflicts:
#	crates/gitlawb-node/src/db/mod.rs
2026-08-10 15:08:30 -05:00
beardthelion d4e5ecf299 fix(review): bound the probe's channel waits and stop overstating the handshake
Cross-model review round, findings corroborated by the in-process reliability
reviewer.

Both channel waits were unbounded. If the reader thread never reached its read,
or never returned after the peer close, the test wedged the binary instead of
failing, and cargo test has no per-test timeout to rescue it. Both are now
recv_timeout with bounds far above any real scheduling delay, so hitting one is
a genuine defect report rather than a flake, and the failure names which wait
stalled.

The handshake paragraph also overstated what the handshake buys. It listed "the
reader thread not yet scheduled to reach the read at all" among the states the
handshake removes, which reads as though the whole scheduling window is closed.
It is not: the handshake removes thread spawn from the window, and a reader
descheduled between its own Started send and its read is still a false-green
path. That path is now stated plainly instead of being left to the following
paragraph to correct.

Re-verified after the change: probe RED 50/50 under a deleted
set_nonblocking(false) at full-suite parallelism, RED on the named message under
a deleted set_read_timeout, 50/50 green unmutated, fmt and clippy clean.
2026-08-10 13:55:53 -05:00
beardthelion 4d9bd5a18a fix(review): assert the read-timeout postcondition and drop the socket clone
Two findings from review of the probe test.

The helper has two postconditions and only one was guarded. Deleting
set_read_timeout left every test green while a stalled peer would park the
shim's only accept-loop thread forever, since handle_conn runs inline in that
loop. std exposes read_timeout() as a getter, so this half is checkable exactly
with no timing component. Confirmed load-bearing: deleting the line reds the
probe on the new message.

The probe also normalized one socket and then read from a try_clone of it, so
it only proved anything if blocking mode is shared across the two. On Linux it
is, because try_clone is dup(2) and O_NONBLOCK lives on the shared open file
description. On Windows try_clone goes through WSADuplicateSocketW and WSASocketW
(std os/windows/io/socket.rs), which mints a new socket descriptor, and whether
FIONBIO follows it is a question this test would have depended on without being
able to answer. It would have failed silently and green, on the one platform
where the underlying defect is real. The clone was unnecessary, so the reader
now owns the socket the helper normalized and the question does not arise.

Also correct two overstatements in the probe's own doc comment. It called
itself deterministic while its verdict is a 250ms window, and it described the
residual false-green path as "the few instructions between the signal and the
call" when the window is a full cross-thread round trip: signal, read, return,
channel send, and this thread waking. Both now say what the code does.

Re-verified after the change: probe RED 50/50 under a deleted
set_nonblocking(false) at full-suite parallelism, RED on the named message under
a deleted set_read_timeout, and 50/50 green unmutated. The end-to-end test came
in at 39/50 on this run against the first mutation, a third sample alongside 43
and 47, so its doc comment now reports the spread instead of two numbers.
2026-08-10 13:46:43 -05:00
beardthelion 6f58f35c95 test(git-remote): make the non-blocking-accept guard deterministic
The regression test added alongside the shim fix could pass with the fix
removed. It synchronized on a channel send that happens after accept() but
before handle_conn runs, so the request bytes could already be buffered by
the time the shim read them, and the read then succeeded on a non-blocking
socket. Measured at 43 and 47 of 50 full-suite runs catching a deleted
set_nonblocking(false), so it went false-green roughly one run in eight.

Extract the normalization into normalize_accepted_stream and assert its
postcondition directly, with no client and no request-arrival race. The new
probe puts a connected socket into non-blocking mode, calls the helper, and
checks that a reader parks instead of returning WouldBlock. Blocking mode has
no getter to query: std exposes none and Winsock has no FIONBIO getter at all,
so the property has to be observed through behavior.

The reader signals Started immediately before its read and the main thread
waits for that signal unbounded before opening the 250ms window. Without the
handshake the window would also be satisfied by a thread that has not been
scheduled yet, which would reintroduce the same silent hole one level down.
Peer close comes after the assertion, never before; closing first unblocks the
read and fails the test even when the helper is correct.

Also correct a platform claim this PR had in five places. The bug is not
Windows-only. An accepted socket inherits the listener's flag on BSD-derived
stacks, so macOS and the BSDs are exposed too; Linux is the exception because
std uses accept4 there, which does not inherit file status flags
(rust-lang/rust#67027). A macOS contributor could hit this today while the
comments said it was impossible. The macOS half is documented rather than
executed, and the comments now say so.

Verification, full-suite runs at default parallelism:
  helper body mutated (set_nonblocking(false) deleted): probe RED 50/50
  call site mutated (helper call deleted):              probe GREEN 0/50,
                                                        integration test 45
                                                        and 46 of 50
  unmutated:                                            50/50 green

The call-site row is the probe's blind spot, measured rather than assumed: it
calls the helper directly, so only the integration test notices a deleted
call, and it does so at a sampled rate rather than reliably. That residual is
recorded in the integration test's doc comment.
2026-08-10 12:20:48 -05:00
Gravirei c68e1b07f8 fix(test): match SHA-256 advisory-lock key in receive-pack deadline test
The test reproduces acquire_write's advisory-lock key derivation locally so
a held session lock can stall the handler. repo_store's advisory_lock_key
now uses SHA-256, so the DefaultHasher copy no longer collided and the
stall never happened. Update the local copy to the SHA-256 derivation.
2026-08-10 22:09:10 +06:00
Gravirei 9f340eac5e Merge upstream/main into fix/issue-210-stable-advisory-lock-key
Resolve conflicts: keep the SHA-256 advisory-lock key change alongside
upstream's connection-affine RepoWriteGuard rework and its F1/F3b/F4/U8
test suite; RUN-A-NODE.md keeps both the updated Public URL row and the
rolling-upgrade advisory-lock row.
2026-08-10 21:31:38 +06:00
beardthelion 8d7f95fd63 fix(git-remote): normalize the shim's accepted socket to blocking (#192)
start_shim sets the listener non-blocking so its accept loop can poll the
stop flag. On unix accept() hands back a blocking socket, so that never
showed; on Windows the accepted socket inherits the listener's FIONBIO
state and std does not reset it, so every connection arrived non-blocking.

handle_conn reads with read_line(..).unwrap_or(0), which turns the
resulting WouldBlock into "peer closed" and returns without writing a
response, dropping the connection and aborting the client mid-request. The
shim accepts as soon as the handshake completes, strictly before the
request bytes arrive, so the window was open on every connection and
closed only because the 5ms accept poll usually lagged the client. That is
what made real_git_withheld_shaped_first_post fail on windows-latest with
os error 10053 and zero POSTs recorded.

Normalizing at handle_conn entry also restores the 30s read timeout on the
next line, which a non-blocking socket silently made meaningless.

The regression test forces the Windows socket shape explicitly, so it is
load-bearing on every platform rather than only the one that exhibits the
bug, and it establishes the accept-before-request interleaving with a
handshake rather than a sleep: a timing-based version cannot flake red but
can flake green under load, which would make the guard vacuous.
2026-08-10 10:14:43 -05:00
Ayush7614 c29a91af6b fix(node): map ipfs/arweave DB outages to 503 via bare ? (#251)
Rebased onto current main. Keep budget-timeout wrappers on get_by_cid;
convert sqlx failures with Into<AppError> (not AppError::Internal) so
pool closed/timeouts become 503 db_unavailable. Same for list_pins and
arweave anchors, with closed-pool regressions.
2026-08-10 20:19:51 +05:30
Kevin Codex 241b36633b Merge pull request #247 from Ayush7614/fix/226-opaque-internal-errors
fix(node): opaque AppError::Internal and AppError::Db HTTP bodies (#226)
2026-08-10 21:43:42 +08:00
Kevin Codex 425ebf40e7 Merge pull request #309 from euxaristia/fix/strict-ed25519-verification
fix(core): enforce strict RFC 8032 Ed25519 signature verification
2026-08-10 21:41:57 +08:00
beardthelion ea2219b2cb fix(node): bound the legacy-repair read by the pin batch deadline
repair_legacy_provider_cid built its own deadline from git_timeout and never
consulted the batch deadline, but both pin loops call it with a pin_semaphore
permit held. At shipped defaults that is a 600s git_service_timeout_secs against
a 120s PIN_BATCH_BUDGET, so one legacy row whose cat-file wedges could hold a
global pin slot for five times the budget the batch is supposed to cost and
starve every other repo's pin work. The loop's budget gate cannot help: it runs
at the top of the next iteration and cannot preempt a call already in flight.

The helper now takes the deadline rather than deriving it, since its two kinds
of caller can afford different holds. Both pin loops clamp to
min(batch deadline, now + git_timeout). The boot sweep holds no permit and has
no batch to overrun, so it keeps the plain git_timeout.

The Kubo side has had this since the repair landed and the Pinata side inherited
it from the lockstep work in the previous commit, so both get a test: a wedged
cat-file with a 60s git_timeout and a 2s batch budget must return on the batch
order. Both were RED at 62s before the clamp and are load-bearing by mutation.

Also stops the read-bound comments implying both arms bind in a default
deployment. At shipped defaults the batch deadline is always the tighter one;
the git_timeout arm is what an operator who lowers that knob gets.
2026-08-10 07:43:53 -05:00
euxaristiaandcairn-code 1fcbbbd89c fix(attest): use strict Ed25519 verification with regression test
Co-Authored-By: cairn-code <[email protected]>
2026-08-10 07:38:40 -04:00
euxaristiaandcairn-code 871b8610cf test(core): cover strict Ed25519 verification
Add a regression test for verify_strict: a signature forged under a weak (small-order) public key satisfies the verification equation but must be rejected. The non-strict verify path accepts it.

Co-Authored-By: cairn-code <[email protected]>
2026-08-10 07:38:40 -04:00
euxaristiaandcairn-code a6c13be15b fix(core): use strict Ed25519 signature verification
Use verify_strict from ed25519-dalek for identity signature checks, matching the crate's recommended RFC 8032 strict verification mode.

Co-Authored-By: cairn-code <[email protected]>
2026-08-10 07:38:40 -04:00
Ayush7614 af821e09bf test(node): disposition peer-readiness fixture writers in ledger
Same #248 test-only INSERT INTO peers fixtures as on #254; record them
in LEDGER so test (stable)/beta pass on the rebased #247 head.
2026-08-10 15:51:49 +05:30
Ayush7614 d192028ee1 fix(node): opaque AppError::Internal and AppError::Db HTTP bodies (#226)
Rebased onto current main. Keep Overloaded/UnresolvableDid arms; make
Internal and non-unavailable Db responses opaque with server-side logs,
and pin exact JSON in unit + anonymous route tests.
2026-08-10 15:51:49 +05:30
beardthelion 4681f54327 fix(node): bring the Pinata pin path back into lockstep with the Kubo one
Two gaps remained after the main merge closed the unbounded-read one. The Kubo
skip branch repairs a legacy provider-CID row to the raw-content resolver key on
re-push and the Pinata skip branch did not, so a Pinata-first node left those
rows unresolvable until the deferred sweep caught them. And the Pinata object
read was bounded only by the batch deadline, so one wedged cat-file could hold
the pin permit for the whole budget.

Both are fixed by threading git_timeout into pinata::pin_new_objects, which it
needed anyway: repair_legacy_provider_cid takes a Duration, and the read bound
is now min(batch deadline, now + git_timeout), the same shape the Kubo side
uses. repair_legacy_provider_cid becomes pub(crate).

The returned pairs still diverge on purpose and the twin docs say why: Kubo omits
a record-failed pin because its return is log-only, Pinata keeps it because its
return feeds the announcement cid_map. Pinata's return behavior is unchanged.

Four tests drive pinata::pin_new_objects rather than the Kubo twin, because a
repair call merely present in pinata.rs proves nothing about the Pinata lane:
the repair runs on the skip branch, the codec cost gate reads no bytes for a
row that is already canonical, a failed repair does not break the batch, and the
upload path never runs the repair. The two must-not guards have no obtainable
pre-fix RED, so they are proven by mutation instead: removing the cost gate and
moving the repair onto the upload path each turn one red.
2026-08-10 02:14:10 -05:00
beardthelion f839fb9246 fix(node): stop reporting a Kubo pin whose DB record never landed
On record_pinned_cid_with_source exhausting its retries, pin_new_objects warned
and then pushed the object onto the returned vector anyway, so the post-push log
told operators an object was pinned when the durable index had no row for it and
the resolver could not serve that CID. The push is now conditional on the record
succeeding.

The twins deliberately diverge here and the old comment said the opposite, that
the push mirrored pinata to keep them structurally identical. The Kubo return is
log-only, so dropping a record-failed pin costs nothing. The pinata return is a
real input: api/repos.rs builds the sha-to-cid map from it, which drives
upsert_branch_cid and the p2p ref-update gossip, so omitting rows there would
silently drop branch-to-CID records. Both comments now say so, since the next
reader's instinct will be to re-align them.

Test asserts the record-failed object is absent from the return while the add
mock still fired, and that a second object is still attempted rather than the
batch breaking. Verified RED against the unconditional push before the fix, and
load-bearing by re-injecting it.
2026-08-10 01:34:04 -05:00
beardthelion 8f3a52175a docs(node): correct the walk-cap and db-pool operator docs
Two doc drifts jatmn flagged on #173, both docs-only.

The /ipfs walk cap is min(internal ceiling, GITLAWB_IPFS_MAX_REPOS_WALKED),
where the internal ceiling is MAX_PIN_SOURCES + 1 = 17. config.rs, README, and
.env.example all described a max() floor instead, so an operator reading them
would think the default of 64 was the effective ceiling and that raising the
knob bought headroom. Neither is true. The runtime is not changed: two tests set
the knob to 1 and depend on the cap actually being 1, so wiring the documented
floor would make the knob inert and break them.

db_max_connections has defaulted to 48 for a while and four comments still said
20. Two of them did arithmetic on it. main.rs and repo_store.rs justified the
separate advisory-lock pool by "20 is below 32", which is now false and made the
separation look like a consequence of the defaults; it is structural, since a
push holds its connection for the whole receive-pack at any pool size. Also adds
a RUN-A-NODE troubleshooting entry for the boot failure an in-place upgrade hits
when it is still configured with the old pool size of 20.
2026-08-10 01:24:46 -05:00
beardthelion 9f74811269 Merge origin/main into fix/issue-135-ipfs-cid-tree-gate
#174 landed on main at 5981257, so the integration merge this branch carried is
now redundant with main's own history. Five files conflicted, and two more
auto-merged into states that did not compile.

store.rs: main migrated the bounded-read family from a per-call timeout:
Duration to a shared deadline: Instant. The auto-merge took main's
batch_check_probe alongside this branch's object_type_bounded, which compose,
and produced two read_object_bounded definitions with incompatible signatures
and no conflict marker. Took main's convention throughout and re-applied this
branch's additions on top: object_size_bounded, the wedged-child reaper test,
and the doc enrichments.

ipfs_pin.rs: the pin read is bounded by the earlier of the batch deadline
(#174) and the object's own git_timeout (#173). Neither implies the other and
both are load-bearing: with only the batch deadline, one wedged cat-file holds
the pin permit for the whole 120s budget; with only git_timeout, a batch of
slow reads runs past the budget. Each direction was verified by injecting it
and observing the RED.

api/repos.rs: kept main's choice of the literal "git" over state.git_bin for
the pin path's cat-file, since that knob is the walk binary and tests point it
at a fake. The parameter stays so the seam is still drivable from a test, and
the reason now sits at the call site.

main.rs: dropped main's free sweep_rate_limiters, superseded by
AppState::sweep_rate_limiters, which sweeps the same limiters plus
ipfs_work_rate_limiter. Both call sites already used the method.

db/mod.rs: dispositioned main's two new peers-table test writers in the
completeness ledger.

Migration numbering is unchanged. v17 is byte-identical on both sides, and
main's own reservation comment already allots 18-23 to this work.
2026-08-10 01:08:46 -05:00
Kevin Codex 225644a3ea Merge pull request #248 from MikeTomlin19/agent/fix-peer-readiness-176
fix(node): use readiness for peer liveness
2026-08-10 12:17:20 +08:00
Kevin Codex 6668525f41 Merge pull request #286 from Gitlawb/fix/icaptcha-release-versioning
fix(release): bump icaptcha-client with the rest of the workspace
2026-08-10 12:11:19 +08:00
Kevin Codex 5fdf155c7f Merge pull request #192 from Gitlawb/fix/issue-117-multiround-fetch-deadlock
fix(git-remote): drive multi-round fetch as a v0 stateless-RPC client loop (#117)
2026-08-10 12:10:26 +08:00
Kevin Codex 8d38043a4c Merge pull request #290 from Gitlawb/fix/273-peer-authority-gate
fix(node): bind a peer row to its DID so only its keyholder can repoint it (#273)
2026-08-10 12:02:39 +08:00
mike 337442e449 test: pin gossip readiness persistence 2026-08-09 23:12:12 -04:00
mike 74f7554894 fix(node): bound peer failure tracking 2026-08-09 23:10:59 -04:00
mike 6bd8f41212 fix(node): harden peer readiness state 2026-08-09 23:10:59 -04:00
mike 5af14c203d fix(node): bound legacy readiness fallback 2026-08-09 23:09:55 -04:00
mike 547f9aed10 fix(node): preserve legacy peer compatibility 2026-08-09 23:09:55 -04:00
mike 1a841a99e2 fix(node): use readiness for peer liveness 2026-08-09 23:09:55 -04:00
Gravirei a27a267871 fix(gl): require non-empty error message and use ephemeral port in whoami tests (#220) 2026-08-06 13:18:29 +06:00
Gravirei b4007a5e00 fix(gl): assert whoami 404 verdict and preserve transport error cause (#220) 2026-08-06 11:58:11 +06:00
Gravirei a329174175 fix(gl): assert 404 verdict and harden transport error handling in whoami (#220) 2026-08-06 11:57:23 +06:00
Gravirei fd78cfa3ad fix(gl): only report registered: false on 404 in whoami (#220) 2026-08-06 11:39:15 +06:00
Gravirei 3cc15cf724 fix(gl): only report registered: false on 404 in whoami (#220) 2026-08-06 11:39:07 +06:00
Gravirei 16cc3b37e8 fix(gl): only report registered: false on 404 in whoami (#220) 2026-08-06 11:39:07 +06:00
Gravirei 3f41df5868 fix(gl): only report registered: false on 404 in whoami (#220) 2026-08-06 11:39:07 +06:00
Gravirei b0e5cc0ee0 fix(gl): only report registered: false on 404 in whoami (#220) 2026-08-06 11:39:07 +06:00
Gravirei 12bc759ea6 fix(gl): only report registered: false on 404 in whoami (#220) 2026-08-06 11:39:07 +06:00
Gravirei 10e3492825 fix(gl): only report registered: false on 404 in whoami (#220) 2026-08-06 11:39:07 +06:00
Gravirei c10f2ca255 fix(gl): only report registered: false on 404 in whoami (#220) 2026-08-06 11:39:07 +06:00
Gravirei 75b0df8f1c fix(whoami): update registered status handling on 404 response 2026-08-06 11:39:07 +06:00
Gravirei 3add5356e3 fix(gl): only report registered: false on 404 in whoami (#220) 2026-08-06 11:38:47 +06:00
Gravirei fdd11030dc fix(gl): only report registered: false on 404 in whoami (#220) 2026-08-06 11:38:47 +06:00
Gravirei 89e10d0ff1 fix(repo_store): use SHA-256 for stable advisory-lock key (#210)
DefaultHasher is not frozen by the Rust standard and has shifted across
toolchain versions, so the same (owner_slug, repo_name) produced a
different i64 advisory-lock key on different builds and broke
cross-machine write-exclusion for the same repo.

Hash with SHA-256 instead and pin the result with a golden-value test,
plus a one-axis-at-a-time differential test so a regression that drops
either parameter is caught. Document the SHA-256 re-keying caveat on
acquire_write and in the RUN-A-NODE.md operator checklist (drain writes
or cut over single-node during a rolling upgrade).
2026-08-05 19:26:44 +06:00
beardthelion 5df7d2afb3 test(gl): pin the local-add print selection, not just its helper
local_add_refusal had six unit tests and the wiring had none, so
replacing the whole match at the call site with an unconditional "Added
to local peer list." left every test green. That is the shape the store
already names: a helper test plus a status-mapping test does not prove
the call site.

Extract the selection into local_add_report, following the same
print-selection-helper precedent used for the certificate DID hint, and
assert both directions: a refusal never yields the success line, and a
200 never yields the warning. Collapsing the match to always return the
success line now turns that red. The call site is left as a two-line
dispatch carrying no logic.
2026-08-03 20:38:53 -05:00
beardthelion 8313c10c5b test(node): make the peers-write scan see a statement split across lines
The scan matched a verb-plus-table needle per line, so it was whitespace
and case sensitive. The identical statement written across lines walked
straight past it, and that form is how the longer queries in this file
are already written, so it is the shape a new writer most likely takes.

Normalize the whole file before matching, mapping offsets back to line
numbers so each statement is still attributed to the function issuing
it. The scan is split into a helper so the property can be asserted
against a synthetic source: the tree contains no multi-line peers write
today, so a test that only ran against the real tree would prove nothing.
Fixtures derive from the needles at runtime for the reason the needles
themselves do, which is that a literal here is found by the scan of this
very file.
2026-08-03 20:38:53 -05:00
beardthelion d6434b287b fix(node): scope the unproven DID check to the insert, not to the DID
The check ran before anything looked at whether the row existed, so it
refused a pure liveness refresh of a row already in the table. Nothing is
protected by that: the row is already there, and an identical-URL refresh
changes no authority. What it did instead was freeze every peer a
deployed database already holds from before this gate, since upsert_peer
validated nothing then and the handler accepted any parseable DID. Those
rows take a 400 on every unsigned announce and cannot be corrected by a
signed one either, because no key resolves.

Now the DID is judged only when no row exists, which is the only place
refusing it prevents anything. The read-then-write race is benign: if a
row appears in between, the write becomes an update and the existing
repoint guard still refuses any http_url change.
2026-08-03 20:38:53 -05:00
beardthelion 895fcf05d8 fix(gl): check the peer's status before parsing its announce reply
The capped read means a peer answering with a body past the cap leaves
truncated JSON behind, and parsing that with ? before the status check
turned a 503 into "invalid JSON response", losing the status the user
needs. A non-JSON error page from a caller-chosen peer URL does the same.
The failure check now runs first and falls back to the body text when
there is no message field. Bound by a test that drives the command
against a mock peer, since reordering the two statements compiles and
passes every unit test in the file.
2026-08-03 20:38:53 -05:00
beardthelion 8b3aa1decf fix(gl): defang and bound node-supplied text before it reaches the terminal
The peer-add refusal warning added earlier in this branch copied the
node's message straight into stderr, and both announce bodies were read
unbounded, so a hostile --node endpoint, a compromised local node, or a
MITM on plain http could put ANSI or OSC escapes into the user's
terminal. The accepted-announce block had the same hole on the success
path, where the DID and URL come from a peer whose URL the caller chose.

sanitize_node_msg and read_body_capped move from sync.rs to http.rs,
which already owns the wire boundary, and both paths in peer.rs now use
them. Only the display is sanitized: the values forwarded to the local
node stay verbatim, since that is data rather than terminal output.
2026-08-03 20:38:53 -05:00
beardthelion e8f091e1c4 fix(node): give the unresolvable-DID refusal its own error code
The split reported the cause in the message but left the code as
bad_request, so a client telling this apart from the other validation
failures on the announce route had to substring-match prose. The auth
middleware already answers unresolvable_did for the same failure on a
request's keyid; the peer-write path now answers the same way. Status is
unchanged at 400, and an unsupported DID method keeps bad_request.
2026-08-03 20:38:53 -05:00
beardthelion df2a1797d1 fix(node): report an unresolvable did:key distinctly from an unsupported method
The unproven insert path folded every to_verifying_key failure into
UnsupportedDidMethod, so an oversize, malformed, or non-ed25519 did:key
was told its DID method is unsupported, which is false and leaves the
caller nothing to act on. Split the arm by cause: a parse or resolution
failure now returns UnresolvableDid carrying the underlying reason, in
the same wording the signed path already uses, while a foreign method
keeps methodNotSupported. Every input stays refused in the 400 class.
2026-08-03 20:38:53 -05:00
beardthelion ebb93e3231 test(gl): pin the local-add refusal warning to the node's production wording
The refusal test fed local_add_refusal an invented message, so it could
never go red when the node's wording changed. Drive it with the Display
of PeerWriteDenied::UnprovenRepoint and assert the same substring the
node-side pin asserts against a real router response.
2026-08-03 20:38:53 -05:00
beardthelion b6ebaa4b1e fix(gl): surface the node's refusal instead of reporting a peer add that did not happen
The announce route now answers 403 when an unproven caller tries to
repoint an existing peer. gl peer add discarded the local node's reply
with let _ and printed the success line unconditionally, so a refused add
reported as a completed one. The refusal is split into a small function so
the path is assertable, and it is tested for a 403 with a message, a
refusal with no body, and the accepted case. The local add stays best
effort: a transport failure warns rather than failing the command, which
is what the surrounding code already assumed.

The README described the staged rollout as accepting unsigned peers on
these routes, which is no longer the whole rule. With signed peer writes
off, an unsigned announce may still register an unseen peer and refresh a
row whose URL is unchanged, but changing an existing peer's http_url needs
a signature from that peer's own DID, and only a did:key can be registered
unsigned.
2026-08-03 20:38:53 -05:00
beardthelion 73bf132c1a fix(node): resolve the key on the unproven insert path, and bound the decode
The gate restricting unproven inserts to did:key checked the method label
only. Did::from_str validates that the method is one of key, web, or
gitlawb and never inspects the key material, so did:key:notarealkey passed
and created exactly the row nobody can ever correct that the gate exists
to prevent. It now resolves the key the way the signature path does.

Six test fixtures using invented did:key strings stopped inserting once
the gate became real. That failure is the evidence it now reads key
material rather than a prefix, and it means those fixtures were exercising
a path no real DID could take. They carry derivable values now.

Resolving the key put a quadratic base58 decode on a route that accepts
anonymous callers by default. Measured through the router, a 64,000
character did:key cost 8.05s against 4.6ms for the previous label check,
on a tokio worker with no spawn_blocking, and a 1.5 MB body reaches the
handler. The bound sits in to_verifying_key so it also covers the
HTTP-signature keyid, which reads an untrusted string through the same
function. An ed25519 method-id is a fixed 48 characters, so 64 is slack.

Its test asserts the error names the length rather than merely being an
error, because an oversized id fails either way and a check for is_err()
would pass with the guard removed, having already paid the cost.

Three last_ping_ok assertions were vacuous: the seed helper never granted
reachability, so they compared false against false and would have held
against code that never cleared the flag. The helper grants it now and
asserts the seed took. Nothing was hiding behind them.
2026-08-03 20:38:53 -05:00
beardthelion 7cdfb843ff test(node): bind the peer-authority gate and ledger its writers (#273)
Drives the must-not set through the production router with real RFC 9421
signatures rather than injected extensions, so the signed path is exercised
as a caller would reach it, in both require_signed_peer_writes modes.

Covers the allowed cases and every rejection: unsigned insert of an unseen
DID, unsigned repoint refused with the row byte-identical after, a
signature from the row's own DID accepted, a signature from a different
DID refused, an identical-URL re-announce refreshing only last_seen, and a
did:web announce refused with no row created.

The signature-from-a-different-DID case is the RUSTSEC-2022-0009 shape,
where libp2p-core accepted a valid signature without checking it derived
the claimed identity. It asserts the key is bound to the row rather than
to anything the request carries.

The announce handler's keyid-mismatch branch had no test at all and
returned 400. It now returns 403 and is tested: a caller denied for want
of proof of control over the targeted row gets the same response class
whether or not it presented a signature.

The completeness ledger derives its set from every writer of the peers
table rather than from upsert_peer's callers, because a caller scan cannot
see a future writer that issues its own SQL. A second table records the
call-site authority for main.rs's bootstrap announce-back, which issues no
SQL and so can never appear in the writer ledger; its authority choice has
no runtime test in this change and is recorded as such.

The type system carries the guard: the authority parameter has no default,
so a new caller cannot omit it. The scan is the backstop for a raw
statement that bypasses upsert_peer entirely, and stays minimal rather
than copying the handler-directory apparatus, which does not apply to a
scan over SQL literals.
2026-08-03 20:38:53 -05:00
beardthelion 14481df288 fix(node): bind a peer row to its DID at the upsert_peer boundary (#273)
upsert_peer ended in an unconditional ON CONFLICT DO UPDATE SET http_url,
so any unauthenticated caller could repoint any peer. A repointed row
steers the sync worker's git remote, the post-receive notify fan-out,
trigger_sync, and the public GET /api/v1/resolve/{did}, none of which
consult the reachability flag that #270's stopgap resets.

An unsigned write may now insert an unseen DID but may never change an
existing row's http_url; changing one requires a verified signature from
that row's own DID. The gate is at the db boundary rather than in the
announce handler because there is a second writer: the bootstrap
announce-back in main.rs reads a peer's own JSON response body with no
proof of anything, and now declares itself unproven, which makes it
insert-only by construction.

The proven variant carries the DID it proves and upsert_peer compares it
against the row being written. A bare proven/unproven flag would say that
something was proven without saying which DID, leaving the real check in
the handler and reproducing RUSTSEC-2022-0009, where a valid signature was
accepted without checking it derived the claimed identity.

An unproven insert is restricted to did:key. Only did:key can ever
authenticate here, so a did:web or did:gitlawb row created unsigned would
become unwritable by anyone once this rule lands.

The unproven path derives its refusal from rows_affected, because an
ON CONFLICT DO UPDATE with a guarding WHERE reports zero rows rather than
raising, and a silent no-op is the fail-open this gate exists to prevent.
The refusal is a typed error the handler maps to 403, downcast out of the
anyhow chain the way sqlx errors already are, so it does not render as a
500.

Observed at c83cbc5 before the fix: an unsigned announce naming an
existing peer returned 200 OK and left the attacker's URL stored. That
test is the RED this change turns green.

Two of #270's reachability tests now drive the proven path. Under this
rule an unproven repoint is refused, so they can no longer express a URL
change through it; they assert the same behavior as before.
2026-08-03 20:38:53 -05:00
beardthelion 6a405f65f1 fix(release): scope the annotation check to the [package] table
The coverage gate matched `# x-release-please-version` anywhere in a manifest,
so an annotated `version` under any other table satisfied it. A crate could
leave the version cargo actually reads unannotated, pass the gate, and still
freeze at release time, with release-please rewriting the decoy line instead:
the exact failure this gate exists to catch.

Scope the match to the [package] table.
2026-08-03 09:45:17 -05:00
beardthelion 9bba8f9934 test(node): restore the ipfs-max-repos-walked range test the #174 merge dropped
The merge kept the knob and its clap range but carried over only the
default-value assertion, so nothing asserted that 0 and 1048577 are
rejected. Restored verbatim from the #174 side, beside its
ipfs_max_repo_visits sibling.

Load-bearing: widening the range to 0..=2_097_152 turns it red.
2026-08-03 01:16:52 -05:00
beardthelion a0e0e0bf1b Merge remote-tracking branch 'origin/fix/served-git-concurrency-cap' into fix/issue-135-ipfs-cid-tree-gate 2026-08-03 00:52:56 -05:00
beardthelion aeb6895141 test(node): stop the repos.rs structural gates splitting at the wrong marker
Three gates scanned the "production half" of api/repos.rs by splitting at the
first cfg(test) attribute. Adding test-only items above the code they check
moved that split point above every line being scanned, so all three failed
looking for production code that was still right there.

They now split at the test module, which is what they meant. This is the same
defect already fixed in the ipfs gate during the merge, and no split-on-the-
attribute site remains.

Both behavior-bearing gates were re-proven load-bearing: F2 goes red when the
Pinata object set is re-derived before the pin permit is taken, and the tail
gate goes red when release stops consuming the same success flag. F3 is a
presence check whose marker cannot be removed by a single compiling edit, so it
is not mutation-proven here.
2026-08-02 20:16:58 -05:00
beardthelion d1fc9bc835 fix(node): retry the coalesced drain's re-read instead of losing the push
The drain read the repo row and its visibility rules once each. Either read
failing dropped that coalesced push's pins and recovery copy for good: the repo
read returned None on error, and the rules read collapsed through ok(), which
makes the repo read as not announceable. Both fail closed, so nothing leaked,
but nothing sweeps the lost work up either. A single database blip was enough.

resolve_drain_object_list now refreshes through a bounded retry: three
attempts, 50ms doubling, and either read failing is transient. A missing repo
row is the one terminal answer, so it releases at once and spends no retry
budget. Exhaustion logs at ERROR with the repo id and attempt count, so giving
up is observable rather than silent.

The coalescing loop is untouched. It already has the shape the retry needs: a
refresh that gives up skips the pin call, and the next finish_or_take_pending
picks up anything that coalesced while the retry was running.

Eight restored tests cover it, written and observed red against the seam-only
tree first. The five the plan predicted red came up red for the reasons it
predicted, and the three controls stayed green. All eight are mutation-proven:
no-retry, an off-by-one bound, gone burning budget, a rules error read as
empty, an unfiltered candidate set, a duplicated refresh, break-on-giveup, and
a pending slot that never drains each turn their own test red.
2026-08-02 20:16:58 -05:00
beardthelion 8557a12e59 test(node): add a test-only fault seam for the drain's two re-reads
A real Postgres pool will not fail on demand, so the drain's error arms have no
way to be driven from a test. This ports the injection table the removed
requeue loop used, keyed on the task's repo id, which is a fresh uuid per test
so parallel cases cannot cross-inject.

The two reads now go through drain_get_repo and drain_list_rules, whose bodies
consult the seam under cfg(test) and then make the same call as before. Error
handling is untouched: the repo read keeps its arms, the rules read keeps its
ok() collapse, and no retry exists yet. The suite is unchanged at 856, which is
the point.

inject and counters carry a dead_code allow until the tests that call them
land.
2026-08-02 18:50:03 -05:00
beardthelion 979d3c45bb test(node): restore the four u3 drain tests onto the coalescer
These covered post-push work being requeued rather than dropped, and were
removed in the merge because they drove the replaced requeue loop. The
behaviour they check is still there, so they re-seam onto the coalescer without
any production change.

The seam moved in three ways. The inflight key is the repo identity key, not
the row id. A coalescing push now merges its (old, new) tip pairs into the
pending slot, so a test that passes an empty vec merges nothing and the drain
never laps: every coalesce carries real commit oids. And the full-scan arm is
reached through the pending-slot overflow, so the leak test coalesces 1025
pairs and asserts the slot degraded to FullScan before the run.

run_encrypt_pin_task_for_test builds the task context the way the production
spawn site does and takes a real owner and name, because the drain re-fetches
by owner and name; a blank name would resolve Gone on every lap.

All four are mutation-proven: drain-does-nothing, stale-empty-rules,
full-scan-skips-the-fail-closed-filter, and a leaked inflight key each turn
their own test red.
2026-08-02 18:39:33 -05:00
beardthelion 3ca35d216d test(node): restore three guard tests the #174 merge dropped
The merge deleted coverage that no longer compiled against the reconciled
designs, rather than re-seaming it. Three of those tests have counterparts in
the merged tree after all:

unlock_error_connection_close_is_bounded and
write_guard_dropped_off_runtime_disposes_the_connection came back with
close_conn_bounded and the Drop backstop, which the lock-pool design needed
after all (the after_release hook cannot free a lock when the unlock errors on
a live session). Both now observe the store's derived lock pool rather than the
pool handed to for_testing, which is where the guard's connection actually
lives.

get_by_cid_caps_repos_walked_knob_bounds_the_walks was orphaned when its body
was spliced into a sibling test. It drives GITLAWB_IPFS_MAX_REPOS_WALKED, which
is the binding cap only now that the gate takes the tighter of the two walk
ceilings.

The one test not restored is write_guard_release_when_not_locked: the guard has
a single construction site and it sits below the lock-or-bail, so a guard that
never took the lock cannot exist.
2026-08-02 18:39:33 -05:00
beardthelion e5c1da2bc6 Merge #174 into the CID tree-gate branch
Reconciles the served-git concurrency cap with the /ipfs CID tree gate. The two
branches independently reworked the same three seams, so most of this is picking
one mechanism per seam rather than taking a side per file.

/ipfs walk admission: keeps this branch's provenance-first resolver and its lazy
legacy scan, but drops the detached serve task in favour of #174's shared
Arc<WalkAdmission>, cloned into each spawn_blocking. Both close the permit-release
bypass; the Arc does it without leaving an abandoned request's full legacy scan
running against a held slot. Ported in with it: the request-budget gate at all four
stages with each child deadline clamped to the remainder, the walk timeout computed
inside the closure so queue delay is charged, taint sources in place of a single
truncated flag, and the deterministic-fault 500 gated on nothing else having tainted.

The two per-request ceilings are now composed rather than one shadowing the other.
The visit ceiling was unwired by the merge and the walk cap answered to only one of
the two knobs; both are honoured, the tighter one winning.

Object probe: object_type_bounded takes a Duration and returns ProbeError, so this
branch's call shape keeps #174's absence-vs-unreadable-store discriminator. A clean
`missing` from cat-file is byte-identical whether the object is absent or the pack
is unreadable, and the discriminator is store readability, not git's wording.

Write lock: the advisory-lock pool's after_release hook does not cover an unlock that
errors on a live session. Measured: a poisoned connection returned to the pool still
held the lock 15s later, because the hook's own unlock_all fails the same way. So the
guard now disposes of that connection, in release and in the Drop backstop.

Post-push replication goes to #174's coalescer. This branch's requeue_faults
integration suite has no equivalent seam on that design and is not carried over; it
is recoverable from the pre-merge head.

Also clamps the provenance path's three per-source lookups to the request budget.
They run while the walk permits are held, which is the stall hazard the preload
clamp already covered, and the structural guard found them because it now requires
every occurrence to be wrapped instead of assuming exactly one.

1346 tests pass across the workspace; fmt and clippy are clean under --locked. The
four re-anchored structural guards were mutation-tested and all four go red on the
defect they name.
2026-08-02 16:09:57 -05:00
beardthelion 4f424693b8 fix(release): bump icaptcha-client with the rest of the workspace
icaptcha-client became a workspace member in #138, after #130 last wrote
release-please-config.json, and nothing updated extra-files. release-please
rewrites a version line only when the line carries the annotation AND the
manifest is listed, so the crate was skipped every release and sat at 0.4.0
while the other five walked to 0.7.1.

Nothing broke: every intra-workspace path dep is bare, so no version
requirement could go unsatisfied; no crate is published to a registry; and
the crate reads no CARGO_PKG_VERSION of its own. The cost was the workspace
Cargo.toml comment asserting a lockstep rule the config did not keep, which
is what the next person adding a crate would read and copy.

Add the missing entry, and gate the rule so it cannot silently lapse again.
The gate derives its required set from `cargo metadata`, so a new crate reds
CI whether or not its author knows the gate exists, and it covers all three
release-silent shapes: a member absent from extra-files, a member whose
version line lost the annotation, and an extra-files path that is no longer
a member.

Next release moves icaptcha-client 0.4.0 to the current workspace version in
one step. Nothing depends on that version, so the jump is cosmetic.
2026-07-30 08:54:33 -05:00
beardthelion 219fbdbdba test(node): cover the sweep's boot wiring and prove an interrupted push never reaches the upload
Two claims from earlier commits were argued rather than executed. Both are now
observed.

The legacy-CID sweep's spawn sat inline in main() where no test could reach it,
so the batch size, the inter-batch delay, the detachment and the shutdown select
were all reasoned. The block moves verbatim into `spawn_legacy_cid_sweep`, which
a test calls directly. It seeds rows whose keys are already canonical, so the
cost gate skips each and the persisted cursor reads out walk progress cleanly,
then asserts the cursor stops at the configured batch boundary, the call returns
without blocking, the task is still alive inside its delay, and shutdown ends it.
Never spawning, reading the batch or delay from the wrong knob, or removing the
shutdown branch each turns it red. What remains reasoned is the single call line
inside main(), down from a twenty-six line block.

The claim that a disconnect-interrupted push never uploads to Tigris was
structural: the tests build the store with no Tigris client, so a counter inside
the client arm could never move and would have proved nothing. The counter now
sits at the upload decision, before the client is consulted, and the disconnect
test asserts zero while the success test asserts exactly one. That positive
control is what makes the zero meaningful, and it is itself load-bearing:
releasing the success path with failure semantics drops it to zero, and adding a
Drop impl that uploads raises the disconnect count to one.

The counter observes reaching the upload site, not an S3 request. TigrisClient
wraps a concrete SDK client with no injectable endpoint, and plumbing one through
would be out of proportion to the risk here.
2026-07-27 21:44:31 -05:00
beardthelion d85416127b fix(node): shed 503 from the issue and PR write paths when the lock pool is exhausted
The lock-pool work gave `acquire_write` a distinct exhaustion error and mapped it
to 503 with Retry-After, but only on the push handler. The three other callers,
both issue writes and the PR merge, still turned it into a generic git error and
returned 500. Those are exactly the callers that hold no concurrency permit, so
they are the likeliest to meet an exhausted pool in the first place.

All three now go through the push handler's helper rather than repeating the
mapping, so the log level and the message cannot drift between the four sites.
Each has a test that occupies the single connection of a one-slot pool with a
guard on a different repo, so the failure is pool capacity rather than
advisory-lock contention, and asserts status 503 plus retry-after through
IntoResponse. Reverting the mapping at one site turns only that site's test red.

Worth knowing for review, and left alone deliberately: close_issue takes the
write guard before its author check, so this shed is reachable by any
authenticated caller whether or not they could close the issue. That ordering
predates this change and reordering the handler is not this commit's business,
but it does mean the 503 discloses that the repo's write pool is busy.
2026-07-27 21:14:55 -05:00
beardthelion ef9b7165cf fix(node): keep the incompleteness marker until a source is really recorded, and let the sweep retry
Four follow-ups from an adversarial pass over the previous commits.

The marker clear ran unconditionally inside `record_pin_source`'s transaction,
but its INSERT is `ON CONFLICT DO NOTHING` guarded by the cap, so it records
nothing when the repo is already a source. That is the common case, not a rare
one: the pin path calls `record_pin_source` for every already-pinned object, and
on a requeue pass that is the whole-repo enumeration. So a re-push from the repo
that was already recorded cleared a marker set by a different repo's failure and
the public copy went back to 404ing. The clear is now gated on the insert
actually adding a row. A genuine record from a third repo still clears the
marker, which needs a per-(oid, repo) table to close and is documented rather
than papered over.

The sweep advanced its cursor before every skip, which stops a wedge but made a
transient skip permanent: once the walk reached the end of the table the cursor
parked there and no later boot read another row. On a Tigris-backed node the
common skip is "repo is not on local disk", so the sweep would repair whatever
happened to be warm at boot and then never run again, and the new advertise
filter turned those rows from "advertised but 404" into "never advertised, never
repaired". Repairs now report whether a skip was retryable or terminal, and a run
that skipped anything retryable rewinds the cursor once, after the walk, so the
next run re-walks. A permanently unrepairable row is terminal and does not
trigger a rewind, so this cannot become a hot loop.

The sweep also reached its repo paths through the raw join helper, skipping the
three-layer traversal barrier `RepoStore::local_path` applies. A test with a repo
row named `../../escapee` showed it reading and repairing bytes from outside
`repos_dir`. That barrier is now extracted and shared, so the two callers cannot
drift. And the repair's synchronous `git cat-file` moved under `spawn_blocking`,
matching the existing treatment of the withheld walk, so a wedged read no longer
parks a worker thread for the full service timeout.

Finally the resolver's two source-set queries ran ahead of the work-budget peek
that exists to shed a spent caller before doing work. The peek now runs first. A
throttled caller whose set turns out complete gets 429 where it used to get 404,
which is the honest answer since its search never ran, and it removes an oracle
that let a throttled caller tell a complete source set from an incomplete one.
2026-07-27 18:29:01 -05:00
beardthelion 752fd19b6d fix(node): stop the lock pool from being occupied by spinners and racing the reaper
Three regressions from the previous four commits, each demonstrated before being
fixed.

The advisory-lock retry loop checked its connection out before spinning, so a
caller that lost the lock held a pool connection through all sixty one-second
attempts. The old pool-per-query shape returned it between iterations, so a
spinner occupied nothing. That matters more than the pool sizing suggested:
`acquire_write` has three non-push callers (two in issues, one in pulls) that
hold no concurrency permit, and the issue routes carry no rate limit, so any
self-minted DID could fire enough concurrent closes at one repo to strand the
pool and fail authenticated pushes on every repo. The checkout now happens per
attempt and a losing attempt returns the connection before sleeping, so only the
winner retains one. Lock-pool exhaustion also carries its own error now and sheds
503 with Retry-After instead of surfacing as a 500.

Holding the lock on one connection then meant a client disconnect released it
early: the handler future dies mid-receive-pack, the guard drops without
`release`, and `after_release` frees the lock while the detached reaper is still
giving the process group its SIGTERM grace. A second writer was admitted 90ms
after the drop, well inside that window, which is the invariant smart_http
already documents. The write guard now rides the AdmissionGuard into the reaper,
the same seam the admission permits already use, so the lock outlives the group.
The success path still reclaims the guard and releases it synchronously, and a
dropped guard performs no Tigris upload, so an interrupted push cannot publish a
half-applied repo.

The requeue loop's exit on an exhausted re-read discarded a push that coalesced
during the retry window: `requeue_or_release` had cleared the dirty bit without
marking the guard released, so `Drop` removed the key and no pass was ever
attempted. It now falls through with nothing to replicate and lets the tail's
atomic check-and-clear decide. A lap only happens when a push actually coalesced
and each pays a full bounded re-read, so a sustained outage still terminates,
which is tested with ten thousand injected faults under a watchdog rather than
argued.

Also clamps the lock pool's derived size, which previously followed
max_concurrent_git_pushes up to its million-connection ceiling, and documents the
node's total connection budget on that knob.
2026-07-27 17:47:56 -05:00
beardthelion 412d04e12d fix(node): sweep legacy provider-CID pins instead of waiting for a re-push
`repair_legacy_provider_cid` had exactly one trigger: the already-pinned skip
branch inside `pin_new_objects`, which a push only reaches when it re-carries the
object. Normal git negotiation omits objects the node already has, so on an
upgraded node that push generally never arrives. Meanwhile `list_pinned_cids`
kept advertising the stored provider CID that this branch's `/ipfs/{cid}`
deliberately withholds, so the node handed out CIDs it would then refuse to
serve, and previously pinned data stayed unresolvable.

Two halves, both needed. A detached background sweep walks `pinned_cids` in
bounded batches with an inter-batch delay, resolves each row's repo from its
recorded provenance, and reuses the existing repair (and its cost gate, so a row
already keyed on a raw CIDv1 reads no bytes). Migration v16 adds a single-row
cursor table so a restart mid-walk resumes rather than rewinding; the cursor
advances on every row read, before any skip, so an unrepairable row cannot wedge
the walk. A row whose bytes are gone, whose provenance is NULL, or whose repo is
not on local disk is left exactly as it is. And `list_pinned_cids` now omits rows
whose key is not a raw CIDv1, so the window before the sweep catches up
advertises nothing the resolver withholds.

The sweep resolves paths directly rather than through `repo_store.acquire`, which
would pull cold repos back from Tigris and turn a repair pass into a bulk
restore. Defaults are deliberately quiet: 64 rows per batch, 60s between batches.

Ten tests cover the repair, the bytes-gone and unrepairable arms, the cost gate,
the batch bound, cursor resumption, both states of one row through the advertise
filter, and the degenerate empty and no-legacy-rows tables. Removing the filter,
the batch bound, the cursor advance, the bytes-gone early return, or the
inter-batch sleep each turns a test red.

One pre-existing test used placeholder resolver keys that the new filter drops;
its values are now real raw CIDv1s with every assertion unchanged.
2026-07-27 15:53:26 -05:00
beardthelion ff189ceab9 fix(node): record a failed pin-source write so the resolver keeps its fallback
The resolver treated a non-empty, below-cap pin-source set as proof that every
source had been recorded, and skipped the bounded scan on that basis. But
`record_pin_source` is best effort at every call site: the ipfs_pin sites were
warn-only once their retries exhausted, and both pinata sites had no retry at
all. An object first pinned from a private repo and later pushed from a public
one whose source write failed keeps a set naming only the private source, so
`GET /ipfs/{cid}` returned 404 for an object the public repo would have served.

Migration v15 adds `pinned_cids.pin_sources_incomplete`, NOT NULL DEFAULT FALSE
so every existing row reads as complete. A record that fails outright sets it; a
later successful record clears it, inside the same transaction as the insert so
the two paths cannot drift. `needs_scan` now ORs the marker in, short-circuiting
after the existing empty and at-cap checks so it costs nothing on the serve path.
Pinata's two bare calls now share `retry_db_record` with ipfs_pin. The ipfs_pin
first-pin path additionally records the CID and its source in one transaction;
Pinata's remains two retried calls.

INV-10 is preserved: a complete set still tail-404s with no preload, asserted on
the existing preload counter. The fallback is not an authorization bypass, and
the test for that asserts the scan actually ran before checking the denial, so
the denial cannot pass vacuously. Dropping the marker term from the gate, forcing
it true, or removing the clear each turns a test red.

The marker is per-object rather than per-(object, repo): a successful record from
one repo clears a marker set by another's failure. That is the conservative
direction, since the marker only ever adds the fallback back.
2026-07-27 14:40:03 -05:00
beardthelion 2961ba73ec fix(node): retry the requeue re-read so a coalesced push is not silently dropped
`requeue_or_release` clears the dirty flag atomically with the decision to loop,
which is correct and stays as it is. The problem was what the pass did next: a
failed `get_repo_by_id` fell into an Err arm that yielded no rules, and
`list_visibility_rules(...).ok()` collapsed its own error the same way. With no
rules, `replication_withheld_set` returns `(false, None)`, the pass skipped its
work entirely, and the loop exited having consumed the dirty bit that stood for
the coalesced push. There is no reconciliation sweep to re-derive it, so that
push's objects were never pinned and its withheld blobs never sealed.

`requeue_refresh_state` now separates the three cases the old code collapsed.
`Ok(Some)` proceeds. `Ok(None)` means the repo really is gone and releases
without touching the retry budget. `Err` from either read is transient and is
retried with doubling backoff up to three attempts. A rules-read error is no
longer indistinguishable from a repo that has no rules.

Exhausting the retries still drops the pass, which is the pre-existing residual,
but it now logs at error level with the repo id and attempt count instead of
disappearing behind a warning. Six tests cover the retried-then-lands path, the
bounded-exhaustion path, repo-gone, the rules-read arm, freshness of the applied
rules, and the coalescing property that had to survive. Reverting the Err arm,
restoring the `.ok()`, raising the bound, or downgrading the error log each turns
a test red.
2026-07-27 13:52:44 -05:00
beardthelion a4f7d5dffb fix(node): own the repo write lock on one connection so cancellation cannot leak it
`acquire_write` took `pg_try_advisory_lock` through the shared pool, then awaited
the Tigris exists/download path before constructing `RepoWriteGuard`. Two bugs
fell out of that. The `tokio::time::timeout` this branch added around the call
can drop the future inside that window, after the lock is held and before the
only unlock (`RepoWriteGuard::release`) exists, wedging every later push to the
repo. And because a session advisory lock belongs to the connection that took
it, running the lock and the unlock through the pool lets them land on different
connections: the unlock returns false and leaks, while a competing acquire that
draws the holding connection re-enters the lock and two writers run against the
same repo concurrently.

`RepoStore` now owns a dedicated lock pool built with an `after_release` hook
that runs `pg_advisory_unlock_all()`. `acquire_write` checks out one connection,
takes the lock on it, and moves it into the guard, which unlocks on that same
connection. sqlx's `PoolConnection::drop` spawns `return_to_pool`, which runs the
hook, so a connection dropped by cancellation still clears its locks; the unlock
is asynchronous with respect to the drop, which the tests account for. The pool
is separate from the main query pool because a push holds its connection for the
whole receive-pack and `db_max_connections` (20) sits below
`max_concurrent_git_pushes` (32).

Seven tests cover the cancellation window, the same-session release, and the
serialization the pool-mismatch bug was breaking. Both the cancellation
regression and the concurrent-writer case were observed red before the fix, and
removing the `after_release` hook or moving the unlock back to the pool turns
them red again.
2026-07-27 13:22:45 -05:00
beardthelion cc41d37bef fix(node): bound the post-push pin and repair reads so a wedge cannot pin the coalescing key
A cross-model adversarial pass found that run_post_push_replication holds the
per-repo coalescing key until requeue_or_release, but pin_new_objects reached it
only after two unbounded git reads: the U7 legacy-repair read and the
pre-existing pin read, both plain Command::output with no teardown. A wedged
cat-file on a stuck backend hung the task forever, so the key was held until
process death and later pushes only marked it dirty without spawning a
replacement (the same class this PR closed on the /ipfs serve path).

Add read_object_bounded (the bounded twins under one shared deadline) and use it
at both sites; a timeout skips that object and lets the task proceed to
requeue_or_release. Adds a wedge-reap regression proven load-bearing two ways.
2026-07-20 21:06:10 -05:00
beardthelion 9cef54ea18 fix(review): share one deadline across the /ipfs size+content reads
Code review found each candidate's size and content reads were each granted a
full git_service_timeout, so a single served candidate could hold the /ipfs
walk permit for 2x the timeout. Share one deadline across both stages (mirrors
build_filtered_pack) so the read holds admission for one timeout total, and
correct the admission-bound comment: the worst case is O(candidates x timeout)
bounded by the walk concurrency cap, not a single deadline.
2026-07-20 20:27:35 -05:00
beardthelion 15bdc5e64a fix(node): opportunistically repair legacy provider-CID pins
Before this PR the pin path stored the provider CID (Kubo dag-pb / Pinata) in
pinned_cids.cid, which the new resolver recomputes-and-404s, while
/api/v1/ipfs/pins still advertises the stale value. Repair a legacy row in the
already-pinned skip branch: gate on a cheap stored-CID codec check (only a
non-raw-codec CID reads bytes, so raw rows keep the DB-only skip cost),
recompute the raw CID, and rewrite it while stashing the old value in a new v14
legacy_provider_cid column (distinct from pinata_cid, which gates the Pinata
pin-skip). Rows whose bytes are gone stay withheld. Documents that the deferred
one-shot sweep, not this opportunistic path, fully retires the window.
2026-07-20 20:11:42 -05:00
beardthelion c04c4a5da4 fix(node): retry pin-source and pinned-CID recording on transient errors
The three warn-only record sites in the detached post-push task dropped a
transient DB failure silently, leaving a permanently incomplete pin-source set
that makes the resolver 404 a valid public copy. Wrap them in a bounded retry
(inserts are idempotent via ON CONFLICT DO NOTHING); on exhaustion the warn
still fires so behavior degrades to today's. Documents the residual crash/outage
window that only a reconciliation sweep retires.
2026-07-20 19:50:23 -05:00
beardthelion 20c338eca5 fix(node): split /ipfs work accounting off the route rate limiter
The route middleware charged ipfs_rate_limiter once per request while the
legacy scan and provenance walk charged the same bucket per probe/walk, so a
one-probe request cost two tokens and GITLAWB_IPFS_RATE_LIMIT=1 admitted the
request at the route then 429'd it at the pre-scan peek.

Add a bounded ipfs_work_rate_limiter, move the in-scan, per-walk, and pre-scan
charges onto it, and leave the route limiter as the pure once-per-request
brake. The work-budget capacity derives from the route limit with a floor of
one full legacy-probe budget per window (no new operator knob), so a
default-config deep search never self-throttles. Adds it to the periodic sweep,
updates all three AppState construction sites, and rewrites the contradictory
limiter comments.
2026-07-20 19:41:57 -05:00
beardthelion 86fc2763a3 fix(node): wire GITLAWB_IPFS_MAX_REPOS_WALKED to the legacy-probe budget
The knob was parsed and documented as the per-request /ipfs fan-out cap but
never read in production: AppState seeded the legacy-probe budget from a fixed
256 constant, so setting the knob did nothing and an operator could still incur
256 acquire/probe operations per request.

Seed ipfs_max_legacy_probes from the knob (default raised to 256 to preserve
shipped behavior); leave the history-walk ceiling on its MAX_PIN_SOURCES+1
constant so a provenanced request is never truncated into a false 503. Updates
the README and .env.example to the shipped behavior and adds cap-honored and
plumbing tests.
2026-07-20 19:17:12 -05:00
beardthelion 6222775bb1 fix(node): requeue coalesced pushes instead of dropping them
Per-repo coalescing dropped every push that arrived while a repo's encryption
task was in flight, and no reconciliation ever processed it, so a withheld
blob added by the coalesced push had its recovery copy permanently absent. The
drop also silently skipped the coalesced push's local-IPFS pin work.

EncryptInflight becomes a dirty-flag map; the detached task loops, and the
atomic check-and-clear sits at the task tail, unconditional on the
has_path_scoped_rule gate and walk success (a public or rules-free repo would
otherwise exit before it ran). Each requeue re-reads repo state fresh and
re-enumerates the pin half through the fail-closed full scan, never bare
list_all_objects, so a coalesced rule change is honored and no withheld or
dangling object leaks. At-most-one-task-per-repo is preserved.
2026-07-20 19:04:55 -05:00
beardthelion b881447b59 fix(node): own /ipfs admission for the life of the walk
get_by_cid held the global walk permit and the per-source permit as handler
locals, so dropping the request future released both while the spawn_blocking
probe, visibility walk, and object read kept running; a cancel-spam or timeout
client could exceed MAX_CONCURRENT_IPFS_WALKS and the per-source cap, and the
bare cat-file probe/read children had no teardown at all.

Move the gated serve pipeline into a detached task that owns an AdmissionGuard,
awaited by the handler, so admission releases only after the bounded work is
gone; back the /ipfs probe and read with run_bounded_git twins for
process-group teardown. Adds cancel-mid-walk, timeout-reap, and cancel-spam
regressions plus two load-bearing INV-22 rows. Also reflows one U1 test line to
satisfy cargo fmt.
2026-07-20 18:34:09 -05:00
beardthelion de2ad5a83d fix(node): hold read admission through the filtered-pack reaper
The path-scoped upload-pack branch kept the read and per-caller permits as
handler locals and called upload_pack_excluding with no AdmissionGuard, so a
cancelled filtered clone released admission while KillGroupOnDrop was still
reaping the git group, bypassing the read and per-caller concurrency caps.

drive_git_child now returns the disarmed guard on success so one guard rides
both build_filtered_pack stages (rev-list then pack-objects); the handler builds
the guard from the permits as the plain path does. Adds a disconnect regression
and an INV-22 gate row, both proven load-bearing.
2026-07-20 17:54:49 -05:00
beardthelion 1e2f0d1dd6 fix(node): close the /ipfs pin-source griefing hole via a bounded scan fallback
An attacker who pins a public object from MAX_PIN_SOURCES repos before the
legitimate public source registers fills pin_repo_sources (record_pin_source is
first-N-wins and drops later sources silently); the buried public source is then
never reached because a non-empty provenance set suppressed the legacy scan, so
anon GET /ipfs/{cid} 404s forever for a public object (re-breaks F1).

U1: db::pin_sources_at_cap reports whether pin_repo_sources is at MAX_PIN_SOURCES
for an oid (the only observable signal that a servable source may have been
dropped, since the write cap never overshoots).

U2: get_by_cid falls back to the bounded legacy scan on a provenance miss when the
set is empty OR at_cap. The scan gates every repo through the real per-caller gate,
so it finds the buried public copy. A complete (non-full) set still fast-404s, so
ordinary denials never fan out to O(repos) (INV-10/F3); the fallback honors the
is_throttled peek.

Regression ipfs_cid_buried_public_source_still_serves_via_scan_fallback: 404 before,
200 after; pin_sources_at_cap_flips_at_max covers the boundary.
2026-07-17 10:15:56 -05:00
beardthelion 107abf9145 fix(node): align ipfs_pin return with pinata; add Retry-After to SearchIncomplete
- ipfs_pin::pin_new_objects returns the provider Hash (not the raw resolver key),
  matching the pinata twin's contract; the DB cid stays the raw content CID. The
  return is logging-only here, so this is drift-avoidance, not a functional fix.
- AppError::SearchIncomplete now carries Retry-After: 1 like Overloaded, so both
  retryable 503s from the /ipfs handler advertise the retry hint consistently.
2026-07-17 10:15:56 -05:00
beardthelion bb3ecc8f30 fix(node): tie the /ipfs walk ceiling to MAX_PIN_SOURCES + 1
A provenanced object has a bounded source set (first-pinner + up to
MAX_PIN_SOURCES additional). With the walk ceiling at 16 == MAX_PIN_SOURCES, an
authorizing public source sorting after 16 path-scoped denials was never reached:
the ceiling truncated the search and returned a false 503 for a readable object.
Raise the ceiling to MAX_PIN_SOURCES + 1 so the whole bounded provenance set is
always tried; the legacy scan stays bounded by MAX_LEGACY_PROBES_PER_REQUEST.
2026-07-17 10:15:56 -05:00
beardthelion dbadd243b4 fix(node): bound the /ipfs cat-file probes under the walk permit; backfill pinata provenance
P2-E: the object-type probe and the F6 size+read ran bare git cat-file inside
spawn_blocking while the /ipfs walk permit was held, so a wedged cat-file could
pin the global walk slot for the request's life. Bound both under
git_service_timeout_secs; on timeout free the slot (truncated -> retryable 503)
and skip the repo.

P2-D: the pinata already-pinned branch now backfills NULL first-pinner
provenance in lockstep with the ipfs_pin skip branch (consistency; the object
was already resolvable via the pin_repo_sources union).
2026-07-17 10:15:56 -05:00
beardthelion 332a5785ca fix(node): close grok P1 findings on the #173/#174 IPFS integration
P1-A: gate_and_serve held the /ipfs walk permit across a bare repo_store.acquire;
wrap it in git_acquire_timeout_secs (mirroring #174's own handler) so a cold/hung
Tigris acquire cannot pin the global walk slot. On expiry skip the repo and mark
the search truncated (retryable 503, not a false 404).

P1-B: the local-IPFS pin path recorded Kubo's provider Hash as pinned_cids.cid;
for objects above the block size that is a dag-pb root that does not hash the raw
content, so GET /ipfs/{cid} listed then 404'd them under the F2 integrity check.
Record the locally-computed raw-content CID, mirroring the pinata twin.
2026-07-17 10:15:56 -05:00
beardthelion b6bc69898c feat(node): integrate the #173 IPFS CID tree-gate onto the #174 walk-concurrency base
Rebased onto #174 (fix/served-git-concurrency-cap). #173's incremental history
cannot replay commit-by-commit because #174 rewrote the same handler files in
parallel; this single commit carries the fully-integrated tree (identical to the
verified merge result), with the follow-up fixes as separate commits on top.

Brings in #173: GET /ipfs/{cid} CID->oid resolution via pinned_cids, per-caller
path-scoped blob/tree/commit-tag gating (#135/#173 F1-F6), and the pin-source
provenance table. Adapts #173's tree/commit-tag walks to #174's run_bounded_git
so the held /ipfs walk-concurrency permit is duration-safe (F5).
2026-07-17 10:15:56 -05:00
65 changed files with 42072 additions and 1827 deletions

No files matched your search

+56 -10
View File
@@ -24,11 +24,26 @@ DATABASE_URL=postgresql://gitlawb:changeme@localhost:5432/gitlawb
# ── Database pool & startup resilience ────────────────────────────────────
# Maximum connections in the PostgreSQL pool. A cap, not a floor —
# connections open lazily. Size against the DB server's max_connections,
# remembering admin tooling opens its own pool. Each concurrent write pins one
# connection for its whole duration (the connection-affine advisory lock), so the
# node REJECTS at boot any value below GITLAWB_MAX_CONCURRENT_GIT_PUSHES + 8
# headroom — keep this comfortably above that (default 48 for pushes 32).
# remembering admin tooling opens its own pool.
GITLAWB_DB_MAX_CONNECTIONS=48
# Maximum connections in the DEDICATED advisory-lock pool, separate from the
# pool above. Every in-flight repo write pins one connection here for its whole
# duration, so this is a hard ceiling on simultaneous writes node-wide: size it
# to expected peak concurrent writers, not small. Keeping it separate is what
# stops a push burst from starving ordinary request handlers. Budget
# (GITLAWB_DB_MAX_CONNECTIONS + this) per node against the server's
# max_connections, times node count, plus admin tooling.
GITLAWB_DB_LOCK_POOL_MAX_CONNECTIONS=40
# Upper bound, in seconds, on any object-storage transfer that runs while a
# per-repo write lock is HELD (the archive download inside acquire_write and the
# upload inside release). These were free before the lock's connection was
# pinned to the guard; now an unbounded stall holds a lock-pool slot, and enough
# stalls deny every write on the node. The bound applies PER SPAN and there are
# two (the acquire-side refresh, which covers the existence check and download
# together, and the release-side upload), so worst-case slot occupancy is about
# twice this value plus the git work between them. Read it together with the pool
# size above and with GITLAWB_GIT_SERVICE_TIMEOUT_SECS.
GITLAWB_LOCK_HELD_TRANSFER_TIMEOUT_SECS=300
# Seconds a request waits for a pool connection before failing with 503.
GITLAWB_DB_ACQUIRE_TIMEOUT_SECS=5
# Upper bound on each startup connect+migrate attempt, in seconds. Keep it
@@ -94,9 +109,14 @@ GITLAWB_REQUIRE_SIGNED_PEER_WRITES=false
# Require the authenticated pusher to be the repo owner on git-receive-pack.
# A valid did:key signature is authentication, not authorization: anyone can
# sign as their own DID. When true, pushes from a non-owner DID are rejected.
# Keep false until the repo owner is ready for owner-only writes.
GITLAWB_ENFORCE_OWNER_PUSH=false
# mint a key, derive its DID and sign, so with this off every signed caller may
# push to every repository, private ones included. On by default.
#
# Set to false only for a rolling upgrade whose pushers are not yet the repo
# owner. Note that delegated and CI keys count as non-owners: UCAN git/push is
# verified but not yet honored for authorization, so they cannot push while this
# is on. See docs/RUN-A-NODE.md.
GITLAWB_ENFORCE_OWNER_PUSH=true
# Comma-separated libp2p multiaddrs.
# Example: /ip4/1.2.3.4/udp/7546/quic-v1/p2p/12D3KooW...
@@ -191,8 +211,9 @@ GITLAWB_MAX_CONCURRENT_READS_PER_CALLER=16
# many WRITE-pool slots. This one is load-bearing for push availability, since
# it is acquired before the global write permit: without it, one host minting
# disposable did:key identities could open enough slow pushes to monopolize the
# write pool and 503 every other source (owner enforcement defaults off, and
# the push rate limiter caps arrival rate, not in-flight concurrency).
# write pool and 503 every other source (the push rate limiter caps arrival
# rate, not in-flight concurrency, and owner enforcement can be turned off for
# a rolling upgrade).
# Keyed on the resolved source IP, never the DID, so a DID farm does not defeat
# them; keying granularity follows GITLAWB_TRUSTED_PROXY like the read cap above.
@@ -213,10 +234,26 @@ GITLAWB_MAX_CONCURRENT_IPFS_WALKS=32
# per-caller caps via GITLAWB_TRUSTED_PROXY; reject-before-insert bounded map).
# Default 4.
GITLAWB_IPFS_WALK_PER_SOURCE=4
# Max legacy (NULL-provenance) repos probed per single /ipfs request, bounding the
# scan-fallback fan-out (git cat-file per candidate repo) for an anonymous caller. A
# truncated scan sheds a retryable 503, never a false 404. Default 256.
GITLAWB_IPFS_MAX_LEGACY_PROBES=256
# Max repo ROWS one /ipfs request's legacy scan may read from the database. The probe
# ceiling above only counts once a probe runs, and quarantined or private repos are
# denied before that, so without this an all-denying inventory paged the whole repo
# table for one anonymous request. A truncated scan sheds a retryable 503 carrying a
# sealed `continuation` token; echo it as ?scan= to resume, so a holder buried past the
# ceiling is reachable in ceil(repos / ceiling) + 1 requests. Raising this also raises
# the per-caller /ipfs work allowance, since each page is charged to it. Lowering it
# sharpens a coarse oracle: laddering to the end reveals the node's total repo count to
# within one ceiling. Default 2048.
GITLAWB_IPFS_MAX_LEGACY_SCAN_ROWS=2048
# Max EXPENSIVE path-scope visibility walks per single /ipfs request (only a
# blob in a path-scoped repo costs a full-history walk). Over-cap repos are
# skipped without a verdict and the scan continues; if the object is then found
# nowhere the request sheds a retryable 503 instead of a false 404. Default 64.
# nowhere the request sheds a retryable 503 instead of a false 404. The effective
# cap is the tighter of this value and the node's internal per-request ceiling of
# 17, so values above 17 have no effect; lower values do tighten it. Default 64.
GITLAWB_IPFS_MAX_REPOS_WALKED=64
# Ceiling on repos one /ipfs request may VISIT past the visibility gate. Each
# visit costs a repo acquire — on a Tigris cache miss a full archive download,
@@ -237,6 +274,15 @@ GITLAWB_IPFS_MAX_REPO_VISITS=1024
# Must be 1..=3153600000 (100 years): the node derives an Instant deadline from
# this value, and a larger one cannot be represented. Default 600.
GITLAWB_IPFS_REQUEST_BUDGET_SECS=600
# Shorter budget (seconds) for the pre-walk CID resolve: the lookup that maps a
# requested CID to its git oid(s), which runs while the scarce walk admission is
# already held. A well-formed CID with no pin row does no probe and no walk work,
# so without this a stalled lookup could hold a walk slot for the whole request
# budget while nothing walked. The effective deadline is the lesser of this and
# the remaining request budget; walk and probe work stay on the request budget,
# so a slow but progressing scan is never shed by it.
# Must be 1..=3153600000 (100 years). Default 10.
GITLAWB_IPFS_RESOLVE_BUDGET_SECS=10
# Max /ipfs/{cid} requests per client IP per hour (route flood brake, distinct
# from the concurrency caps above). 0 disables. Default 600.
GITLAWB_IPFS_RATE_LIMIT=600
+32
View File
@@ -296,3 +296,35 @@ jobs:
- name: Check gitlawb-core dependency allowlist
run: bash scripts/check-gitlawb-core-deps.sh
# release-please only rewrites a crate's version when the line carries the
# `# x-release-please-version` annotation AND the manifest is listed under
# extra-files in release-please-config.json. Miss either half and the crate
# is skipped silently at release time, freezing at its creation version:
# icaptcha-client was added as a workspace member after the config was last
# written and sat at 0.4.0 through four releases before anyone noticed.
# Exhaustive-by-construction, like the allowlist gate above: the required set
# is derived from `cargo metadata`, so a newly added crate reds CI whether or
# not its author knew this gate existed.
release-versioning:
name: release versioning coverage
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Check out repository
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- name: Set up Rust toolchain
uses: dtolnay/rust-toolchain@b3b07ba8b418998c39fb20f53e8b695cdcc8de1b # stable
with:
toolchain: stable
- name: Cache cargo
uses: Swatinem/rust-cache@98c8021b550208e191a6a3145459bfc9fb29c4c0 # v2.8.0
with:
key: release-versioning
- name: Check every workspace crate is release-managed
run: bash scripts/check-release-versioning.sh
+4 -2
View File
@@ -31,7 +31,9 @@ crates/
├── gitlawb-core/ crypto primitives (DID, CID, HTTP sigs, UCAN, ref certs)
├── gitlawb-node/ axum HTTP server, git smart HTTP, P2P, GraphQL
├── gl/ CLI — identity, repos, MCP server, Base L2 names
└── git-remote-gitlawb/ git remote helper for gitlawb:// URLs
├── git-remote-gitlawb/ git remote helper for gitlawb:// URLs
├── gitlawb-attest/ provenance attestations for ref-update certificates
└── icaptcha-client/ client for the iCaptcha proof-of-intelligence service
docs/ Operator guides
scripts/ Build helpers
```
@@ -106,7 +108,7 @@ cargo test -p gitlawb-node
- **TypeScript SDK** (`@gitlawb/sdk`) — client library for the HTTP API
- **Python SDK** (`gitlawb`) — for ML/agent pipeline integration
- **UCAN chain validation** — complete the auth middleware
- **UCAN authorization** — add trusted issuer anchoring, capability checks, and authorization-aware revocation
- **Filecoin storage tier** — wire up cold storage deals
- **Documentation** — guides, tutorials, API examples
- **Node operators** — run a public node and report issues
Generated
+10 -9
View File
@@ -2630,7 +2630,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7ab67060fc6b8ef687992d439ca0fa36e7ed17e9a0b16b25b601e8757df720de"
dependencies = [
"data-encoding",
"syn 2.0.117",
"syn 1.0.109",
]
[[package]]
@@ -3002,7 +3002,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
dependencies = [
"libc",
"windows-sys 0.61.2",
"windows-sys 0.52.0",
]
[[package]]
@@ -3427,6 +3427,7 @@ dependencies = [
"sha2",
"thiserror 2.0.18",
"tokio",
"url",
"uuid",
"zeroize",
]
@@ -3542,9 +3543,9 @@ dependencies = [
[[package]]
name = "h2"
version = "0.4.13"
version = "0.4.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2f44da3a8150a6703ed5d34e164b875fd14c2cdab9af1252a9a1020bde2bdc54"
checksum = "839c0e8a181239723652be9062bb56ca5bf5f64011f73b623f6f4fc59086a228"
dependencies = [
"atomic-waker",
"bytes",
@@ -3863,7 +3864,7 @@ dependencies = [
"libc",
"percent-encoding",
"pin-project-lite",
"socket2 0.6.3",
"socket2 0.5.10",
"tokio",
"tower-service",
"tracing",
@@ -5431,7 +5432,7 @@ dependencies = [
"quinn-udp",
"rustc-hash",
"rustls",
"socket2 0.6.3",
"socket2 0.5.10",
"thiserror 2.0.18",
"tokio",
"tracing",
@@ -5468,7 +5469,7 @@ dependencies = [
"cfg_aliases",
"libc",
"once_cell",
"socket2 0.6.3",
"socket2 0.5.10",
"tracing",
"windows-sys 0.52.0",
]
@@ -5900,7 +5901,7 @@ dependencies = [
"errno",
"libc",
"linux-raw-sys",
"windows-sys 0.61.2",
"windows-sys 0.52.0",
]
[[package]]
@@ -6794,7 +6795,7 @@ dependencies = [
"getrandom 0.4.2",
"once_cell",
"rustix",
"windows-sys 0.61.2",
"windows-sys 0.52.0",
]
[[package]]
+3
View File
@@ -53,6 +53,9 @@ chrono = { version = "0.4", features = ["serde"] }
uuid = { version = "1", features = ["v4"] }
# http client
reqwest = { version = "0.12", features = ["blocking", "json", "multipart", "rustls-tls"], default-features = false }
# URL parsing (what reqwest::Url re-exports, so the shared redirect predicate can
# take a parsed URL without pulling reqwest into gitlawb-core)
url = "2"
# HMAC
hmac = "0.12"
+68 -10
View File
@@ -19,7 +19,7 @@ The mission is simple: once code is pushed to the network, it should not disappe
## What is in this repository?
This is a Rust workspace with four crates:
This is a Rust workspace with six crates:
| Crate | Purpose |
|---|---|
@@ -27,6 +27,8 @@ This is a Rust workspace with four crates:
| `gl` | The Gitlawb CLI for identity, repos, issues, PRs, bounties, tasks, peers, node status, MCP, and setup flows. |
| `git-remote-gitlawb` | Git remote helper for `gitlawb://` URLs, so normal `git clone`, `git fetch`, and `git push` can talk to Gitlawb nodes. |
| `gitlawb-core` | Shared primitives: Ed25519 identities, `did:key`, CIDs, RFC 9421 HTTP signatures, certificates, and UCAN tokens. |
| `gitlawb-attest` | Pluggable external provenance attestations for ref-update certificates. |
| `icaptcha-client` | Client for the iCaptcha proof-of-intelligence flow used to protect spam-prone writes. |
---
@@ -54,6 +56,7 @@ Good today:
- Bare git repository storage.
- Git smart-HTTP clone/fetch/push.
- RFC 9421-signed writes.
- Repository and path-scoped visibility enforcement for repository and Git content reads, with 404-shaped repository denials.
- DID identities.
- `gl` CLI workflows.
- libp2p peer discovery/gossip foundation.
@@ -63,11 +66,13 @@ Good today:
Known limitations:
- Private repository read enforcement is not wired yet. Treat public nodes as public infrastructure unless you restrict access at your proxy/firewall.
- UCAN chain validation and revocation are not complete.
- Repository write authorization is not capability-complete yet; HTTP signatures prove identity, not full authorization policy.
- Repository write authorization is not secure by default: `GITLAWB_ENFORCE_OWNER_PUSH` defaults to `false` for compatibility, so a valid HTTP Signature identifies a pusher but does not enforce owner-only pushes.
- UCAN proof chains are validated when supplied, but UCAN capabilities are not consulted by write authorization and the root issuer is not independently trust-anchored. UCANs therefore do not yet grant scoped collaborator access.
- Agent lifecycle revocation is not enforced by HTTP Signature authorization; do not rely on removing or revoking an agent record to block a compromised signer.
- Read visibility is not a blanket data-classification boundary: task, IPFS-pin, and Arweave-anchor listings are not repository-gated; withheld path names can be visible to a root reader; and later visibility changes cannot retract content already announced or externally anchored.
- Peer writes are signed by upgraded nodes, but strict signed-peer enforcement is opt-in during rolling upgrades.
- GraphQL mutations need mutation-aware auth before becoming a public write surface.
- Current GraphQL mutations require an authenticated signer, but there is no mutation-specific guardrail that prevents a future mutation from omitting that check.
- Pull-request review comments do not yet have threaded line-level anchors, and merges do not enforce approval requirements.
See:
@@ -224,6 +229,53 @@ Notes:
sends the bearer token **only** to your configured origin, never to a URL a
node advertises, so a hostile node can't capture the key or redirect the solve.
### Fetching an object by CID
```bash
gl ipfs list # CIDs this node has pinned
gl ipfs get bafkrei... > object.bin # object bytes on stdout
```
Objects that were pinned before the node started recording which repo they came
from are found by scanning its repo inventory, and that scan stops at the
per-request ceilings in the [Configuration](#configuration) table. A stopped scan
answers 503 with a resume token instead of a false "not found", and `gl ipfs get`
follows the token automatically: up to 8 resumes after the first request, so at
most 9 calls to the node, waiting between attempts for as long as the node's
`Retry-After` asks and never longer than 5 seconds.
The whole ladder runs under a 60 second wall-clock deadline. The deadline bounds
the search, not the download: each attempt gets the time left on it to produce
response headers, and once an object is found its bytes stream outside that
deadline. They are not unbounded, though. The client's own 30 second HTTP timeout
is a total request timeout, running from the start of a request until its body
has finished, so a transfer still going 30 seconds after its request began is cut
off. Waits between attempts never run past the deadline either, so a single run
spends at most around 90 seconds on the network: the deadline plus the 30 second
timeout covering the last attempt. Writing the object out sits outside both
bounds, so piping into a reader that stops reading can hold the command open
longer than that.
Two node-side brakes end a ladder early and are reported rather than retried
around. A 429 is terminal, because the node's rate-limit window is an hour and
that wait cannot be honored inside one invocation; a transient overload (a 503
carrying no incomplete-scan code) is retried on the token already held, under the
same cap, clamp and deadline. The per-IP fanout brake can also stop a ladder well
short of the 9 calls, so automatic resumption is not a guarantee of reaching the
object.
When a bound stops the ladder with a usable token in hand, the command prints the
token and the invocation that continues from it before exiting nonzero:
```txt
resume from where this stopped: gl ipfs get bafkrei... --scan <token>
```
Run that to carry on from where the scan stopped. Re-running without `--scan`
restarts at the first row, reproduces the same truncation and spends the node's
per-IP budget again, so the token is the only thing that makes progress. Tokens
are valid for an hour.
---
## Architecture
@@ -309,7 +361,7 @@ POST /api/v1/peers/announce
POST /api/v1/sync/notify
```
When `GITLAWB_REQUIRE_SIGNED_PEER_WRITES=false`, unsigned legacy peers are accepted on those two routes, but signed requests are verified when signature headers are present. Once all live peers upgrade, operators can set:
When `GITLAWB_REQUIRE_SIGNED_PEER_WRITES=false`, unsigned legacy peers are accepted on those two routes, but signed requests are verified when signature headers are present. Staged rollout relaxes who may announce, not who owns a peer row. An unsigned announce may register a previously unseen peer, and it may refresh an existing row whose `http_url` is unchanged, but changing an existing peer's `http_url` requires an RFC 9421 signature from that peer's own DID and is refused with 403 otherwise. An unsigned announce can only register a `did:key`, since that is the only method whose verifying key can be resolved and therefore the only kind of row anyone could ever correct through the signed path; any other DID is refused with 400. Once all live peers upgrade, operators can set:
```bash
GITLAWB_REQUIRE_SIGNED_PEER_WRITES=true
@@ -340,7 +392,8 @@ Important node settings:
| `GITLAWB_BOOTSTRAP_PEERS` | Comma-separated HTTP peer URLs. |
| `GITLAWB_P2P_BOOTSTRAP` | Comma-separated libp2p multiaddrs. |
| `GITLAWB_BOOTSTRAP_DISABLE_SEEDS` | Disable embedded seed peers for isolated dev/test networks. |
| `GITLAWB_REQUIRE_SIGNED_PEER_WRITES` | Require signed peer announce/sync writes. |
| `GITLAWB_REQUIRE_SIGNED_PEER_WRITES` | Require signed peer announce/sync writes. Defaults to `false` during the staged rollout below. |
| `GITLAWB_ENFORCE_OWNER_PUSH` | Require the authenticated pusher to be the repo owner on `git-receive-pack`. **Defaults to `true`.** A `did:key` signature is authentication, not authorization — anyone can mint a key and sign — so with this off every signed caller may push to every repository, private ones included. Delegated and CI keys count as non-owners: a UCAN `git/push` capability is verified but not yet honored for authorization, so they cannot push while this is on. Set `false` only for a rolling upgrade; see [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md). |
| `GITLAWB_AUTO_SYNC` | Enable automatic sync from known peers. |
| `GITLAWB_MAX_PACK_BYTES` | Max git pack body size for smart-HTTP routes. |
| `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack, receive-pack, or `info/refs` advertisement may run before it is aborted (504). Default 600. Also bounds the withheld-blob classification walk (on both the upload-pack serve and receive-pack replication paths) and the push-side pin-candidate discovery (`rev-list` / `cat-file`), each reaped via process-group teardown at the deadline. On the path-scoped upload-pack path the classification walk and the pack serve share ONE deadline, so this value bounds their combined duration rather than granting each stage a full budget: a walk that consumes it leaves the serve nothing and the clone gets a 504. Serving large path-scoped repos may therefore need a higher value than they did when each stage was budgeted separately. Accepted range is 1 to 3153600000 (100 years), since the node derives deadlines from this value and a larger one cannot be represented. |
@@ -352,9 +405,12 @@ Important node settings:
| `GITLAWB_REPO_LEASE_MAX_WAITERS` | Max pushes parked at once waiting for the same repo's write lease. Each waiter pins its buffered pack body, so this bounds that memory for a hot repo; past the cap the newest push sheds a 503 + Retry-After instead of queueing. Pushes to other repos are unaffected, and the lease holder is not counted. Default 8. |
| `GITLAWB_MAX_CONCURRENT_IPFS_WALKS` | Max concurrent `GET /ipfs/{cid}` visibility walks across all callers (own pool, disjoint from the served-git pools); over-cap sheds 503. Default 32. |
| `GITLAWB_IPFS_WALK_PER_SOURCE` | Max concurrent `/ipfs` walks a single source IP may hold. Default 4. |
| `GITLAWB_IPFS_MAX_REPOS_WALKED` | Max expensive path-scope visibility walks per `/ipfs/{cid}` request; over-cap repos are skipped and the scan continues, shedding a retryable 503 (not a false 404) if the object is then found nowhere. Default 64. |
| `GITLAWB_IPFS_MAX_LEGACY_PROBES` | Max legacy (NULL-provenance) repos probed per `/ipfs/{cid}` request, bounding the scan-fallback fan-out. A truncated scan returns a retryable 503, not a false 404. Default 256. |
| `GITLAWB_IPFS_MAX_LEGACY_SCAN_ROWS` | Max repo rows one `/ipfs/{cid}` request's legacy scan may read from the database. The probe ceiling above only starts counting once a probe runs, and quarantined or private repos are denied before that, so this is what bounds a scan over an inventory that denies the caller everywhere. A truncated scan sheds a retryable 503 carrying an opaque `continuation` token; echoing it as `?scan=` resumes the scan where it stopped. Every per-request ceiling on this path (rows, probes, visits, retained rule bytes) mints one, so each request advances the ladder by at least the rows it read and a holder buried past a ceiling is reached in a bounded number of requests, `ceil(repos / ceiling) + 1` when this row ceiling is the one that binds. No ceiling ever produces a 404. Every page is charged to the caller's `/ipfs` work allowance, so raising this raises that allowance too. Lowering it sharpens an oracle: because a truncation emits a token and a completed wrap does not, laddering to the end reveals the node's total repo count (private and quarantined included) to within one ceiling. Default 2048. |
| `GITLAWB_IPFS_MAX_REPOS_WALKED` | Max expensive path-scope visibility walks a `/ipfs/{cid}` request may run per phase; over-cap repos are skipped and the scan continues, shedding a retryable 503 (not a false 404) if the object is then found nowhere. The effective cap is the tighter of this knob and the node's internal history-walk ceiling of 17 (`MAX_PIN_SOURCES + 1`), so a value above 17 has no effect while a value below it does tighten the cap. It is charged per phase: the provenance lookup and the legacy-scan fallback get separate equal budgets, so one request can run up to twice the cap in total. Default 64. |
| `GITLAWB_IPFS_MAX_REPO_VISITS` | Ceiling on repos one `/ipfs/{cid}` request may visit (acquire + probe) past the visibility gate. Also the worst-case per-request Tigris fetch count. On exhaustion the scan stops with a retryable 503. Default 1024. |
| `GITLAWB_IPFS_REQUEST_BUDGET_SECS` | Absolute wall-clock budget for one admitted `/ipfs/{cid}` request's acquire+walk lifetime. Per-stage clamps bound the acquire and walk stages to the remaining budget, and no stage starts once it is exhausted; the scan then stops with a retryable 503. The object-type probe and content-read `cat-file` subprocesses are budget-checked before starting and each also run under their own deadline (the lesser of `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` and the remaining budget), reaped via process-group teardown, so a hung `cat-file` cannot hold the request's walk slot past it. One hang path is still unbounded: the probe's object-store readability check is a plain filesystem sweep with nothing to reap, so a wedged filesystem can hold the slot past the deadline. Default 600. Accepted range is 1 to 3153600000 (100 years), since the node derives a deadline from this value and a larger one cannot be represented. |
| `GITLAWB_IPFS_RESOLVE_BUDGET_SECS` | Shorter budget for the pre-walk CID resolve inside an admitted `/ipfs/{cid}` request: the lookup that maps the requested CID to its git oid(s), which runs while the scarce walk admission is already held. A well-formed CID with no pin row does no probe and no walk work, so without this it could hold a walk slot for the whole request budget while nothing walked, and enough such requests shed every real retrieval at admission. The effective deadline is the lesser of this and the remaining request budget, so a value above `GITLAWB_IPFS_REQUEST_BUDGET_SECS` degrades to the request budget. Only the resolve is on this clock; walk and probe work stay on the request budget, so a slow but progressing scan is never shed by it. Default 10. Accepted range is 1 to 3153600000 (100 years). |
| `GITLAWB_IPFS_RATE_LIMIT` | Max `/ipfs/{cid}` requests per client IP per hour (route flood brake). 0 disables. Default 600. |
| `GITLAWB_TIGRIS_BUCKET` | Optional S3/Tigris shared repo storage bucket. |
| `GITLAWB_PINATA_JWT` | Optional Pinata/IPFS warm-storage pinning. |
@@ -362,6 +418,8 @@ Important node settings:
Production note: change the default Postgres password before exposing a node publicly.
Legacy-pin window: releases before the CID-resolver work stored the provider CID (Kubo dag-pb / Pinata) as a pinned object's resolver key. The `/ipfs/{cid}` resolver now recomputes the raw-content CID from the object bytes and refuses to serve a key that does not match, so `GET /api/v1/ipfs/pins` can still advertise an unrepaired legacy CID that 404s. Such a row is repaired opportunistically the next time a push carries the object again (its key is rewritten to the raw CID, the old value kept in `legacy_provider_cid`), but git negotiation omits objects the node already has, so most legacy rows never re-enter a push delta. A deferred one-shot startup sweep, not this opportunistic path, is what fully retires the advertise-then-404 window. Rows whose object bytes are gone stay withheld.
---
## Optional node staking
@@ -466,8 +524,8 @@ Short-term priorities:
2. Add Docker and installer smoke tests.
3. Improve operator docs and `gl doctor` checks.
4. Harden peer writes and publish the signed-peer rollout plan.
5. Implement repo write authorization: owner checks, protected branches, and UCAN capability checks.
6. Implement private-read enforcement or remove private repo affordances until it exists.
5. Close default-open write authorization and wire trusted UCAN delegation into repository permissions.
6. Add threaded, line-level pull-request discussions and enforce approval requirements on merges.
7. Add metrics for pushes, fetches, pack sizes, peer sync, failed auth, and webhooks.
Product direction:
+32 -22
View File
@@ -28,10 +28,11 @@ We will acknowledge receipt within 48 hours and aim to release a fix within 14 d
- Every git object is content-addressed via CIDv1 (SHA-256)
- Tamper-evident by construction — a modified object changes its CID
**UCAN capability tokens**
- Bootstrap UCAN tokens issued at registration
- Capability-scoped: `git:push`, `git:fetch`, `issue:create`, `pr:open`
- JWT-format tokens with expiry
**UCAN token validation**
- Bootstrap UCAN tokens are issued at registration.
- A supplied token's signature, audience, expiry, and proof-chain attenuation are validated.
- Tokens use a signed JSON wire format with expiry.
- Capability grants are not yet consulted by repository write authorization; see the limitations below.
**Smart contracts (Base Sepolia testnet)**
- `GitlawbDIDRegistry` — on-chain DID → document registry
@@ -48,21 +49,20 @@ We will acknowledge receipt within 48 hours and aim to release a fix within 14 d
---
## Known Limitations (Planned for v0.2)
## Known Limitations
These are **documented, accepted limitations** for the current live release and should be prioritized without breaking existing nodes during rolling upgrades.
These are documented limitations of the current live release. They should be prioritized without breaking existing nodes during rolling upgrades.
### UCAN chain validation
- The auth middleware verifies HTTP Signatures and token structure, but does not yet walk the full UCAN delegation chain.
- **Impact:** A node cannot yet enforce fine-grained capability delegation. Currently, any registered agent with a valid HTTP Signature can push.
- **Mitigation:** Keep write endpoints signed, treat public nodes as public infrastructure, and treat trust scores as soft rate-limiting signals rather than authorization.
- **Fix target:** v0.2
### Repository write authorization defaults
- `git-receive-pack` verifies HTTP Signatures, but `GITLAWB_ENFORCE_OWNER_PUSH` defaults to `false` for compatibility during rollout.
- **Impact:** With the default setting, a valid signature authenticates the pusher but does not require that DID to be the repository owner.
- **Mitigation:** Set `GITLAWB_ENFORCE_OWNER_PUSH=true` on nodes where owner-only pushes are required. Confirm that every legitimate pusher uses the owner DID before enabling it.
### UCAN revocation
- Issued UCAN tokens cannot be revoked before expiry.
- **Impact:** If a keypair is compromised, the attacker retains access until the UCAN expires (default: 30 days).
- **Mitigation:** Regenerate your identity (`gl identity new --force`) and re-register to issue a new UCAN. Until revocation/blocklisting is implemented, operators should remove compromised DIDs directly from their local database.
- **Fix target:** v0.2
### UCAN delegation and revocation
- The middleware validates a supplied UCAN's complete proof chain, but a root token is accepted without an independently trusted issuer anchor. `Ucan::can` is not yet used by write handlers, so a UCAN does not grant scoped repository access.
- Agent lifecycle revocation is not checked by HTTP Signature authorization. Removing or revoking an agent record does not itself block a compromised DID from authenticating.
- **Impact:** Do not use UCANs for collaborator permissions or rely on agent-record revocation as a key-compromise response.
- **Mitigation:** Keep sensitive deployments behind operational network controls and enable owner-only push enforcement where it fits the deployment until trusted delegation and authorization-aware revocation are implemented.
### git-receive-pack authentication
- The `git-receive-pack` endpoint enforces HTTP Signature auth. Plain Git smart-HTTP clients do not generate those headers, so the `git-remote-gitlawb` helper is required for pushes.
@@ -71,10 +71,20 @@ These are **documented, accepted limitations** for the current live release and
- **Fix target:** v0.2
### Private repository reads
- Repository records have an `is_public` field and the node exposes `GITLAWB_PUBLIC_READ`, but per-repository private-read enforcement is not wired in the current live release.
- **Impact:** Do not store private repositories or secrets on public nodes.
- **Mitigation:** Run isolated nodes for non-public data and restrict network access at the reverse proxy or firewall layer.
- **Fix target:** v0.2
- Repository and path-scoped visibility checks are enforced for repository API and Git content reads. A denied whole-repository or root read returns the same 404 shape as a missing repository, so the denial does not reveal private-repository existence.
- Sparse-clone support exposes withheld path globs to callers who may read the repository root. Do not put sensitive information in withheld path names.
- `GET /api/v1/tasks`, `/api/v1/ipfs/pins`, and `/api/v1/arweave/anchors` are not repository-gated. Task records include a UCAN token; pin and anchor listings expose object and ref metadata.
- Changing a repository's visibility controls future serving, but cannot retract ref metadata or configured external pins and anchors already announced while the repository was public. Do not push secrets to an announceable repository.
- **Impact:** Visibility policies protect the repository and Git content routes they gate, not every metadata endpoint or previously published content.
- **Remaining boundary:** This read control does not address the independent write-authorization and UCAN-delegation limitations described above.
### Pull-request review enforcement
- Pull-request review comments are not yet threaded or line-anchored, and merges do not enforce required approvals.
- **Impact:** Teams must use their own review policy or external controls for merge approval requirements.
### GraphQL mutation coverage
- Existing GraphQL mutations require an authenticated signer, but a mutation-specific source-level guardrail has not yet been added for future mutations.
- **Impact:** A new mutation could accidentally omit its signer check without an explicit test fence.
### Peer route hardening rollout
- Peer announce and sync notification routes accept signed requests and verify DID matches when a signature is present.
@@ -101,7 +111,7 @@ These are **documented, accepted limitations** for the current live release and
| Key storage | PKCS#8 PEM, 0600 permissions |
| Content hashing | SHA-256 via CIDv1 |
| HTTP Signatures | RFC 9421 (Ed25519 + SHA-256 Content-Digest) |
| UCAN tokens | JWT (Ed25519 signatures) |
| UCAN tokens | Signed JSON object (Ed25519 signature) |
| On-chain | ECDSA secp256k1 (Base L2 / Ethereum) |
---
@@ -109,7 +119,7 @@ These are **documented, accepted limitations** for the current live release and
## Threat Model
gitlawb is designed to be secure against:
- **Unauthorized writes** — HTTP Signature auth on all write endpoints
- **Unauthenticated writes** — HTTP Signature auth on protected write endpoints; see Known Limitations for owner and capability authorization gaps
- **Tampered git objects** — CIDv1 content addressing detects modification
- **Identity spoofing** — DIDs derived from public keys, unforgeable without the private key
- **Centralized takedown** — no single point of control; data on IPFS + Arweave
+1 -1
View File
@@ -11,7 +11,7 @@ name = "git-remote-gitlawb"
path = "src/main.rs"
[dependencies]
gitlawb-core = { path = "../gitlawb-core" }
gitlawb-core = { path = "../gitlawb-core", features = ["redirect"] }
anyhow = { workspace = true }
reqwest = { workspace = true }
tracing = { workspace = true }
+488 -42
View File
@@ -199,9 +199,7 @@ fn handle_connect<R: Read>(
other => bail!("unsupported git service: {other}"),
}
let client = reqwest::blocking::Client::builder()
.timeout(std::time::Duration::from_secs(300))
.build()?;
let client = build_http_client()?;
// ── Phase 1: ref advertisement (GET /info/refs?service=<service>) ─────────
//
@@ -320,6 +318,58 @@ fn handle_connect<R: Read>(
)
}
// ── HTTP client ───────────────────────────────────────────────────────────────
/// Total request timeout. A pack transfer can be large and slow, so this is far
/// wider than the CLI's.
const HTTP_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(300);
/// The one client both phases use, with the redirect policy the signing surfaces
/// need.
///
/// Every request this client sends can carry RFC 9421 `Signature` and
/// `Signature-Input` headers: a push signs from the first request, and a fetch of a
/// private repo signs on the retry. reqwest strips only `Authorization`, `Cookie`,
/// `Proxy-Authorization` and `WWW-Authenticate` across hosts, so under the default
/// `Policy::limited(10)` those signature headers rode a 302 to whatever origin the
/// node named, and on a 307/308 the pack body went with them. Scope the follow to the
/// origin that issued the redirect AND to an identical request-target, which is the
/// same predicate `gl` uses.
fn build_http_client() -> Result<reqwest::blocking::Client> {
Ok(reqwest::blocking::Client::builder()
.timeout(HTTP_TIMEOUT)
.redirect(reqwest::redirect::Policy::custom(same_origin_redirect))
.build()?)
}
/// Refuse any redirect that leaves the issuing origin or rewrites the request-target,
/// and bound the chain.
///
/// Refusal is `stop`, not `error`: the 3xx comes back as an ordinary response and
/// the caller reports it through the status path it already has.
///
/// `Policy::custom` replaces reqwest's built-in limit, so the chain bound is
/// restated. It is not what makes the request finite (`HTTP_TIMEOUT` covers the whole
/// chain); it is what keeps a node that redirects to itself from costing a request
/// per round trip until that timeout. `>` and not `>=` because reqwest pushes the
/// redirecting URL onto `previous` before consulting the policy, which is how
/// `Policy::limited` reads the same counter.
fn same_origin_redirect(attempt: reqwest::redirect::Attempt<'_>) -> reqwest::redirect::Action {
let Some(previous) = attempt.previous().last() else {
// Unreachable through reqwest, which pushes the redirecting URL first, but
// the safe reading of "cannot prove same-origin" is to refuse.
return attempt.stop();
};
if attempt.previous().len() > gitlawb_core::redirect::MAX_REDIRECTS {
return attempt.stop();
}
if gitlawb_core::redirect::may_follow(previous, attempt.url()) {
attempt.follow()
} else {
attempt.stop()
}
}
// ── Smart-protocol request builders ───────────────────────────────────────────
const USER_AGENT: &str = "git/2.0 git-remote-gitlawb/0.1.0";
@@ -858,6 +908,438 @@ mod tests {
String::from_utf8_lossy(&buf).into_owned()
}
/// The signed headers must not survive a redirect off the node's origin, on
/// EITHER phase.
///
/// This is the binary git runs for `clone`, `fetch` and `push`. It built its
/// client with a timeout and nothing else, so it ran reqwest's default
/// `Policy::limited(10)`, and reqwest's cross-host header stripping covers
/// `Authorization`, `Cookie`, `Proxy-Authorization` and `WWW-Authenticate` only.
/// `Signature` and `Signature-Input` came straight through. A push signs from the
/// first request, so a hostile node answering 302 was handed a working credential,
/// and a 307/308 would have taken the pack body along.
///
/// Two mockito servers are two ports on one host, which is the boundary this
/// policy draws. The second server answers everything and expects nothing, with a
/// second mock matching on the `signature` header also at zero, so a followed
/// redirect fails whether or not the signature came with it. Phase 1 is driven
/// with a 302 and Phase 2 with a 308, the status that would carry the body.
///
/// MUTATION (RED): drop the `.redirect(...)` line from `build_http_client`.
#[test]
fn signed_requests_do_not_follow_a_redirect_off_the_node_origin() {
let kp = Keypair::generate();
let client = build_http_client().unwrap();
let mut elsewhere = mockito::Server::new();
let never = elsewhere
.mock("GET", mockito::Matcher::Any)
.with_status(200)
.with_body("bytes from the redirect target")
.expect(0)
.create();
let never_post = elsewhere
.mock("POST", mockito::Matcher::Any)
.with_status(200)
.expect(0)
.create();
let signature_seen_get = elsewhere
.mock("GET", mockito::Matcher::Any)
.match_header("signature", mockito::Matcher::Any)
.with_status(200)
.expect(0)
.create();
let signature_seen_post = elsewhere
.mock("POST", mockito::Matcher::Any)
.match_header("signature", mockito::Matcher::Any)
.with_status(200)
.expect(0)
.create();
let mut node = mockito::Server::new();
let repo_base = format!("{}/zOwner/myrepo", node.url());
let bounce_get = node
.mock("GET", mockito::Matcher::Regex(r"/info/refs".to_string()))
.with_status(302)
.with_header("location", &format!("{}/info/refs", elsewhere.url()))
.expect(1)
.create();
let bounce_post = node
.mock(
"POST",
mockito::Matcher::Regex(r"/git-receive-pack$".to_string()),
)
.with_status(308)
.with_header("location", &format!("{}/git-receive-pack", elsewhere.url()))
.expect(1)
.create();
let refs_url = format!("{repo_base}/info/refs?service=git-receive-pack");
let advertisement = build_advertisement_request(&client, &refs_url, Some(&kp))
.send()
.unwrap();
let body = b"0009done\n".to_vec();
let post_url = format!("{repo_base}/git-receive-pack");
let pack_post =
build_pack_post_request(&client, &post_url, "git-receive-pack", &body, Some(&kp))
.body(body.clone())
.send()
.unwrap();
// The other origin first: it is the assertion that names the leak, and it must
// be the one that speaks when a followed redirect makes every one of these
// fail at once.
never.assert();
never_post.assert();
signature_seen_get.assert();
signature_seen_post.assert();
bounce_get.assert();
bounce_post.assert();
assert_eq!(
advertisement.status(),
302,
"a refused redirect stops rather than errors, so the caller sees the 3xx"
);
assert!(
!advertisement
.text()
.unwrap()
.contains("bytes from the redirect target"),
"the redirect target's bytes must never reach the caller"
);
assert_eq!(
pack_post.status(),
308,
"the pack POST stops at the redirect too"
);
}
/// A same-origin hop that REWRITES the request-target is refused, even though the
/// origin never changes.
///
/// The signature binds `@path` as the literal path-and-query the helper signed, and
/// the node rebuilds it from the URI it received. An `info/refs` to `info/refs/`
/// bounce therefore presents a signature over a target the node never saw, so a
/// clone or push behind such a proxy 401s. Refusing the hop hands the caller the
/// 3xx that names what happened instead.
///
/// The target mock expects zero hits and is asserted: mockito only checks
/// `.expect(N)` when `.assert()` runs, so an unbound or unasserted mock passes
/// vacuously.
#[test]
fn a_same_origin_path_changing_redirect_is_refused() {
let kp = Keypair::generate();
let client = build_http_client().unwrap();
let mut node = mockito::Server::new();
let bounce = node
.mock("GET", "/zOwner/myrepo/info/refs")
.with_status(301)
.with_header("location", "/zOwner/myrepo/info/refs/")
.expect(1)
.create();
let target = node
.mock("GET", "/zOwner/myrepo/info/refs/")
.with_status(200)
.with_body("bytes from the rewritten target")
.expect(0)
.create();
let refs_url = format!("{}/zOwner/myrepo/info/refs", node.url());
let resp = build_advertisement_request(&client, &refs_url, Some(&kp))
.send()
.unwrap();
assert_eq!(
resp.status(),
301,
"a refused redirect stops rather than errors, so the caller sees the 3xx"
);
assert!(
!resp
.text()
.unwrap()
.contains("bytes from the rewritten target"),
"the rewritten target's bytes must never reach the caller"
);
bounce.assert();
target.assert();
}
/// The other direction, and the only same-origin hop still followed: a redirect
/// that re-issues the IDENTICAL request-target.
///
/// Without this the policy could be tightened to `Policy::none()` and nothing in
/// this crate would notice. It is also the first executed coverage of the chain
/// bound in `build_http_client`'s policy closure: the route answers 301 pointing at
/// itself, so bounded it is hit once for the original request plus `MAX_REDIRECTS`
/// follows and the call returns the 301 (a refused hop stops rather than errors).
/// Unbounded it would spin until `HTTP_TIMEOUT`, costing the node a request per
/// round trip. Its twin is `a_self_redirect_stops_at_the_chain_bound` in
/// `crates/gl/src/http.rs`.
#[test]
fn an_identical_target_redirect_is_still_followed_up_to_the_chain_bound() {
let kp = Keypair::generate();
let client = build_http_client().unwrap();
let mut node = mockito::Server::new();
let loop_route = node
.mock("GET", "/zOwner/myrepo/info/refs")
.with_status(301)
.with_header("location", "/zOwner/myrepo/info/refs")
.expect(gitlawb_core::redirect::MAX_REDIRECTS + 1)
.create();
let refs_url = format!("{}/zOwner/myrepo/info/refs", node.url());
let resp = build_advertisement_request(&client, &refs_url, Some(&kp))
.send()
.expect("the bound must end the chain, not the timeout");
assert_eq!(
resp.status(),
301,
"the chain ends by refusing the next hop, so the last 3xx is what comes back"
);
loop_route.assert();
}
// ── the node's own verification, run over what the helper actually sent ──
/// What the verifying mock made of a request that reached it.
///
/// The empty slot (`None`) is its own state and means the mock was never hit at
/// all, which is what the refusal test asserts. It must stay distinguishable from
/// [`Verdict::WrongIdentity`], because "nobody verified anything" and "something
/// verified against the wrong key" are opposite findings.
///
/// The payloads are read through `Debug` in the assertion messages and nowhere
/// else, which the dead-code pass does not count; they carry the detail that makes
/// a failure legible, so they stay.
#[derive(Debug)]
#[allow(dead_code)]
enum Verdict {
/// The chain accepted the signature AND the key it resolved is the test's DID.
Accepted,
/// The chain refused it. Carries the error so a failure reads as the actual
/// rejection rather than a bare hit count.
Rejected(String),
/// The chain accepted a signature made by somebody else. A key resolved from
/// the parsed `key_id` is read out of the artifact under verification, so an
/// accept on it alone proves consistency, never authenticity.
WrongIdentity { expected: String, got: String },
}
/// The node's `require_signature` verification, over a request this crate did not
/// necessarily build: parse the headers, recompute the content-digest from the
/// body, rebuild the signing string over `@method`/`@path`/`content-digest`,
/// Ed25519-verify. Returns the DID the signature resolved to, so a caller can pin
/// the identity.
///
/// Its twin is the hand-copy in `crates/gl/src/http.rs`, which cannot import this
/// module (this is a binary crate's test module). Keep the two textually identical
/// apart from the mockito seam around them, so an edit to one is visibly an edit to
/// both.
///
/// The production verifier both copies mirror is `crate::auth::require_signature` in
/// `crates/gitlawb-node/src/auth/mod.rs`. This is a re-implementation, not a call, so
/// an edit to that middleware has to land here too: otherwise the copies drift and
/// this test keeps passing against a rule the node has stopped applying.
///
/// It asserts internally, which is deliberate but constrains its callers: inside
/// `with_body_from_request` those assertions fire on the server thread and reach
/// the client as a transport error, not as a recorded verdict. So the identity
/// check lives in the caller as a [`Verdict`] variant, never as an assert in here.
fn node_verifies(
method: &str,
path_and_query: &str,
body: &[u8],
sig_input: &str,
sig_header: &str,
content_digest: &str,
) -> anyhow::Result<String> {
use gitlawb_core::http_sig::{build_signing_string, compute_content_digest, HttpSignature};
use gitlawb_core::identity::verify;
use std::collections::HashMap;
let sig = HttpSignature::parse(sig_input, sig_header)?;
sig.check_created()?;
assert!(
sig.missing_components().is_empty(),
"signature must cover all required components"
);
assert_eq!(sig.alg, "ed25519");
assert_eq!(
content_digest,
compute_content_digest(body),
"content-digest must match the body"
);
let vk = sig.key_id.to_verifying_key()?;
let mut values = HashMap::new();
values.insert("@method".to_string(), method.to_uppercase());
values.insert("@path".to_string(), path_and_query.to_string());
values.insert("content-digest".to_string(), content_digest.to_string());
let sig_params_value = sig_input.strip_prefix("sig1=").unwrap_or(sig_input);
let components: Vec<&str> = sig.components.iter().map(String::as_str).collect();
let signing_string = build_signing_string(&components, sig_params_value, &values)?;
let sig_array: [u8; 64] = sig.signature_bytes.as_slice().try_into()?;
verify(&vk, signing_string.as_bytes(), &sig_array)?;
Ok(sig.key_id.to_string())
}
/// Pull a header value off a received mockito request, or explain which one the
/// helper failed to send.
fn received_header(req: &mockito::Request, name: &str) -> String {
req.header(name)
.first()
.unwrap_or_else(|| panic!("the helper sent no {name} header"))
.to_str()
.unwrap()
.to_string()
}
/// Run [`node_verifies`] over a GET that arrived at the mock and record what the
/// node would have made of it, pinned to `expected_did`.
fn record_get_verdict(
req: &mockito::Request,
expected_did: &str,
slot: &std::sync::Arc<std::sync::Mutex<Option<Verdict>>>,
) {
let verdict = match node_verifies(
"GET",
req.path_and_query(),
b"",
&received_header(req, "signature-input"),
&received_header(req, "signature"),
&received_header(req, "content-digest"),
) {
Ok(did) if did == expected_did => Verdict::Accepted,
Ok(did) => Verdict::WrongIdentity {
expected: expected_did.to_string(),
got: did,
},
Err(e) => Verdict::Rejected(e.to_string()),
};
*slot.lock().unwrap() = Some(verdict);
}
/// The finding's repro, now a guard: a rewritten same-origin target must never
/// receive the advertisement's signature, and the proof is the node's own
/// verification, not a hit count.
///
/// Post-fix the hop is refused, so the slot stays empty. Pre-fix the hop is
/// followed and the slot records the Ed25519 rejection of a signature made over
/// `.../info/refs?service=git-upload-pack` and presented at `.../info/refs/...`,
/// which is the 401 an operator behind such a proxy actually sees. The verdict is
/// asserted first, so a failure speaks about verification rather than about
/// reachability.
///
/// Its paired positive control is
/// `a_direct_signed_advertisement_verifies_under_the_node_verifier`: without it, an
/// empty slot would be satisfied just as well by a harness that can never record
/// anything.
#[test]
fn a_rewritten_target_never_receives_the_signature() {
let kp = Keypair::generate();
let expected_did = kp.did().to_string();
let client = build_http_client().unwrap();
let slot = std::sync::Arc::new(std::sync::Mutex::new(None::<Verdict>));
let mut node = mockito::Server::new();
let bounce = node
.mock("GET", "/zOwner/myrepo/info/refs?service=git-upload-pack")
.with_status(301)
.with_header(
"location",
"/zOwner/myrepo/info/refs/?service=git-upload-pack",
)
.expect(1)
.create();
let recorder = slot.clone();
let did_for_target = expected_did.clone();
let target = node
.mock("GET", "/zOwner/myrepo/info/refs/?service=git-upload-pack")
.with_status(200)
.with_body_from_request(move |req| {
record_get_verdict(req, &did_for_target, &recorder);
Vec::new()
})
.expect(0)
.create();
let refs_url = format!(
"{}/zOwner/myrepo/info/refs?service=git-upload-pack",
node.url()
);
let resp = build_advertisement_request(&client, &refs_url, Some(&kp))
.send()
.unwrap();
let verdict = slot.lock().unwrap().take();
assert!(
verdict.is_none(),
"the node's own verifier must never see this request: the signature covers \
/zOwner/myrepo/info/refs?service=git-upload-pack and the rewritten target \
adds a trailing slash, so what arrives there is a stale request-target; \
recorded verdict: {verdict:?}"
);
assert_eq!(
resp.status(),
301,
"the caller sees the 3xx, not the rewritten target's answer"
);
bounce.assert();
target.assert();
}
/// The positive control for the test above, and the proof that the helper signs the
/// query it sends.
///
/// A direct signed advertisement GET, no redirect anywhere, through the same
/// verifying mock. The verdict must be `Accepted`, which is what makes the refusal
/// test's empty slot attributable to the refusal rather than to a harness that
/// cannot record. The advertisement URL carries `?service=`, so a helper that
/// signed the bare path would land here as `Rejected`.
#[test]
fn a_direct_signed_advertisement_verifies_under_the_node_verifier() {
let kp = Keypair::generate();
let expected_did = kp.did().to_string();
let client = build_http_client().unwrap();
let slot = std::sync::Arc::new(std::sync::Mutex::new(None::<Verdict>));
let mut node = mockito::Server::new();
let recorder = slot.clone();
let did_for_target = expected_did.clone();
let route = node
.mock("GET", "/zOwner/myrepo/info/refs?service=git-upload-pack")
.with_status(200)
.with_body_from_request(move |req| {
record_get_verdict(req, &did_for_target, &recorder);
Vec::new()
})
.expect(1)
.create();
let refs_url = format!(
"{}/zOwner/myrepo/info/refs?service=git-upload-pack",
node.url()
);
let resp = build_advertisement_request(&client, &refs_url, Some(&kp))
.send()
.unwrap();
assert_eq!(resp.status(), 200);
let verdict = slot.lock().unwrap().take();
assert!(
matches!(verdict, Some(Verdict::Accepted)),
"a direct signed advertisement must verify under the node's own chain and \
resolve to {expected_did}, or the refusal test's empty slot proves \
nothing; recorded verdict: {verdict:?}"
);
route.assert();
}
/// The regression that round-1 missed: the Phase-2 `git-upload-pack` POST was
/// left unsigned, so an owner's fetch of a private repo cleared the (now signed)
/// advertisement and then 404'd on the pack POST. Drive BOTH request builders
@@ -1137,49 +1619,13 @@ mod tests {
/// to end (sign here, verify with the node's verifier), not reasoned.
#[test]
fn client_signature_verifies_under_node_verification_for_both_services() {
use gitlawb_core::http_sig::{build_signing_string, compute_content_digest, HttpSignature};
use gitlawb_core::identity::verify;
use std::collections::HashMap;
let kp = Keypair::generate();
let client = reqwest::blocking::Client::new();
let body = b"0009done\n".to_vec();
// Re-implements the node's require_signature verification (auth/mod.rs):
// parse headers, recompute content-digest from the body, rebuild the signing
// string over @method/@path/content-digest, Ed25519-verify. Ok iff the node
// would accept it.
let node_verifies = |method: &str,
path_and_query: &str,
body: &[u8],
sig_input: &str,
sig_header: &str,
content_digest: &str|
-> anyhow::Result<()> {
let sig = HttpSignature::parse(sig_input, sig_header)?;
sig.check_created()?;
assert!(
sig.missing_components().is_empty(),
"signature must cover all required components"
);
assert_eq!(sig.alg, "ed25519");
assert_eq!(
content_digest,
compute_content_digest(body),
"content-digest must match the body"
);
let vk = sig.key_id.to_verifying_key()?;
let mut values = HashMap::new();
values.insert("@method".to_string(), method.to_uppercase());
values.insert("@path".to_string(), path_and_query.to_string());
values.insert("content-digest".to_string(), content_digest.to_string());
let sig_params_value = sig_input.strip_prefix("sig1=").unwrap_or(sig_input);
let components: Vec<&str> = sig.components.iter().map(String::as_str).collect();
let signing_string = build_signing_string(&components, sig_params_value, &values)?;
let sig_array: [u8; 64] = sig.signature_bytes.as_slice().try_into()?;
verify(&vk, signing_string.as_bytes(), &sig_array)?;
Ok(())
};
// The verification chain is [`node_verifies`], the test-module helper the
// redirect verdict tests share. Same primitives the node's require_signature
// runs, over the request-target this crate transmits.
// @path exactly as the node reconstructs it from the request it receives.
let path_and_query = |req: &reqwest::blocking::Request| match req.url().query() {
Some(q) => format!("{}?{}", req.url().path(), q),
+174 -27
View File
@@ -162,16 +162,28 @@ fn start_shim(repo: PathBuf, mode: ShimMode) -> Shim {
}
}
fn handle_conn(stream: TcpStream, repo: &Path, mode: ShimMode, posts: &AtomicUsize) {
// `start_shim` makes the LISTENER non-blocking to poll its stop flag. On
// Windows the accepted socket inherits that (Winsock FIONBIO inheritance;
// std's `accept` does not reset it), on unix it does not. Every read below
// is a blocking read whose errors collapse to "peer closed", so an inherited
// WouldBlock would drop the connection with no response and abort the client
// mid-request. Normalize before reading; this also restores the read timeout
// on the next line, which a non-blocking socket silently makes meaningless.
/// `start_shim` makes the LISTENER non-blocking to poll its stop flag, and an
/// accepted socket can inherit that flag.
///
/// Observed on Windows: Winsock hands back a socket carrying the listener's
/// FIONBIO state and std's `accept` does not reset it. Expected on macOS and the
/// BSDs for the same reason, since the accept4 man page describes inheritance as
/// the canonical BSD behavior, though nothing in this repo's CI exercises that
/// half. Linux is the exception, because std uses `accept4` there and it does
/// not inherit file status flags (rust-lang/rust#67027).
///
/// Every read in `handle_conn` is a blocking read whose errors collapse to "peer
/// closed", so an inherited WouldBlock would drop the connection with no
/// response and abort the client mid-request. Normalize before reading; this
/// also restores the read timeout, which a non-blocking socket silently makes
/// meaningless.
fn normalize_accepted_stream(stream: &TcpStream) {
stream.set_nonblocking(false).ok();
stream.set_read_timeout(Some(Duration::from_secs(30))).ok();
}
fn handle_conn(stream: TcpStream, repo: &Path, mode: ShimMode, posts: &AtomicUsize) {
normalize_accepted_stream(&stream);
let mut reader = BufReader::new(stream);
// Request line.
@@ -757,26 +769,160 @@ fn build_divergent_repos(shared_commits: usize) -> Repos {
}
}
/// `normalize_accepted_stream` must leave the socket in blocking mode.
///
/// This asserts the helper's postconditions directly: no client, no request, and
/// no race over whether bytes arrive before the shim reads them. It removes the
/// byte-arrival race entirely, which is what the end-to-end test cannot do. It is
/// not literally deterministic, and the paragraph below says exactly what is left.
///
/// The read timeout has a getter, so it is checked exactly. Blocking mode does
/// not: std exposes none, and Winsock has no FIONBIO getter at all (`ioctlsocket`
/// documents FIONBIO with no output type, so it is set-only), so that half has to
/// be observed through behavior: a blocking socket with no data pending parks its
/// reader, a non-blocking one returns `WouldBlock` at once.
///
/// The reader owns the very socket the helper normalized. Do not reintroduce a
/// `try_clone` here: on Windows that mints a new descriptor through
/// `WSADuplicateSocketW` rather than duplicating a handle, and whether FIONBIO
/// state follows the new descriptor is a question this test would then depend on
/// and could not answer. Owning the original removes the question.
///
/// The `Started` handshake is load-bearing and must not be removed. It SHRINKS
/// the vacuity window; it does not close it. Read that literally, because the
/// looser version of this sentence is the mistake this whole test exists to
/// correct. Without the handshake the window includes thread spawn itself, which
/// is unbounded and can be very slow on a loaded runner. With it, spawn is out of
/// the window entirely and what remains is the cross-thread round trip: the
/// signal, the read call, its `WouldBlock` return, the channel send, and this
/// thread waking from `recv_timeout`.
///
/// So the residual false-green path is a reader descheduled anywhere along that
/// round trip for more than 250ms, notably between its own `Started` send and its
/// `read`. That path is real and this comment does not claim otherwise. What the
/// design does buy is that the guard cannot flake RED, and that no arrangement of
/// bytes on the wire can fool it, only the scheduler. Measured 50 of 50 RED under
/// a mutation deleting the normalization, at full-suite parallelism; that bounds
/// the false-green rate, it does not make it zero.
///
/// Teardown order is a constraint, not a preference: the peer close must come
/// AFTER the timeout assertion. Closing first unblocks the parked read, the
/// outcome arrives, and the assertion fails even when the helper is correct.
#[test]
fn normalize_accepted_stream_leaves_the_socket_blocking() {
enum Probe {
Started,
Outcome(std::io::Result<usize>),
}
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
let client = TcpStream::connect(addr).unwrap();
let (accepted, _) = listener.accept().unwrap();
// The state Windows hands the shim. Forced here so the property is checked
// on every platform rather than only where the OS supplies it.
accepted.set_nonblocking(true).unwrap();
normalize_accepted_stream(&accepted);
// The helper's other postcondition, and the one std lets us check exactly.
// Without this, deleting the timeout leaves every test here green while a
// stalled peer parks the shim's only accept-loop thread forever.
assert_eq!(
accepted.read_timeout().unwrap(),
Some(Duration::from_secs(30)),
"normalize_accepted_stream must restore the read timeout, not only blocking mode"
);
let (tx, rx) = std::sync::mpsc::channel();
// Move, never clone: see the try_clone note above.
let mut reader = accepted;
let probe = std::thread::spawn(move || {
let mut buf = [0u8; 1];
tx.send(Probe::Started).unwrap();
let outcome = reader.read(&mut buf);
let _ = tx.send(Probe::Outcome(outcome));
});
// Bounded, not `recv()`. A bare recv here turns "the reader never ran" into a
// hung test binary rather than a failing test, and cargo test has no per-test
// timeout to rescue it. The bound is far above any real scheduling delay, so
// hitting it is a genuine defect report, never a flake.
match rx
.recv_timeout(Duration::from_secs(30))
.expect("reader thread never signalled Started (died, or never scheduled)")
{
Probe::Started => {}
Probe::Outcome(_) => panic!("reader reported an outcome before its Started signal"),
}
match rx.recv_timeout(Duration::from_millis(250)) {
// The read is still parked, which is what blocking mode means.
Err(std::sync::mpsc::RecvTimeoutError::Timeout) => {}
Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => {
panic!("the reader thread exited instead of parking in its read")
}
Ok(Probe::Started) => panic!("received a second Started signal"),
Ok(Probe::Outcome(outcome)) => {
assert_eq!(
outcome.as_ref().err().map(std::io::Error::kind),
Some(std::io::ErrorKind::WouldBlock),
"the read returned instead of parking, but not with WouldBlock \
({outcome:?}); that is a different failure than the one this test \
guards, so do not read it as a non-blocking socket"
);
panic!(
"accepted socket is still non-blocking after \
normalize_accepted_stream: the read returned WouldBlock \
immediately instead of parking"
);
}
}
// Only now: EOF unblocks the parked read so the thread can be joined. Bounded
// for the same reason as the handshake wait above; the socket's own 30s read
// timeout is the other backstop, and it is asserted rather than assumed.
drop(client);
match rx
.recv_timeout(Duration::from_secs(60))
.expect("reader did not return after the peer closed")
{
Probe::Outcome(outcome) => assert_eq!(
outcome.ok(),
Some(0),
"expected EOF once the peer closed, not payload"
),
Probe::Started => panic!("received a second Started signal"),
}
probe.join().unwrap();
}
/// The shim must answer a connection that arrives in non-blocking mode.
///
/// `start_shim` sets the LISTENER non-blocking so the accept loop can poll the
/// stop flag. On unix the socket handed back by `accept()` is blocking, so that
/// is invisible; on Windows, Winsock has the accepted socket inherit the
/// listener's FIONBIO state and std does not reset it (`accept` is a bare
/// `c::accept`), so every accepted connection is non-blocking there. The shim
/// reads with `read_line(..).unwrap_or(0)`, which collapses `WouldBlock` into
/// "peer closed" and returns WITHOUT writing a response, aborting the client
/// mid-request. Because the shim accepts as soon as the handshake completes,
/// strictly before the request bytes arrive, that window is open on every
/// connection and closes only because the accept poll usually lags the client.
/// End-to-end realism check through the real `handle_conn` request path. The
/// deterministic guarantee lives in
/// `normalize_accepted_stream_leaves_the_socket_blocking`, which asserts the
/// helper's postcondition with no client at all; see `normalize_accepted_stream`
/// for why an accepted socket can arrive non-blocking in the first place. The
/// shim reads with `read_line(..).unwrap_or(0)`, which collapses `WouldBlock`
/// into "peer closed" and returns WITHOUT writing a response, aborting the
/// client mid-request.
///
/// Force the Windows shape explicitly so this is checked on every platform
/// rather than only the one that exhibits it. The accept-before-request
/// interleaving is the losing one, so it is established by a handshake rather
/// than a sleep: a timing-based version cannot flake RED, but it CAN flake
/// GREEN on a loaded runner (bytes already buffered, so the read succeeds even
/// while non-blocking), which would make this guard vacuous exactly where it
/// is supposed to bite.
/// This test is BEST EFFORT and known to be nondeterministic. Its handshake
/// fires after `accept()` but before `handle_conn` runs, so it narrows the
/// accept-before-request window without closing it: the request bytes can
/// already be buffered by the time the shim reads, and the read then succeeds
/// even with the normalization removed. Against a deleted
/// `set_nonblocking(false)` it caught the defect in 39, 43, and 47 of 50 across
/// three full-suite samples. The spread is the point: it is a sampled tendency,
/// not a property, so do not quote a percentage off it. Running it alone on an
/// idle machine flattered it to 48 of 50; the full-suite numbers are the honest
/// ones, because that is how it ships.
///
/// It is still the only coverage for the helper's CALL SITE. The deterministic
/// probe calls `normalize_accepted_stream` directly, so deleting the call from
/// `handle_conn` leaves that probe green (0 of 50 in both samples) and only this
/// test notices, at 45 and 46 of 50.
#[test]
fn shim_answers_a_connection_that_arrives_non_blocking() {
let repos = build_divergent_repos(1);
@@ -798,8 +944,9 @@ fn shim_answers_a_connection_that_arrives_non_blocking() {
Err(e) => panic!("accept failed: {e}"),
}
};
// What Windows hands the shim. On unix this is the only way to reach
// that state, so the assertion below is load-bearing here too.
// What Windows hands the shim, and what macOS is expected to. On Linux
// this is the only way to reach that state, so forcing it is what makes
// the assertions below run on every platform.
stream.set_nonblocking(true).unwrap();
accepted_tx.send(()).unwrap();
handle_conn(stream, &repo, ShimMode::Normal, &posts);
+37 -2
View File
@@ -16,7 +16,7 @@
//! by exact match.
use base64::{engine::general_purpose::URL_SAFE_NO_PAD as B64U, Engine};
use ed25519_dalek::{Signature, Signer, SigningKey, Verifier, VerifyingKey};
use ed25519_dalek::{Signature, Signer, SigningKey, VerifyingKey};
use serde::{de::DeserializeOwned, Deserialize, Serialize};
use crate::error::{Error, Result};
@@ -111,7 +111,7 @@ impl Attestation {
.try_into()
.map_err(|_| Error::Signature("signature must be 64 bytes".to_string()))?;
let sig = Signature::from_bytes(&sig_bytes);
vk.verify(&bytes, &sig)
vk.verify_strict(&bytes, &sig)
.map_err(|e| Error::Signature(format!("ed25519: {e}")))?;
Ok(vk)
@@ -554,4 +554,39 @@ mod tests {
let err = tampered.verify_signature(cert_hash).unwrap_err();
assert!(matches!(err, Error::Signature(_)));
}
/// Regression guard for strict verification: a signature forged under a
/// weak (small-order) public key satisfies the verification equation
/// but must be rejected. The identity point is such a key: with R the
/// identity and S = 0, [S]B - [k]A is the identity for any message.
#[test]
fn verify_rejects_weak_key_signature() {
let mut weak_key_bytes = [0u8; 32];
weak_key_bytes[0] = 1; // compressed identity point (y = 1, x = 0)
let weak_vk = VerifyingKey::from_bytes(&weak_key_bytes).unwrap();
assert!(weak_vk.is_weak(), "identity point must be a weak key");
let cert_hash = sample_cert_hash();
let mut forged = [0u8; 64];
forged[0] = 1; // R = identity point, S = 0
let forged_sig = B64U.encode(forged);
// Build an attestation with the weak key as signer and the forged
// signature. The weak-key check happens at verify time.
let mut att = dummy_attestation(&SigningKey::generate(&mut OsRng), cert_hash);
let mut buf = Vec::with_capacity(ED25519_MULTICODEC.len() + 32);
buf.extend_from_slice(&ED25519_MULTICODEC);
buf.extend_from_slice(&weak_key_bytes);
att.signer = format!(
"did:key:{}",
multibase::encode(multibase::Base::Base58Btc, &buf)
);
att.sig = forged_sig;
let err = att.verify_signature(cert_hash).unwrap_err();
assert!(
matches!(err, Error::Signature(_)),
"signature under a weak (small-order) public key must be rejected"
);
}
}
+7
View File
@@ -22,6 +22,7 @@ multihash-codetable = { workspace = true }
cid = { workspace = true }
chrono = { workspace = true }
uuid = { workspace = true }
url = { workspace = true, optional = true }
zeroize = { version = "1", features = ["derive"] }
pkcs8 = { version = "0.10", features = ["pem", "std"] }
curve25519-dalek = "4"
@@ -30,3 +31,9 @@ chacha20poly1305 = "0.10"
[dev-dependencies]
tokio = { workspace = true }
# Not optional here: the redirect matrix must run under a bare
# `cargo test -p gitlawb-core`, with no feature selected.
url = { workspace = true }
[features]
redirect = ["dep:url"]
+35 -2
View File
@@ -6,6 +6,8 @@
//! The schema is frozen at v1. All fields are mandatory for forward compatibility.
//! Nodes that receive a certificate with an unknown version MUST reject it.
use std::collections::HashSet;
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use uuid::Uuid;
@@ -135,10 +137,14 @@ impl RefUpdateCert {
/// Check if this certificate satisfies a threshold of valid signatures
/// from the provided set of authorized maintainer DIDs.
///
/// Counts distinct signer DIDs, not signature entries: a repeated
/// signature from the same maintainer counts once.
pub fn satisfies_threshold(&self, maintainers: &[Did], threshold: usize) -> Result<bool> {
let valid = self.verify_all()?;
let count = valid.iter().filter(|d| maintainers.contains(d)).count();
Ok(count >= threshold)
let distinct_signers: HashSet<&Did> =
valid.iter().filter(|d| maintainers.contains(d)).collect();
Ok(distinct_signers.len() >= threshold)
}
/// Validate the certificate structure (not signatures).
@@ -344,6 +350,33 @@ mod tests {
assert!(!cert.satisfies_threshold(&maintainers, 1).unwrap());
}
#[test]
fn satisfies_threshold_rejects_duplicated_signature() {
let kp1 = Keypair::generate();
let kp2 = Keypair::generate();
let kp3 = Keypair::generate();
let repo_did = kp1.did();
let mut cert = RefUpdateCert::new(
repo_did,
"refs/heads/main".to_string(),
dummy_hash('0'),
dummy_hash('a'),
1,
&kp1,
)
.unwrap();
// Copy-paste the only real signature onto the certificate a second
// time. Same signer, same valid signature, still one real signer.
let dup = cert.signatures[0].clone();
cert.signatures.push(dup);
let maintainers = vec![kp1.did(), kp2.did(), kp3.did()];
// Two signature entries but a single distinct signer must not
// satisfy a 2-of-3 threshold.
assert!(!cert.satisfies_threshold(&maintainers, 2).unwrap());
}
#[test]
fn threshold_zero_is_always_satisfied() {
let kp = Keypair::generate();
+45
View File
@@ -64,6 +64,19 @@ impl Cid {
}
}
/// True when `s` parses as a CIDv1 with the raw codec — the exact shape
/// [`Cid::from_git_object_bytes`] produces and the `/ipfs` resolver looks up.
/// A legacy provider CID (Kubo dag-pb, Pinata CIDv0) parses to a different
/// version or codec and returns `false`, marking it an opportunistic-repair
/// candidate. Decidable from the string alone (no object bytes), so the pin path
/// can gate the byte-read/recompute cost on it and leave non-legacy rows at the
/// existing DB-only skip cost. An unparseable string is non-canonical (`false`).
pub fn is_raw_cidv1(s: &str) -> bool {
s.parse::<CidGeneric<64>>()
.map(|c| c.version() == cid::Version::V1 && c.codec() == RAW)
.unwrap_or(false)
}
impl fmt::Display for Cid {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.0)
@@ -177,6 +190,38 @@ mod tests {
assert!(result.is_err());
}
#[test]
fn is_raw_cidv1_classifies_codec_from_string() {
// The canonical resolver key: CIDv1 + raw codec → not a repair candidate.
let raw = Cid::from_git_object_bytes(b"blob 5\0hello");
assert!(
is_raw_cidv1(raw.as_str()),
"from_git_object_bytes output is CIDv1/raw"
);
// A CIDv0 (Pinata dag-pb legacy shape) → repair candidate.
assert!(
!is_raw_cidv1("QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG"),
"a CIDv0 dag-pb value is a legacy-repair candidate"
);
// A CIDv1 with the dag-pb codec (the Kubo above-block-size root) over the
// same multihash → still a repair candidate (codec, not just version).
let parsed = raw.as_str().parse::<CidGeneric<64>>().unwrap();
const DAG_PB: u64 = 0x70;
let dagpb = CidGeneric::<64>::new_v1(DAG_PB, *parsed.hash()).to_string();
assert!(
!is_raw_cidv1(&dagpb),
"a CIDv1 dag-pb value is a legacy-repair candidate"
);
// Garbage is non-canonical.
assert!(
!is_raw_cidv1("not-a-cid"),
"an unparseable string is non-canonical"
);
}
#[test]
fn sha256_hex_of_empty_input_is_well_known() {
// SHA-256("") is a fixed constant; verifies the hasher is wired correctly.
+61
View File
@@ -80,6 +80,19 @@ impl Did {
)));
}
// Refuse an oversized id before decoding. base58 decoding is quadratic
// in its input, and both callers of this function take the string from
// an untrusted request: the unauthenticated peer-announce gate, and the
// HTTP-signature keyid. An ed25519 did:key method-id is a fixed 48
// characters, so this bound is slack rather than a behavior change, and
// it has to sit ahead of the decode to be worth anything.
const MAX_METHOD_ID_LEN: usize = 64;
if self.method_id().len() > MAX_METHOD_ID_LEN {
return Err(Error::InvalidDid(
"did:key method-specific id too long".to_string(),
));
}
let (_, bytes) =
multibase::decode(self.method_id()).map_err(|e| Error::InvalidDid(e.to_string()))?;
@@ -308,4 +321,52 @@ mod tests {
assert_eq!(Did::web("example.com").method_id(), "example.com");
assert_eq!(Did::gitlawb("z6MkSomeKey").method_id(), "z6MkSomeKey");
}
/// An oversized method-specific id is refused on its length, before the
/// base58 decode runs.
///
/// base58 decoding is quadratic in the input, and `to_verifying_key` is
/// reached from two places that take an attacker-supplied string: the
/// unauthenticated peer-announce gate, and the HTTP-signature keyid. A
/// 64k-character DID measured 8s through the router without this guard
/// against 4.6ms with it, on a tokio worker with no spawn_blocking, so a
/// single caller could buy CPU-seconds by the thousand.
///
/// Asserting the MESSAGE is what makes this load-bearing: an oversized id
/// errors either way, so a test that only checked `is_err()` would pass
/// with the guard removed, having merely paid the cost it exists to avoid.
#[test]
fn to_verifying_key_rejects_an_oversized_method_id_before_decoding() {
let oversized = format!("did:key:z{}", "z".repeat(64_000));
let did: Did = oversized.parse().expect("method is still `key`");
let start = std::time::Instant::now();
let err = did.to_verifying_key().expect_err("must be refused");
let elapsed = start.elapsed();
assert!(
err.to_string().contains("too long"),
"must be refused on length, not after the decode: {err}"
);
assert!(
elapsed < std::time::Duration::from_secs(1),
"refusal must not pay the quadratic decode: took {elapsed:?}"
);
}
/// The bound must not reject a real key. Every `did:key` this project
/// mints is a fixed 48 characters, so the limit is slack, not a behavior
/// change.
#[test]
fn to_verifying_key_still_accepts_a_real_did_key() {
use ed25519_dalek::SigningKey;
let signing = SigningKey::from_bytes(&[7u8; 32]);
let did = Did::from_verifying_key(&signing.verifying_key());
assert_eq!(
did.method_id().len(),
48,
"did:key method-id is fixed-width"
);
did.to_verifying_key().expect("a real did:key must resolve");
}
}
+23 -2
View File
@@ -78,10 +78,9 @@ impl Keypair {
/// Verify an Ed25519 signature.
pub fn verify(verifying_key: &VerifyingKey, msg: &[u8], sig_bytes: &[u8; 64]) -> Result<()> {
use ed25519_dalek::Verifier;
let sig = Signature::from_bytes(sig_bytes);
verifying_key
.verify(msg, &sig)
.verify_strict(msg, &sig)
.map_err(|_| Error::SignatureInvalid)
}
@@ -209,6 +208,28 @@ mod tests {
);
}
#[test]
fn verify_rejects_weak_key_signature() {
// Regression guard for strict verification: a signature forged under a
// weak (small-order) public key satisfies the verification equation
// but must be rejected. The identity point is such a key: with R the
// identity and S = 0, [S]B - [k]A is the identity for any message.
let mut weak_key_bytes = [0u8; 32];
weak_key_bytes[0] = 1; // compressed identity point (y = 1, x = 0)
let weak_vk = VerifyingKey::from_bytes(&weak_key_bytes).unwrap();
assert!(weak_vk.is_weak(), "identity point must be a weak key");
let msg = b"forged message";
let mut forged = [0u8; 64];
forged[0] = 1; // R = identity point, S = 0
let result = verify(&weak_vk, msg, &forged);
assert!(
result.is_err(),
"signature under a weak (small-order) public key must be rejected"
);
}
#[test]
fn signed_payload_round_trip() {
let kp = Keypair::generate();
+9
View File
@@ -5,7 +5,16 @@ pub mod encrypt;
pub mod error;
pub mod http_sig;
pub mod identity;
// `url` is the one dependency here that drags a tail (idna, then the icu
// crates), and gitlawb-core is allowlisted to stay embeddable. Every client that
// needs this predicate already parses URLs, so they opt in and nothing else
// pays. `test` is in the cfg so `cargo test -p gitlawb-core` still compiles and
// runs the matrix below with no feature selected; without it the tests would
// silently not run, which is the failure this module exists to prevent.
#[cfg(any(feature = "redirect", test))]
pub mod redirect;
pub mod sanitize;
pub mod scan_token;
pub mod ucan;
pub use error::Error;
+241
View File
@@ -0,0 +1,241 @@
//! Redirect policy shared by every gitlawb HTTP client that signs its requests.
//!
//! Both `gl` (async) and `git-remote-gitlawb` (blocking) attach RFC 9421
//! `Signature` and `Signature-Input` headers, and reqwest strips only
//! `Authorization`, `Cookie`, `Proxy-Authorization` and `WWW-Authenticate` when a
//! redirect crosses hosts. A signature would survive that hop, and it binds
//! `@method`, `@path` and `content-digest` with no authority component, so a node
//! answering 302 could hand a working credential to a host of its choosing and read
//! as the caller anywhere until the clock-skew window closes. On a 307/308 the
//! request body goes along with it, which for the remote helper is the pack.
//!
//! That same missing authority component is why the predicate also pins the
//! request-target: `@path` is signed as the client sent it and verified as the node
//! received it, so a same-origin hop that rewrites the path or the query (a
//! trailing-slash or query normalization) makes the signature cover a target the
//! node never saw and the read 401s. Only a hop that re-issues the identical target,
//! an http-to-https upgrade being the one that matters in practice, is followed.
//!
//! The decision lives here rather than in either client because the two used to
//! disagree: `gl` was scoped to the origin while the remote helper, the binary that
//! actually runs `git clone gitlawb://`, still ran reqwest's default and followed
//! anywhere. One predicate is what keeps a future third client from repeating that.
//!
//! The type is `url::Url`, which is what `reqwest::Url` re-exports, so both clients
//! pass their attempt URLs straight in.
/// Longest redirect chain followed. `reqwest::redirect::Policy::custom` replaces
/// reqwest's built-in limit, so the bound has to be restated by every client that
/// installs a custom policy; the value is reqwest's own default.
///
/// This counts FOLLOWS, matching `Policy::limited`: reqwest pushes the redirecting
/// URL onto `previous` before consulting the policy, and `Limit(max)` refuses once
/// `previous.len() > max`, so a caller comparing against this constant must use `>`
/// too or it permits one hop fewer than it says.
pub const MAX_REDIRECTS: usize = 10;
/// Follow a redirect only when it stays on the origin that issued it AND re-issues
/// the identical request-target.
///
/// `Policy::none()` would have been the simpler answer, but one same-origin redirect
/// shape is legitimate here: a node fronted by a proxy that upgrades http to https,
/// or otherwise re-issues the same path and query, so the policy is scoped rather
/// than switched off.
///
/// Path and query must match exactly, and that is the request-target clause rather
/// than an origin one. `@path` is signed as the client sent it and verified as the
/// node received it, so a hop that rewrites either half leaves a signature covering
/// a target nobody asked for and the node answers 401. Refusing the hop turns a
/// confusing 401 into the 3xx that names what actually happened. One policy covers
/// signed and unsigned callers alike, for the same reason the predicate is shared:
/// two rules would drift.
///
/// The clause pins `@path`, and only `@path`. A gitlawb signature also covers
/// `@method` and `content-digest`, and both of those are still open on a followed
/// hop: on a 301, 302 or 303, reqwest 0.12.28 delegates to tower-http's
/// `FollowRedirect`, which rewrites a POST to a GET and empties the body
/// (tower-http-0.6.8 `src/follow_redirect/mod.rs:273-285`), while its
/// `drop_payload_headers` removes only `Content-Type`, `Content-Length`,
/// `Content-Encoding` and `Transfer-Encoding`. So `Signature`, `Signature-Input` and
/// `Content-Digest` ride along on a request that no longer has the method or the body
/// they were computed over. Only 307 and 308 preserve both. This predicate returning
/// true therefore makes a GET-shaped hop safe to replay against the node's verifier
/// and says nothing about a bodied one: a signed write must not rely on it alone.
///
/// `Url::query` is `None` for `/a` and `Some("")` for `/a?`, and those are two
/// different request-targets on the node side too, so the comparison is strict and
/// needs no special case.
///
/// Host and port must match exactly. Port is compared as `Url::port`, which is
/// `None` for a scheme's default port, so http -> https on the same host compares
/// equal while http -> http on a different port does not. A downgrade from https to
/// http is refused as well: the target is the same host, but the signature would go
/// out in cleartext, which is the same credential leak by a slower route.
///
/// Host comparison rides on `url`'s parse-time normalization (lowercasing and IDN
/// -> punycode), so the spellings an attacker reaches for do not open a gap. That
/// is a property of the parsed `Url`, not of this function, which is why the test
/// matrix pins it: a move to raw string comparison would silently lose it.
pub fn may_follow(previous: &url::Url, next: &url::Url) -> bool {
let same_origin = next.host_str() == previous.host_str() && next.port() == previous.port();
let same_target = next.path() == previous.path() && next.query() == previous.query();
let downgraded = previous.scheme() == "https" && next.scheme() != "https";
same_origin && same_target && !downgraded
}
#[cfg(test)]
mod tests {
use super::*;
/// Every branch of the decision, both ways, plus the spellings that would slip
/// past a comparison less careful than `Url`'s own normalization.
#[test]
fn may_follow_covers_each_origin_branch() {
let url = |s: &str| url::Url::parse(s).unwrap();
let cases: &[(&str, &str, bool, &str)] = &[
(
"http://node.example/a",
"http://node.example/b",
false,
"same origin but the path changed: @path is signed as sent and verified as received",
),
(
"http://node.example/a",
"https://node.example/a",
true,
"http to https on one host with an identical request-target: both ports are \
the scheme default, so the proxy upgrade is still followed",
),
(
"https://node.example/a",
"https://node.example/a/",
false,
"trailing-slash normalization changes the request-target, so it is refused",
),
(
"https://node.example:8443/a",
"https://node.example:8443/a",
true,
"same explicit port",
),
(
"http://node.example/a",
"http://attacker.example/a",
false,
"different host",
),
(
"http://node.example/a",
"http://node.example:8080/a",
false,
"same host, different port",
),
(
"https://node.example/a",
"http://node.example/a",
false,
"https downgraded to cleartext on the same host",
),
(
"https://node.example/a",
"http://node.example:443/a",
false,
"a downgrade dressed up as the https port",
),
// The rows below pass today because `Url::parse` normalizes the host, not
// because anything here compares case-insensitively or decodes IDN. They
// are the variants an attacker reaches for, so they are pinned: swapping
// this predicate for a raw string comparison must break the suite.
// Each of these pairs an IDENTICAL path on both sides, deliberately. The
// request-target clause below would make every one of them false on the
// path alone, and a row that is false for two reasons has stopped pinning
// either. With the paths equal, the host or port comparison is the only
// thing left that can decide them.
(
"https://node.example/a",
"https://NODE.EXAMPLE/a",
true,
"same host in a different case: parse lowercases it",
),
(
"https://node.example/a",
"https://node.example./a",
false,
"a trailing dot is a different host to url, so the redirect is refused",
),
(
"https://exämple.test/a",
"https://xn--exmple-cua.test/a",
true,
"unicode host and its punycode spelling are one host after parse",
),
(
"https://node.example/a",
"https://user:[email protected]/a",
true,
"userinfo is not part of the origin: same host, still followed",
),
(
"https://node.example/a",
"https://[email protected]/a",
false,
"the node's name smuggled into userinfo: the host is the attacker's",
),
(
"https://node.example/a",
"https://attacker.example/a#node.example",
false,
"the node's name pushed into the fragment: the host is the attacker's",
),
// The request-target clause, both directions. `@path` is the only thing a
// gitlawb signature binds the request to, so a hop that rewrites it hands
// the node a signature over a target it never received.
(
"https://node.example/a?x=1",
"https://node.example/a?x=1",
true,
"identical request-target: the same-origin hop that is still followed",
),
(
"https://node.example/a?x=1",
"https://node.example/a?x=2",
false,
"same path but a different query: the request-target covers the query too",
),
(
"https://node.example/a",
"https://node.example/a?x=1",
false,
"a query added where there was none",
),
(
"http://node.example/a",
"http://node.example/a?",
false,
"an empty query added where there was none: a missing query and an empty \
one are different request-targets",
),
(
"https://node.example/a?x=1",
"https://node.example/a",
false,
"the query dropped where there was one: the comparison is symmetric, and \
nothing else in the matrix pins that direction",
),
(
"https://node.example/a#x",
"https://node.example/a#y",
true,
"a fragment-only difference is still followed: a fragment never reaches \
the wire, so it is no part of the request-target the node verifies",
),
];
for (previous, next, expected, why) in cases {
assert_eq!(
may_follow(&url(previous), &url(next)),
*expected,
"{previous} -> {next} ({why})"
);
}
}
}
+499
View File
@@ -0,0 +1,499 @@
//! Sealed continuation token for the node's bounded legacy CID scan (INV-13).
//!
//! The `/ipfs/{cid}` resolver's legacy scan stops at a row ceiling and sheds a
//! retryable 503. To let a holder buried past that ceiling still be reached, the
//! shed carries the scan position so the caller can echo it back and resume. The
//! whole point of the design is that the node keeps NO server-side scan state: the
//! position rides in the caller's token.
//!
//! That makes the token an EMITTED continuation derived from a FETCHED row, and on
//! a scan that served nothing every fetched row is by construction a private or
//! quarantined repo the caller may not read. The row's `created_at` leaks its
//! creation time and its `id` carries the owner's DID, so both halves are withheld
//! fields and the token must be CONFIDENTIAL, not merely tamper-evident:
//!
//! * AEAD-sealed (XChaCha20-Poly1305), never base64-of-plaintext and never
//! signed plaintext. Integrity is not confidentiality.
//! * A fresh `OsRng` nonce on EVERY seal. Under a stream cipher a repeated nonce
//! means repeated keystream, and an attacker who can force the node to seal a
//! position whose plaintext they know XORs two tokens and recovers a withheld
//! row's fields in full, strictly worse than emitting plaintext.
//! * FIXED-WIDTH plaintext. AEAD ciphertext is plaintext-length plus the tag, and
//! every field of a scan position varies in length, so a variable encoding would
//! make token LENGTH a side channel for the sealed row (a short name under a
//! short owner vs a long one) and for the candidate oid's width (40 hex on a
//! sha1 repo, 64 on a sha256 one). Every token this module mints is byte-identical
//! in length. Each field is padded to its own separate width, which is a per-field
//! constant and so still leaks nothing about a given row or candidate.
//! * The canonical CID as associated data, so a token minted while scanning for
//! one CID does not authenticate when replayed against another.
//!
//! Every failure to open (wrong key, tampered bytes, wrong CID, expired, malformed)
//! returns the same `None`. The caller treats that as "no token" and starts at the
//! front, so no failure class is distinguishable and the token is no oracle.
use base64::{engine::general_purpose::URL_SAFE_NO_PAD as B64URL, Engine};
use chacha20poly1305::{
aead::{Aead, KeyInit, OsRng, Payload},
XChaCha20Poly1305, XNonce,
};
use rand::RngCore;
/// Keyset position of the last row a truncated scan fetched.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ScanPosition {
/// The row's raw stored `created_at` text, the first half of the keyset cursor.
pub created_at_key: String,
/// The row's `id`, the tiebreaking half of the keyset cursor.
pub id: String,
/// The oid hex of the CANDIDATE this position resumes. One CID can map to several
/// git oids, so a bare row cursor names a row without naming whose walk it belongs
/// to. The candidate is named by IDENTITY rather than by position in the candidate
/// list: that list is ordered by hex and mutates between rungs, so an index would
/// name a different candidate the moment anything is pinned or unpinned.
pub sha256_hex: String,
}
/// Plaintext version byte, so a future layout change is a clean open-failure
/// (treated as absent) rather than a misparse.
///
/// Bumped to 2 when the two halves stopped sharing one width (see [`ID_WIDTH`]), and
/// to 3 when the position gained the candidate oid it resumes (see [`OID_WIDTH`]).
/// A token minted under an earlier layout is a different length and a different
/// framing, so it opens to `None` and the caller restarts at the front, which is the
/// safe direction: a misparse would resume at a fabricated row and skip coverage.
const VERSION: u8 = 3;
/// Byte width the `created_at` half is padded to. Every value stored here is a
/// serialized timestamp, about 30 bytes, so 64 is roomy for the field's whole domain.
/// It is deliberately NOT widened to match [`ID_WIDTH`]: padding both halves to the id
/// width would nearly double every token for a field that can never use the space.
const CREATED_WIDTH: usize = 64;
/// Byte width the `id` half is padded to.
///
/// The bound is set by the WRITERS, not by what a typical id happens to look like.
/// `upsert_mirror_repo` builds `repos.id` as `{owner}/{name}`, and the slug validators
/// in the node's `repo_store` admit an owner of up to 255 bytes and a name of up to
/// 100, so 356 bytes is reachable through the ordinary write path and repo names are
/// peer-controllable. 384 clears that with margin.
///
/// Under-sizing this is not a cosmetic bug. A row at a truncation boundary whose id
/// exceeds the width fails the seal, the handler sheds a 503 with no continuation, and
/// a tokenless shed is byte-identical to the wrapped-scan response whose contract is
/// "the absence of a token means the ladder is over". The boundary row is deterministic
/// for a stable inventory, so every retry reproduces it and every row past it becomes
/// permanently unreachable. Anything past the width still fails loudly rather than
/// silently truncating a cursor into one that resumes at the wrong row.
const ID_WIDTH: usize = 384;
/// Byte width the candidate oid half is padded to.
///
/// Git mints exactly two oid widths and this field carries BOTH. A production repo is
/// created by `store::init_bare` with `git init --bare --object-format=sha1`, so its
/// oids are 40 hex; only the sha256 test fixtures mint 64. The field is therefore
/// length-prefixed like the two row halves rather than a bare fixed 64: a 64-only
/// field would fail every seal on a real deployment, and a failed seal sheds a
/// tokenless 503 that is byte-identical to "your ladder is over".
///
/// The padding to 64 is what keeps the WIDTH off the wire. Without it a 40-hex token
/// is 24 bytes shorter than a 64-hex one, and token length would say which object
/// format the holder's repo uses.
const OID_WIDTH: usize = 64;
/// `version | created_len:u16 | created[CREATED_WIDTH] | id_len:u16 | id[ID_WIDTH] | oid_len:u16 | oid[OID_WIDTH] | expires:i64`
const PLAINTEXT_LEN: usize = 1 + 2 + CREATED_WIDTH + 2 + ID_WIDTH + 2 + OID_WIDTH + 8;
/// Nonce width for XChaCha20-Poly1305.
const NONCE_LEN: usize = 24;
/// A fresh random 32-byte sealing key from the OS CSPRNG.
pub fn new_key() -> [u8; 32] {
let mut key = [0u8; 32];
OsRng.fill_bytes(&mut key);
key
}
// The two halves below are deliberately separate and adjacent. The framing pair owns
// "every token is the same length"; the AEAD pair owns "the contents are confidential
// and CID-bound". Keeping them apart is what lets each property be exercised (and
// broken) without disturbing the other.
/// Encode a position into the FIXED-WIDTH plaintext:
/// `version | created_len:u16 | created[CREATED_WIDTH] | id_len:u16 | id[ID_WIDTH] | oid_len:u16 | oid[OID_WIDTH] | expires:i64`
///
/// The padding is the point. AEAD ciphertext is plaintext-length plus the tag, and every
/// field of a scan position varies in length, so a length-prefixed encoding with no
/// padding would make token LENGTH a side channel for the sealed row and for the
/// candidate's object format. Each field is padded to its OWN fixed width, which keeps
/// every minted token the same length while letting the id half carry the range the
/// write path actually admits.
fn encode_position(pos: &ScanPosition, expires_at_unix: i64) -> anyhow::Result<Vec<u8>> {
let mut out = vec![0u8; PLAINTEXT_LEN];
out[0] = VERSION;
let mut at = 1;
for (field, width) in [
(pos.created_at_key.as_bytes(), CREATED_WIDTH),
(pos.id.as_bytes(), ID_WIDTH),
(pos.sha256_hex.as_bytes(), OID_WIDTH),
] {
if field.len() > width {
// Loud rather than truncating: a clipped cursor resumes at the wrong row and
// silently skips coverage, which is the availability half of the bug this
// token exists to fix.
anyhow::bail!(
"scan token field is {} bytes, over the {width}-byte fixed width",
field.len()
);
}
out[at..at + 2].copy_from_slice(&(field.len() as u16).to_le_bytes());
at += 2;
out[at..at + field.len()].copy_from_slice(field);
at += width;
}
out[at..at + 8].copy_from_slice(&expires_at_unix.to_le_bytes());
Ok(out)
}
/// Decode what [`encode_position`] wrote. `None` on any structural mismatch.
fn decode_position(bytes: &[u8]) -> Option<(ScanPosition, i64)> {
if bytes.len() != PLAINTEXT_LEN || bytes[0] != VERSION {
return None;
}
let mut at = 1;
let mut fields = [const { String::new() }; 3];
for (slot, width) in fields.iter_mut().zip([CREATED_WIDTH, ID_WIDTH, OID_WIDTH]) {
let len = u16::from_le_bytes([bytes[at], bytes[at + 1]]) as usize;
at += 2;
if len > width {
return None;
}
*slot = String::from_utf8(bytes[at..at + len].to_vec()).ok()?;
at += width;
}
let expires_at = i64::from_le_bytes(bytes[at..at + 8].try_into().ok()?);
let [created_at_key, id, sha256_hex] = fields;
// A zero-length candidate is a third state the encoder never mints. The front-of-table
// sentinel is empty ROW halves with a real oid, so an empty oid would hand the resume
// path a candidate that names nothing; refuse it like every other malformed frame.
if sha256_hex.is_empty() {
return None;
}
Some((
ScanPosition {
created_at_key,
id,
sha256_hex,
},
expires_at,
))
}
/// AEAD-seal `plaintext` under `key`, bound to `cid`, framed as `nonce || ciphertext`.
fn seal_bytes(key: &[u8; 32], cid: &str, plaintext: &[u8]) -> anyhow::Result<Vec<u8>> {
let cipher = XChaCha20Poly1305::new_from_slice(key)
.map_err(|e| anyhow::anyhow!("scan token key: {e}"))?;
// A FRESH nonce per seal, from the OS CSPRNG. Under a stream cipher a repeated nonce
// repeats the keystream, and two tokens sealed under one nonce XOR to the difference
// of their plaintexts, which recovers a withheld row in full when the attacker can
// force one of the two positions. This draw is the property the whole confidentiality
// claim rests on.
let mut nonce = [0u8; NONCE_LEN];
OsRng.fill_bytes(&mut nonce);
let sealed = cipher
.encrypt(
XNonce::from_slice(&nonce),
Payload {
msg: plaintext,
// The canonical CID as associated data: a token minted while scanning for
// one CID does not authenticate against another, so it cannot be replayed
// to seed a different scan.
aad: cid.as_bytes(),
},
)
.map_err(|e| anyhow::anyhow!("scan token seal: {e}"))?;
let mut out = Vec::with_capacity(NONCE_LEN + sealed.len());
out.extend_from_slice(&nonce);
out.extend_from_slice(&sealed);
Ok(out)
}
/// Open what [`seal_bytes`] framed. `None` on any failure, including a wrong `cid`.
fn open_bytes(key: &[u8; 32], cid: &str, raw: &[u8]) -> Option<Vec<u8>> {
if raw.len() <= NONCE_LEN + 16 {
return None;
}
let (nonce, sealed) = raw.split_at(NONCE_LEN);
let cipher = XChaCha20Poly1305::new_from_slice(key).ok()?;
cipher
.decrypt(
XNonce::from_slice(nonce),
Payload {
msg: sealed,
aad: cid.as_bytes(),
},
)
.ok()
}
/// Seal `pos` under `key`, bound to `cid`, expiring at `expires_at_unix`.
///
/// Returns the base64url (no pad) token. Errors only when a field exceeds
/// its own fixed width ([`CREATED_WIDTH`], [`ID_WIDTH`], [`OID_WIDTH`]) or the AEAD
/// itself fails, never silently truncates.
pub fn seal_scan_token(
key: &[u8; 32],
cid: &str,
pos: &ScanPosition,
expires_at_unix: i64,
) -> anyhow::Result<String> {
let plaintext = encode_position(pos, expires_at_unix)?;
Ok(B64URL.encode(seal_bytes(key, cid, &plaintext)?))
}
/// Open a token minted by [`seal_scan_token`] under the same key and CID.
///
/// `None` for every failure class alike (wrong key, tampered, foreign CID, expired,
/// malformed, wrong version), so the caller can treat all of them as "absent" without
/// leaking which one occurred.
pub fn open_scan_token(
key: &[u8; 32],
cid: &str,
token: &str,
now_unix: i64,
) -> Option<ScanPosition> {
let raw = B64URL.decode(token).ok()?;
let plaintext = open_bytes(key, cid, &raw)?;
let (pos, expires_at) = decode_position(&plaintext)?;
if now_unix >= expires_at {
return None;
}
Some(pos)
}
#[cfg(test)]
mod tests {
use super::*;
/// A production-shaped candidate: `git init --bare --object-format=sha1`, so 40 hex.
const OID_40: &str = "a1b2c3d4e5f60718293a4b5c6d7e8f9012345678";
/// A test-fixture-shaped candidate: the sha256 repos the suite creates mint 64 hex.
const OID_64: &str = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
fn pos(created: &str, id: &str) -> ScanPosition {
pos_for(created, id, OID_40)
}
fn pos_for(created: &str, id: &str, sha256_hex: &str) -> ScanPosition {
ScanPosition {
created_at_key: created.to_string(),
id: id.to_string(),
sha256_hex: sha256_hex.to_string(),
}
}
/// Seal a hand-built plaintext through the AEAD half, so a test can frame bytes the
/// encoder would never mint (an old version byte, a zero-length oid) and still exercise
/// the real open path.
fn seal_raw(key: &[u8; 32], cid: &str, plaintext: &[u8]) -> String {
B64URL.encode(seal_bytes(key, cid, plaintext).unwrap())
}
/// Byte offset of the oid length prefix inside the plaintext, derived from the widths
/// rather than hardcoded so a width change moves it with the layout.
const OID_LEN_AT: usize = 1 + 2 + CREATED_WIDTH + 2 + ID_WIDTH;
/// Scenario 1: both of git's oid widths round trip. The 40-hex case is the PRODUCTION
/// shape (`git init --bare --object-format=sha1`), and an all-64 fixture suite would
/// never exercise it, which is exactly how a fixed-64 field would ship broken.
#[test]
fn round_trips_at_both_oid_widths() {
let key = new_key();
for hex in [OID_40, OID_64] {
let p = pos_for("2020-01-01T00:00:03+00:00", "z6MkOwner/private-repo", hex);
let t =
seal_scan_token(&key, "bafkcid", &p, 1 << 40).expect("both oid widths must seal");
assert_eq!(
open_scan_token(&key, "bafkcid", &t, 0),
Some(p),
"a {}-hex candidate must open to the identical position",
hex.len()
);
}
}
/// Scenario 2: a token framed under the OLD version opens to `None`, never a misparse.
/// Both legs matter: the version-2 plaintext was a different LENGTH, and a future
/// same-length layout would only be caught by the version byte itself.
#[test]
fn a_prior_layout_version_opens_to_none() {
let key = new_key();
// The version-2 layout verbatim: no oid field, so 461 bytes.
let mut old = vec![0u8; 1 + 2 + CREATED_WIDTH + 2 + ID_WIDTH + 8];
old[0] = 2;
assert_eq!(old.len(), 461, "the version-2 plaintext was 461 bytes");
assert_eq!(
open_scan_token(&key, "bafkcid", &seal_raw(&key, "bafkcid", &old), 0),
None,
"a version-2 token must open to None so the caller restarts at the front"
);
// Same length, stale version byte: only the version check can refuse this one.
let mut stamped = encode_position(
&pos("2020-01-01T00:00:03+00:00", "z6MkOwner/private-repo"),
1 << 40,
)
.unwrap();
stamped[0] = VERSION - 1;
assert_eq!(
open_scan_token(&key, "bafkcid", &seal_raw(&key, "bafkcid", &stamped), 0),
None,
"a stale version byte must open to None even at the current width"
);
}
/// Scenario 3: token length is invariant across candidate VALUE and candidate WIDTH.
/// The padding is what keeps the oid width off the wire: without it a 40-hex token is
/// 24 bytes shorter than a 64-hex one and the length says which repo format the
/// holder uses.
#[test]
fn token_length_is_invariant_across_oid_widths() {
let key = new_key();
let created = "2020-01-01T00:00:00+00:00";
let id = "z6MkOwner/private-repo";
let short =
seal_scan_token(&key, "bafkcid", &pos_for(created, id, OID_40), 1 << 40).unwrap();
let long =
seal_scan_token(&key, "bafkcid", &pos_for(created, id, OID_64), 1 << 40).unwrap();
assert_eq!(
short.len(),
long.len(),
"a 40-hex and a 64-hex candidate must mint tokens of identical length, or the \
oid width is a side channel"
);
// The absolute width, pinned by execution rather than by arithmetic on paper. The
// gl client's mock fixtures hardcode this number (`TOKEN_LEN` in
// crates/gl/src/ipfs_cmd.rs), and nothing in that crate seals a real token, so this
// assertion is the only executable check that the two agree.
assert_eq!(
short.len(),
756,
"24 nonce + {PLAINTEXT_LEN} plaintext + 16 tag, base64url no pad"
);
}
/// Scenario 4: the front-of-table sentinel. Empty row halves with a REAL candidate
/// round trip, which is what lets a seal say "this candidate, no row cursor yet".
#[test]
fn empty_row_fields_round_trip_with_a_real_candidate() {
let key = new_key();
let p = pos_for("", "", OID_40);
let t =
seal_scan_token(&key, "bafkcid", &p, 1 << 40).expect("the front sentinel must seal");
assert_eq!(open_scan_token(&key, "bafkcid", &t, 0), Some(p));
}
/// Scenario 6, third leg: a zero-length oid is a distinguishable third state that the
/// encoder never mints, and accepting it would hand the sentinel machinery a candidate
/// naming nothing. The decode path refuses it.
#[test]
fn a_zero_length_candidate_opens_to_none() {
let key = new_key();
let mut plaintext = encode_position(
&pos("2020-01-01T00:00:03+00:00", "z6MkOwner/private-repo"),
1 << 40,
)
.unwrap();
plaintext[OID_LEN_AT..OID_LEN_AT + 2].copy_from_slice(&0u16.to_le_bytes());
assert_eq!(
open_scan_token(&key, "bafkcid", &seal_raw(&key, "bafkcid", &plaintext), 0),
None,
"a zero-length candidate must open to None, not to a position naming nothing"
);
}
/// Scenario 6, second leg: an oid past the 64-byte width fails the seal loudly rather
/// than being clipped into a hex that names a different candidate.
#[test]
fn an_oid_over_the_fixed_width_fails_loudly() {
let key = new_key();
let p = pos_for(
"2020-01-01T00:00:03+00:00",
"z6MkOwner/private-repo",
&"a".repeat(OID_WIDTH + 1),
);
assert!(
seal_scan_token(&key, "bafkcid", &p, 1 << 40).is_err(),
"an over-wide candidate must fail the seal, never be truncated into a hex that \
resumes the wrong candidate"
);
}
#[test]
fn round_trips_under_the_same_key_and_cid() {
let key = new_key();
let p = pos("2020-01-01T00:00:03+00:00", "z6MkOwner/private-repo");
let t = seal_scan_token(&key, "bafkcid", &p, 1 << 40).unwrap();
assert_eq!(open_scan_token(&key, "bafkcid", &t, 0), Some(p));
}
#[test]
fn every_failure_class_opens_to_none() {
let key = new_key();
let other = new_key();
let p = pos("2020-01-01T00:00:03+00:00", "z6MkOwner/private-repo");
let t = seal_scan_token(&key, "bafkcid", &p, 1 << 40).unwrap();
assert_eq!(open_scan_token(&other, "bafkcid", &t, 0), None, "wrong key");
assert_eq!(
open_scan_token(&key, "bafkOTHER", &t, 0),
None,
"foreign CID"
);
assert_eq!(
open_scan_token(&key, "bafkcid", &t, 1 << 41),
None,
"expired"
);
assert_eq!(open_scan_token(&key, "bafkcid", "!!not b64", 0), None);
assert_eq!(open_scan_token(&key, "bafkcid", "", 0), None);
let mut flipped: Vec<u8> = t.bytes().collect();
let last = flipped.len() - 1;
flipped[last] = if flipped[last] == b'A' { b'B' } else { b'A' };
assert_eq!(
open_scan_token(&key, "bafkcid", &String::from_utf8(flipped).unwrap(), 0),
None,
"tampered"
);
}
/// The id half must clear the LARGEST repo id the node's own write path admits,
/// not merely a typical one. `upsert_mirror_repo` builds `repos.id` as
/// `{owner}/{name}`, and the slug validators in `repo_store` admit 255 bytes of
/// owner and 100 of name, so 356 is reachable. A width under that turns the
/// boundary row into a seal failure, which sheds a tokenless 503 that is
/// byte-identical to "your ladder is over" and strands every row past it forever.
#[test]
fn round_trips_a_repo_id_at_the_write_paths_maximum() {
let key = new_key();
let id = format!("{}/{}", "o".repeat(255), "n".repeat(100));
assert_eq!(id.len(), 356, "255 owner + '/' + 100 name");
let p = pos("2020-01-01T00:00:03+00:00", &id);
let t = seal_scan_token(&key, "bafkcid", &p, 1 << 40)
.expect("a repo id the write path admits must seal, never fail the width");
assert_eq!(open_scan_token(&key, "bafkcid", &t, 0), Some(p));
}
#[test]
fn a_field_over_the_fixed_width_fails_loudly() {
let key = new_key();
let p = pos("2020-01-01T00:00:03+00:00", &"x".repeat(ID_WIDTH + 1));
assert!(
seal_scan_token(&key, "bafkcid", &p, 1 << 40).is_err(),
"an over-wide field must fail the seal, never be truncated into a cursor \
that resumes at the wrong row"
);
}
}
+49 -2
View File
@@ -26,14 +26,61 @@ pub async fn list_anchors(
Query(q): Query<ListAnchorsQuery>,
) -> Result<Json<serde_json::Value>> {
let limit = q.limit.min(200);
// Bare `?` so connection-class sqlx failures downcast to `AppError::Db` and
// map to 503 `db_unavailable` (not 500 via `.map_err(AppError::Internal)`) (#251).
let anchors = state
.db
.list_arweave_anchors(q.repo.as_deref(), limit)
.await
.map_err(crate::error::AppError::Internal)?;
.await?;
Ok(Json(serde_json::json!({
"anchors": anchors,
"count": anchors.len(),
})))
}
#[cfg(test)]
mod closed_pool_tests {
use super::*;
use axum::http::{Request, StatusCode};
use axum::Router;
use serde_json::Value;
use sqlx::PgPool;
use tower::ServiceExt;
/// #251: a closed pool on /api/v1/arweave/anchors must be 503 db_unavailable.
#[sqlx::test]
async fn list_anchors_closed_pool_returns_503_db_unavailable(pool: PgPool) {
let state = crate::test_support::test_state(pool.clone()).await;
pool.close().await;
let resp = Router::new()
.route("/api/v1/arweave/anchors", axum::routing::get(list_anchors))
.with_state(state)
.oneshot(
Request::builder()
.uri("/api/v1/arweave/anchors")
.body(axum::body::Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(
resp.status(),
StatusCode::SERVICE_UNAVAILABLE,
"closed-pool outage must be retryable 503, not 500"
);
let bytes = axum::body::to_bytes(resp.into_body(), usize::MAX)
.await
.expect("read body");
let v: Value = serde_json::from_slice(&bytes).expect("json body");
assert_eq!(
v,
serde_json::json!({
"error": crate::error::DB_UNAVAILABLE_CODE,
"message": crate::error::DB_UNAVAILABLE_MESSAGE,
})
);
}
}
+14 -1
View File
@@ -1440,7 +1440,8 @@ mod ref_updates_feed_tests {
// A DB error in the gate fails closed as 500, not swallowed into an empty 200 (the
// regression the old get_repo().ok().flatten() allowed). Inject by dropping a
// column get_repo selects so its query errors.
// column get_repo selects so its query errors. Also pin the anonymous body to
// the exact opaque object (#226).
#[sqlx::test]
async fn repo_events_db_error_fails_closed_500(pool: PgPool) {
let state = test_state(pool.clone()).await;
@@ -1463,6 +1464,18 @@ mod ref_updates_feed_tests {
StatusCode::INTERNAL_SERVER_ERROR,
"a DB error must fail closed (500), never serve an empty 200"
);
let body = axum::body::to_bytes(resp.into_body(), usize::MAX)
.await
.expect("read body");
let body = String::from_utf8_lossy(&body);
let v: serde_json::Value = serde_json::from_str(&body).expect("json body");
assert_eq!(
v,
serde_json::json!({
"error": "db_error",
"message": crate::error::DB_ERROR_MESSAGE,
})
);
}
// Symmetric to the gate DB-error test: a DB error in the CERT fetch (after the gate
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+25 -5
View File
@@ -208,11 +208,15 @@ mod authz_guard {
(issues, "create_issue", "authorize_repo_read("),
(bounties, "create_bounty", "authorize_repo_read("),
(repos, "fork_repo", "authorize_repo_read("),
// get_by_cid gates each iterated repo row directly via visibility_check
// (KTD2a: it must NOT route through authorize_repo_read's fuzzy re-resolve).
(ipfs, "get_by_cid", "visibility_check("),
// #94 sibling read surfaces: gate private-repo metadata on read
// visibility (public repos stay anonymous; private repos 404).
// get_by_cid resolves each candidate (provenance path + legacy scan) through
// the shared `gate_and_serve`; the gate markers themselves are asserted
// below. This row proves the delegation is real, so the gate is actually
// reached rather than dead code. The delegated gate still calls
// `visibility_check` directly and never `authorize_repo_read`, so it keeps
// the property the pre-merge marker enforced: no fuzzy re-resolve.
(ipfs, "get_by_cid", "gate_and_serve("),
// Sibling read surfaces: gate private-repo metadata on read visibility
// (public repos stay anonymous; private repos 404).
(replicas, "list_replicas", "authorize_repo_read("),
(protect, "list_protected_branches", "authorize_repo_read("),
(labels, "list_labels", "authorize_repo_read("),
@@ -250,6 +254,22 @@ mod authz_guard {
"visibility::require_owner must use did_matches for DID-safe owner matching"
);
// The CID read surface (#173) enforces its gate inside the shared
// `gate_and_serve`, which BOTH the provenance path and the legacy scan call, so
// the markers must live there (the get_by_cid row above only proves delegation).
// The repo's own "/" visibility check (KTD2a — never authorize_repo_read's fuzzy
// re-resolve) and the quarantine hard-drop BEFORE visibility (INV-11) are both
// load-bearing: removing either re-opens a leak on the provenance path.
let gate_body = fn_body(ipfs, "gate_and_serve");
assert!(
gate_body.contains("visibility_check("),
"gate_and_serve must gate the CID read surface via visibility_check (KTD2a)"
);
assert!(
gate_body.contains("if quarantined"),
"gate_and_serve must hard-drop a quarantined repo before the visibility gate (INV-11)"
);
for (src, func, marker) in rows {
let body = fn_body(src, func);
assert!(
+795 -20
View File
@@ -17,9 +17,39 @@ use gitlawb_core::did::Did;
use serde::{Deserialize, Serialize};
use crate::auth::AuthenticatedDid;
use crate::db::{PeerWriteAuthority, PeerWriteDenied};
use crate::error::{AppError, Result};
use crate::state::AppState;
/// Map an `upsert_peer` failure onto its response class.
///
/// The boundary's denials arrive inside an anyhow chain, so they are downcast
/// back out the way `From<anyhow::Error> for AppError` already recovers sqlx
/// errors. Without this a refusal renders as a 500, which is the mis-rendered
/// denial this gate exists to avoid. An authority refusal is 403; naming a DID
/// method that can never authenticate, or a DID whose verifying key cannot be
/// resolved, is a fact about the input's form, so both are the 400 validation
/// class, and the unresolvable one carries its own code so it reads the same to
/// a client as the auth middleware's answer for an unresolvable keyid. The match
/// stays exhaustive with no wildcard: the compile error a new denial causes here
/// is what finds every mapping site.
fn peer_write_error(err: anyhow::Error) -> AppError {
match err.downcast::<PeerWriteDenied>() {
Ok(denied) => match &denied {
PeerWriteDenied::UnsupportedDidMethod { .. } => {
AppError::BadRequest(denied.to_string())
}
PeerWriteDenied::UnresolvableDid { .. } => {
AppError::UnresolvableDid(denied.to_string())
}
PeerWriteDenied::UnprovenRepoint { .. } | PeerWriteDenied::ProofDidMismatch { .. } => {
AppError::Forbidden(denied.to_string())
}
},
Err(other) => AppError::from(other),
}
}
#[derive(Debug, Deserialize)]
pub struct AnnounceRequest {
/// The DID of the announcing node
@@ -187,18 +217,30 @@ pub async fn announce(
.did
.parse()
.map_err(|e: gitlawb_core::Error| AppError::BadRequest(e.to_string()))?;
if let Some(Extension(auth)) = auth {
// The extension's DID is cryptographically derived, never body-supplied:
// auth/mod.rs resolves the verifying key FROM the keyid DID itself, so what
// is carried through to the upsert boundary below is a proof, not a claim.
let proven_did: Option<String> = if let Some(Extension(auth)) = auth {
// 403, not 400: this caller proved control of some DID and spent that
// proof on a different DID's row, which is a refusal of authority over
// the targeted row: the same denial as an unsigned repoint, and two
// callers denied for the same reason must not get different response
// classes. The 400 class stays for input-form failures: a malformed
// DID, a non-public URL, a self-announce, and a DID method that can
// never authenticate.
if auth.0 != announced_did.to_string() {
return Err(AppError::BadRequest(
return Err(AppError::Forbidden(
"Signature keyid must match announced DID".into(),
));
}
Some(auth.0)
} else {
tracing::warn!(
did = %announced_did,
"accepted unsigned peer announce; set GITLAWB_REQUIRE_SIGNED_PEER_WRITES=true after all peers upgrade"
);
}
None
};
// Validate the URL is a public http(s) endpoint. The announce route is
// reachable unauthenticated (until all peers sign), so without this an
@@ -228,7 +270,20 @@ pub async fn announce(
));
}
state.db.upsert_peer(&req.did, &req.http_url).await?;
// The authority declaration, not a re-statement of the check above: the
// boundary carries the proven DID and re-checks it against the row itself,
// so the keyid comparison above is an early rejection rather than the only
// gate. The other writer (the bootstrap announce-back in main.rs) never
// reaches this handler at all.
let authority = match proven_did.as_deref() {
Some(did) => PeerWriteAuthority::Proven(did),
None => PeerWriteAuthority::Unproven,
};
state
.db
.upsert_peer(&req.did, &req.http_url, authority)
.await
.map_err(peer_write_error)?;
tracing::info!(did = %req.did, url = %req.http_url, "peer announced");
@@ -443,21 +498,15 @@ pub async fn ping_peer(
.find(|p| p.did == did)
.ok_or_else(|| AppError::RepoNotFound(format!("peer {did} not found")))?;
// Async ping
let url = format!("{}/health", peer.http_url.trim_end_matches('/'));
// Use the shared no-redirect client: bare `reqwest::get` follows redirects,
// so a peer could answer with `302 -> http://127.0.0.1/` and turn the ping
// into an SSRF probe.
let ok = state
.http_client
.get(&url)
.send()
.await
.map(|r| r.status().is_success())
.unwrap_or(false);
let _ = state.db.mark_peer_ping(&did, ok).await;
// Use DB-aware readiness rather than process liveness: `/health` stays 200
// during a mid-life database outage, while `/ready` reports 503. A 404-only
// fallback keeps pre-readiness peers compatible. The shared helper also
// preserves the no-redirect SSRF guard on peer-controlled URLs.
let ok = crate::ping_peer_readiness(&state.http_client, &peer.http_url).await;
// This anonymous diagnostic endpoint is intentionally read-only. The
// periodic gossip loop owns the persisted federation-reachability gate and
// requires consecutive failures before marking a peer unreachable.
Ok(Json(serde_json::json!({
"did": did,
"http_url": peer.http_url,
@@ -564,7 +613,7 @@ mod tests {
use axum::body::Body;
use axum::extract::ConnectInfo;
use axum::http::{header, Method, Request, StatusCode};
use axum::routing::post;
use axum::routing::{get, post};
use axum::{middleware, Extension, Router};
use gitlawb_core::http_sig::sign_request;
use gitlawb_core::identity::Keypair;
@@ -770,12 +819,84 @@ mod tests {
let did = Keypair::generate().did().to_string();
state
.db
.upsert_peer(&did, "https://peer.example.com")
.upsert_peer(
&did,
"https://peer.example.com",
crate::db::PeerWriteAuthority::Unproven,
)
.await
.unwrap();
did
}
#[sqlx::test]
async fn manual_ping_uses_readiness_without_mutating_federation_gate(pool: PgPool) {
let mut server = mockito::Server::new_async().await;
let ready = server
.mock("GET", "/ready")
.with_status(503)
.expect(1)
.create_async()
.await;
let health = server
.mock("GET", "/health")
.with_status(200)
.expect(0)
.create_async()
.await;
let mut state = test_state(pool.clone()).await;
state.http_client =
Arc::new(crate::build_http_client().expect("production HTTP client should build"));
let did = Keypair::generate().did().to_string();
let now = chrono::Utc::now().to_rfc3339();
sqlx::query(
"INSERT INTO peers (did, http_url, last_seen, last_ping_ok, announced_at)
VALUES ($1, $2, $3, TRUE, $3)",
)
.bind(&did)
.bind(server.url())
.bind(now)
.execute(&pool)
.await
.unwrap();
let response = Router::new()
.route("/api/v1/peers/{did}/ping", get(super::ping_peer))
.with_state(state.clone())
.oneshot(
Request::builder()
.method(Method::GET)
.uri(format!("/api/v1/peers/{did}/ping"))
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let body = axum::body::to_bytes(response.into_body(), usize::MAX)
.await
.unwrap();
let json: serde_json::Value = serde_json::from_slice(&body).unwrap();
assert_eq!(json["reachable"], false);
ready.assert_async().await;
health.assert_async().await;
let peer = state
.db
.list_peers()
.await
.unwrap()
.into_iter()
.find(|peer| peer.did == did)
.unwrap();
assert!(
peer.last_ping_ok,
"an anonymous manual sample must not rewrite the federation gate"
);
}
#[sqlx::test]
async fn sync_notify_unsigned_flood_is_throttled(pool: PgPool) {
// notify still accepts an unsigned known-peer notification (rolling-upgrade
@@ -1155,6 +1276,57 @@ mod tests {
assert_eq!(resp.status(), StatusCode::OK);
}
/// U1 (#273). An unsigned announce must not repoint an existing peer's row.
///
/// Kills the unconditional `ON CONFLICT(did) DO UPDATE SET http_url = $2` in
/// `Db::upsert_peer`. The failure message carries BOTH observed values,
/// because the gate is the observed rewrite (2xx AND the attacker URL
/// stored), not merely a failing assertion: a 429 from the peer-write brake,
/// a fixture URL `is_public_http_url` rejects, or a DB error would all read
/// as "assertion failed" while proving nothing about this defect.
#[sqlx::test]
async fn unsigned_announce_cannot_repoint_an_existing_peer(pool: PgPool) {
const HONEST: &str = "https://honest-peer.example.com";
const ATTACKER: &str = "https://attacker.example.com";
let state = test_state(pool).await;
let did = Keypair::generate().did().to_string();
state
.db
.upsert_peer(&did, HONEST, crate::db::PeerWriteAuthority::Unproven)
.await
.unwrap();
let stored_url = |peers: Vec<crate::db::PeerRecord>| {
peers.into_iter().find(|p| p.did == did).map(|p| p.http_url)
};
assert_eq!(
stored_url(state.db.list_peers().await.unwrap()).as_deref(),
Some(HONEST),
"seed did not take; the case below would pass vacuously"
);
// Clone before build_router consumes state, so the row can be read back.
let db = state.db.clone();
let body = format!(r#"{{"did":"{did}","http_url":"{ATTACKER}"}}"#);
let resp = crate::server::build_router(state)
.oneshot(unsigned_post(
"/api/v1/peers/announce",
&body,
"203.0.113.77:5000",
))
.await
.unwrap();
let status = resp.status();
let stored = stored_url(db.list_peers().await.unwrap())
.expect("peer row vanished; neither the allowed nor the denied outcome");
assert_eq!(
stored, HONEST,
"unsigned announce repointed an existing peer (observed status {status}, stored http_url {stored})"
);
}
// Build a request carrying a REAL RFC 9421 signature (not just an injected
// extension), so it passes require_signature on the full production router.
fn real_signed_trigger(peer: &str) -> Request<Body> {
@@ -1265,4 +1437,607 @@ mod tests {
"a 429 on a sync route must not say 'push': {s:?}"
);
}
// ── #273: the announce authority contract ─────────────────────────────────
//
// U4 drives the handler mounted alone, because signed_request_as injects an
// AuthenticatedDid extension without a real signature and require_signature
// would reject that on the full router. U5 then proves the same properties
// through build_router with REAL RFC 9421 signatures, where a signature can
// be pointed at a row it does not own.
const OWNER_URL: &str = "https://owner-peer.example.com";
const MOVED_URL: &str = "https://moved-peer.example.com";
const HIJACK_URL: &str = "https://hijacker.example.com";
const WEB_DID: &str = "did:web:squatter.example.com";
const KEYID_MISMATCH: &str = "Signature keyid must match announced DID";
/// Every column a consumer sees, so a case that claims "unchanged" compares
/// the whole row rather than the one field it happened to think of.
type PeerSnapshot = (String, String, Option<String>, bool, String);
async fn snapshot(db: &crate::db::Db, did: &str) -> Option<PeerSnapshot> {
db.list_peers()
.await
.unwrap()
.into_iter()
.find(|p| p.did == did)
.map(|p| {
(
p.did,
p.http_url,
p.last_seen,
p.last_ping_ok,
p.announced_at,
)
})
}
/// Seed a peer row that has EARNED reachability, asserting the seed took,
/// so no case below can pass vacuously against a row that was never written
/// or against a flag that was already false. Without the granted ping a
/// "must be unreachable after this" assertion compares false to false and
/// stays green with the production code that clears the flag removed.
async fn seed_peer_row(state: &AppState, did: &str, url: &str) -> PeerSnapshot {
state
.db
.upsert_peer(did, url, crate::db::PeerWriteAuthority::Unproven)
.await
.expect("seed");
state.db.mark_peer_ping(did, true).await.expect("seed ping");
let seeded = snapshot(&state.db, did).await.expect("seed did not take");
assert_eq!(seeded.1, url, "seed did not take");
assert!(
seeded.3,
"the seeded row must start reachable, or a later unreachable assertion is vacuous"
);
seeded
}
/// Status, error code, and message together, so a denial is asserted by its
/// exact class and stated reason rather than by "not 2xx".
async fn status_and_error(resp: axum::response::Response) -> (StatusCode, String, String) {
let status = resp.status();
let bytes = axum::body::to_bytes(resp.into_body(), usize::MAX)
.await
.unwrap();
let v: serde_json::Value = serde_json::from_slice(&bytes).unwrap_or_else(|_| {
panic!(
"error body must be JSON: {}",
String::from_utf8_lossy(&bytes)
)
});
let field = |k: &str| v[k].as_str().unwrap_or_default().to_string();
(status, field("error"), field("message"))
}
fn announce_body(did: &str, url: &str) -> String {
format!(r#"{{"did":"{did}","http_url":"{url}"}}"#)
}
fn announce_only(state: AppState) -> Router {
Router::new()
.route("/api/v1/peers/announce", post(super::announce))
.with_state(state)
}
fn announce_as(did: &str, body: &str) -> Request<Body> {
signed_request_as(
did,
Method::POST,
"/api/v1/peers/announce",
Body::from(body.to_string()),
)
}
/// U4 scenario 1, and the first test the keyid branch has ever had: a caller
/// who proved control of one DID must not announce another's. Kills
/// neutralizing the handler's keyid comparison, and kills demoting this
/// denial back to the 400 validation class where the unsigned repoint below
/// is a 403 for the same reason.
#[sqlx::test]
async fn announce_keyid_mismatch_is_a_403_denial(pool: PgPool) {
let state = test_state(pool).await;
let victim = Keypair::generate().did().to_string();
let hijacker = Keypair::generate().did().to_string();
let before = seed_peer_row(&state, &victim, OWNER_URL).await;
let resp = announce_only(state.clone())
.oneshot(announce_as(&hijacker, &announce_body(&victim, HIJACK_URL)))
.await
.unwrap();
let (status, code, message) = status_and_error(resp).await;
assert_eq!(
status,
StatusCode::FORBIDDEN,
"a proof spent on another DID's row is an authority refusal: {message}"
);
assert_eq!(code, "forbidden", "denial body must name the 403 class");
assert!(
message.contains(KEYID_MISMATCH),
"the denial must name the keyid mismatch, got {message:?}"
);
assert_eq!(
snapshot(&state.db, &victim).await.expect("row vanished"),
before,
"the stored row must be byte-identical after a refused announce"
);
}
/// U4 scenario 2: the allowed signed case. A caller whose keyid IS the row
/// repoints it. Kills a gate that refuses proven writes too.
#[sqlx::test]
async fn announce_with_matching_keyid_repoints_its_own_row(pool: PgPool) {
let state = test_state(pool).await;
let owner = Keypair::generate().did().to_string();
seed_peer_row(&state, &owner, OWNER_URL).await;
let resp = announce_only(state.clone())
.oneshot(announce_as(&owner, &announce_body(&owner, MOVED_URL)))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let after = snapshot(&state.db, &owner).await.expect("row vanished");
assert_eq!(after.1, MOVED_URL, "a proven repoint must land");
}
/// U4 scenario 3: an unsigned caller cannot move an existing row, and the
/// refusal renders as a 403 with its reason, never as a 500 or as success.
#[sqlx::test]
async fn announce_unsigned_repoint_is_a_403_denial(pool: PgPool) {
let state = test_state(pool).await;
let victim = Keypair::generate().did().to_string();
let before = seed_peer_row(&state, &victim, OWNER_URL).await;
let resp = announce_only(state.clone())
.oneshot(unsigned_post(
"/api/v1/peers/announce",
&announce_body(&victim, HIJACK_URL),
"203.0.113.90:5000",
))
.await
.unwrap();
let (status, code, message) = status_and_error(resp).await;
assert_eq!(
status,
StatusCode::FORBIDDEN,
"an unproven repoint is an authority refusal: {message}"
);
assert_eq!(code, "forbidden", "denial body must name the 403 class");
assert!(
message.contains("unproven announce cannot change an existing peer"),
"the denial must name the repoint refusal, got {message:?}"
);
assert_eq!(
snapshot(&state.db, &victim).await.expect("row vanished"),
before,
"the stored row must be byte-identical after a refused repoint"
);
}
/// U4 scenarios 4 and 5, the allowed unsigned class: discovery of an unseen
/// did:key still inserts, and an honest peer re-announcing the IDENTICAL URL
/// still refreshes liveness. The URL is reused byte for byte because the
/// comparison is exact, so a trailing slash would make this pass as a
/// repoint rejection instead. Kills a gate widened until it breaks
/// discovery or honest re-announces.
#[sqlx::test]
async fn announce_unsigned_insert_and_identical_url_reannounce_are_accepted(pool: PgPool) {
let state = test_state(pool).await;
let did = Keypair::generate().did().to_string();
let body = announce_body(&did, OWNER_URL);
let resp = announce_only(state.clone())
.oneshot(unsigned_post(
"/api/v1/peers/announce",
&body,
"203.0.113.91:5000",
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK, "discovery must stay open");
let before = snapshot(&state.db, &did).await.expect("row not inserted");
assert_eq!(before.1, OWNER_URL);
assert!(!before.3, "a never-probed peer inserts unreachable");
let resp = announce_only(state.clone())
.oneshot(unsigned_post(
"/api/v1/peers/announce",
&body,
"203.0.113.92:5000",
))
.await
.unwrap();
assert_eq!(
resp.status(),
StatusCode::OK,
"an identical-URL re-announce is a liveness signal, not a repoint"
);
let after = snapshot(&state.db, &did).await.expect("row vanished");
assert_eq!(after.1, OWNER_URL, "URL untouched");
assert_eq!(after.3, before.3, "reachability untouched");
assert_eq!(after.4, before.4, "announced_at untouched");
let parse = |s: &Option<String>| {
chrono::DateTime::parse_from_rfc3339(s.as_deref().expect("last_seen is set")).unwrap()
};
assert!(
parse(&after.2) > parse(&before.2),
"a same-URL re-announce must advance last_seen"
);
}
/// U4 scenario 6 (R8): a DID method that can never authenticate is refused
/// in the 400 validation class, with nothing written. This one IS about the
/// input's form, not the caller's authority, which is why it is not a 403.
#[sqlx::test]
async fn announce_of_a_did_web_is_a_400_validation_failure(pool: PgPool) {
let state = test_state(pool).await;
let resp = announce_only(state.clone())
.oneshot(unsigned_post(
"/api/v1/peers/announce",
&announce_body(WEB_DID, OWNER_URL),
"203.0.113.93:5000",
))
.await
.unwrap();
let (status, code, message) = status_and_error(resp).await;
assert_eq!(
status,
StatusCode::BAD_REQUEST,
"an unsupported DID method is a validation failure: {message}"
);
assert_eq!(code, "bad_request");
assert!(
message.contains("methodNotSupported"),
"the rejection must carry the methodNotSupported shape, got {message:?}"
);
assert!(
snapshot(&state.db, WEB_DID).await.is_none(),
"a rejected announce must leave no row behind"
);
}
/// The sibling of the case above, and the one it must not swallow: the
/// method IS did:key, so methodNotSupported would be a false statement
/// about the input. The value is the W3C secp256k1 did:key test vector,
/// which decodes cleanly and is simply not an ed25519 key. Still the 400
/// validation class, still nothing written.
#[sqlx::test]
async fn announce_of_an_unresolvable_did_key_is_a_400_with_the_cause(pool: PgPool) {
let state = test_state(pool).await;
let did = "did:key:zQ3shokFTS3brHcDQrn82RUDfCZESWL1ZdCEJwekUDPQiYBme";
let resp = announce_only(state.clone())
.oneshot(unsigned_post(
"/api/v1/peers/announce",
&announce_body(did, OWNER_URL),
"203.0.113.99:5000",
))
.await
.unwrap();
let (status, code, message) = status_and_error(resp).await;
assert_eq!(
status,
StatusCode::BAD_REQUEST,
"an unresolvable did:key is a validation failure: {message}"
);
assert_eq!(
code, "unresolvable_did",
"the code must name the cause, the same way the signed path does for a keyid it \
cannot resolve; a client distinguishing this from the other 400s on this route \
must not have to substring-match the message: {message:?}"
);
assert!(
message.contains("cannot resolve DID"),
"the rejection must report resolution, not method support, got {message:?}"
);
assert!(
snapshot(&state.db, did).await.is_none(),
"a rejected announce must leave no row behind"
);
}
// ── U5: the same matrix through the production router, real signatures ────
/// An announce carrying a REAL RFC 9421 signature, signed AS `kp` while the
/// body names whatever DID the caller chooses. That split is the point: it
/// is what lets a valid signature be pointed at a row it does not own,
/// which is the RUSTSEC-2022-0009 shape.
fn real_signed_announce(kp: &Keypair, body: &str, peer: &str) -> Request<Body> {
let path = "/api/v1/peers/announce";
let s = sign_request(kp, "POST", path, body.as_bytes());
let mut req = Request::builder()
.method(Method::POST)
.uri(path)
.header(header::CONTENT_TYPE, "application/json")
.header("content-digest", s.content_digest)
.header("signature-input", s.signature_input)
.header("signature", s.signature)
.body(Body::from(body.to_string()))
.unwrap();
req.extensions_mut()
.insert(ConnectInfo(peer.parse::<SocketAddr>().unwrap()));
req
}
/// U5 scenario 1: the lowest-privilege caller there is, fully
/// unauthenticated, still seeds an unseen did:key row through the whole
/// production stack.
#[sqlx::test]
async fn router_unsigned_announce_inserts_an_unseen_did(pool: PgPool) {
let state = test_state(pool).await;
let did = Keypair::generate().did().to_string();
let db = state.db.clone();
let resp = crate::server::build_router(state)
.oneshot(unsigned_post(
"/api/v1/peers/announce",
&announce_body(&did, OWNER_URL),
"203.0.113.94:5000",
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let row = snapshot(&db, &did).await.expect("row not inserted");
assert_eq!(row.1, OWNER_URL);
}
/// U5 scenario 2: the defect this change closes, driven end to end. An
/// unsigned repoint is refused with its reason and the row does not move.
/// Kills reverting the unproven arm to an unconditional DO UPDATE.
#[sqlx::test]
async fn router_unsigned_announce_cannot_repoint_an_existing_row(pool: PgPool) {
let state = test_state(pool).await;
let victim = Keypair::generate().did().to_string();
let before = seed_peer_row(&state, &victim, OWNER_URL).await;
let db = state.db.clone();
let resp = crate::server::build_router(state)
.oneshot(unsigned_post(
"/api/v1/peers/announce",
&announce_body(&victim, HIJACK_URL),
"203.0.113.95:5000",
))
.await
.unwrap();
// The row assertion leads: the property is that the row does not move.
// A response-class regression is a second, separate failure below.
let (status, code, message) = status_and_error(resp).await;
assert_eq!(
snapshot(&db, &victim).await.expect("row vanished"),
before,
"the stored row must be byte-identical after a refused unsigned repoint \
(observed status {status})"
);
assert_eq!(status, StatusCode::FORBIDDEN, "observed body: {message}");
assert_eq!(code, "forbidden");
}
/// U5 scenario 3: the allowed signed case through the real verification
/// middleware. The row was seeded under this keypair's own did:key, so the
/// signature names the row it moves.
#[sqlx::test]
async fn router_signature_from_the_rows_own_did_repoints_it(pool: PgPool) {
let state = test_state(pool).await;
let owner = Keypair::generate();
let owner_did = owner.did().to_string();
seed_peer_row(&state, &owner_did, OWNER_URL).await;
let db = state.db.clone();
let resp = crate::server::build_router(state)
.oneshot(real_signed_announce(
&owner,
&announce_body(&owner_did, MOVED_URL),
"203.0.113.96:5000",
))
.await
.unwrap();
assert_eq!(
resp.status(),
StatusCode::OK,
"a real signature from the row's own DID must be admitted"
);
let row = snapshot(&db, &owner_did).await.expect("row vanished");
assert_eq!(row.1, MOVED_URL, "a proven repoint must land");
assert!(
!row.3,
"a repointed peer must re-earn reachability even when the repoint is signed"
);
}
/// U5 scenario 4, the RUSTSEC-2022-0009 shape: a signature that verifies
/// perfectly, spent on a row it does not own. The key must be bound to the
/// ROW, not to anything the request carries. The assertion names the keyid
/// denial specifically, because the boundary's own DID comparison would
/// also refuse this write, so a status-only assertion would stay green with
/// the handler branch neutralized.
#[sqlx::test]
async fn router_signature_from_another_did_cannot_repoint_this_row(pool: PgPool) {
let state = test_state(pool).await;
let victim = Keypair::generate().did().to_string();
let hijacker = Keypair::generate();
let before = seed_peer_row(&state, &victim, OWNER_URL).await;
let db = state.db.clone();
let resp = crate::server::build_router(state)
.oneshot(real_signed_announce(
&hijacker,
&announce_body(&victim, HIJACK_URL),
"203.0.113.97:5000",
))
.await
.unwrap();
let (status, code, message) = status_and_error(resp).await;
assert_eq!(
status,
StatusCode::FORBIDDEN,
"a valid signature over another DID's row must be refused: {message}"
);
assert_eq!(code, "forbidden");
assert!(
message.contains(KEYID_MISMATCH),
"the denial must name the keyid mismatch, got {message:?}"
);
assert_eq!(
snapshot(&db, &victim).await.expect("row vanished"),
before,
"the stored row must be byte-identical after a mismatched signature"
);
}
/// U5 scenario 5: an honest unsigned peer re-announcing the identical URL
/// through the full stack refreshes last_seen and moves nothing else. The
/// seeded string is reused byte for byte, since the comparison is exact.
#[sqlx::test]
async fn router_unsigned_identical_url_reannounce_refreshes_last_seen(pool: PgPool) {
let state = test_state(pool).await;
let did = Keypair::generate().did().to_string();
let before = seed_peer_row(&state, &did, OWNER_URL).await;
let db = state.db.clone();
let resp = crate::server::build_router(state)
.oneshot(unsigned_post(
"/api/v1/peers/announce",
&announce_body(&did, OWNER_URL),
"203.0.113.98:5000",
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let after = snapshot(&db, &did).await.expect("row vanished");
assert_eq!(after.1, OWNER_URL, "URL untouched");
assert_eq!(after.3, before.3, "reachability untouched");
assert_eq!(after.4, before.4, "announced_at untouched");
let parse = |s: &Option<String>| {
chrono::DateTime::parse_from_rfc3339(s.as_deref().expect("last_seen is set")).unwrap()
};
assert!(
parse(&after.2) > parse(&before.2),
"a same-URL re-announce must advance last_seen"
);
}
/// U5 scenario 6 (R8) through the full stack: a did:web announce is refused
/// in the validation class and writes nothing. Such a row could never be
/// corrected by anyone, since only a did:key can ever satisfy the proven
/// path. Kills neutralizing the method gate on the unproven arm.
#[sqlx::test]
async fn router_unsigned_announce_of_a_did_web_is_refused(pool: PgPool) {
let state = test_state(pool).await;
let db = state.db.clone();
let resp = crate::server::build_router(state)
.oneshot(unsigned_post(
"/api/v1/peers/announce",
&announce_body(WEB_DID, OWNER_URL),
"203.0.113.99:5000",
))
.await
.unwrap();
let (status, code, message) = status_and_error(resp).await;
assert_eq!(
status,
StatusCode::BAD_REQUEST,
"an unsupported DID method must be refused: {message}"
);
assert_eq!(code, "bad_request");
assert!(
message.contains("methodNotSupported"),
"the rejection must carry the methodNotSupported shape, got {message:?}"
);
assert!(
snapshot(&db, WEB_DID).await.is_none(),
"a rejected announce must leave no row behind"
);
}
/// U5, flag-on mode: with require_signed_peer_writes the write group moves
/// under add_auth_layers, so the unsigned caller dies at the auth layer with
/// a 401 instead of reaching the handler's 403. The authority rules
/// themselves do not change: a signature over another DID's row is still
/// refused, and the row's own key still moves it. The flag is applied BEFORE
/// build_router, which reads it at build time.
#[sqlx::test]
async fn announce_authority_matrix_in_signed_writes_mode(pool: PgPool) {
let mut state = test_state(pool).await;
require_signed_peer_writes(&mut state);
let owner = Keypair::generate();
let owner_did = owner.did().to_string();
let hijacker = Keypair::generate();
let before = seed_peer_row(&state, &owner_did, OWNER_URL).await;
let db = state.db.clone();
let router = crate::server::build_router(state);
let resp = router
.clone()
.oneshot(unsigned_post(
"/api/v1/peers/announce",
&announce_body(&owner_did, HIJACK_URL),
"203.0.113.100:5000",
))
.await
.unwrap();
assert_eq!(
resp.status(),
StatusCode::UNAUTHORIZED,
"with the flag on, an unsigned announce dies at the auth layer"
);
assert_eq!(
snapshot(&db, &owner_did).await.expect("row vanished"),
before,
"the row must be byte-identical after the 401"
);
let resp = router
.clone()
.oneshot(real_signed_announce(
&hijacker,
&announce_body(&owner_did, HIJACK_URL),
"203.0.113.101:5000",
))
.await
.unwrap();
let (status, _, message) = status_and_error(resp).await;
assert_eq!(
status,
StatusCode::FORBIDDEN,
"a signature over another DID's row is still refused: {message}"
);
assert!(message.contains(KEYID_MISMATCH), "got {message:?}");
assert_eq!(
snapshot(&db, &owner_did).await.expect("row vanished"),
before,
"the row must be byte-identical after the mismatched signature"
);
let resp = router
.oneshot(real_signed_announce(
&owner,
&announce_body(&owner_did, MOVED_URL),
"203.0.113.102:5000",
))
.await
.unwrap();
assert_eq!(
resp.status(),
StatusCode::OK,
"the row's own key still moves it with the flag on"
);
assert_eq!(
snapshot(&db, &owner_did).await.expect("row vanished").1,
MOVED_URL
);
}
}
+107 -2
View File
@@ -213,7 +213,7 @@ pub async fn merge_pr(
.repo_store
.acquire_write(&record.owner_did, &record.name)
.await
.map_err(|e| AppError::Git(e.to_string()))?;
.map_err(|e| crate::api::repos::acquire_write_app_error(&e, &name))?;
let disk_path = guard.path().to_path_buf();
let merger_did = auth.0;
let merge_result = store::merge_branch(
@@ -225,7 +225,10 @@ pub async fn merge_pr(
);
// Always release the advisory lock — even on error; upload to Tigris only on success.
guard.release(merge_result.is_ok()).await;
// Short-circuit on a refused publish before the PR is marked merged and
// before the webhook fires. Both are irreversible announcements of a merge
// commit that only exists on this node's disk.
guard.release(merge_result.is_ok()).await.into_result()?;
let merge_sha = merge_result.map_err(|e| AppError::Git(e.to_string()))?;
@@ -424,3 +427,105 @@ pub async fn list_comments(
let comments = state.db.list_pr_comments(&pr.id).await?;
Ok(Json(serde_json::json!({ "comments": comments })))
}
/// #173 F1 follow-up: merge holds no admission permit, so an exhausted write-lock
/// pool must shed 503 + Retry-After.
#[cfg(test)]
mod lock_pool_shed_tests {
use super::*;
use axum::response::IntoResponse;
use sqlx::PgPool;
fn seed_repo(owner_did: &str, name: &str) -> crate::db::RepoRecord {
let now = Utc::now();
crate::db::RepoRecord {
id: Uuid::new_v4().to_string(),
name: name.to_string(),
owner_did: owner_did.to_string(),
description: None,
is_public: true,
default_branch: "main".to_string(),
created_at: now,
updated_at: now,
disk_path: format!("/tmp/{name}"),
forked_from: None,
machine_id: None,
}
}
#[sqlx::test]
async fn merge_pr_lock_pool_exhaustion_sheds_503_not_500(pool: PgPool) {
let owner = "did:key:zMERGELOCKPOOLOWNERAAAAAAAAAAAAAAAAAAAAA";
let mut state = crate::test_support::test_state(pool.clone()).await;
state.repo_store = crate::git::repo_store::RepoStore::new(
std::path::PathBuf::from("/tmp/gitlawb-pulls-lockpool"),
None,
crate::git::repo_store::build_lock_pool(&pool, 1, std::time::Duration::from_secs(1)),
std::time::Duration::from_secs(300),
);
let repo = seed_repo(owner, "lp-merge");
let repo_id = repo.id.clone();
state.db.create_repo(&repo).await.expect("seed repo");
let now = Utc::now().to_rfc3339();
state
.db
.create_pr(&PullRequest {
id: Uuid::new_v4().to_string(),
repo_id: repo_id.clone(),
number: 1,
title: "lp".to_string(),
body: None,
author_did: owner.to_string(),
source_branch: "feature".to_string(),
target_branch: "main".to_string(),
status: "open".to_string(),
merged_by_did: None,
merged_at: None,
created_at: now.clone(),
updated_at: now,
})
.await
.expect("seed open PR");
let held = state
.repo_store
.acquire_write(owner, "other-repo")
.await
.expect("the first write takes the only lock-pool connection");
let shed = merge_pr(
State(state.clone()),
Extension(AuthenticatedDid(owner.to_string())),
Path((owner.to_string(), "lp-merge".to_string(), 1)),
)
.await;
let err = shed.expect_err("an exhausted lock pool must fail the call");
let resp = err.into_response();
assert_eq!(
resp.status(),
StatusCode::SERVICE_UNAVAILABLE,
"merge_pr: an exhausted write-lock pool must shed 503, not a 500 git error"
);
assert_eq!(
resp.headers()
.get("retry-after")
.map(|v| v.to_str().unwrap()),
Some("1"),
"merge_pr: a capacity shed must tell the client when to retry"
);
let _ = held.release(false).await;
let admitted = merge_pr(
State(state.clone()),
Extension(AuthenticatedDid(owner.to_string())),
Path((owner.to_string(), "lp-merge".to_string(), 1)),
)
.await;
assert!(
!matches!(admitted, Err(AppError::Overloaded(_))),
"with the lock pool free, merge_pr must not be shed as capacity; got {:?}",
admitted.err()
);
}
}
File diff suppressed because it is too large. Load diff
+12 -1
View File
@@ -490,6 +490,7 @@ mod tests {
use clap::Parser;
let keypair = Keypair::generate();
let scan_token_key = crate::state::AppState::derive_scan_token_key(&keypair);
let (ref_tx, _) = tokio::sync::broadcast::channel(1);
let (task_tx, _) = tokio::sync::broadcast::channel(1);
let pool = sqlx::postgres::PgPoolOptions::new()
@@ -516,6 +517,17 @@ mod tests {
rate_limiter: RateLimiter::new(100, Duration::from_secs(60)),
create_ip_rate_limiter: RateLimiter::new(1000, Duration::from_secs(3600)),
push_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
close_issue_rate_limiter: RateLimiter::new(120, Duration::from_secs(3600)),
ipfs_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
ipfs_work_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
ipfs_max_history_walks: crate::api::ipfs::MAX_HISTORY_WALKS_PER_REQUEST,
ipfs_max_legacy_probes: crate::api::ipfs::MAX_LEGACY_PROBES_PER_REQUEST,
ipfs_legacy_scan_page_rows: crate::api::ipfs::LEGACY_SCAN_PAGE_ROWS,
ipfs_max_legacy_scan_rows: crate::api::ipfs::MAX_LEGACY_SCAN_ROWS_PER_REQUEST,
ipfs_max_legacy_scan_rule_bytes:
crate::api::ipfs::MAX_LEGACY_SCAN_RULE_BYTES_PER_REQUEST,
ipfs_scan_token_key: Arc::new(scan_token_key),
ipfs_max_served_object_bytes: crate::api::ipfs::MAX_SERVED_OBJECT_BYTES,
push_limiter_trust: crate::rate_limit::TrustedProxy::None,
sync_trigger_rate_limiter: RateLimiter::new(60, Duration::from_secs(3600)),
peer_write_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
@@ -534,7 +546,6 @@ mod tests {
git_ipfs_walk_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_ipfs_walk_per_caller:
crate::rate_limit::PerCallerConcurrency::with_default_max_keys(16),
ipfs_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
git_bin: "git".to_string(),
}
}
+605 -43
View File
@@ -1,13 +1,13 @@
use clap::Parser;
use std::path::PathBuf;
/// Upper bound on `git_service_timeout_secs` and `ipfs_request_budget_secs`, in seconds
/// (100 years).
/// Upper bound on `git_service_timeout_secs`, `ipfs_request_budget_secs`, and
/// `ipfs_resolve_budget_secs`, in seconds (100 years).
///
/// Two consumers now, so a future tightening moves both. `ipfs_request_budget_secs`
/// derives only the `Instant` addition in `get_by_cid`, not the lease-steal multiply
/// below, but it shares this ceiling because the defect class and the "set it very large
/// to disable" contract are the same.
/// Three consumers now, so a future tightening moves all of them. `ipfs_request_budget_secs`
/// and `ipfs_resolve_budget_secs` derive only the `Instant` addition in `get_by_cid`, not the
/// lease-steal multiply below, but they share this ceiling because the defect class and the
/// "set it very large to disable" contract are the same.
///
/// The knob is not just stored, it is arithmetic input: the write path derives the
/// per-repo lease steal bound from it (`* 2 + 60`), and #174 routed it into
@@ -78,10 +78,26 @@ pub struct Config {
/// Require the authenticated pusher to be the repo owner on `git-receive-pack`.
/// Authentication (a valid did:key signature) is not authorization on its own:
/// any party can sign as their own DID. When true, pushes whose authenticated
/// DID is not the repo owner are rejected. Keep false during rolling upgrades;
/// flip it on once owners are ready for owner-only writes.
#[arg(long, env = "GITLAWB_ENFORCE_OWNER_PUSH", default_value_t = false)]
/// any party can mint a did:key and sign as it, so with this off every signed
/// caller may push to every repository, private ones included. On by default.
///
/// Turn it off only for a rolling upgrade whose pushers are not yet the repo
/// owner. Both `GITLAWB_ENFORCE_OWNER_PUSH=false` and `--enforce-owner-push
/// false` disable it, and the bare `--enforce-owner-push` form still means
/// `true`.
///
/// The value-taking action is what makes the CLI form parse at all: as a
/// presence-only flag, `--enforce-owner-push false` is an "unexpected argument"
/// error. The env form resolved correctly either way, so it is the CLI escape
/// hatch this buys, not the environment one.
#[arg(
long,
env = "GITLAWB_ENFORCE_OWNER_PUSH",
action = clap::ArgAction::Set,
num_args = 0..=1,
default_value_t = true,
default_missing_value = "true"
)]
pub enforce_owner_push: bool,
/// URL of local IPFS/Kubo node HTTP API (e.g. http://127.0.0.1:5001)
@@ -236,11 +252,9 @@ pub struct Config {
/// Maximum connections in the PostgreSQL pool. This is a cap, not a floor
/// (connections open lazily). Size against the database server's
/// max_connections, remembering admin tooling opens its own pool. Each
/// concurrent write pins one pooled connection for its whole duration (the
/// advisory lock in `repo_store::acquire_write` is connection-affine), so this
/// must exceed `max_concurrent_git_pushes` by `DB_POOL_APP_HEADROOM` or slow
/// pushes starve every other DB path — enforced by `Config::validate`.
/// max_connections, remembering admin tooling opens its own pool. Writes pin
/// the dedicated lock pool, not this one; this pool still needs enough headroom
/// for ordinary request handlers and metadata paths (`DB_POOL_APP_HEADROOM`).
#[arg(
long,
env = "GITLAWB_DB_MAX_CONNECTIONS",
@@ -249,6 +263,46 @@ pub struct Config {
)]
pub db_max_connections: u32,
/// Maximum connections in the dedicated advisory-lock pool, which is separate
/// from the main pool above.
///
/// Size this against the expected peak number of concurrent distinct-repo
/// writers, NOT small. Every in-flight repo write pins one connection here for
/// its whole duration (the write, its metadata tail, and the bounded archive
/// upload), so this value is a hard ceiling on simultaneous writes node-wide.
/// Keeping it separate from GITLAWB_DB_MAX_CONNECTIONS is what stops a push
/// burst from starving ordinary request handlers; the cost is that
/// (main pool + lock pool) must fit inside the database server's
/// max_connections, times the number of nodes, plus admin tooling.
#[arg(
long,
env = "GITLAWB_DB_LOCK_POOL_MAX_CONNECTIONS",
default_value_t = 40,
value_parser = clap::value_parser!(u32).range(1..)
)]
pub db_lock_pool_max_connections: u32,
/// Upper bound, in seconds, on any single object-storage transfer that runs
/// while the per-repo advisory lock is HELD.
///
/// Two bounded spans exist, and the bound applies per span. The acquire-side
/// refresh in `acquire_write` covers the existence HEAD and the download
/// together under ONE budget (it runs after the lock is taken and before the
/// guard is constructed), and the archive upload in `release` gets its own.
/// Worst-case slot occupancy is therefore about twice this value plus the git
/// work between them, not one times this value. Both used to be free, because the lock's
/// connection was returned to the pool immediately; now that a write guard
/// pins a lock-pool connection for its whole lifetime, an unbounded transfer
/// holds that slot, and enough stalled transfers deny every write on the node.
/// This is the bound that keeps a stall from becoming an outage.
#[arg(
long,
env = "GITLAWB_LOCK_HELD_TRANSFER_TIMEOUT_SECS",
default_value_t = 300,
value_parser = clap::value_parser!(u64).range(1..)
)]
pub lock_held_transfer_timeout_secs: u64,
/// Maximum time a request waits for a pool connection before failing with
/// 503, in seconds. Bounds queueing when the database is slow or down.
#[arg(
@@ -341,6 +395,16 @@ pub struct Config {
/// Default: 32. Must be between 1 and 1_048_576 (the ceiling keeps the value
/// under tokio's `Semaphore` permit limit so an oversized value is a clean CLI
/// error rather than a boot-time panic).
///
/// CONNECTION BUDGET. A push holds a Postgres connection from the node's separate
/// advisory-lock pool (`GITLAWB_DB_LOCK_POOL_MAX_CONNECTIONS`, default 40) for
/// the whole receive-pack. That pool must be at least this value plus
/// `DB_LOCK_POOL_NON_PUSH_HEADROOM` so admitted pushes and non-push mutations
/// can each pin a lock-pool connection for their whole duration. Size BOTH
/// pools against the database server's `max_connections`: the main pool
/// (`GITLAWB_DB_MAX_CONNECTIONS`, default 48) serves ordinary handlers, and the
/// lock pool serves writes. Raising this knob does not raise the lock pool;
/// set `GITLAWB_DB_LOCK_POOL_MAX_CONNECTIONS` explicitly.
#[arg(
long,
env = "GITLAWB_MAX_CONCURRENT_GIT_PUSHES",
@@ -452,6 +516,54 @@ pub struct Config {
)]
pub ipfs_walk_per_source: usize,
/// Per-request ceiling on the number of legacy (NULL-provenance) repos the
/// `/ipfs/{cid}` resolver's scan fallback will PROBE (`acquire` + `git cat-file
/// -t`) before giving up. The provenance path targets its recorded sources; the
/// legacy scan, absent this bound, fans one anonymous request out to O(repos)
/// subprocess spawns and cold-cache fetches for a CID enumerable from the public
/// pins index. A truncated scan surfaces as a retryable 503, never a false 404.
/// Wired into `AppState::ipfs_max_legacy_probes` at construction. This knob does
/// not govern the history-walk ceiling; see `ipfs_max_repos_walked` for that.
/// Must be between 1 and 1_048_576. Default: 256.
#[arg(
long,
env = "GITLAWB_IPFS_MAX_LEGACY_PROBES",
default_value_t = crate::api::ipfs::MAX_LEGACY_PROBES_PER_REQUEST as usize,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub ipfs_max_legacy_probes: usize,
/// Per-request ceiling on how many repo ROWS the `/ipfs/{cid}` resolver's legacy
/// scan may fetch from the database. The probe ceiling above only starts counting
/// once a probe runs, and the two denial classes that dominate a hostile inventory
/// (quarantine, and a root-scope visibility deny) return before a probe or a visit
/// is spent, so without this an all-denying node paged its ENTIRE repo table for one
/// anonymous request while holding a scarce walk permit.
///
/// Reach bound: a holder buried past the ceiling is servable in
/// `ceil(repos / ceiling) + 1` token-echoing retries. A truncated scan sheds a
/// retryable 503 carrying a sealed continuation token; the caller echoes it as
/// `?scan=` and the scan resumes where it stopped. No server-side scan state.
///
/// Floor coupling: raising this knob raises every caller's per-window `/ipfs` work
/// allowance whenever the route limit sits below the derived floor, because the
/// floor must fit one full deep scan's page toll (see `AppState::ipfs_work_budget`).
///
/// Tuning DOWN trade: token presence is a coarse inventory-size oracle. A ceiling
/// truncation emits a token and a wrapped scan does not, so laddering to the
/// `scan-wrapped` taint tells an anonymous caller the node's total repo count,
/// private and quarantined included, to within one ceiling. Tolled and coarse at
/// the 2048 default; it sharpens as the ceiling is lowered.
///
/// Must be between 1 and 1_048_576. Default: 2048.
#[arg(
long,
env = "GITLAWB_IPFS_MAX_LEGACY_SCAN_ROWS",
default_value_t = crate::api::ipfs::MAX_LEGACY_SCAN_ROWS_PER_REQUEST,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub ipfs_max_legacy_scan_rows: usize,
/// Upper bound on the number of EXPENSIVE visibility walks
/// (`allowed_blob_set_for_caller_bounded`, a full-history git walk in a
/// blocking thread) a single `/ipfs/{cid}` request may run. Only a blob in a
@@ -463,6 +575,22 @@ pub struct Config {
/// sheds a retryable 503 + Retry-After rather than misreport existing content
/// absent with a 404. The handler still short-circuits the moment it serves.
/// Must be between 1 and 1_048_576. Default: 64.
///
/// The effective ceiling is the TIGHTER of this knob and the node's internal
/// history-walk ceiling, `MAX_PIN_SOURCES + 1` = 17 (see
/// `api::ipfs::MAX_HISTORY_WALKS_PER_REQUEST` and the `min()` that combines the
/// two in the resolver). Setting this above 17 changes nothing, because the
/// internal ceiling already binds. Setting it below 17 does lower the cap: the
/// constant side of the `min()` is what keeps a request from being truncated
/// before its whole bounded provenance source set has been tried, so an operator
/// who goes under it is choosing a tighter cap that can 503 a provenanced
/// request, which is allowed.
///
/// That combined cap is charged PER PHASE, not per request: the provenance phase and
/// the legacy-scan fallback each get their own equal budget, so one request can run
/// up to twice it in total (see `MAX_HISTORY_WALKS_PER_REQUEST`, which explains why
/// the split is what keeps the fallback from inheriting a provenance phase's spent
/// remainder).
#[arg(
long,
env = "GITLAWB_IPFS_MAX_REPOS_WALKED",
@@ -531,14 +659,86 @@ pub struct Config {
)]
pub ipfs_request_budget_secs: u64,
/// Budget for the PRE-WALK CID resolve inside `get_by_cid`, in seconds: the
/// `oids_for_cid` lookup that maps the requested CID to its git oid(s), which runs
/// while the scarce walk admission (the global pool permit plus the per-source
/// sub-permit) is already held.
///
/// It exists because that one await decides whether the request does any admitted
/// work at all. A syntactically valid CID with no `pinned_cids` row runs zero probes
/// and zero walks, so under a stalled or saturated pool it would otherwise occupy a
/// walk slot for the whole `ipfs_request_budget_secs` window (600s by default) while
/// nothing is walking, and enough distinct source keys doing that reject every real
/// `/ipfs` retrieval at admission. The other repair, resolving the CID before taking
/// admission, was rejected: admission stays FIRST so an anonymous flood sheds before
/// touching the database at all, and moving the read ahead of it would let arbitrarily
/// many unadmitted permissionless callers stack concurrent DB queries.
///
/// The effective deadline is the lesser of this and the remaining request budget, so a
/// value larger than `ipfs_request_budget_secs` degrades to the request budget rather
/// than extending it. Only the resolve is on this clock; every later stage stays on the
/// full request budget, because from the second oid candidate on those run after real
/// probe and walk work and a short deadline anchored at admission would shed a
/// legitimately slow but progressing scan.
///
/// Must be positive, and no larger than `GIT_SERVICE_TIMEOUT_SECS_MAX`, for the same
/// representability reason as the request budget above: `get_by_cid` derives the
/// resolve deadline as `Instant::now() + Duration::from_secs(this)`, and that addition
/// panics on overflow in release builds too. Default: 10s.
#[arg(
long,
env = "GITLAWB_IPFS_RESOLVE_BUDGET_SECS",
default_value_t = 10,
value_parser = clap::value_parser!(u64).range(1..=GIT_SERVICE_TIMEOUT_SECS_MAX)
)]
pub ipfs_resolve_budget_secs: u64,
/// Per-client-IP rate limit for `GET /ipfs/{cid}`, in requests per hour. The
/// route is publicly reachable (`optional_signature`) and each request can drive
/// a full-history git walk, so it carries a per-IP flood brake in addition to the
/// concurrency cap above (a rate limit bounds request *rate*, the semaphore
/// bounds concurrent slow holds — different axes). Keyed on the resolved client
/// IP via `GITLAWB_TRUSTED_PROXY`. `0` disables. Default: 600.
///
/// This is the pure once-per-request ROUTE brake. The resolver's internal
/// per-probe/per-walk WORK budget is a SEPARATE bucket whose capacity is DERIVED
/// from this value (`AppState::ipfs_work_budget`), not a knob of its own; `0` here
/// disables that derived bucket too.
#[arg(long, env = "GITLAWB_IPFS_RATE_LIMIT", default_value_t = 600)]
pub ipfs_rate_limit: usize,
/// Rows the legacy provider-CID repair sweep reads per batch (U4, #173).
///
/// The sweep walks every `pinned_cids` row on the node once, repairing rows that
/// releases before this branch keyed on a PROVIDER CID (Kubo dag-pb / Pinata CIDv0)
/// instead of the raw-content resolver key. This bounds one batch, so the sweep can
/// never turn into a single unbounded table scan competing with request traffic.
/// Conservative on purpose: paired with the inter-batch delay below the default is
/// ~64 rows per minute, which finishes a large pin set in hours of idle background
/// work rather than one expensive burst. Must be between 1 and 100_000.
#[arg(
long,
env = "GITLAWB_PIN_REPAIR_SWEEP_BATCH",
default_value_t = 64,
value_parser = clap::builder::RangedU64ValueParser::<i64>::new().range(1..=100_000)
)]
pub pin_repair_sweep_batch: i64,
/// Seconds the legacy provider-CID repair sweep sleeps between batches (U4, #173).
///
/// Each batch costs an indexed range scan plus, for the legacy rows in it, a
/// `git cat-file` per row. The delay is what keeps that off the DB's and the disk's
/// critical path: the sweep is repairing rows that have been unresolvable since the
/// upgrade, so finishing slowly is fine and finishing fast at the cost of live
/// traffic is not. `0` disables the pause (test and one-off operational use only).
/// Must be between 0 and 86_400.
#[arg(
long,
env = "GITLAWB_PIN_REPAIR_SWEEP_DELAY_SECS",
default_value_t = 60,
value_parser = clap::builder::RangedU64ValueParser::<u64>::new().range(0..=86_400)
)]
pub pin_repair_sweep_delay_secs: u64,
}
impl Config {
@@ -562,25 +762,37 @@ impl Config {
/// the pool must clear the concurrent-write cap by at least this margin.
pub const DB_POOL_APP_HEADROOM: u32 = 8;
/// Lock-pool slots reserved for non-push `acquire_write` callers (issue create,
/// close, merge) so a saturated push budget cannot turn ordinary mutations into
/// lock-pool exhaustion on unrelated repositories.
pub const DB_LOCK_POOL_NON_PUSH_HEADROOM: u32 = 8;
/// Cross-field boot validation. Single-field ranges are enforced by clap; this
/// catches combinations that ship a denial-of-service under otherwise-valid
/// values. Call once at startup and fail fast on `Err`.
pub fn validate(&self) -> Result<(), String> {
// A write pins one pooled connection for its whole duration (the
// connection-affine advisory lock in repo_store::acquire_write), and
// concurrent writes are capped at max_concurrent_git_pushes. If the pool
// does not exceed that cap by DB_POOL_APP_HEADROOM, a burst of slow pushes
// drains every connection and every other DB path 503s. (#174 F1)
let floor = (self.max_concurrent_git_pushes as u64) + (Self::DB_POOL_APP_HEADROOM as u64);
if (self.db_max_connections as u64) < floor {
// Concurrent git writes pin the dedicated lock pool for their whole
// duration (connection-affine advisory lock in acquire_write). The main
// pool no longer carries that occupancy.
let lock_floor = self
.max_concurrent_git_pushes
.saturating_add(Self::DB_LOCK_POOL_NON_PUSH_HEADROOM as usize);
if (self.db_lock_pool_max_connections as usize) < lock_floor {
return Err(format!(
"GITLAWB_DB_MAX_CONNECTIONS ({}) must be at least max_concurrent_git_pushes ({}) \
+ {} headroom = {}: each concurrent write pins one pooled connection for its whole \
duration, so a smaller pool lets a burst of slow pushes starve every other DB path.",
self.db_max_connections,
"GITLAWB_DB_LOCK_POOL_MAX_CONNECTIONS ({}) must be at least \
max_concurrent_git_pushes ({}) plus {} for non-push mutations \
that share the lock pool",
self.db_lock_pool_max_connections,
self.max_concurrent_git_pushes,
Self::DB_POOL_APP_HEADROOM,
floor
Self::DB_LOCK_POOL_NON_PUSH_HEADROOM
));
}
let main_floor = Self::DB_POOL_APP_HEADROOM as u64;
if (self.db_max_connections as u64) < main_floor {
return Err(format!(
"GITLAWB_DB_MAX_CONNECTIONS ({}) must be at least {} for non-git \
database paths",
self.db_max_connections, main_floor
));
}
Ok(())
@@ -591,6 +803,25 @@ impl Config {
mod tests {
use super::*;
#[test]
fn lock_pool_size_defaults_to_40_and_rejects_zero() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).db_lock_pool_max_connections,
40
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--db-lock-pool-max-connections", "8"])
.db_lock_pool_max_connections,
8
);
// A zero-sized lock pool would deny every write, so clap must reject it
// rather than let a node boot into a state where no repo can be written.
assert!(
Config::try_parse_from(["gitlawb-node", "--db-lock-pool-max-connections", "0"])
.is_err()
);
}
#[test]
fn git_service_timeout_defaults_to_600_and_rejects_zero() {
assert_eq!(
@@ -771,6 +1002,35 @@ mod tests {
);
}
/// U4 (#173): the repair sweep's bounds are conservative by default and a batch of
/// 0 (a sweep that walks nothing and never terminates) is a CLI error, not a
/// runtime hang. The delay does accept 0, for tests and one-off operational runs.
#[test]
fn pin_repair_sweep_knobs_default_conservatively() {
let c = Config::parse_from(["gitlawb-node"]);
assert_eq!(c.pin_repair_sweep_batch, 64);
assert_eq!(c.pin_repair_sweep_delay_secs, 60);
assert!(Config::try_parse_from(["gitlawb-node", "--pin-repair-sweep-batch", "0"]).is_err());
assert!(
Config::try_parse_from(["gitlawb-node", "--pin-repair-sweep-batch", "100001"]).is_err()
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--pin-repair-sweep-batch", "8"])
.pin_repair_sweep_batch,
8
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--pin-repair-sweep-delay-secs", "0"])
.pin_repair_sweep_delay_secs,
0
);
assert!(
Config::try_parse_from(["gitlawb-node", "--pin-repair-sweep-delay-secs", "86401"])
.is_err()
);
}
#[test]
fn ipfs_walk_per_source_defaults_and_rejects_out_of_range() {
assert_eq!(Config::parse_from(["gitlawb-node"]).ipfs_walk_per_source, 4);
@@ -786,6 +1046,195 @@ mod tests {
);
}
/// The legacy-probe budget and the expensive-walk cap are SEPARATE knobs with
/// different defaults. They were one field until the probe budget and the walk cap
/// were split apart, so assert both defaults here: a future collapse back into one
/// field silently gives one of the two the other's default.
#[test]
fn ipfs_probe_and_walk_knobs_default_apart_and_reject_out_of_range() {
let default = Config::parse_from(["gitlawb-node"]);
assert_eq!(default.ipfs_max_legacy_probes, 256, "legacy-probe budget");
assert_eq!(default.ipfs_max_repos_walked, 64, "expensive-walk cap");
assert_eq!(
Config::parse_from(["gitlawb-node", "--ipfs-max-legacy-probes", "8"])
.ipfs_max_legacy_probes,
8
);
// 0 would probe no repos (serve nothing); clap must reject it.
assert!(Config::try_parse_from(["gitlawb-node", "--ipfs-max-legacy-probes", "0"]).is_err());
assert!(
Config::try_parse_from(["gitlawb-node", "--ipfs-max-legacy-probes", "1048577"])
.is_err()
);
assert!(Config::try_parse_from(["gitlawb-node", "--ipfs-max-repos-walked", "0"]).is_err());
}
/// The `GITLAWB_IPFS_MAX_LEGACY_PROBES` knob must actually reach the legacy-probe
/// budget it advertises: production seeds `ipfs_max_legacy_probes` from this helper,
/// so the knob is a no-op unless the helper reflects it. RED while the helper returns
/// the hardcoded `MAX_LEGACY_PROBES_PER_REQUEST` (256 regardless of the knob), GREEN
/// once it reads the knob.
#[test]
fn ipfs_max_legacy_probes_wires_the_legacy_probe_budget() {
use crate::state::AppState;
// Knob set to 1 → a one-probe legacy budget.
let one = Config::parse_from(["gitlawb-node", "--ipfs-max-legacy-probes", "1"]);
assert_eq!(
AppState::ipfs_legacy_probe_budget(&one),
1,
"the knob must control the legacy-probe budget, not be ignored"
);
// Unset knob preserves the shipped 256-probe behaviour.
let default = Config::parse_from(["gitlawb-node"]);
assert_eq!(
AppState::ipfs_legacy_probe_budget(&default),
256,
"the default knob keeps the shipped 256-probe budget"
);
assert_eq!(
AppState::ipfs_legacy_probe_budget(&default),
crate::api::ipfs::MAX_LEGACY_PROBES_PER_REQUEST,
"the default budget equals the constant it replaced"
);
// Ceiling guard: the knob never governs the history-walk ceiling, which must
// stay at MAX_PIN_SOURCES + 1 or a provenanced full source set false-503s.
assert!(
crate::api::ipfs::MAX_HISTORY_WALKS_PER_REQUEST > crate::db::MAX_PIN_SOURCES as u32,
"the history-walk ceiling is independent of the repos-walked knob"
);
}
/// The `/ipfs` work-budget capacity is DERIVED from the route limit (R6, KTD6), with
/// a hard floor of one complete COMBINED resolution per window: the provenance
/// phase's walk term plus a full legacy search (the effective
/// `ipfs_max_legacy_probes` plus the row ceiling's page toll). This guards the
/// derived default so a single default-config deep search never self-throttles
/// mid-scan and recreates the F6 admit-then-429 for a legitimate caller. A
/// `RateLimiter` sized to the derived budget must admit the whole budget back to
/// back.
#[test]
fn ipfs_work_budget_derives_from_route_limit_and_clears_the_probe_floor() {
use crate::state::AppState;
// Default config: derived work budget = max(route 600, probe budget 256) = 600,
// comfortably above the 256-probe floor.
let default = Config::parse_from(["gitlawb-node"]);
let budget = AppState::ipfs_work_budget(&default);
assert_eq!(budget, 600, "default derives max(route 600, probe 256)");
assert!(
budget >= AppState::ipfs_legacy_probe_budget(&default) as usize,
"the work budget must clear one full legacy search per window"
);
// Tight route limit (1): the floor lifts the work budget to one complete
// COMBINED resolution, the 256-probe budget PLUS the page toll a 2048-row
// ceiling costs at 128 rows per page (16) PLUS the provenance phase's walk term
// min(17, 64) = 17, so 289, NOT down to 1. The provenance walks come off the
// same bucket before the fallback runs, so a floor without that term hands the
// legacy search a bucket the provenance phase already spent from. This case
// also carries the walk term's ABOVE-constant direction: the repos-walked knob
// is at its default 64, so `MAX_HISTORY_WALKS_PER_REQUEST` (17) is what binds.
let tight = Config::parse_from(["gitlawb-node", "--ipfs-rate-limit", "1"]);
assert_eq!(
AppState::ipfs_work_budget(&tight),
289,
"a tight route limit is floored at probes + pages + walks \
(256 + 16 + min(17, 64) = 17), not clamped to 1"
);
// The walk term's BELOW-constant direction: a repos-walked knob under the
// history-walk constant is what the resolver's own `walk_cap` min() selects, so
// it is what the floor must carry too. 256 + 16 + min(17, 3) = 275.
let narrow_walk = Config::parse_from([
"gitlawb-node",
"--ipfs-rate-limit",
"1",
"--ipfs-max-repos-walked",
"3",
]);
assert_eq!(
AppState::ipfs_work_budget(&narrow_walk),
275,
"the walk term takes min(17, repos-walked 3) = 3, the resolver's own \
walk_cap, so the floor is 256 + 16 + 3"
);
// Raised probe budget lifts the floor with it (the work budget tracks the
// effective probe budget, not the constant). The walk cap here is a SECOND
// below-constant proof at a different pair of values: min(17, 7) = 7, and the
// probe knob is raised at the same time so a floor that folded the two terms
// together (they were one field before the split) reads visibly wrong rather
// than plausibly right.
let raised = Config::parse_from([
"gitlawb-node",
"--ipfs-rate-limit",
"10",
"--ipfs-max-legacy-probes",
"1000",
"--ipfs-max-repos-walked",
"7",
]);
assert_eq!(
AppState::ipfs_work_budget(&raised),
1023,
"the floor tracks the operator-raised legacy-probe budget (1000) plus the \
default row ceiling's page toll (16) plus the walk term min(17, 7) = 7"
);
// The scan-rows knob is coupled to the floor too, and this EXECUTES the coupling
// rather than describing it: every page the ceiling permits is charged to the
// caller's work bucket, so a raised ceiling that did not lift the floor would
// 429 an honest caller part-way down their own token ladder. 4096 rows at 128
// rows per page is 32 pages, so the floor is 256 + 32 + the default walk term
// of 17.
let wide_scan = Config::parse_from([
"gitlawb-node",
"--ipfs-rate-limit",
"10",
"--ipfs-max-legacy-scan-rows",
"4096",
]);
assert_eq!(
AppState::ipfs_work_budget(&wide_scan),
305,
"raising the row ceiling must raise the work floor by the pages it buys \
(256 probes + 4096/128 = 32 pages + min(17, 64) = 17 walks), or a full \
deep scan self-throttles"
);
// 0 route limit disables the derived bucket too (a 0-capacity limiter admits all).
let disabled = Config::parse_from(["gitlawb-node", "--ipfs-rate-limit", "0"]);
assert_eq!(
AppState::ipfs_work_budget(&disabled),
0,
"route limit 0 disables the derived work bucket alongside the route brake"
);
// Behavioral floor: a limiter sized to the derived (tight-route) budget admits
// a whole combined resolution's worth of charges back to back for one source,
// then sheds the next.
let budget = AppState::ipfs_work_budget(&tight);
let limiter =
crate::rate_limit::RateLimiter::new(budget, std::time::Duration::from_secs(3600));
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
rt.block_on(async {
for i in 0..budget {
assert!(
limiter.check("1.2.3.4").await,
"charge {i} of one full combined resolution must be admitted (no mid-scan throttle)"
);
}
assert!(
!limiter.check("1.2.3.4").await,
"the probe past the derived budget is shed"
);
});
}
#[test]
fn ipfs_max_repos_walked_defaults_and_rejects_out_of_range() {
assert_eq!(
@@ -840,6 +1289,52 @@ mod tests {
);
}
#[test]
fn ipfs_resolve_budget_secs_defaults_to_10_and_rejects_zero() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).ipfs_resolve_budget_secs,
10
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--ipfs-resolve-budget-secs", "3"])
.ipfs_resolve_budget_secs,
3
);
// 0 would shed every /ipfs request at the pre-walk resolve (unconditional
// 503); clap must reject it.
assert!(
Config::try_parse_from(["gitlawb-node", "--ipfs-resolve-budget-secs", "0"]).is_err()
);
// The ceiling is shared with the request budget: at the max it parses and the
// derived deadline is still representable, past it clap rejects.
let at_max = Config::try_parse_from([
"gitlawb-node",
"--ipfs-resolve-budget-secs",
&GIT_SERVICE_TIMEOUT_SECS_MAX.to_string(),
])
.expect("the documented maximum must parse");
assert_eq!(
at_max.ipfs_resolve_budget_secs,
GIT_SERVICE_TIMEOUT_SECS_MAX
);
assert!(std::time::Instant::now()
.checked_add(std::time::Duration::from_secs(
at_max.ipfs_resolve_budget_secs
))
.is_some());
for over in [GIT_SERVICE_TIMEOUT_SECS_MAX + 1, u64::MAX] {
assert!(
Config::try_parse_from([
"gitlawb-node",
"--ipfs-resolve-budget-secs",
&over.to_string(),
])
.is_err(),
"{over} is past the representable ceiling and must be rejected at parse time"
);
}
}
/// #174 (RED-before/GREEN-after): the upper bound is what keeps the deadline derived
/// from this knob in range. `get_by_cid` builds the request budget as
/// `Instant::now() + Duration::from_secs(this)` (api/ipfs.rs), and that addition is an
@@ -936,41 +1431,108 @@ mod tests {
);
}
/// #174 F1: a connection-affine write lock pins a pooled connection per
/// concurrent write, so the pool must clear `max_concurrent_git_pushes` by
/// `DB_POOL_APP_HEADROOM` or a push burst starves every other DB path.
/// `validate()` must reject an under-sized pool at boot.
/// #174 F1: concurrent git writes pin the dedicated lock pool, not the main
/// pool. `validate()` must reject an under-sized lock pool at boot.
#[test]
fn db_pool_must_clear_the_git_push_cap() {
// Shipped defaults validate (48 >= 32 + 8).
// Shipped defaults validate (lock pool 40 >= pushes 32 + non-push headroom 8).
Config::parse_from(["gitlawb-node"])
.validate()
.expect("default config must validate");
// An under-sized pool relative to the push cap is rejected (20 < 32 + 8).
let under = Config::parse_from([
// An under-sized lock pool relative to the push cap is rejected.
let under_lock = Config::parse_from([
"gitlawb-node",
"--db-max-connections",
"20",
"--db-lock-pool-max-connections",
"16",
"--max-concurrent-git-pushes",
"32",
]);
assert!(
under.validate().is_err(),
"db_max_connections 20 below max_concurrent_git_pushes 32 + headroom must be rejected"
under_lock.validate().is_err(),
"db_lock_pool_max_connections below max_concurrent_git_pushes + headroom must be rejected"
);
// Exactly at the floor validates (40 == 32 + 8).
let at_floor = Config::parse_from([
// Exactly at the push cap without non-push headroom is rejected.
let push_only = Config::parse_from([
"gitlawb-node",
"--db-lock-pool-max-connections",
"32",
"--max-concurrent-git-pushes",
"32",
]);
assert!(
push_only.validate().is_err(),
"db_lock_pool_max_connections equal to max_concurrent_git_pushes must be rejected"
);
// Main pool can be smaller than pushes + headroom when the lock pool carries writes.
let split = Config::parse_from([
"gitlawb-node",
"--db-max-connections",
"16",
"--db-lock-pool-max-connections",
"40",
"--max-concurrent-git-pushes",
"32",
]);
assert!(
at_floor.validate().is_ok(),
"db_max_connections at the floor (pushes + headroom) must validate"
split.validate().is_ok(),
"a small main pool with a large lock pool must validate"
);
// Main pool still needs the app headroom floor.
let under_main = Config::parse_from(["gitlawb-node", "--db-max-connections", "4"]);
assert!(
under_main.validate().is_err(),
"db_max_connections below DB_POOL_APP_HEADROOM must be rejected"
);
}
/// The DECLARED default, read off the parser rather than out of a parse.
///
/// `Config::parse_from` consults the process environment, so on a host that
/// exports `GITLAWB_ENFORCE_OWNER_PUSH=false` a parse-based assertion says
/// nothing about what this crate declares — it reports the operator's setting.
/// Asserting the declaration is the env-independent form, and it is the one that
/// actually fails if someone flips `default_value_t` back.
#[test]
fn enforce_owner_push_is_declared_true_independent_of_the_environment() {
use clap::CommandFactory;
let cmd = Config::command();
let arg = cmd
.get_arguments()
.find(|a| a.get_id() == "enforce_owner_push")
.expect("the argument must exist");
assert_eq!(
arg.get_default_values(),
["true"],
"owner-only push must be the declared default; a node started with no \
configuration cannot accept a push from a self-minted key"
);
}
/// The flip must not strand an operator mid-upgrade.
///
/// Turning the gate on is a breaking change for any deployment whose pushers are
/// not yet the repo owner, so the escape hatch has to keep working. As a
/// presence-only flag `--enforce-owner-push false` is not "false", it is an
/// "unexpected argument" error; the value-taking action is what makes that form
/// parse.
///
/// This pins the CLI form only. The environment form resolved to `false` under
/// the presence-only declaration too, so it is not what this change fixed, and it
/// is not exercised here because the process environment is global and these
/// tests run in parallel.
#[test]
fn enforce_owner_push_stays_disableable_for_rolling_upgrades() {
assert!(
!Config::parse_from(["gitlawb-node", "--enforce-owner-push", "false"])
.enforce_owner_push,
"operators must still be able to opt out during a rolling upgrade"
);
assert!(
Config::parse_from(["gitlawb-node", "--enforce-owner-push", "true"]).enforce_owner_push
);
}
}
File diff suppressed because it is too large. Load diff
+252 -14
View File
@@ -35,12 +35,31 @@ pub enum AppError {
#[error("invalid request: {0}")]
BadRequest(String),
/// A DID was well-formed enough to carry to a resolver but no verifying key
/// could be derived from it. Its own code rather than plain `bad_request`
/// because the auth middleware already answers `unresolvable_did` for the
/// same failure on a request's keyid, and a client should not have to
/// substring-match a message to tell this apart from the other validation
/// failures on the same route.
#[error("unresolvable did: {0}")]
UnresolvableDid(String),
#[error("too many requests: {0}")]
TooManyRequests(String),
#[error("incomplete: {0}")]
Incomplete(String),
/// A bounded search that could not complete. `continuation`, when present, is the
/// sealed scan position the caller echoes as `?scan=` to resume where the search
/// stopped (#173 round 13, F2). It is AEAD-sealed at the mint site, never plaintext:
/// the row it names is by construction one the caller was denied (INV-13).
#[error("search incomplete: {message}")]
SearchIncomplete {
message: String,
continuation: Option<String>,
},
#[error("git error: {0}")]
Git(String),
@@ -50,6 +69,15 @@ pub enum AppError {
#[error("server overloaded: {0}")]
Overloaded(String),
#[error("repository is busy")]
RepoBusy,
#[error("repository is temporarily unavailable")]
RepoUnavailable,
#[error("repository write was fenced by a concurrent publish")]
RepoWriteFenced,
#[error("database error: {0}")]
Db(#[from] sqlx::Error),
@@ -63,6 +91,14 @@ pub enum AppError {
pub const DB_UNAVAILABLE_CODE: &str = "db_unavailable";
pub const DB_UNAVAILABLE_MESSAGE: &str = "database is temporarily unavailable";
/// Generic client-facing message for `AppError::Internal`. The real error is
/// logged server-side; never put sqlx/anyhow detail in the HTTP body (#226).
pub const INTERNAL_ERROR_MESSAGE: &str = "an internal error occurred";
/// Generic client-facing message for non-unavailable `AppError::Db`. Query /
/// schema errors stay in logs; the HTTP body must not leak them (#226).
pub const DB_ERROR_MESSAGE: &str = "a database error occurred";
/// Connection-level sqlx failures that mean the database is unreachable right
/// now (retryable, 503), as opposed to server-reported query errors.
fn db_unavailable(e: &sqlx::Error) -> bool {
@@ -83,7 +119,31 @@ impl From<anyhow::Error> for AppError {
fn from(err: anyhow::Error) -> Self {
match err.downcast::<sqlx::Error>() {
Ok(sql) => AppError::Db(sql),
Err(err) => AppError::Internal(err),
// Lock contention is transient and ordinary, so it must not land as a
// 500. The internal message names the owner slug and repo, so the
// variant carries nothing: the detail stays in the log at the raise
// site and the client gets a fixed retryable body.
Err(err) => match err.downcast::<crate::git::repo_store::LockPoolBusy>() {
Ok(_) => AppError::Overloaded("git write locks at capacity, retry shortly".into()),
Err(err) => match err.downcast::<crate::git::repo_store::RepoBusy>() {
Ok(_) => AppError::RepoBusy,
// Same reasoning one rung down: a refused under-lock refresh is a
// transient storage condition, and its internal message names the
// owner slug and repo, so the variant carries nothing.
Err(err) => match err.downcast::<crate::git::repo_store::RepoUnavailable>() {
Ok(_) => AppError::RepoUnavailable,
// And one more rung: a publish the store refused twice is
// transient in the same way, and the retry is the client's
// to make. The variant carries nothing for the same reason
// as the two above.
Err(err) => match err.downcast::<crate::git::repo_store::RepoWriteFenced>()
{
Ok(_) => AppError::RepoWriteFenced,
Err(err) => AppError::Internal(err),
},
},
},
},
}
}
}
@@ -135,16 +195,51 @@ impl IntoResponse for AppError {
// IcaptchaProofRequired is handled above (it carries extra headers/fields).
AppError::IcaptchaProofRequired { .. } => unreachable!("handled before this match"),
AppError::BadRequest(msg) => (StatusCode::BAD_REQUEST, "bad_request", msg.clone()),
AppError::UnresolvableDid(msg) => {
(StatusCode::BAD_REQUEST, "unresolvable_did", msg.clone())
}
AppError::TooManyRequests(msg) => {
(StatusCode::TOO_MANY_REQUESTS, "rate_limited", msg.clone())
}
AppError::Incomplete(msg) => {
(StatusCode::UNPROCESSABLE_ENTITY, "incomplete", msg.clone())
}
// A bounded search that could not complete (the CID resolver hit its
// legacy-probe or walk ceiling), distinct from the 404 that asserts a
// definitive not-found: absence was NOT proven, so the caller should
// retry rather than treat it as gone (#173, F2). 503, retryable.
AppError::SearchIncomplete { message, .. } => (
StatusCode::SERVICE_UNAVAILABLE,
"search_incomplete",
message.clone(),
),
AppError::Git(msg) => (StatusCode::INTERNAL_SERVER_ERROR, "git_error", msg.clone()),
// 504, distinct from the 500 git_error and from the read-gate's 404 /
// the auth 401, so the client can tell a deadline from a failure.
AppError::Timeout(msg) => (StatusCode::GATEWAY_TIMEOUT, "git_timeout", msg.clone()),
// 503 with a FIXED body: the caller should retry, and must not be told
// which repo is contended or for how long.
AppError::RepoBusy => (
StatusCode::SERVICE_UNAVAILABLE,
"repo_busy",
"repository is busy — retry".into(),
),
// 503 with a FIXED body for the same reason: the caller should retry, and
// must not be told which repo could not be refreshed or why.
AppError::RepoUnavailable => (
StatusCode::SERVICE_UNAVAILABLE,
"repo_unavailable",
"repository is temporarily unavailable, retry".into(),
),
// 503 with a FIXED body again, and its own code: the caller should
// retry, but the condition is not contention, so a client that
// distinguishes them should be able to. The body must not say which
// repo lost its publish or to whom.
AppError::RepoWriteFenced => (
StatusCode::SERVICE_UNAVAILABLE,
"repo_write_fenced",
"repository changed underneath this write, retry".into(),
),
AppError::Db(e) if db_unavailable(e) => (
StatusCode::SERVICE_UNAVAILABLE,
DB_UNAVAILABLE_CODE,
@@ -156,24 +251,61 @@ impl IntoResponse for AppError {
AppError::Overloaded(msg) => {
(StatusCode::SERVICE_UNAVAILABLE, "overloaded", msg.clone())
}
AppError::Db(e) => (StatusCode::INTERNAL_SERVER_ERROR, "db_error", e.to_string()),
AppError::Internal(e) => (
StatusCode::INTERNAL_SERVER_ERROR,
"internal_error",
e.to_string(),
),
// Opaque body + server log: bare `?` on sqlx paths becomes `AppError::Db`
// via `From`, so this arm (not `Internal`) is the common leak for open
// routes like GET /api/v1/repos and GET /api/v1/peers (#226).
AppError::Db(e) => {
tracing::error!(error = %e, "database error");
(
StatusCode::INTERNAL_SERVER_ERROR,
"db_error",
DB_ERROR_MESSAGE.into(),
)
}
// Opaque body: handlers that map with `.map_err(AppError::Internal)`
// (e.g. GET /ipfs/{cid}) land here; other DB failures usually hit `Db`.
// Log `{e:#}` so context-wrapped anyhow chains keep the leaf cause
// (Display alone is only the outermost layer; see api/repos.rs).
AppError::Internal(e) => {
tracing::error!(error = %format!("{e:#}"), "internal error");
(
StatusCode::INTERNAL_SERVER_ERROR,
"internal_error",
INTERNAL_ERROR_MESSAGE.into(),
)
}
};
let body = Json(json!({
let mut body = json!({
"error": code,
"message": message,
}));
});
// A truncated CID search may carry the sealed position the caller echoes as
// `?scan=` to resume. Rendered as a third body field, present only when the
// shed actually left something to resume: a wrapped scan and a throttled
// request both omit it, and its ABSENCE is what tells a caller the ladder is
// over. It is opaque ciphertext; see `gitlawb_core::scan_token`.
if let AppError::SearchIncomplete {
continuation: Some(token),
..
} = &self
{
body["continuation"] = json!(token);
}
let mut resp = (status, body).into_response();
// Overloaded advertises when to retry. It rides the shared tail above for
// its body/status, so the header is attached here rather than in a bespoke
// early return — keeping the variant handled in exactly one place.
if matches!(self, AppError::Overloaded(_)) {
let mut resp = (status, Json(body)).into_response();
// Both retryable 503s advertise when to retry: Overloaded (capacity shed) and
// SearchIncomplete (a bounded CID search cut short by a cap — retry may complete
// it). They ride the shared tail above for body/status, so the header is attached
// here rather than in bespoke early returns, keeping each variant handled once.
if matches!(
self,
AppError::Overloaded(_)
| AppError::SearchIncomplete { .. }
| AppError::RepoBusy
| AppError::RepoUnavailable
| AppError::RepoWriteFenced
) {
resp.headers_mut().insert(
axum::http::header::RETRY_AFTER,
axum::http::HeaderValue::from_static("1"),
@@ -211,4 +343,110 @@ mod tests {
"1"
);
}
/// #226: raw sqlx/DB detail must never appear in the Internal 500 body.
#[tokio::test]
async fn internal_error_body_is_opaque() {
use serde_json::{json, Value};
let leak = "error returned from database: relation \"repos\" does not exist";
let resp = AppError::Internal(anyhow::anyhow!("{leak}")).into_response();
assert_eq!(resp.status(), StatusCode::INTERNAL_SERVER_ERROR);
let bytes = axum::body::to_bytes(resp.into_body(), usize::MAX)
.await
.expect("read body");
let v: Value = serde_json::from_slice(&bytes).expect("json body");
// Exact object: a new `detail` field with different sensitive text must
// also fail, not only a repeat of the original error string.
assert_eq!(
v,
json!({
"error": "internal_error",
"message": INTERNAL_ERROR_MESSAGE,
})
);
}
/// #226: `AppError::Db` query errors (the common `?` path) must also be opaque.
#[tokio::test]
async fn db_error_body_is_opaque() {
use serde_json::{json, Value};
let resp = AppError::Db(sqlx::Error::Protocol(
"error returned from database: column \"is_public\" does not exist".into(),
))
.into_response();
assert_eq!(resp.status(), StatusCode::INTERNAL_SERVER_ERROR);
let bytes = axum::body::to_bytes(resp.into_body(), usize::MAX)
.await
.expect("read body");
let v: Value = serde_json::from_slice(&bytes).expect("json body");
assert_eq!(
v,
json!({
"error": "db_error",
"message": DB_ERROR_MESSAGE,
})
);
}
/// Connection-level failures must stay 503 `db_unavailable`, not collapse
/// into the opaque 500 `db_error` arm if `db_unavailable` loses a variant.
#[tokio::test]
async fn db_pool_timeout_stays_503_unavailable() {
use serde_json::{json, Value};
let resp = AppError::Db(sqlx::Error::PoolTimedOut).into_response();
assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE);
let bytes = axum::body::to_bytes(resp.into_body(), usize::MAX)
.await
.expect("read body");
let v: Value = serde_json::from_slice(&bytes).expect("json body");
assert_eq!(
v,
json!({
"error": DB_UNAVAILABLE_CODE,
"message": DB_UNAVAILABLE_MESSAGE,
})
);
}
/// #251: bare `?` on anyhow-wrapped sqlx relies on this downcast so a
/// closed pool becomes 503 `db_unavailable`, not 500 `internal_error`.
#[test]
fn pool_closed_via_anyhow_from_is_503_db_unavailable() {
let err: AppError = anyhow::Error::from(sqlx::Error::PoolClosed).into();
let resp = err.into_response();
assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE);
}
#[test]
fn lock_pool_busy_via_anyhow_from_is_503_overloaded() {
let err: AppError = anyhow::Error::new(crate::git::repo_store::LockPoolBusy).into();
let resp = err.into_response();
assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE);
assert_eq!(
resp.headers().get("retry-after").unwrap().to_str().unwrap(),
"1"
);
}
#[test]
fn repo_busy_unavailable_and_fenced_advertise_retry_after() {
for err in [
AppError::RepoBusy,
AppError::RepoUnavailable,
AppError::RepoWriteFenced,
] {
let resp = err.into_response();
assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE);
assert_eq!(
resp.headers().get("retry-after").unwrap().to_str().unwrap(),
"1"
);
}
}
}
+42
View File
@@ -61,6 +61,27 @@ pub fn create_issue(repo_path: &Path, issue_id: &str, json: &str) -> Result<()>
Ok(())
}
/// Remove a single issue ref while the write guard still holds the advisory lock,
/// after a definite publish refusal. Best-effort: a failed delete leaves a
/// local-only orphan, which is still better than telling the client to retry into a
/// duplicate id. Not used on `UploadUnknowable`, where the PUT may still land.
pub fn delete_issue_ref(
git_bin: &str,
repo_path: &Path,
issue_id: &str,
deadline: std::time::Instant,
) -> Result<()> {
let ref_name = format!("refs/gitlawb/issues/{issue_id}");
crate::git::visibility_pack::run_bounded_git(
git_bin,
&["update-ref", "-d", &ref_name],
repo_path,
b"",
deadline,
)?;
Ok(())
}
/// List all issue refs and return their JSON content.
pub fn list_issues(repo_path: &Path) -> Result<Vec<String>> {
// List all refs under refs/gitlawb/issues/
@@ -278,6 +299,27 @@ mod tests {
assert!(result.unwrap_err().to_string().contains("ambiguous"));
}
#[test]
fn delete_issue_ref_removes_a_created_ref() {
let dir = TempDir::new().unwrap();
init_repo(&dir);
let full_id = "eee88888-0000-0000-0000-000000000000";
create_issue(
dir.path(),
full_id,
r#"{"id":"eee88888-0000-0000-0000-000000000000","status":"open"}"#,
)
.unwrap();
delete_issue_ref(
"git",
dir.path(),
full_id,
std::time::Instant::now() + std::time::Duration::from_secs(30),
)
.unwrap();
assert_eq!(resolve_issue_id(dir.path(), full_id).unwrap(), None);
}
#[test]
fn test_close_issue_via_prefix() {
let dir = TempDir::new().unwrap();
File diff suppressed because it is too large. Load diff
+166 -10
View File
@@ -14,13 +14,13 @@ use tokio::process::Command;
/// the work they admitted, so admission is released only when that work is truly
/// done — not the instant the handler future drops on a client disconnect.
///
/// A move-only wrapper: no methods beyond construction and `Drop`. The handler
/// MOVEs its permits in and keeps no copy (a retained copy would drop early and
/// release admission the moment the future is dropped, defeating the guard). It is
/// threaded into `drive_git_child`, whose [`KillGroupOnDrop`] moves it into the
/// detached reaper on disconnect, so both permits drop only after the process group
/// is confirmed reaped (`kill(-pgid,0)==ESRCH`) rather than while the group is still
/// alive holding PIDs past the concurrency cap (#174 P1-a, plain-spawn residual).
/// A drop-only wrapper: nothing here inspects what it holds. The handler MOVEs its
/// permits in and keeps no copy (a retained copy would drop early and release admission
/// the moment the future is dropped, defeating the guard). It is threaded into
/// `drive_git_child`, whose [`KillGroupOnDrop`] moves it into the detached reaper on
/// disconnect, so both permits drop only after the process group is confirmed reaped
/// (`kill(-pgid,0)==ESRCH`) rather than while the group is still alive holding PIDs past
/// the concurrency cap (#174 P1-a, plain-spawn residual).
///
/// The `be0cdd6` path-scoped upload-pack walk already applies this discipline by
/// moving its permits into the `spawn_blocking`; this generalizes it to the plain
@@ -33,6 +33,8 @@ pub struct AdmissionGuard {
// 'static` so the guard can move into the detached reaper task.
_global: Option<Box<dyn Send + 'static>>,
_caller: Option<Box<dyn Send + 'static>>,
// Any further work-scoped hold that must outlive the process group; see `with_hold`.
_hold: Option<Box<dyn Send + 'static>>,
// Per-repo write lease (#174 U2/F3), `Some` ONLY on the receive-pack write path
// (via [`with_lease`](Self::with_lease)); `None` on every read path and every
// non-receive-pack write path. It rides this guard into `KillGroupOnDrop`'s detached
@@ -54,10 +56,26 @@ impl AdmissionGuard {
Self {
_global: Some(Box::new(global)),
_caller: caller.map(|c| Box::new(c) as Box<dyn Send + 'static>),
_hold: None,
_lease: None,
}
}
/// Attach a further hold that must not be released until the process group is
/// reaped, and ride it through the same seam as the permits.
///
/// The push handler uses this for the repo WRITE LOCK (#173 F2). Its
/// `guard.release(..)` line is only reached if `receive_pack` returns, so on a client
/// disconnect the lock used to be freed by the dropped future while the detached
/// reaper was still giving the group its SIGTERM grace, admitting a second
/// `receive-pack` on the same repo. Carrying the lock here holds it until the group
/// is ESRCH-confirmed gone, which is the same invariant the timeout path already
/// keeps ("a caller releasing a write lock can't race them", `reap_group_on_timeout`).
pub fn with_hold(mut self, hold: impl Send + 'static) -> Self {
self._hold = Some(Box::new(hold));
self
}
/// Attach the per-repo write lease (#174 U2/F3). Called ONLY on the receive-pack
/// write path, so the lease rides the disconnect reaper; read paths never call this.
pub fn with_lease(mut self, lease: crate::state::RepoWriteLease) -> Self {
@@ -92,7 +110,7 @@ pub async fn info_refs(
.arg("--stateless-rpc")
.arg("--advertise-refs")
.arg(repo_path);
// No request body — advertise-refs does not read stdin.
// No request body: advertise-refs does not read stdin.
let (stdout, admission) =
drive_git_child(command, Bytes::new(), timeout, "advertise-refs", admission).await?;
// Single-stage op: the advertisement's group is reaped by now; release admission
@@ -659,14 +677,15 @@ pub async fn build_filtered_pack(
command
.args(["pack-objects", "--stdout"])
.current_dir(repo_path);
drive_git_child(
let (out, admission) = drive_git_child(
command,
Bytes::from(data),
deadline.saturating_duration_since(Instant::now()),
"pack-objects",
admission,
)
.await
.await?;
Ok((out, admission))
}
/// Serve a clone/fetch with the withheld blobs removed from the response pack.
@@ -1880,6 +1899,143 @@ mod tests {
);
}
// #174 U1 (R2, KTD3, RED-before/GREEN-after): the path-scoped filtered-pack serve
// must hold read + per-caller admission until its pack-objects process group is
// reaped on a client disconnect, exactly as the plain upload_pack path does. Before
// the fix build_filtered_pack took no AdmissionGuard and the handler's `_hold`
// permits dropped the instant the request future was dropped, so disconnect-spam on
// a path-scoped repo could hold PIDs past the concurrency cap while the permits were
// already free (#174 P1-a, on the filtered path the plain path had already closed).
//
// A real AdmissionGuard built from two owned semaphore permits rides
// rev-list -> pack-objects. We drive the future until pack-objects has forked its
// grandchild (the streaming pack-writer stand-in), assert the permits are still held
// mid-serve, then DROP the future (client disconnect) and assert the permits are
// released only AFTER the group is ESRCH-confirmed gone — never while it is alive.
// Goes RED if the guard is not threaded into the pack-objects stage (it would then
// drop after rev-list, freeing the permits mid-serve or on the bare future drop).
#[cfg(unix)]
#[tokio::test]
async fn filtered_pack_holds_admission_until_group_reaped_on_disconnect() {
use std::sync::Arc;
use tokio::sync::Semaphore;
let tmp = tempfile::TempDir::new().unwrap();
let pidfile = tmp.path().join("pids");
// rev-list returns one oid fast; pack-objects forks a grandchild (the streaming
// writer stand-in), records leader+grandchild pids, then hangs so the future
// parks mid-serve with the guard owned by the pack-objects KillGroupOnDrop. The
// grandchild inherits (holds open) the stdout pipe, so drive_git_child's
// read_to_end blocks and the future stays pending until we drop it.
let body = format!(
"#!/bin/sh\ncase \"$1\" in\n rev-list) echo deadbeefdeadbeefdeadbeefdeadbeefdeadbeef ;;\n pack-objects) sleep 300 &\nprintf '%s\\n%s\\n' \"$$\" \"$!\" > \"{}\"\nwait ;;\n *) exit 1 ;;\nesac\n",
pidfile.display()
);
let git_bin = write_fake_git(tmp.path(), &body);
let withheld = HashSet::new();
// Retry the fake-git spawn race like the sibling disconnect test; each attempt
// gets a FRESH semaphore so a dropped losing attempt can't skew the winning
// attempt's permit accounting. Keep the winning attempt's future PENDING so the
// drop below exercises the client-disconnect teardown.
let (fut, sem, leader, grandchild) = {
let mut attempt = 0u64;
loop {
attempt += 1;
let _ = std::fs::remove_file(&pidfile);
// Semaphore(4): two owned permits model the handler's global-read +
// per-caller admission, leaving 2 available while the op is in flight.
let sem = Arc::new(Semaphore::new(4));
let g = sem.clone().try_acquire_owned().unwrap();
let c = sem.clone().try_acquire_owned().unwrap();
let admission = AdmissionGuard::new(g, Some(c));
let mut fut = Box::pin(build_filtered_pack(
git_bin.to_str().unwrap(),
tmp.path(),
&withheld,
Duration::from_secs(60),
Some(admission),
));
// Advance the future a slice at a time until the fake records its pids
// (i.e. pack-objects is running). `Ok(_)` means the future returned
// before the pidfile appeared (spawn error / early exit); stop polling
// then, since re-polling a completed future panics.
let mut pids = None;
for _ in 0..500 {
let finished = tokio::time::timeout(Duration::from_millis(10), &mut fut)
.await
.is_ok();
if let Some(p) = read_two_pids(&pidfile) {
pids = Some(p);
break;
}
if finished {
break;
}
}
match pids {
Some((l, gch)) => break (fut, sem, l, gch),
None => {
// Transient spawn miss: drop the still-armed future so its guard
// reaps anything that spawned (and returns the permits), then
// back off before retrying.
drop(fut);
assert!(
attempt < FAKE_GIT_RETRY_ATTEMPTS,
"fake git failed to reach the pack-objects stage after \
{FAKE_GIT_RETRY_ATTEMPTS} attempts (persistent failure, \
not a transient parallel-runner miss)"
);
tokio::time::sleep(Duration::from_millis(
FAKE_GIT_BACKOFF_STEP_MS * attempt,
))
.await;
}
}
}
};
let _cleanup = ReapOnPanic(vec![leader, grandchild]);
assert!(alive(grandchild), "grandchild must be running mid-serve");
// Mid-serve: the pack-objects stage owns the guard, so the two permits are still
// held. A build that dropped the guard after rev-list (unthreaded pack-objects
// stage) would have freed them here.
assert_eq!(
sem.available_permits(),
2,
"admission permits must be held while the filtered serve is in flight"
);
// Client disconnect: drop the request future. The pack-objects KillGroupOnDrop
// must tear the group down AND hold the permits until the group is
// ESRCH-confirmed gone, releasing them only then.
drop(fut);
let mut released_while_alive = false;
let mut released_after_reap = false;
for _ in 0..500 {
let released = sem.available_permits() == 4;
let group_alive = alive(grandchild);
if released && group_alive {
released_while_alive = true;
}
if released && !group_alive {
released_after_reap = true;
break;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
assert!(
!released_while_alive,
"admission permits were released while the git group was still alive — the \
path-scoped concurrency-cap bypass (#174 P1-a) is open"
);
assert!(
released_after_reap,
"admission permits must be released once the group is reaped on disconnect"
);
}
// ── F1: filtered-serve admission threaded through both pack stages ──────
//
// The handler-level disconnect regression lives in api/repos.rs
+275 -1
View File
@@ -431,7 +431,9 @@ fn batch_check_probe(
/// never on any English `fatal:` wording (KTD-4): a genuinely-absent object is the only
/// `Ok(None)` (404) path. A probe that could not honestly examine the store is a
/// [`ProbeError`], split by object-store readability into `Transient` (retryable 503)
/// and `Deterministic` (terminal 500) so the serve path can shed the right status.
/// and `Deterministic` (terminal 500) so the serve path can shed the right status. The
/// deadline itself arrives as a `Transient` fault, so the handler marks the search
/// truncated rather than reporting a false not-found (#173 round-10 R1/KTD2).
pub fn object_type_bounded(
git_bin: &str,
repo_path: &Path,
@@ -599,6 +601,63 @@ pub(crate) fn object_store_readable_store_wide(repo_path: &Path) -> bool {
object_store_readable(repo_path, "")
}
/// Bounded `git cat-file -s` size read for the `GET /ipfs/{cid}` serve path (#173
/// round-10, R1/KTD2): reads the object size WITHOUT its content (so an oversized object
/// is rejected before it is buffered, #173 F6), under
/// [`run_bounded_git`](crate::git::visibility_pack::run_bounded_git) so a wedged size
/// read is reaped at `deadline` instead of pinning the held /ipfs walk admission.
/// `Ok(n)` on success, and otherwise a [`ProbeError`] in the same vocabulary the type and
/// content stages use: a reaped child is `Transient` (retryable), and every other failure
/// goes through [`classify_store_fault`], so it is `Transient` when the object store is
/// not readable and `Deterministic` when it is.
///
/// There is deliberately NO absence value (#173 round 12). This returned
/// `Ok(None)` for every non-timeout failure, which `gate_and_serve` reads as a verified
/// absence and does not taint the search for, so a corrupt object, an unreadable pack, or
/// a failed spawn handed an authorized caller a definitive 404 instead of the retryable
/// 503 tail. Absence is also not this stage's question: the caller has already had a
/// `Present` verdict from [`object_type_bounded`], so an object that cannot be sized here
/// is a fault, not a not-found, and the one honest exception (a concurrent gc between the
/// two stages) is classified by store readability like any other. Making the absence
/// unrepresentable is what keeps the next caller from reintroducing the swallow.
///
/// Takes the shared `deadline` rather than its own timeout so a caller that pairs this
/// size check with a later read spends ONE budget across the pair, not one per stage.
pub fn object_size_bounded(
git_bin: &str,
repo_path: &Path,
sha256_hex: &str,
deadline: std::time::Instant,
) -> std::result::Result<u64, ProbeError> {
match crate::git::visibility_pack::run_bounded_git(
git_bin,
&["cat-file", "-s", sha256_hex],
repo_path,
b"",
deadline,
) {
Ok(out) => {
let text = String::from_utf8_lossy(&out);
text.trim().parse::<u64>().map_err(|e| {
// git exited 0 but did not print a size: the store answered something
// this code cannot read, which is a fault and never an absence.
classify_store_fault(
repo_path,
sha256_hex,
anyhow::anyhow!("unparseable `cat-file -s` output {:?}: {e}", text.trim()),
)
})
}
// The watchdog reaped the child at `deadline`; retryable whatever the store looks
// like, so it is routed before readability gets a say (same rule as the content
// stage in `read_object_bounded`).
Err(e) if e.is::<crate::git::smart_http::GitServiceTimeout>() => {
Err(ProbeError::Transient(e))
}
Err(e) => Err(classify_store_fault(repo_path, sha256_hex, e)),
}
}
/// Bounded, reaped variant of [`read_object_content`] for the async `/ipfs` serve
/// path (#174 F3). Same teardown guarantees as [`object_type_bounded`].
pub fn read_object_content_bounded(
@@ -811,6 +870,7 @@ pub fn merge_branch(
}
/// Resolve a repo disk path: {repos_dir}/{owner_slug}/{repo_name}.git
#[allow(dead_code)] // exercised from test_support and state tests; production uses validated_repo_disk_path
pub fn repo_disk_path(repos_dir: &Path, owner_did: &str, repo_name: &str) -> PathBuf {
// Sanitize the DID for use as a directory name
let owner_slug = owner_did.replace([':', '/'], "_");
@@ -866,6 +926,92 @@ mod tests {
);
}
/// #173 round-10 (KTD2): `object_type_bounded` reaps a wedged `cat-file` child at its
/// deadline instead of blocking on it to natural exit, so a hung probe cannot pin the
/// /ipfs walk admission the owning task holds. A fake `git` records its pid and sleeps
/// far past the 1s deadline; the `run_bounded_git` watchdog (SIGTERM -> grace ->
/// SIGKILL of the process group) must kill it well before that natural exit, and the
/// call must surface `GitServiceTimeout`. REVERT PROOF (RED): swap the twin's
/// `run_bounded_git` for the bare `Command::output()` and the wedged child stays alive
/// past the deadline — the mid-flight liveness poll below reads it still running.
#[cfg(unix)]
#[test]
fn object_type_bounded_reaps_wedged_child_at_deadline() {
use std::time::Duration;
let tmp = tempfile::TempDir::new().unwrap();
let pidfile = tmp.path().join("catfile.pid");
// `cat-file` records its own pid then sleeps 8s (>> the 1s deadline) so the probe
// is genuinely wedged; the watchdog is what must end it.
let body = format!(
"#!/bin/sh\n\
case \"$1\" in\n\
cat-file) echo $$ > \"{}\"; sleep 8 ;;\n\
*) : ;;\n\
esac\n\
exit 0\n",
pidfile.display()
);
let git_path = tmp.path().join("fakegit");
std::fs::write(&git_path, &body).unwrap();
{
use std::os::unix::fs::PermissionsExt;
let mut perm = std::fs::metadata(&git_path).unwrap().permissions();
perm.set_mode(0o755);
std::fs::set_permissions(&git_path, perm).unwrap();
}
let repo = tmp.path().to_path_buf();
let git = git_path.to_str().unwrap().to_string();
let alive = |pid: i32| unsafe { libc::kill(pid, 0) == 0 };
// The bounded probe blocks until the watchdog tears the child down, so run it on
// a worker thread and poll for the reap from here.
let handle = std::thread::spawn(move || {
let deadline = std::time::Instant::now() + Duration::from_secs(1);
super::object_type_bounded(&git, &repo, "deadbeef", deadline)
});
let mut pid = None;
for _ in 0..500 {
if let Some(p) = std::fs::read_to_string(&pidfile)
.ok()
.and_then(|s| s.trim().parse::<i32>().ok())
{
pid = Some(p);
break;
}
std::thread::sleep(Duration::from_millis(10));
}
let pid = pid.expect("the fake cat-file must have spawned and recorded its pid");
// Past the 1s deadline + SIGTERM grace but well before the 8s natural exit: the
// watchdog must already have reaped the wedged group. A bare, unbounded read would
// leave it running here — the load-bearing RED.
std::thread::sleep(Duration::from_secs(3));
let reaped = !alive(pid);
// Defensive reap so a RED run leaks no orphan.
unsafe {
libc::kill(pid, libc::SIGKILL);
}
assert!(
reaped,
"object_type_bounded must reap the wedged cat-file child at the deadline, \
not leave it running to its natural exit"
);
let res = handle.join().expect("probe thread joins");
let err = res.expect_err("a deadline overrun must be an error, not a value");
// A reaped deadline is a retryable fault, not a verdict about the object, so it
// must arrive as Transient (-> 503) carrying GitServiceTimeout.
let super::ProbeError::Transient(inner) = &err else {
panic!("a deadline overrun must be a Transient probe fault, got: {err:?}");
};
assert!(
inner.is::<crate::git::smart_http::GitServiceTimeout>(),
"a deadline overrun must surface GitServiceTimeout, got: {inner:?}"
);
}
/// #174 F5 (RED-before/GREEN-after): a packed object whose pack/idx is unreadable
/// makes `git cat-file -t` emit "could not get object info" — byte-identical to a
/// genuine miss. `object_type_bounded` must report absence ONLY when the object
@@ -1371,6 +1517,134 @@ mod tests {
);
}
/// #173 round 12 (jatmn): the SIZE probe must use the same absence-versus-fault
/// vocabulary the type probe does. It mapped every non-timeout failure to `Ok(None)`,
/// which `gate_and_serve` reads as a verified absence and does not taint, so a
/// corrupt object, an unreadable pack, or a failed spawn ended the search cleanly and
/// handed an authorized caller a definitive 404 instead of the retryable 503 tail.
///
/// A corrupt loose object is the same fixture the type stage uses for this, and it is
/// the honest case: the object EXISTS, the type probe says so, and only the size read
/// fails. RED before the change (`Ok(None)`).
#[cfg(unix)]
#[test]
fn object_size_bounded_corrupt_loose_object_is_fault_not_absence() {
use std::os::unix::fs::PermissionsExt;
let td = tempfile::TempDir::new().unwrap();
let work = td.path().join("sizecorrupt");
std::fs::create_dir_all(&work).unwrap();
let g = |args: &[&str]| {
assert!(
Command::new("git")
.args(args)
.current_dir(&work)
.status()
.unwrap()
.success(),
"git {args:?}"
);
};
g(&["init", "-q", "--object-format=sha256", "."]);
g(&["config", "user.email", "t@t"]);
g(&["config", "user.name", "t"]);
std::fs::write(work.join("f.txt"), b"loose object content\n").unwrap();
g(&["add", "f.txt"]);
g(&["commit", "-qm", "c1"]);
let blob = String::from_utf8(
Command::new("git")
.args(["rev-parse", "HEAD:f.txt"])
.current_dir(&work)
.output()
.unwrap()
.stdout,
)
.unwrap()
.trim()
.to_string();
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
// Healthy first: the probe reads the real size, so the fault case below is not
// passing for the trivial reason that this fixture never worked.
let healthy = super::object_size_bounded("git", &work, &blob, deadline);
assert!(
matches!(healthy, Ok(n) if n > 0),
"a readable object reports its size; got {healthy:?}"
);
let obj = work.join(".git/objects").join(&blob[0..2]).join(&blob[2..]);
let mut perms = std::fs::metadata(&obj).unwrap().permissions();
perms.set_mode(0o644);
std::fs::set_permissions(&obj, perms).unwrap();
std::fs::write(&obj, b"garbage not a zlib stream").unwrap();
let res = super::object_size_bounded("git", &work, &blob, deadline);
assert!(
res.is_err(),
"a corrupt object must surface as a probe fault, never as an absence the \
resolver renders as a clean 404; got {res:?}"
);
}
/// #173 round 12 (jatmn), the transient arm: a store this process cannot read is the
/// retryable case, so the size probe classifies it `Transient` exactly as the type
/// probe does. Distinguishing the two arms is the whole point of routing through
/// `classify_store_fault` rather than returning one undifferentiated error.
#[cfg(unix)]
#[test]
fn object_size_bounded_unreadable_store_is_transient() {
use std::os::unix::fs::PermissionsExt;
let td = tempfile::TempDir::new().unwrap();
let work = td.path().join("sizeunreadable");
std::fs::create_dir_all(&work).unwrap();
let g = |args: &[&str]| {
assert!(
Command::new("git")
.args(args)
.current_dir(&work)
.status()
.unwrap()
.success(),
"git {args:?}"
);
};
g(&["init", "-q", "--object-format=sha256", "."]);
g(&["config", "user.email", "t@t"]);
g(&["config", "user.name", "t"]);
std::fs::write(work.join("f.txt"), b"loose object content\n").unwrap();
g(&["add", "f.txt"]);
g(&["commit", "-qm", "c1"]);
let blob = String::from_utf8(
Command::new("git")
.args(["rev-parse", "HEAD:f.txt"])
.current_dir(&work)
.output()
.unwrap()
.stdout,
)
.unwrap()
.trim()
.to_string();
// Make this oid's loose fan-out unreadable: git fails, and the store cannot
// certify absence, so the fault is retryable rather than terminal.
let fanout = work.join(".git/objects").join(&blob[0..2]);
let mut perms = std::fs::metadata(&fanout).unwrap().permissions();
perms.set_mode(0o000);
std::fs::set_permissions(&fanout, perms).unwrap();
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
let res = super::object_size_bounded("git", &work, &blob, deadline);
let mut restore = std::fs::metadata(&fanout).unwrap().permissions();
restore.set_mode(0o755);
std::fs::set_permissions(&fanout, restore).unwrap();
assert!(
matches!(res, Err(super::ProbeError::Transient(_))),
"an unreadable object store is the retryable arm; got {res:?}"
);
}
/// #174 F5/U4: a corrupt LOOSE object makes `git cat-file --batch-check` print
/// `<oid> missing` on stdout (exit 0) yet emit `error:` diagnostics on stderr. The
/// clean-`missing` absence path must NOT fire here — the `error:` line disqualifies
+460 -44
View File
@@ -5,12 +5,41 @@
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::atomic::AtomicBool;
use std::sync::{Arc, Mutex, OnceLock};
use anyhow::{Context, Result};
use aws_sdk_s3::Client as S3Client;
use tracing::{debug, info};
/// The precondition an upload is fenced on.
///
/// Object storage is the only place a fence can hold. Dropping the future of an
/// in-flight PUT does not cancel the request the server is already processing,
/// so no amount of local locking stops an abandoned writer's bytes from landing
/// after a successor has published. A conditional PUT the store itself refuses
/// is what actually stops it.
#[derive(Clone, Debug)]
pub enum UploadPrecondition {
/// Publish only if the stored object is still the generation we observed.
IfMatch(String),
/// Publish only if nothing is stored under the key yet.
IfAbsent,
/// No fence. Last writer wins.
Unconditional,
}
/// Why an upload failed, split so a caller can tell "someone else already
/// published this key" (expected, and dropping our bytes is the correct
/// outcome) from a real storage failure.
#[derive(Debug, thiserror::Error)]
pub enum UploadError {
#[error("upload precondition lost (HTTP {status})")]
PreconditionLost { status: u16 },
#[error(transparent)]
Other(#[from] anyhow::Error),
}
/// Wrapper around the S3 client with the configured bucket.
#[derive(Clone)]
pub struct TigrisClient {
@@ -31,21 +60,29 @@ impl TigrisClient {
})
}
/// Test-only constructor with an explicit S3 endpoint, region, and static
/// credentials — no env-var reads, so parallel tests cannot race each other's
/// `AWS_*` environment the way the env-based `new` would. Lets a test point
/// the client at a non-routable endpoint to exercise acquire-stall paths.
/// Build a client pointed at an arbitrary endpoint, for tests.
///
/// The production constructor reads the endpoint and credentials from the
/// environment, which a test cannot steer without mutating process-global
/// state. This takes both explicitly so a test can aim the client at a
/// closed port and get a prompt transport error out of `exists()`.
///
/// `RetryConfig::disabled()` is load-bearing, not tidiness: the SDK's default
/// policy retries a connection refusal with backoff, which turns each failing
/// call into seconds of waiting.
#[cfg(test)]
pub(crate) async fn for_testing_with_endpoint(bucket: &str, endpoint_url: &str) -> Self {
let creds = aws_sdk_s3::config::Credentials::new("test", "test", None, None, "test");
let config = aws_config::defaults(aws_config::BehaviorVersion::latest())
.endpoint_url(endpoint_url)
.region(aws_config::Region::new("auto"))
.credentials_provider(creds)
.load()
.await;
pub fn for_testing_with_endpoint(bucket: &str, endpoint: &str) -> Self {
use aws_sdk_s3::config::{retry::RetryConfig, Credentials, Region};
let config = aws_sdk_s3::config::Config::builder()
.endpoint_url(endpoint)
.credentials_provider(Credentials::new("test", "test", None, None, "test"))
.region(Region::new("auto"))
.retry_config(RetryConfig::disabled())
.behavior_version_latest()
.build();
Self {
s3: S3Client::new(&config),
s3: S3Client::from_conf(config),
bucket: bucket.to_string(),
}
}
@@ -77,8 +114,47 @@ impl TigrisClient {
}
}
/// Upload a local bare repo directory to Tigris as a tar.zst archive.
pub async fn upload(&self, owner_slug: &str, repo_name: &str, local_path: &Path) -> Result<()> {
/// Read the ETag of a repo archive, or `None` when nothing is stored under
/// the key. The ETag identifies the generation a later conditional upload
/// can fence itself on.
///
/// Separate from `exists` rather than folded into it: `exists` has callers
/// that only want the boolean, and widening its return type would churn
/// every one of them for no benefit.
pub async fn head_etag(&self, owner_slug: &str, repo_name: &str) -> Result<Option<String>> {
let key = Self::repo_key(owner_slug, repo_name);
match self
.s3
.head_object()
.bucket(&self.bucket)
.key(&key)
.send()
.await
{
Ok(out) => Ok(Some(
out.e_tag()
.context(format!("tigris HEAD {key}: hit carried no ETag"))?
.to_string(),
)),
Err(e) => {
if e.as_service_error().is_some_and(|e| e.is_not_found()) {
Ok(None)
} else {
Err(anyhow::anyhow!("tigris HEAD {key}: {e}"))
}
}
}
}
/// Upload a local bare repo directory to Tigris as a tar.zst archive,
/// fenced by `precondition`.
pub async fn upload(
&self,
owner_slug: &str,
repo_name: &str,
local_path: &Path,
precondition: UploadPrecondition,
) -> std::result::Result<(), UploadError> {
let key = Self::repo_key(owner_slug, repo_name);
debug!(key = %key, path = %local_path.display(), "uploading repo to tigris");
@@ -93,15 +169,57 @@ impl TigrisClient {
let body = aws_sdk_s3::primitives::ByteStream::from(archive_bytes);
self.s3
let mut req = self
.s3
.put_object()
.bucket(&self.bucket)
.key(&key)
.body(body)
.content_type("application/zstd")
.send()
.await
.context(format!("tigris PUT {key}"))?;
.content_type("application/zstd");
match &precondition {
UploadPrecondition::IfMatch(etag) => req = req.if_match(etag),
UploadPrecondition::IfAbsent => req = req.if_none_match("*"),
UploadPrecondition::Unconditional => {}
}
if let Err(e) = req.send().await {
// `PutObjectError` models no PreconditionFailed variant (its arms are
// EncryptionTypeMismatch, InvalidRequest, InvalidWriteOffset,
// TooManyParts, Unhandled), so a refused precondition arrives as
// `Unhandled` and matching the enum would classify it as a generic
// failure. The raw HTTP status off the response is the only place the
// answer actually lives.
//
// Read it via `raw_response()`, not a `ServiceError`-only match: the
// SDK exposes the raw response for BOTH `ServiceError` and
// `ResponseError`, and a refused conditional PUT whose error body the
// SDK cannot parse (malformed XML, premature close) surfaces as
// `ResponseError`. Matching only `ServiceError` would classify that
// unparsable 409/412 as a generic failure, and `RepoWriteGuard::release`
// would log-and-succeed instead of taking the supersede retry,
// acknowledging a write that was definitively not published.
let status = e.raw_response().map(|raw| raw.status().as_u16());
// 412 is always a lost precondition. 409 is one only when we asked
// for create-only, which is how S3-compatible stores report "the key
// already exists". Everything else, 404 included, is a real failure:
// archive keys are never deleted (`delete` has no callers), so a 404
// here means something permanent like a missing bucket or a
// misrouted endpoint, and reporting that as a lost precondition
// would tell a caller to expect a successor that does not exist.
let lost = match status {
Some(412) => true,
Some(409) => matches!(precondition, UploadPrecondition::IfAbsent),
_ => false,
};
if lost {
return Err(UploadError::PreconditionLost {
status: status.expect("a lost precondition came from a status"),
});
}
return Err(UploadError::Other(
anyhow::Error::new(e).context(format!("tigris PUT {key}")),
));
}
info!(key = %key, "uploaded repo to tigris");
Ok(())
@@ -112,10 +230,32 @@ impl TigrisClient {
&self,
owner_slug: &str,
repo_name: &str,
local_path: &Path,
local_path: &super::repo_store::ValidatedRepoDiskPath,
swap_authority: Option<Arc<AtomicBool>>,
) -> Result<()> {
self.download_to(owner_slug, repo_name, local_path, true, swap_authority)
.await
.map(|_| ())
}
/// Download a repo archive from Tigris and extract it, returning the
/// directory that was populated.
///
/// `publish` controls whether the extract is swapped into `target` in place
/// (the live-path mutation used by writes; returns `target`) or unpacked
/// into a fresh temp directory under `target`'s parent (a non-mutating
/// snapshot read; returns the temp dir, which the caller owns and cleans
/// up). The snapshot form never touches the live repo path.
pub async fn download_to(
&self,
owner_slug: &str,
repo_name: &str,
target: &super::repo_store::ValidatedRepoDiskPath,
publish: bool,
swap_authority: Option<Arc<AtomicBool>>,
) -> Result<DownloadExtract> {
let key = Self::repo_key(owner_slug, repo_name);
debug!(key = %key, path = %local_path.display(), "downloading repo from tigris");
debug!(key = %key, path = %target.as_path().display(), "downloading repo from tigris");
let resp = self
.s3
@@ -133,21 +273,62 @@ impl TigrisClient {
.context("reading tigris response body")?
.into_bytes();
// Extract tar.zst to local path
tokio::task::spawn_blocking({
let local_path = local_path.to_path_buf();
move || decompress_repo(&data, &local_path)
// The snapshot temp dir is decided HERE, in the async layer, before the
// extraction runs. Cleanup ownership moves into the blocking task so a
// cancelled async future cannot drop it while extraction is still running.
let snapshot_tmp = if publish {
None
} else {
let parent = target.parent().context("snapshot path has no parent")?;
std::fs::create_dir_all(parent).context("creating parent dir")?;
let file_name = target
.file_name()
.context("snapshot path has no file name")?
.to_string_lossy();
Some(parent.join(format!(
".{file_name}.tmp-snapshot.{}",
uuid::Uuid::new_v4()
)))
};
// Extract tar.zst to a directory.
let extracted = tokio::task::spawn_blocking({
let target = target.clone();
let snapshot_tmp = snapshot_tmp.clone();
move || -> Result<DownloadExtract> {
let result = (|| -> Result<DownloadExtract> {
if publish {
decompress_repo(&data, &target, swap_authority.as_ref())?;
return Ok(DownloadExtract::Published(()));
}
// Non-mutating snapshot: unpack into the temp dir decided above.
// The live repo path is never touched.
let tmp_dir = snapshot_tmp.expect("snapshot path was decided above");
std::fs::create_dir_all(&tmp_dir).context("creating temp extract dir")?;
let unpack = (|| -> Result<()> {
let decoder = zstd::stream::Decoder::new(&data[..])?;
let mut archive = tar::Archive::new(decoder);
archive.unpack(&tmp_dir).context("unpacking tar.zst")?;
Ok(())
})();
if let Err(e) = unpack {
let _ = std::fs::remove_dir_all(&tmp_dir);
return Err(e);
}
Ok(DownloadExtract::Snapshot(TempSnapshotDir { path: tmp_dir }))
})();
result
}
})
.await
.context("extract task panicked")?
.context("extracting repo")?;
info!(key = %key, path = %local_path.display(), "downloaded repo from tigris");
Ok(())
info!(key = %key, path = %target.as_path().display(), "downloaded repo from tigris");
Ok(extracted)
}
/// Delete a repo archive from Tigris.
#[allow(dead_code)]
pub async fn delete(&self, owner_slug: &str, repo_name: &str) -> Result<()> {
let key = Self::repo_key(owner_slug, repo_name);
self.s3
@@ -180,7 +361,7 @@ fn compress_repo(repo_path: &Path) -> Result<Vec<u8>> {
/// `decompress_repo`. Concurrent extractions unpack into isolated temp dirs in
/// parallel, but the final `remove_dir_all` + `rename` must not interleave for
/// the same `local_path`, or they race to a nondeterministic overwrite/failure.
fn publish_lock(local_path: &Path) -> Arc<Mutex<()>> {
pub(crate) fn publish_lock(local_path: &Path) -> Arc<Mutex<()>> {
// KNOWN LIMITATION: this map is never evicted — one (PathBuf, Arc<Mutex>)
// entry accrues per distinct repo path for the process lifetime. Bounded by
// the number of repos a node hosts, so it's negligible for normal use, but
@@ -193,6 +374,35 @@ fn publish_lock(local_path: &Path) -> Arc<Mutex<()>> {
.clone()
}
/// Owns a snapshot temp dir from extraction through handoff to [`RepoSnapshot`].
/// Dropped when the `spawn_blocking` join result is abandoned, so cancellation
/// before the outer future resumes still removes the directory.
pub(crate) struct TempSnapshotDir {
path: PathBuf,
}
impl TempSnapshotDir {
pub(crate) fn into_repo_snapshot(self) -> super::repo_store::RepoSnapshot {
let path = self.path.clone();
std::mem::forget(self);
super::repo_store::RepoSnapshot::from_owned_path(path)
}
}
impl Drop for TempSnapshotDir {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.path);
}
}
/// What [`TigrisClient::download_to`] produced.
pub enum DownloadExtract {
/// Published into the validated live repo path.
Published(()),
/// Unpacked into a throwaway temp dir that cleans up on drop until adopted.
Snapshot(TempSnapshotDir),
}
/// Decompress a tar.zst byte vector into a local directory.
///
/// Extraction is atomic with respect to `local_path`: the archive is unpacked
@@ -200,11 +410,16 @@ fn publish_lock(local_path: &Path) -> Arc<Mutex<()>> {
/// fully succeeds. A corrupt or truncated archive therefore can never clobber a
/// good existing copy at `local_path` — on failure we discard the temp dir and
/// leave `local_path` exactly as it was.
fn decompress_repo(data: &[u8], local_path: &Path) -> Result<()> {
let parent = local_path.parent().context("repo path has no parent")?;
fn decompress_repo(
data: &[u8],
local_path: &super::repo_store::ValidatedRepoDiskPath,
swap_authority: Option<&Arc<AtomicBool>>,
) -> Result<()> {
let live = local_path.as_path();
let parent = live.parent().context("repo path has no parent")?;
std::fs::create_dir_all(parent).context("creating parent dir")?;
let file_name = local_path
let file_name = live
.file_name()
.context("repo path has no file name")?
.to_string_lossy();
@@ -229,17 +444,218 @@ fn decompress_repo(data: &[u8], local_path: &Path) -> Result<()> {
return Err(e);
}
// Swap the freshly-extracted repo into place. rename within the same parent
// is effectively atomic, but most platforms refuse to rename onto a
// non-empty dir, so remove the old copy first. Serialize this per repo path:
// concurrent extractions unpack into isolated temp dirs, but their swaps
// must not interleave or they race to a nondeterministic overwrite/failure.
let lock = publish_lock(local_path);
let _publish = lock.lock().expect("publish lock poisoned");
if local_path.exists() {
std::fs::remove_dir_all(local_path).context("removing stale repo dir")?;
}
std::fs::rename(&tmp_dir, local_path).context("swapping extracted repo into place")?;
// Swap through the validated-path helper so CodeQL sees the barrier before the
// remove/rename sink (`rust/path-injection`).
super::repo_store::swap_extracted_into_validated_repo(local_path, &tmp_dir, swap_authority)?;
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use aws_sdk_s3::primitives::ByteStream;
use futures::FutureExt;
/// The envs the probe needs, all of them, or it does not run.
///
/// `AWS_ENDPOINT_URL_S3` is included on purpose: without it the SDK resolves
/// to real AWS S3, and a probe that passed there would say nothing about
/// Tigris.
fn probe_env() -> Option<String> {
if std::env::var("GITLAWB_TIGRIS_PROBE").ok().as_deref() != Some("1") {
return None;
}
for name in [
"AWS_ACCESS_KEY_ID",
"AWS_SECRET_ACCESS_KEY",
"AWS_ENDPOINT_URL_S3",
] {
if std::env::var(name).is_err() {
eprintln!("tigris conditional-write probe: {name} is unset, skipping");
return None;
}
}
match std::env::var("GITLAWB_TIGRIS_BUCKET") {
Ok(b) if !b.is_empty() => Some(b),
_ => {
eprintln!(
"tigris conditional-write probe: GITLAWB_TIGRIS_BUCKET is unset, skipping"
);
None
}
}
}
/// One conditional PUT, reported as the status that REFUSED it, or `None`
/// when the store accepted the write.
///
/// Accepted is the interesting answer here, not an error: it means the
/// endpoint ignored the header we fenced on.
async fn conditional_put(
s3: &S3Client,
bucket: &str,
key: &str,
body: &'static [u8],
if_match: Option<&str>,
if_none_match: Option<&str>,
) -> Result<Option<u16>, String> {
let mut req = s3
.put_object()
.bucket(bucket)
.key(key)
.body(ByteStream::from_static(body));
if let Some(v) = if_match {
req = req.if_match(v);
}
if let Some(v) = if_none_match {
req = req.if_none_match(v);
}
match req.send().await {
Ok(_) => Ok(None),
Err(e) => match e.raw_response() {
// Same raw-response rule as `upload`: a refused conditional PUT
// whose body the SDK cannot parse surfaces as `ResponseError`, and
// the status has to come off the raw response for both variants.
// Without it, an unparsable 409/412 would report "no HTTP
// response" here and skip the supersede retry.
Some(raw) => Ok(Some(raw.status().as_u16())),
None => Err(format!("conditional PUT {key}: no HTTP response: {e}")),
},
}
}
/// The probe body, written to RETURN its failures rather than panic on
/// them, so the caller's cleanup is reached on every arm.
async fn conditional_write_probe(s3: &S3Client, bucket: &str, key: &str) -> Result<(), String> {
// 1. A plain PUT under a throwaway key.
let seeded = s3
.put_object()
.bucket(bucket)
.key(key)
.body(ByteStream::from_static(b"probe-one"))
.send()
.await
.map_err(|e| format!("seeding PUT {key}: {e}"))?;
// 2. Its ETag, which is the generation the next arm fences against.
let etag = seeded
.e_tag()
.ok_or_else(|| format!("seeding PUT {key} returned no ETag"))?
.to_string();
// 3. A deliberately wrong If-Match. A store honoring it answers 412.
let wrong = format!("\"{}\"", "0".repeat(32));
if etag.trim_matches('"') == wrong.trim_matches('"') {
return Err(format!(
"the seeded ETag {etag} collides with the deliberately wrong one, \
so this arm would prove nothing"
));
}
match conditional_put(s3, bucket, key, b"probe-two", Some(&wrong), None).await? {
Some(412) => {}
Some(status) => {
return Err(format!(
"a stale If-Match must be refused with 412, the endpoint answered {status}"
))
}
None => {
return Err(
"a stale If-Match was ACCEPTED: this endpoint does not honor If-Match, so \
the release fence cannot hold here"
.to_string(),
)
}
}
// 4. If-None-Match `*` over the object that now exists. This arm matters
// MORE than the one above. An ignored If-Match eventually surfaces as
// odd behavior, because a stale writer overwrites and someone notices
// the lost tree. An ignored If-None-Match just returns 200, so a publish
// that should have been fenced lands with no error anywhere: the silent
// no-op the bucket-type caveat on this test describes.
match conditional_put(s3, bucket, key, b"probe-three", None, Some("*")).await? {
// Either status is a pass, and the asymmetry with the If-Match arm
// above mirrors `upload`'s classifier exactly: 412 is always a lost
// precondition, and 409 is one too when we asked for create-only.
// AWS documents 409 for a create-only conflict racing a delete, so a
// store answering it is enforcing the precondition and we already
// handle it. Pinning 412 alone here would fail the probe against a
// backend that is behaving correctly, which sends whoever runs it
// chasing a fault that is not there.
Some(412) | Some(409) => {}
Some(status) => {
return Err(format!(
"create-only over an existing object must be refused with 412 or 409, \
the endpoint answered {status}"
))
}
None => {
return Err(
"If-None-Match * was ACCEPTED over an existing object: this endpoint does \
not honor create-only, so a fenced publish lands silently"
.to_string(),
)
}
}
Ok(())
}
/// Probe the REAL Tigris endpoint for the conditional-write semantics the
/// release fence depends on.
///
/// UNTIL THIS IS RUN AGAINST REAL CREDENTIALS, the fence is verified against
/// vendor documentation and an in-process mock, not against the backend it
/// runs on. The mock implements the semantics we believe Tigris has; it
/// cannot tell us whether Tigris actually has them.
///
/// The bucket matters, not just the endpoint. Tigris documents conditional
/// operations as supported on Single-region and Multi-region buckets only.
/// Global and Dual-region buckets are eventually consistent, and a
/// conditional PUT evaluated against a stale replica would make the fence a
/// silent no-op rather than an error. So point `GITLAWB_TIGRIS_BUCKET` at a
/// throwaway bucket of the SAME type production uses.
///
/// Ignored by default and additionally gated on `GITLAWB_TIGRIS_PROBE=1`,
/// because it writes to a real bucket and costs real requests. Run with:
/// `GITLAWB_TIGRIS_PROBE=1 cargo test -p gitlawb-node --bin gitlawb-node
/// tigris_honors_conditional_writes -- --ignored --nocapture`
#[tokio::test]
#[ignore = "writes to a real Tigris bucket; needs GITLAWB_TIGRIS_PROBE=1 plus credentials"]
async fn tigris_honors_conditional_writes() {
let Some(bucket) = probe_env() else {
eprintln!(
"tigris conditional-write probe: skipped. Set GITLAWB_TIGRIS_PROBE=1, \
GITLAWB_TIGRIS_BUCKET, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and \
AWS_ENDPOINT_URL_S3 to run it."
);
return;
};
let config = aws_config::load_defaults(aws_config::BehaviorVersion::latest()).await;
let s3 = S3Client::new(&config);
// A fresh key per run, so a probe that somehow orphaned an object on an
// earlier run cannot change what this one observes.
let key = format!("probe/conditional-write-{}.bin", uuid::Uuid::new_v4());
// CLEANUP MUST RUN ON EVERY ARM, and a failing assertion is precisely
// the case this probe exists to catch, so the delete cannot sit after
// the checks. The body returns its failures rather than panicking, and
// `catch_unwind` covers the panic an SDK call could still raise; either
// way the delete below is reached before the verdict is re-raised.
let outcome = std::panic::AssertUnwindSafe(conditional_write_probe(&s3, &bucket, &key))
.catch_unwind()
.await;
if let Err(e) = s3.delete_object().bucket(&bucket).key(&key).send().await {
eprintln!("tigris conditional-write probe: cleanup of {key} failed: {e}");
}
match outcome {
Ok(Ok(())) => {}
Ok(Err(msg)) => panic!("tigris conditional-write probe: {msg}"),
Err(payload) => std::panic::resume_unwind(payload),
}
}
}
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+497 -61
View File
@@ -31,6 +31,7 @@ use axum::response::IntoResponse;
use axum::routing::get;
use axum::{Json, Router};
use clap::Parser;
use std::collections::HashSet;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
use tokio::net::TcpListener;
@@ -89,6 +90,7 @@ async fn main() -> Result<()> {
// Load or generate the node's identity keypair
let keypair = load_or_create_keypair(&config)?;
let scan_token_key = AppState::derive_scan_token_key(&keypair);
let node_did = keypair.did();
// One-time metrics init. Must run before any handler that calls into
@@ -286,8 +288,20 @@ async fn main() -> Result<()> {
None
};
let repo_store =
git::repo_store::RepoStore::new(config.repos_dir.clone(), tigris, db.pool().clone());
// Advisory-lock connections come from their own pool: a write guard pins one
// for its whole lifetime, and sharing the application pool would let a push
// burst starve ordinary request handlers.
let lock_pool = db::Db::lock_pool(
&config.database_url,
config.db_lock_pool_max_connections,
std::time::Duration::from_secs(config.db_acquire_timeout_secs),
)?;
let repo_store = git::repo_store::RepoStore::new(
config.repos_dir.clone(),
tigris,
lock_pool,
std::time::Duration::from_secs(config.lock_held_transfer_timeout_secs),
);
// Per-DID limiter for the creation endpoints. Keyed on the authenticated
// DID (attacker-varied), so bound its key set to cap memory.
@@ -332,6 +346,23 @@ async fn main() -> Result<()> {
tracing::warn!("GITLAWB_PUSH_RATE_LIMIT=0 — per-IP push rate limiting disabled");
}
// close_issue drives a full archive download for non-owner author pre-checks.
// Keep it on its own bucket so it cannot drain the receive-pack limit.
let close_issue_limit = std::env::var("GITLAWB_CLOSE_ISSUE_RATE_LIMIT")
.ok()
.and_then(|v| v.trim().parse::<usize>().ok())
.unwrap_or(120);
let close_issue_rate_limiter = rate_limit::RateLimiter::new_bounded(
close_issue_limit,
std::time::Duration::from_secs(3600),
200_000,
);
if close_issue_limit == 0 {
tracing::warn!(
"GITLAWB_CLOSE_ISSUE_RATE_LIMIT=0 — per-IP close_issue rate limiting disabled"
);
}
// Which forwarded header the edge is trusted to set. Default None (trust
// nothing, key on the socket peer). Fly nodes set GITLAWB_TRUSTED_PROXY=fly;
// a node behind Caddy/NGINX sets it to x-forwarded-for.
@@ -381,6 +412,14 @@ async fn main() -> Result<()> {
rate_limiter,
create_ip_rate_limiter,
push_rate_limiter,
close_issue_rate_limiter,
ipfs_max_history_walks: crate::api::ipfs::MAX_HISTORY_WALKS_PER_REQUEST,
ipfs_max_legacy_probes: AppState::ipfs_legacy_probe_budget(&config),
ipfs_legacy_scan_page_rows: crate::api::ipfs::LEGACY_SCAN_PAGE_ROWS,
ipfs_max_legacy_scan_rows: AppState::ipfs_legacy_scan_row_budget(&config),
ipfs_max_legacy_scan_rule_bytes: crate::api::ipfs::MAX_LEGACY_SCAN_RULE_BYTES_PER_REQUEST,
ipfs_scan_token_key: Arc::new(scan_token_key),
ipfs_max_served_object_bytes: crate::api::ipfs::MAX_SERVED_OBJECT_BYTES,
push_limiter_trust,
sync_trigger_rate_limiter,
peer_write_rate_limiter,
@@ -455,6 +494,11 @@ async fn main() -> Result<()> {
std::time::Duration::from_secs(3600),
200_000,
),
ipfs_work_rate_limiter: rate_limit::RateLimiter::new_bounded(
AppState::ipfs_work_budget(&config),
std::time::Duration::from_secs(3600),
200_000,
),
git_bin: "git".to_string(),
};
if config.ipfs_rate_limit == 0 {
@@ -496,7 +540,7 @@ async fn main() -> Result<()> {
loop {
tokio::select! {
_ = tokio::time::sleep(std::time::Duration::from_secs(300)) => {
sweep_rate_limiters(&sweep_state).await;
sweep_state.sweep_rate_limiters().await;
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs() as i64)
@@ -515,6 +559,8 @@ async fn main() -> Result<()> {
});
}
let _legacy_cid_sweep = spawn_legacy_cid_sweep(&state, &config);
let router = server::build_router(state.clone());
// Re-register the socket bound at startup — same fd, so there was never a
// moment with the port closed between the degraded and full servers.
@@ -806,8 +852,8 @@ fn build_degraded_router(node_did: String, db_startup: Arc<DbStartupStatus>) ->
db_startup,
};
// Everything answers 503 with the same body — including /health and
// /ready, so peer pings (which treat any 2xx /health as alive) and
// uptime monitors correctly see a node that cannot serve traffic.
// /ready, so peer readiness probes and uptime monitors correctly see a
// node that cannot serve traffic.
// `/` additionally carries the node identity for probing peers.
Router::new()
.route("/", get(degraded_node_info))
@@ -939,7 +985,7 @@ async fn gossip_task(
}
// Reuse the shared no-redirect client for every gossip outbound call (the
// bootstrap announce POST and the periodic peer /health ping). Peer URLs are
// bootstrap announce POST and the periodic peer /ready ping). Peer URLs are
// attacker-influenceable, so a 3xx to a private address must not be followed.
// Do NOT fall back to reqwest::Client::new(): its default follows redirects
// and would reintroduce the SSRF closed here (#93).
@@ -994,8 +1040,33 @@ async fn gossip_task(
json.get("node_url").and_then(|v| v.as_str()),
) {
if !their_url.is_empty() {
let _ = state.db.upsert_peer(their_did, their_url).await;
tracing::info!(did = %their_did, url = %their_url, "bootstrap peer added");
// Unproven, unconditionally: their_did and
// their_url come straight out of the contacted
// peer's JSON response body, with no signature
// and no proof the claimed DID belongs to the
// peer we reached. This path therefore seeds
// new peers and can never repoint an existing
// row. The upsert's result is matched rather
// than discarded, because "bootstrap peer
// added" printed over a refused write is a
// denial rendering as success. Non-fatal to the
// loop either way.
match state
.db
.upsert_peer(
their_did,
their_url,
db::PeerWriteAuthority::Unproven,
)
.await
{
Ok(()) => {
tracing::info!(did = %their_did, url = %their_url, "bootstrap peer added")
}
Err(e) => {
tracing::warn!(did = %their_did, url = %their_url, err = %e, "bootstrap peer announce-back rejected")
}
}
}
}
}
@@ -1010,6 +1081,8 @@ async fn gossip_task(
// Periodic ping every 5 minutes — exit on shutdown.
let mut interval = tokio::time::interval(std::time::Duration::from_secs(300));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
let mut failed_once: HashSet<String> = HashSet::new();
loop {
tokio::select! {
_ = interval.tick() => {
@@ -1017,10 +1090,13 @@ async fn gossip_task(
Ok(p) => p,
Err(_) => continue,
};
for peer in peers {
let ok = ping_peer_health(&client, &peer.http_url).await;
let _ = state.db.mark_peer_ping(&peer.did, ok).await;
}
gossip_ping_round(
state.db.as_ref(),
client.as_ref(),
&mut failed_once,
peers,
)
.await;
}
_ = shutdown_rx.changed() => {
if *shutdown_rx.borrow() {
@@ -1048,6 +1124,7 @@ mod rate_limiter_sweep_tests {
state.rate_limiter = RateLimiter::new(10, window);
state.create_ip_rate_limiter = RateLimiter::new(10, window);
state.push_rate_limiter = RateLimiter::new(10, window);
state.close_issue_rate_limiter = RateLimiter::new(10, window);
state.sync_trigger_rate_limiter = RateLimiter::new(10, window);
state.peer_write_rate_limiter = RateLimiter::new(10, window);
state.ipfs_rate_limiter = RateLimiter::new(10, window);
@@ -1057,6 +1134,7 @@ mod rate_limiter_sweep_tests {
s.rate_limiter.clone(),
s.create_ip_rate_limiter.clone(),
s.push_rate_limiter.clone(),
s.close_issue_rate_limiter.clone(),
s.sync_trigger_rate_limiter.clone(),
s.peer_write_rate_limiter.clone(),
s.ipfs_rate_limiter.clone(),
@@ -1068,7 +1146,7 @@ mod rate_limiter_sweep_tests {
}
tokio::time::sleep(window * 3).await;
super::sweep_rate_limiters(&state).await;
state.sweep_rate_limiters().await;
for (i, l) in limiters(&state).into_iter().enumerate() {
assert_eq!(l.tracked_keys().await, 0, "limiter {i} was not swept");
@@ -1076,19 +1154,65 @@ mod rate_limiter_sweep_tests {
}
}
/// Evict expired entries from every per-key rate limiter on the state.
///
/// Named and driven off `AppState` so the periodic sweeper stays in step with
/// the limiters the router actually mounts: adding a limiter field and
/// forgetting it here leaves its keys pinned until the map hits `max_keys` and
/// the inline capacity sweep runs (the `/ipfs` limiter was missed this way).
async fn sweep_rate_limiters(state: &AppState) {
state.rate_limiter.cleanup().await;
state.create_ip_rate_limiter.cleanup().await;
state.push_rate_limiter.cleanup().await;
state.sync_trigger_rate_limiter.cleanup().await;
state.peer_write_rate_limiter.cleanup().await;
state.ipfs_rate_limiter.cleanup().await;
/// it.
fn spawn_legacy_cid_sweep(state: &AppState, config: &Config) -> tokio::task::JoinHandle<()> {
let db = state.db.clone();
let repos_dir = config.repos_dir.clone();
let git_bin = state.git_bin.clone();
let git_timeout = std::time::Duration::from_secs(config.git_service_timeout_secs);
let batch = config.pin_repair_sweep_batch;
let delay = std::time::Duration::from_secs(config.pin_repair_sweep_delay_secs);
let mut shutdown_rx = state.subscribe_shutdown();
tokio::spawn(async move {
tokio::select! {
_ = ipfs_pin::run_sweep_rearmed(
&repos_dir, &git_bin, git_timeout, batch, delay,
ipfs_pin::SWEEP_REARM_DELAY, &db,
) => {}
_ = shutdown_rx.changed() => {}
}
})
}
async fn gossip_ping_round(
db: &Db,
client: &reqwest::Client,
failed_once: &mut HashSet<String>,
peers: Vec<db::PeerRecord>,
) {
{
let current_dids: HashSet<&str> = peers.iter().map(|peer| peer.did.as_str()).collect();
failed_once.retain(|did| current_dids.contains(did.as_str()));
}
for peer in peers {
let ok = ping_peer_readiness(client, &peer.http_url).await;
match peer_ping_db_update(failed_once, &peer.did, ok) {
Some(reachable) => {
if let Err(error) = db.mark_peer_ping(&peer.did, reachable).await {
warn!(did = %peer.did, err = %error, "failed to persist peer readiness");
}
}
None => warn!(
did = %peer.did,
"peer readiness probe failed once; preserving previous federation status"
),
}
}
}
/// Decide whether one readiness sample should update the persisted federation
/// gate. A success is authoritative immediately. A failure must repeat on the
/// next gossip tick before it can mark a peer unreachable, so one transient
/// database hiccup does not hide the peer's repositories for five minutes.
fn peer_ping_db_update(failed_once: &mut HashSet<String>, did: &str, ready: bool) -> Option<bool> {
if ready {
failed_once.remove(did);
Some(true)
} else if failed_once.insert(did.to_owned()) {
None
} else {
Some(false)
}
}
/// Build the shared node HTTP client used for every outbound fan-out (sync
@@ -1101,26 +1225,96 @@ async fn sweep_rate_limiters(state: &AppState) {
/// runs, not a hand-rolled equivalent.
fn build_http_client() -> reqwest::Result<reqwest::Client> {
reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(10))
.timeout(OUTBOUND_HTTP_TIMEOUT)
.redirect(reqwest::redirect::Policy::none())
.build()
}
/// Ping a peer's `/health` endpoint and report whether it answered 2xx.
const OUTBOUND_HTTP_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10);
/// Ping a peer's DB-aware `/ready` endpoint and report whether it answered 2xx.
///
/// A 404 falls back to `/health` for compatibility with nodes released before
/// `/ready` existed. Other readiness failures, including 503 during a database
/// outage, fail closed and must not use the liveness-only endpoint. The complete
/// `/ready`-then-`/health` probe shares one timeout budget.
///
/// Takes the client by reference so callers supply the shared, no-redirect
/// `state.http_client`. Peer URLs are attacker-influenceable, so a `3xx` to a
/// private address must not be followed. Do NOT call this with a bare
/// `reqwest::Client::new()`: its default follows redirects and would
/// reintroduce the SSRF this guards against (#93).
async fn ping_peer_health(client: &reqwest::Client, http_url: &str) -> bool {
let url = format!("{}/health", http_url.trim_end_matches('/'));
client
.get(&url)
.send()
.await
.map(|r| r.status().is_success())
.unwrap_or(false)
pub(crate) async fn ping_peer_readiness(client: &reqwest::Client, http_url: &str) -> bool {
ping_peer_readiness_with_timeout(client, http_url, OUTBOUND_HTTP_TIMEOUT).await
}
async fn ping_peer_readiness_with_timeout(
client: &reqwest::Client,
http_url: &str,
timeout: std::time::Duration,
) -> bool {
let base_url = http_url.trim_end_matches('/');
let result = tokio::time::timeout(timeout, async {
let readiness = client.get(format!("{base_url}/ready")).send().await;
match readiness {
Ok(response) if response.status().is_success() => true,
Ok(response) if response.status() == reqwest::StatusCode::NOT_FOUND => {
info!(
peer_url = %http_url,
"peer has no readiness endpoint; falling back to legacy health probe"
);
match client.get(format!("{base_url}/health")).send().await {
Ok(response) if response.status().is_success() => true,
Ok(response) => {
warn!(
peer_url = %http_url,
status = %response.status(),
"legacy peer health probe reported unhealthy"
);
false
}
Err(error) => {
warn!(
peer_url = %http_url,
err = %error,
"legacy peer health probe failed"
);
false
}
}
}
Ok(response) => {
warn!(
peer_url = %http_url,
status = %response.status(),
"peer readiness probe reported unready"
);
false
}
Err(error) => {
warn!(
peer_url = %http_url,
err = %error,
"peer readiness probe failed"
);
false
}
}
})
.await;
match result {
Ok(ready) => ready,
Err(_) => {
warn!(
peer_url = %http_url,
timeout_ms = timeout.as_millis(),
"peer readiness probe timed out"
);
false
}
}
}
fn load_or_create_keypair(config: &Config) -> Result<Keypair> {
@@ -1158,20 +1352,130 @@ fn load_or_create_keypair(config: &Config) -> Result<Keypair> {
#[cfg(test)]
mod gossip_ssrf_tests {
use super::ping_peer_health;
use super::{
gossip_ping_round, peer_ping_db_update, ping_peer_readiness,
ping_peer_readiness_with_timeout,
};
use axum::{http::StatusCode, routing::get, Router};
use sqlx::PgPool;
use std::collections::HashSet;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
use std::time::Duration;
// Build the client exactly as production does (super::build_http_client) so
// these tests bind the redirect guarantee to the real shared client the
// node runs. A regression that makes build_http_client follow redirects
// fails ping_peer_health_does_not_follow_redirect.
// fails ping_peer_readiness_does_not_follow_redirect.
fn production_http_client() -> reqwest::Client {
super::build_http_client().expect("failed to build production http client")
}
// A peer answering `/health` with a 302 toward an internal address must not
#[test]
fn peer_ping_requires_two_failures_before_marking_unreachable() {
let mut failed_once = HashSet::new();
let did = "did:key:z6MkPeer";
assert_eq!(
peer_ping_db_update(&mut failed_once, did, false),
None,
"one transient failure must preserve the persisted federation gate"
);
assert_eq!(
peer_ping_db_update(&mut failed_once, did, false),
Some(false),
"a sustained failure must mark the peer unreachable"
);
assert_eq!(
peer_ping_db_update(&mut failed_once, did, true),
Some(true),
"a success must restore reachability immediately"
);
assert_eq!(
peer_ping_db_update(&mut failed_once, did, false),
None,
"a success must reset the consecutive-failure state"
);
}
#[sqlx::test]
async fn gossip_ping_round_requires_two_failures_before_persisting_unreachable(pool: PgPool) {
let mut server = mockito::Server::new_async().await;
let ready = server
.mock("GET", "/ready")
.with_status(503)
.expect(2)
.create_async()
.await;
let state = crate::test_support::test_state(pool.clone()).await;
let did = "did:key:z6MkGossipHysteresis";
let now = chrono::Utc::now().to_rfc3339();
sqlx::query(
"INSERT INTO peers (did, http_url, last_seen, last_ping_ok, announced_at)
VALUES ($1, $2, $3, TRUE, $3)",
)
.bind(did)
.bind(server.url())
.bind(now)
.execute(&pool)
.await
.unwrap();
let mut failed_once = HashSet::from(["did:key:z6MkRemovedPeer".to_owned()]);
gossip_ping_round(
state.db.as_ref(),
&production_http_client(),
&mut failed_once,
state.db.list_peers().await.unwrap(),
)
.await;
assert!(
!failed_once.contains("did:key:z6MkRemovedPeer"),
"failure tracking must prune peers outside the current snapshot"
);
assert!(
failed_once.contains(did),
"the first failed sample must be retained for the next round"
);
assert!(
state
.db
.list_peers()
.await
.unwrap()
.into_iter()
.find(|peer| peer.did == did)
.unwrap()
.last_ping_ok,
"one transient failure must preserve the persisted federation gate"
);
gossip_ping_round(
state.db.as_ref(),
&production_http_client(),
&mut failed_once,
state.db.list_peers().await.unwrap(),
)
.await;
assert!(
!state
.db
.list_peers()
.await
.unwrap()
.into_iter()
.find(|peer| peer.did == did)
.unwrap()
.last_ping_ok,
"two consecutive failed rounds must persist the peer as unreachable"
);
ready.assert_async().await;
}
// A peer answering `/ready` with a 302 toward an internal address must not
// be followed: the redirect target must never be requested (#93).
#[tokio::test]
async fn ping_peer_health_does_not_follow_redirect() {
async fn ping_peer_readiness_does_not_follow_redirect() {
let mut server = mockito::Server::new_async().await;
let internal = server
.mock("GET", "/internal-metadata")
@@ -1179,6 +1483,154 @@ mod gossip_ssrf_tests {
.expect(0)
.create_async()
.await;
let _ready = server
.mock("GET", "/ready")
.with_status(302)
.with_header("location", &format!("{}/internal-metadata", server.url()))
.create_async()
.await;
let ok = ping_peer_readiness(&production_http_client(), &server.url()).await;
assert!(!ok, "a 302 must not count as a healthy peer");
// expect(0) is enforced only at assert time; this fails if the redirect
// was followed to the internal target.
internal.assert_async().await;
}
#[tokio::test]
async fn ping_peer_readiness_reports_success_on_200() {
let mut server = mockito::Server::new_async().await;
let _ready = server
.mock("GET", "/ready")
.with_status(200)
.create_async()
.await;
let ok = ping_peer_readiness(&production_http_client(), &server.url()).await;
assert!(ok, "a 200 /ready must count as a ready peer");
}
#[tokio::test]
async fn ping_peer_readiness_ignores_liveness_only_health() {
let mut server = mockito::Server::new_async().await;
let health = server
.mock("GET", "/health")
.with_status(200)
.expect(0)
.create_async()
.await;
let ready = server
.mock("GET", "/ready")
.with_status(503)
.expect(1)
.create_async()
.await;
let ok = ping_peer_readiness(&production_http_client(), &server.url()).await;
assert!(!ok, "a peer with an unavailable database must not be ready");
health.assert_async().await;
ready.assert_async().await;
}
#[tokio::test]
async fn ping_peer_readiness_falls_back_for_legacy_peer() {
let mut server = mockito::Server::new_async().await;
let ready = server
.mock("GET", "/ready")
.with_status(404)
.expect(1)
.create_async()
.await;
let health = server
.mock("GET", "/health")
.with_status(200)
.expect(1)
.create_async()
.await;
let ok = ping_peer_readiness(&production_http_client(), &server.url()).await;
assert!(
ok,
"a legacy peer with a healthy /health must remain reachable"
);
ready.assert_async().await;
health.assert_async().await;
}
#[tokio::test]
async fn ping_peer_readiness_legacy_fallback_shares_deadline() {
let health_requests = Arc::new(AtomicUsize::new(0));
let health_requests_for_route = Arc::clone(&health_requests);
let app = Router::new()
.route(
"/ready",
get(|| async {
tokio::time::sleep(Duration::from_millis(100)).await;
StatusCode::NOT_FOUND
}),
)
.route(
"/health",
get(move || {
let health_requests = Arc::clone(&health_requests_for_route);
async move {
health_requests.fetch_add(1, Ordering::Relaxed);
tokio::time::sleep(Duration::from_millis(300)).await;
StatusCode::OK
}
}),
);
let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
.await
.expect("failed to bind delayed legacy peer");
let peer_url = format!(
"http://{}",
listener
.local_addr()
.expect("delayed legacy peer has no local address")
);
let server = tokio::spawn(async move {
axum::serve(listener, app)
.await
.expect("delayed legacy peer failed");
});
let ok = ping_peer_readiness_with_timeout(
&production_http_client(),
&peer_url,
// Each response arrives within 350 ms, but the two serial requests
// cannot both finish within one 350 ms probe budget.
Duration::from_millis(350),
)
.await;
assert!(!ok, "the fallback must not start a fresh timeout budget");
assert_eq!(
health_requests.load(Ordering::Relaxed),
1,
"the delayed 404 should reach the legacy fallback"
);
server.abort();
}
#[tokio::test]
async fn ping_peer_readiness_legacy_fallback_does_not_follow_redirect() {
let mut server = mockito::Server::new_async().await;
let internal = server
.mock("GET", "/internal-metadata")
.with_status(200)
.expect(0)
.create_async()
.await;
let _ready = server
.mock("GET", "/ready")
.with_status(404)
.create_async()
.await;
let _health = server
.mock("GET", "/health")
.with_status(302)
@@ -1186,33 +1638,17 @@ mod gossip_ssrf_tests {
.create_async()
.await;
let ok = ping_peer_health(&production_http_client(), &server.url()).await;
let ok = ping_peer_readiness(&production_http_client(), &server.url()).await;
assert!(!ok, "a 302 must not count as a healthy peer");
// expect(0) is enforced only at assert time; this fails if the redirect
// was followed to the internal target.
assert!(!ok, "the legacy fallback must not follow redirects");
internal.assert_async().await;
}
#[tokio::test]
async fn ping_peer_health_reports_success_on_200() {
let mut server = mockito::Server::new_async().await;
let _health = server
.mock("GET", "/health")
.with_status(200)
.create_async()
.await;
let ok = ping_peer_health(&production_http_client(), &server.url()).await;
assert!(ok, "a 200 /health must count as a healthy peer");
}
// A transport error (nothing listening) must map to unhealthy, never a
// spurious healthy — the .unwrap_or(false) arm.
#[tokio::test]
async fn ping_peer_health_reports_unhealthy_on_connection_error() {
let ok = ping_peer_health(&production_http_client(), "http://127.0.0.1:1").await;
assert!(!ok, "a connection error must count as an unhealthy peer");
async fn ping_peer_readiness_reports_unready_on_connection_error() {
let ok = ping_peer_readiness(&production_http_client(), "http://127.0.0.1:1").await;
assert!(!ok, "a connection error must count as an unready peer");
}
}
+749 -26
View File
@@ -76,42 +76,87 @@ pub async fn pin_object(
/// still needed to read each object's bytes, and `git_bin` names the binary those
/// reads run: the production caller passes the literal `"git"`, and a test passes a
/// fake so the loop's own bound can be driven with a git that never answers.
/// Objects already recorded with a `pinata_cid` are skipped. Returns
/// `(sha_hex, cid)` pairs for each newly pinned object.
/// `git_timeout` is the per-object read bound, the same value and the same role it has
/// in the twin: it bounds both the pin read and the skip branch's opportunistic repair.
/// Objects already recorded with a `pinata_cid` are skipped, and `repo_id` records the
/// pin's provenance (#173). Returns `(sha_hex, provider_cid)` pairs for each newly
/// pinned object: the provider CID is the Pinata gateway CID (used for branch->CID
/// recording and ref-update gossip), NOT the raw resolver-key CID stored in
/// `pinned_cids.cid`.
///
/// # What `batch_budget` does and does not bound
///
/// The loop runs under a `pin_semaphore` permit and that pool defers rather than
/// sheds, so the hold has to be bounded by something other than the pusher's object
/// count. Two things here are:
/// count. Three things here are:
///
/// - this loop's own wall-clock: the deadline is taken once at loop start and
/// checked at the top of every iteration, so no object's work begins with less
/// than the read floor left. It is a gate, not a hard ceiling, since a started
/// iteration still runs to completion;
/// - the git read: `store::read_object_bounded` runs under `spawn_blocking` against the
/// ABSOLUTE batch deadline (not the loop-top remainder, which the `has_pinata_cid`
/// round-trip sitting between the two would push past it), with SIGTERM-then-SIGKILL
/// process-group teardown, so a hung `git cat-file` costs this batch its remaining
/// budget plus one watchdog teardown instead of holding the permit for the child's
/// whole lifetime and blocking a runtime worker while it does.
/// earlier of the ABSOLUTE batch deadline (not the loop-top remainder, which the
/// `has_pinata_cid` round-trip sitting between the two would push past it) and this
/// object's own `git_timeout`, with SIGTERM-then-SIGKILL process-group teardown, so a
/// hung `git cat-file` costs this batch one `git_timeout` plus one watchdog teardown
/// instead of holding the permit for the child's whole lifetime and blocking a runtime
/// worker while it does;
/// - the DB round-trips: every DB operation reachable from inside the region is
/// bounded by the same absolute deadline through `crate::ipfs_pin::db_bounded`,
/// including the two inside `repair_legacy_provider_cid`, which this loop body's own
/// call sites do not show. `retry_db_record` is wrapped as a whole so its ladder
/// cannot multiply one remainder, and the durability writes (the post-upload
/// `record_pinata_cid` and source record, the skip branch's source record, and both
/// incomplete markers) take the floored remainder `max(remaining, DB_RECORD_GRACE)`
/// so a spent budget delays the permit release rather than dropping a write. That
/// floor matters more here than on the twin: `pinned.push` is unconditional, so a
/// dropped record would leave `api::repos` advertising a CID the resolver 404s. A
/// bound is not a rollback, and what an elapsed bound MEANS is a property of the
/// operation, so each site maps that arm from the shape of the call it wrapped. Both
/// source-record sites wrap `record_pin_source`, an explicit transaction whose
/// `tx.commit()` a cancelled future never reaches, so a timeout there definitely did
/// not land and both write the incomplete marker exactly as the definite-error arm
/// does. `record_pinata_cid` is a single autocommit upsert, so ITS timeout is a
/// genuine unknown outcome and is never treated as a failed write. See
/// `crate::ipfs_pin::BoundedDbError::Elapsed`.
///
/// So the LOOP's hold is bounded by roughly `batch_budget`, plus one watchdog
/// teardown and one upload (the shared client's whole-request timeout bounds the
/// upload; `pin_object` takes no per-request override). The PERMIT's hold is NOT
/// bounded by any of this: `api::repos` acquires the permit and then re-derives the
/// object list with `pinata_object_list_for_refs` BEFORE this function is entered,
/// and that walk carries no aggregate deadline. The DB round-trips
/// (`has_pinata_cid`, `record_pinata_cid`) are untimed inside the budgeted region
/// too.
/// teardown, one upload (the shared client's whole-request timeout bounds the
/// upload; `pin_object` takes no per-request override), and the record graces one
/// iteration can chain. `db_record_deadline` re-floors from `Instant::now()` at EVERY
/// call, so the graces inside a single iteration add up rather than sharing one floor:
/// the worst case here is the add path at `deadline + 6s` (`record_pinata_cid`, then
/// `record_pin_source`, then its incomplete marker), against the skip branch's
/// `deadline + 4s`. It does NOT stack per object, because the next iteration's first
/// statement is `batch_budget_gate`, which breaks the batch, so the overrun is one
/// iteration's worth however many objects the push carried. Against the 120s
/// `PIN_BATCH_BUDGET` that is roughly a 5% overrun for the batch, not an unbounded
/// hold. The PERMIT's hold is NOT bounded by any of this, and that stays out of
/// scope: `api::repos` acquires the permit (repos.rs ~2688) and only then re-derives
/// the object list with `pinata_object_list_for_refs` (~2697), BEFORE this function is
/// entered, and that walk carries no aggregate deadline. What is bounded is this
/// loop's own hold, not the permit's total hold and not the semaphore's worst-case
/// queue.
///
/// The twin in `ipfs_pin.rs` shares the budget gate and the bounded read with this
/// loop, so both are at parity again. Change them in lockstep: the skip-if-pinned
/// check, the fault arms, the returned pairs, and the budget handling.
// Eight arguments, one over clippy's threshold: the two the budget adds (`git_bin`,
// `batch_budget`) are what put the read under test injection and under a deadline, and
// grouping them into a struct would only move the same values behind a name the twin in
// `ipfs_pin.rs` does not use. Same allow as the sibling call sites in `api::repos`.
/// The twin in `ipfs_pin.rs` is at parity with this loop on everything that bounds or
/// repairs an object: the shared budget gate, the read bounded by the earlier of the
/// batch deadline and `git_timeout`, the skip branch's opportunistic legacy
/// provider-CID repair, and the DB bound above, which is the SAME helper and the same
/// floor on both sides rather than a copy. Change them in lockstep: the
/// skip-if-pinned check, the provenance and source recording, the fault arms, and the
/// budget handling.
///
/// The RETURNED PAIRS are the one deliberate divergence, and it is not drift. This side
/// pushes a pin whose DB record exhausted its retries, because this return is a real
/// input: `api::repos` builds the sha-to-cid `cid_map` from it, which drives
/// `upsert_branch_cid` and the p2p `publish_ref_update` gossip CID. The twin's return is
/// log-only, so it omits a record-failed pin rather than logging a pin the resolver
/// cannot serve. Moving this side to match would need that consumer moved first.
// Ten arguments, over clippy's threshold: the three the budget and the git seam add
// (`git_bin`, `git_timeout`, `batch_budget`) plus #173's `repo_id` are what put the read
// under test injection and under a deadline, and grouping them into a struct would only
// move the same values behind a name the twin in `ipfs_pin.rs` does not use. Same allow
// as the sibling call sites in `api::repos`.
#[allow(clippy::too_many_arguments)]
pub async fn pin_new_objects(
client: &reqwest::Client,
@@ -119,8 +164,10 @@ pub async fn pin_new_objects(
jwt: &str,
repo_path: &std::path::Path,
git_bin: &str,
git_timeout: Duration,
object_list: Vec<String>,
db: &crate::db::Db,
repo_id: &str,
batch_budget: Duration,
) -> Vec<(String, String)> {
if jwt.is_empty() {
@@ -144,8 +191,124 @@ pub async fn pin_new_objects(
break;
}
match db.has_pinata_cid(&sha).await {
Ok(true) => continue,
// Every DB call from here to the end of the iteration is bounded by the
// ABSOLUTE batch deadline (F3, #173), through the same `db_bounded` helper the
// ipfs_pin twin routes through: this loop runs under the same global pin permit
// and a bare await parked it for the whole stall. The elapsed arm is mapped per
// site below, never as a blanket "existing error arm": a timeout cancels the
// client future but not the statement Postgres is running, so it reports an
// UNKNOWN outcome, not a failed write.
match crate::ipfs_pin::db_bounded(deadline, db.has_pinata_cid(&sha)).await {
Ok(true) => {
// Backfill NULL first-pinner provenance from a known source, in lockstep
// with the ipfs_pin skip branch: a pinata-only node otherwise leaves
// pre-provenance rows' `pinned_cids.repo_id` NULL forever (grok P2-D). The
// resolver still finds the object via the pin_repo_sources union below, so
// this is a consistency backfill, not a correctness fix.
//
// Elapsed here is free to skip: the read costs nothing when it lands late,
// and the backfill's own `AND repo_id IS NULL` guard makes a late-landing
// write idempotent.
match crate::ipfs_pin::db_bounded(deadline, db.provenance_for_oid(&sha)).await {
Ok(None) => {
if let Err(e) = crate::ipfs_pin::db_bounded(
deadline,
db.backfill_pin_provenance(&sha, repo_id),
)
.await
{
tracing::warn!(sha = %sha, err = %e, "failed to backfill pin provenance");
}
}
Ok(Some(_)) => {}
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "DB error reading pin provenance");
}
}
// F1 (#173 round 8): record this repo as an additional source for the
// already-pinned object (mirrors the ipfs_pin skip-branch insert) so the
// resolver can serve a shared object from any pin-path source. U3 (#173):
// retried through the SHARED helper (this was a bare call, so a single
// transient error dropped the source outright) and, on exhaustion, marked
// durably so the resolver keeps the bounded scan fallback for the object.
// The retry ladder is bounded AS A WHOLE, not per attempt: three stalls
// plus their backoff otherwise multiply one remainder by three. Floored at
// DB_RECORD_GRACE because this is a durability write.
match crate::ipfs_pin::db_bounded(
crate::ipfs_pin::db_record_deadline(deadline),
crate::ipfs_pin::retry_db_record(|| db.record_pin_source(&sha, repo_id)),
)
.await
{
Ok(()) => {}
// Elapsed here is a DEFINITE non-write, in lockstep with the twin,
// and for a reason that comes from the operation rather than the
// timeout: `record_pin_source` is an explicit transaction whose
// `tx.commit()` a cancelled future never reaches, so no COMMIT is
// sent and the row cannot have landed. Mark the set incomplete
// exactly as the definite-error arm does; an incomplete-and-unmarked
// set is read as COMPLETE and 404s a copy this repo would serve. The
// marker's cost is bounded (the fallback scan is capped at
// `ipfs_max_legacy_probes` and charges the per-IP work rate limiter
// per probe). The arm stays separate only so the warn tells an
// operator a stalled batch from a scattered per-object failure.
Err(e @ crate::ipfs_pin::BoundedDbError::Elapsed) => {
tracing::warn!(
sha = %sha,
err = %e,
"pin source record did not complete inside the batch deadline; \
a cancelled multi-statement transaction never commits, so the \
source is definitely missing and the set is marked incomplete"
);
if let Err(e) = crate::ipfs_pin::db_bounded(
crate::ipfs_pin::db_record_deadline(deadline),
db.mark_pin_sources_incomplete(&sha, repo_id),
)
.await
{
tracing::warn!(sha = %sha, err = %e, "failed to mark pin sources incomplete");
}
}
// The retries are spent on REAL errors, so this repo is definitely NOT
// in the source set and the set is known incomplete. Floored for the
// same reason the record above is: a spent budget must not drop the
// compensation. The marker write is a single autocommit statement, so
// ITS own elapsed arm is a genuine unknown outcome; nothing branches
// on it, which is why warn-only is right there.
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "failed to record pin source");
if let Err(e) = crate::ipfs_pin::db_bounded(
crate::ipfs_pin::db_record_deadline(deadline),
db.mark_pin_sources_incomplete(&sha, repo_id),
)
.await
{
tracing::warn!(sha = %sha, err = %e, "failed to mark pin sources incomplete");
}
}
}
// R8 (#173 round 10), in lockstep with the ipfs_pin skip branch:
// opportunistically repair a legacy provider-CID row (Kubo dag-pb /
// Pinata) to the raw-content resolver key on this re-push. Cost-gated on
// the stored key's codec, so a non-legacy row reads no bytes. Warn-only:
// a failure leaves the row as-is for a later re-push or the deferred
// one-shot sweep.
// Clamped to the batch deadline, in lockstep with the ipfs_pin twin: this
// runs with the pin permit held, so an unclamped `git_timeout` would let
// one wedged read hold a global pin slot for 600s against a 120s budget.
if let Err(e) = crate::ipfs_pin::repair_legacy_provider_cid(
repo_path,
git_bin,
std::cmp::min(deadline, std::time::Instant::now() + git_timeout),
&sha,
db,
)
.await
{
tracing::warn!(sha = %sha, err = %e, "failed to repair legacy provider CID");
}
continue;
}
Ok(false) => {}
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "DB error checking pinata_cid");
@@ -163,7 +326,16 @@ pub async fn pin_new_objects(
// between the two, so `Instant::now() + budget_left` would land past `deadline` by
// however long the DB took, and under a saturated pool that is the dominant term.
// A slow DB check must not push the read's own bound out.
let read_deadline = deadline;
//
// Bounded by the EARLIER of the batch deadline (#174) and this object's own
// `git_timeout` (#173), the same pair the ipfs_pin twin uses. Both bounds are
// load-bearing and neither implies the other: the batch deadline alone would let
// ONE wedged `cat-file` hold the pin permit for the whole budget, while
// `git_timeout` alone would let a batch of merely-slow reads run past the budget.
// As on the twin, at SHIPPED DEFAULTS the batch deadline is the arm that binds
// (600s git timeout against a 120s budget); the `git_timeout` arm is for an
// operator who tightens that knob below the remaining budget.
let read_deadline = std::cmp::min(deadline, std::time::Instant::now() + git_timeout);
let read_path = repo_path.to_path_buf();
let read_sha = sha.clone();
let read_git = git_bin.to_string();
@@ -224,9 +396,86 @@ pub async fn pin_new_objects(
match pin_object(client, upload_url, jwt, &sha, &data).await {
Ok(cid) if !cid.is_empty() => {
if let Err(e) = db.record_pinata_cid(&sha, &cid).await {
// The resolver key (`pinned_cids.cid`) must be the locally-computed
// raw-content CID, never the provider CID: Pinata wraps the bytes in
// dag-pb/UnixFS, so its returned CID does not hash the raw content and
// must not become an alias `/ipfs/{cid}` serves raw git bytes for (#173).
let raw_cid = gitlawb_core::cid::Cid::from_git_object_bytes(&data).to_string();
// U3 (#173): both records go through the shared retry helper, at parity
// with the ipfs_pin twin. These were bare calls, so one transient DB error
// permanently dropped a pin source.
//
// Both bounds here are FLOORED at DB_RECORD_GRACE (F3, #173). The upload
// runs under the shared client's own ceiling, so a successful one can
// return with ~0 of the batch budget left, and an unfloored bound would
// fail a write that today completes in milliseconds. That costs more on
// this side than on the twin: `pinned.push` below is UNCONDITIONAL, so
// `api/repos.rs` builds its `cid_map` from the pair either way and drives
// `upsert_branch_cid` plus the p2p `publish_ref_update` gossip from it. A
// dropped record therefore makes the node ADVERTISE a CID whose `/ipfs`
// read 404s. If the floored bound still fires, THIS site's outcome really
// is unknown, and unlike the source record below that is a property of
// the operation: `record_pinata_cid` is a single autocommit upsert, so
// the statement Postgres already started can still land after the client
// future is cancelled. The warn names the arm through the error's own
// Display and the site keeps its existing behavior: the pair is still
// returned, and the row may or may not exist.
if let Err(e) = crate::ipfs_pin::db_bounded(
crate::ipfs_pin::db_record_deadline(deadline),
crate::ipfs_pin::retry_db_record(|| {
db.record_pinata_cid(&sha, &raw_cid, &cid, Some(repo_id))
}),
)
.await
{
tracing::warn!(sha = %sha, err = %e, "failed to record pinata_cid in DB");
}
// F1 (#173 round 8): also record the first pinner in pin_repo_sources.
// U3: an exhausted retry marks the set incomplete so the resolver keeps
// the scan fallback rather than 404ing a copy it could serve.
match crate::ipfs_pin::db_bounded(
crate::ipfs_pin::db_record_deadline(deadline),
crate::ipfs_pin::retry_db_record(|| db.record_pin_source(&sha, repo_id)),
)
.await
{
Ok(()) => {}
// Same rule as the skip branch above, and the same reason: this
// wraps `record_pin_source`, an explicit transaction, so a timed-out
// call definitely never committed and the source is definitely
// missing. Mark the set incomplete rather than leaving it incomplete
// and unmarked. Note the contrast with `record_pinata_cid` a few
// lines up: that one is a single autocommit statement, so its
// timeout genuinely is an unknown outcome and it is warn-only.
Err(e @ crate::ipfs_pin::BoundedDbError::Elapsed) => {
tracing::warn!(
sha = %sha,
err = %e,
"pin source record did not complete inside the batch deadline; \
a cancelled multi-statement transaction never commits, so the \
source is definitely missing and the set is marked incomplete"
);
if let Err(e) = crate::ipfs_pin::db_bounded(
crate::ipfs_pin::db_record_deadline(deadline),
db.mark_pin_sources_incomplete(&sha, repo_id),
)
.await
{
tracing::warn!(sha = %sha, err = %e, "failed to mark pin sources incomplete");
}
}
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "failed to record pin source");
if let Err(e) = crate::ipfs_pin::db_bounded(
crate::ipfs_pin::db_record_deadline(deadline),
db.mark_pin_sources_incomplete(&sha, repo_id),
)
.await
{
tracing::warn!(sha = %sha, err = %e, "failed to mark pin sources incomplete");
}
}
}
pinned.push((sha, cid));
}
Ok(_) => {}
@@ -433,8 +682,10 @@ mod tests {
"test-jwt",
&repo_path,
"git",
Duration::from_secs(60),
oids,
&db,
"repo-merge-test",
Duration::from_millis(5500),
),
)
@@ -522,8 +773,10 @@ mod tests {
"test-jwt",
&repo_path,
fake.to_str().unwrap(),
Duration::from_secs(60),
oids,
&db,
"repo-merge-test",
Duration::from_secs(2),
),
)
@@ -574,6 +827,66 @@ mod tests {
);
}
/// U3 scenario 5 (#173): the read is bounded by the EARLIER of the batch deadline and
/// this object's own `git_timeout`, the same pair the ipfs_pin twin uses. The batch
/// budget here is generous (60s) so the budget gate cannot be what ends the call: only
/// the 1s `git_timeout` can. A wedged `git cat-file` that traps SIGTERM and sleeps 30s
/// must therefore be reaped in the `git_timeout` order and the call must return, rather
/// than holding the pin permit for the whole budget.
///
/// RED with `let read_deadline = deadline;` (the pre-U3 bare batch deadline): the read
/// waits out the wedged child, the call runs ~30s, and the outer 20s timeout fires.
#[cfg(unix)]
#[sqlx::test]
async fn pin_new_objects_bounds_the_read_by_git_timeout_not_the_batch_budget(
pool: sqlx::PgPool,
) {
let db = crate::db::Db::for_testing(pool);
db.run_migrations().await.expect("migrations");
let tmp = tempfile::TempDir::new().unwrap();
let repo_path = tmp.path().join("git-timeout.git");
let oids = seed_loose_blobs(&repo_path, 1);
let fake = tmp.path().join("hanging-git");
write_script(&fake, "#!/bin/sh\ntrap '' TERM\necho $$ > pid\nsleep 30\n");
let (_logs, _guard) = capture_logs();
let client = reqwest::Client::new();
let started = std::time::Instant::now();
let pinned = tokio::time::timeout(
Duration::from_secs(20),
pin_new_objects(
&client,
"http://127.0.0.1:9",
"test-jwt",
&repo_path,
fake.to_str().unwrap(),
// The bound under test.
Duration::from_secs(1),
oids,
&db,
"repo-git-timeout",
// Generous, so a call that ends on time ended on `git_timeout`.
Duration::from_secs(60),
),
)
.await
.expect(
"the read must be bounded by git_timeout, not by the batch budget: a wedged git \
cannot hold the pin permit for the whole 60s budget",
);
let elapsed = started.elapsed();
assert!(
pinned.is_empty(),
"a git that never answers cannot produce a pinned object: {pinned:?}"
);
assert!(
elapsed < Duration::from_secs(15),
"elapsed {elapsed:?} must stay in the git_timeout order (1s plus one watchdog \
teardown), not the 60s batch budget"
);
}
/// A `git_bin` wrapper that records every invocation's arguments and then execs the
/// real git, so a test can tell which objects the loop actually attempted. The returned
/// pin list cannot: it is empty both when the loop broke after one object and when it
@@ -650,8 +963,10 @@ mod tests {
"test-jwt",
&repo_path,
&git_bin,
Duration::from_secs(60),
oids,
&db,
"repo-merge-test",
Duration::from_secs(60),
),
)
@@ -722,8 +1037,10 @@ mod tests {
"test-jwt",
&repo_path,
&git_bin,
Duration::from_secs(60),
oids,
&db,
"repo-merge-test",
Duration::from_secs(60),
),
)
@@ -771,8 +1088,10 @@ mod tests {
"test-jwt",
&repo_path,
"git",
Duration::from_secs(60),
oids.clone(),
&db,
"repo-merge-test",
Duration::from_secs(60),
),
)
@@ -798,8 +1117,10 @@ mod tests {
"test-jwt",
&repo_path,
"git",
Duration::from_secs(60),
oids,
&db,
"repo-merge-test",
Duration::from_secs(60),
),
)
@@ -849,8 +1170,10 @@ mod tests {
"",
&repo_path,
fake.to_str().unwrap(),
Duration::from_secs(60),
oids,
&db,
"repo-merge-test",
Duration::from_secs(60),
),
)
@@ -865,6 +1188,406 @@ mod tests {
assert_eq!(requests.load(std::sync::atomic::Ordering::SeqCst), 0);
}
// ── Stalled-DB bound (F3, #173) ───────────────────────────────────────
//
// `api/repos.rs` acquires the GLOBAL `pin_semaphore` for the Pinata
// replication task and holds it across this whole function, the same permit
// the IPFS lane takes. `batch_budget_gate` only gates BETWEEN objects and the
// git read is already clamped, so a bare DB await in the region parked that
// permit for as long as the query was stuck. The tests below drive the stall
// with a `LOCK TABLE .. IN ACCESS EXCLUSIVE MODE` held on a dedicated pooled
// connection, the same technique the ipfs_pin twin's stall tests use, and copy
// their tolerances (a 1.5s budget, an `elapsed < 3s` assertion, a 10s outer
// wrap). The budget is above `PIN_READ_FLOOR` on purpose: below it
// `batch_budget_gate` breaks the batch as the loop body's FIRST statement, so a
// ~1s budget would never reach a DB call and the test would pass with the bound
// deleted.
// ---------------------------------------------------------------------
/// Take an `ACCESS EXCLUSIVE` lock on `table` on a dedicated pooled connection.
/// Every SELECT needs `ACCESS SHARE`, which conflicts, so the next statement
/// touching the table blocks at lock acquisition regardless of row count.
/// Copied from `ipfs_pin.rs`'s test mod rather than shared, since test mods are
/// private, the same way `seed_loose_blobs` and `capture_logs` are.
async fn lock_table(
pool: &sqlx::PgPool,
table: &str,
) -> sqlx::pool::PoolConnection<sqlx::Postgres> {
let mut conn = pool.acquire().await.unwrap();
sqlx::raw_sql(&format!(
"BEGIN; LOCK TABLE {table} IN ACCESS EXCLUSIVE MODE;"
))
.execute(&mut *conn)
.await
.unwrap();
conn
}
async fn rollback(conn: &mut sqlx::pool::PoolConnection<sqlx::Postgres>) {
sqlx::raw_sql("ROLLBACK")
.execute(&mut **conn)
.await
.unwrap();
}
/// Scenario 3: the FIRST DB call in this lane's budgeted region
/// (`has_pinata_cid`) stalls. With the batch deadline bounding it the loop
/// abandons the object, the budget gate then breaks the batch, and the call
/// returns at ~budget having uploaded nothing. Pre-fix the bare await blocks for
/// the lock's whole lifetime, holding the caller's global pin permit with it.
///
/// The upload mock is at `.expect(0)`: a stalled pinned-status check must never
/// fall through to an upload, since that would re-send bytes Pinata may already
/// hold and, worse, return a CID this node then advertises.
#[sqlx::test]
async fn pinata_pin_new_objects_stalled_db_returns_by_budget(pool: sqlx::PgPool) {
let db = crate::db::Db::for_testing(pool.clone());
db.run_migrations().await.expect("migrations");
let tmp = tempfile::TempDir::new().unwrap();
let repo_path = tmp.path().join("pinata_stalled.git");
let oids = seed_loose_blobs(&repo_path, 1);
let mut server = mockito::Server::new_async().await;
let upload = server
.mock("POST", mockito::Matcher::Any)
.with_status(200)
.with_body(r#"{"data":{"cid":"QmShouldNotHappen"}}"#)
.expect(0)
.create_async()
.await;
// Install a log capture even though nothing here asserts on it: `tracing`
// caches a callsite's interest globally the first time it is hit, and a hit
// from a thread with no subscriber caches it as never-interested for the whole
// binary, which silently blinds the sibling tests that DO assert on the batch
// deadline warn.
let (_logs, _log_guard) = capture_logs();
let mut lock = lock_table(&pool, "pinned_cids").await;
let client = reqwest::Client::new();
let started = std::time::Instant::now();
let pinned = tokio::time::timeout(
Duration::from_secs(10),
pin_new_objects(
&client,
&server.url(),
"test-jwt",
&repo_path,
"git",
Duration::from_secs(30),
oids,
&db,
"repo-pinata-stalled",
Duration::from_millis(1500),
),
)
.await
.expect(
"a stalled DB must cost the batch its budget, not the lock's lifetime: the \
bare await hangs past this wrap",
);
let elapsed = started.elapsed();
assert!(
pinned.is_empty(),
"a stalled pinata-status check cannot produce a pinned object: {pinned:?}"
);
assert!(
elapsed < Duration::from_secs(3),
"the batch deadline must end the call at ~budget (1.5s); got {elapsed:?}"
);
upload.assert_async().await;
rollback(&mut lock).await;
}
/// The Pinata half of the timed-out source record, driven rather than argued: the
/// twin's `pin_new_objects_skip_branch_stalled_record_returns_by_budget` covers the
/// Kubo lane and this covers the site that has to change in lockstep with it.
///
/// The object already has a `pinata_cid`, so the loop takes the skip branch and
/// tries to record this repo as an additional source; `pin_repo_sources` is locked
/// for the whole run, so that insert stalls inside `retry_db_record` and the whole
/// ladder elapses against one floored remainder. Two properties:
///
/// - the call still returns promptly, at the record floor rather than the lock's
/// lifetime;
/// - on the TIMEOUT arm the incomplete marker IS written. `record_pin_source` is an
/// explicit transaction, so the cancelled future never reaches `tx.commit()`, no
/// COMMIT is ever sent, and the source definitely did not land. Withholding the
/// marker there would leave the set incomplete AND unmarked, which the resolver
/// reads as complete and 404s a copy this repo would serve.
///
/// `pinned_cids` is deliberately NOT locked, so the marker write itself is free to
/// land and the assertion below is about the branch, not about lock contention.
#[sqlx::test]
async fn pinata_skip_branch_stalled_record_marks_incomplete(pool: sqlx::PgPool) {
let db = crate::db::Db::for_testing(pool.clone());
db.run_migrations().await.expect("migrations");
let tmp = tempfile::TempDir::new().unwrap();
let repo_path = tmp.path().join("pinata_skip_stalled.git");
let oids = seed_loose_blobs(&repo_path, 1);
let sha = oids[0].clone();
// Seed the object as already Pinata-pinned, by a DIFFERENT repo, so the skip
// branch is taken and the source record below is a genuine additional-source
// insert rather than a no-op on the conflict. The resolver key is a canonical
// raw CIDv1 so the opportunistic legacy repair takes its cost gate and reads no
// bytes.
let raw_cid =
gitlawb_core::cid::Cid::from_git_object_bytes(b"pinata skip seed").to_string();
db.record_pinata_cid(&sha, &raw_cid, "QmSeedProviderCid", Some("repo-seed"))
.await
.unwrap();
db.record_pin_source(&sha, "repo-seed").await.unwrap();
let requests = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
let endpoint = delaying_pinata_endpoint(
vec![Duration::from_millis(0)],
std::sync::Arc::clone(&requests),
)
.await;
// Install a log capture even though nothing here asserts on it: `tracing`
// caches a callsite's interest globally the first time it is hit, and a hit
// from a thread with no subscriber caches it as never-interested for the whole
// binary, which silently blinds the sibling tests that DO assert on the batch
// deadline warn.
let (_logs, _log_guard) = capture_logs();
let mut lock = lock_table(&pool, "pin_repo_sources").await;
let client = reqwest::Client::new();
let started = std::time::Instant::now();
let pinned = tokio::time::timeout(
Duration::from_secs(15),
pin_new_objects(
&client,
&endpoint,
"test-jwt",
&repo_path,
"git",
Duration::from_secs(30),
oids,
&db,
"repo-pinata-skip-stalled",
Duration::from_millis(1500),
),
)
.await
.expect(
"the wrapped retry ladder must fit inside one floored remainder: the bare \
retry_db_record hangs past this wrap",
);
let elapsed = started.elapsed();
assert!(
pinned.is_empty(),
"an already-pinned object is skipped, never re-uploaded: {pinned:?}"
);
assert_eq!(
requests.load(std::sync::atomic::Ordering::SeqCst),
0,
"the skip branch must not reach the upload at all"
);
assert!(
elapsed < Duration::from_secs(5),
"the record's floored remainder must end the call promptly; got {elapsed:?}"
);
rollback(&mut lock).await;
drop(lock);
assert!(
db.pin_sources_incomplete(&sha).await.unwrap(),
"a TIMED-OUT `record_pin_source` definitively did not land: it is an explicit \
multi-statement transaction, and the cancelled future never reaches \
`tx.commit()`, so no COMMIT is ever sent and the row cannot exist. The set is \
therefore incomplete, and leaving it UNMARKED is the exact state the marker \
exists to prevent: the resolver reads a non-empty below-cap set as complete \
and 404s a copy this repo would serve"
);
}
/// Scenario 8, the Pinata half of the durability floor. `batch_budget_gate` only
/// guarantees `PIN_READ_FLOOR` before an object STARTS and the upload runs under
/// the shared client's own ceiling, so a successful upload can return with ~0 of
/// the batch budget left. Without the floor the post-upload `record_pinata_cid`
/// would then be failed by a spent deadline, and this lane's `pinned.push` is
/// UNCONDITIONAL: `api/repos.rs` builds `cid_map` from the return and drives
/// `upsert_branch_cid` plus the p2p `publish_ref_update` gossip from it, so a
/// dropped record makes the node advertise a CID whose `/ipfs` read 404s.
///
/// Fixture: a 2s budget, a 1.7s upload, and `pinned_cids` locked from 500ms (well
/// after `has_pinata_cid` has read it, and still well before the upload returns)
/// until 2.4s. The record therefore starts at ~1.72s with ~280ms of budget left
/// and needs ~680ms of lock wait to land, which only the `DB_RECORD_GRACE` floor
/// buys it.
///
/// The lock time is a MARGIN, not a boundary: taking it at 100ms left
/// `has_pinata_cid` racing it on a loaded box, and losing that race makes the read
/// block, time out, and break the batch, which fails on `pinned.len() == 1` for a
/// reason that has nothing to do with the floor. Any time between the
/// `has_pinata_cid` round trip and the upload's 1.7s return proves the same thing.
#[sqlx::test]
async fn pinata_pin_add_with_spent_budget_still_records_cid(pool: sqlx::PgPool) {
let db = crate::db::Db::for_testing(pool.clone());
db.run_migrations().await.expect("migrations");
let tmp = tempfile::TempDir::new().unwrap();
let repo_path = tmp.path().join("pinata_spent_budget.git");
let oids = seed_loose_blobs(&repo_path, 1);
let sha = oids[0].clone();
let requests = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
let endpoint = delaying_pinata_endpoint(
vec![Duration::from_millis(1700)],
std::sync::Arc::clone(&requests),
)
.await;
// Install a log capture even though nothing here asserts on it: `tracing`
// caches a callsite's interest globally the first time it is hit, and a hit
// from a thread with no subscriber caches it as never-interested for the whole
// binary, which silently blinds the sibling tests that DO assert on the batch
// deadline warn.
let (_logs, _log_guard) = capture_logs();
let lock_pool = pool.clone();
let locker = async move {
tokio::time::sleep(Duration::from_millis(500)).await;
let mut conn = lock_table(&lock_pool, "pinned_cids").await;
tokio::time::sleep(Duration::from_millis(1900)).await;
rollback(&mut conn).await;
};
let client = reqwest::Client::new();
let pin = tokio::time::timeout(
Duration::from_secs(20),
pin_new_objects(
&client,
&endpoint,
"test-jwt",
&repo_path,
"git",
Duration::from_secs(30),
oids.clone(),
&db,
"repo-pinata-spent-budget",
Duration::from_millis(2000),
),
);
let (pinned, ()) = tokio::join!(pin, locker);
let pinned = pinned.expect("the floored record must land well inside this wrap");
assert_eq!(
requests.load(std::sync::atomic::Ordering::SeqCst),
1,
"the fixture only proves anything if the upload actually ran"
);
assert!(
db.has_pinata_cid(&sha).await.unwrap(),
"a successful upload whose batch deadline is spent must still land its \
pinned_cids row: this lane pushes the pair unconditionally, so a dropped \
record makes api/repos.rs advertise a CID the resolver cannot serve"
);
assert_eq!(
pinned.len(),
1,
"the uploaded pin must still be returned: {pinned:?}"
);
}
/// The POST-UPLOAD source record's timeout arm, the one site of the three that
/// nothing else executes. The skip-branch twin above and the ipfs_pin lane cover
/// the other two; this arm sits after a SUCCESSFUL `pin_object`, so no skip-branch
/// fixture can reach it.
///
/// Why it has to write the marker at all: `record_pin_source` is an explicit
/// transaction ending in `tx.commit()`, and a cancelled future never gets there, so
/// no COMMIT is sent and the row definitely does not exist. The set is therefore
/// incomplete, and leaving it unmarked is the state the marker exists to prevent.
///
/// Fixture: the object is NOT seeded as Pinata-pinned, so `has_pinata_cid` is false
/// and the run takes the upload path. The mock answers the upload at once, then
/// `pin_repo_sources` is held under `ACCESS EXCLUSIVE` for the whole run, so the
/// post-upload `record_pin_source` blocks and elapses against its floored bound at
/// ~2s. `pinned_cids` is deliberately left UNLOCKED, so both `record_pinata_cid`
/// and the marker write itself are free to land and the assertion is about the arm
/// rather than about my own lock.
///
/// The upload assertion is what keeps this from being vacuous: without it the test
/// would pass just as well if the run never reached the post-upload path at all.
#[sqlx::test]
async fn pinata_post_upload_stalled_record_marks_incomplete(pool: sqlx::PgPool) {
let db = crate::db::Db::for_testing(pool.clone());
db.run_migrations().await.expect("migrations");
let tmp = tempfile::TempDir::new().unwrap();
let repo_path = tmp.path().join("pinata_post_upload.git");
let oids = seed_loose_blobs(&repo_path, 1);
let sha = oids[0].clone();
let mut server = mockito::Server::new_async().await;
let upload = server
.mock("POST", mockito::Matcher::Any)
.with_status(200)
.with_body(r#"{"data":{"cid":"QmPostUploadProviderCid"}}"#)
.expect(1)
.create_async()
.await;
// Install a log capture even though nothing here asserts on it: `tracing`
// caches a callsite's interest globally the first time it is hit, and a hit
// from a thread with no subscriber caches it as never-interested for the whole
// binary, which silently blinds the sibling tests that DO assert on the batch
// deadline warn.
let (_logs, _log_guard) = capture_logs();
let mut lock = lock_table(&pool, "pin_repo_sources").await;
let client = reqwest::Client::new();
let started = std::time::Instant::now();
let pinned = tokio::time::timeout(
Duration::from_secs(20),
pin_new_objects(
&client,
&server.url(),
"test-jwt",
&repo_path,
"git",
Duration::from_secs(30),
oids,
&db,
"repo-pinata-post-upload",
Duration::from_millis(1500),
),
)
.await
.expect(
"the wrapped retry ladder must fit inside one floored remainder: the bare \
retry_db_record hangs past this wrap",
);
let elapsed = started.elapsed();
rollback(&mut lock).await;
drop(lock);
upload.assert_async().await;
assert_eq!(
pinned.len(),
1,
"the upload succeeded, so this lane still returns the pair: {pinned:?}"
);
assert!(
elapsed < Duration::from_secs(8),
"the record's floored remainder must end the call promptly, never at the \
lock's lifetime; got {elapsed:?}"
);
assert!(
db.pin_sources_incomplete(&sha).await.unwrap(),
"the POST-UPLOAD arm must mark the source set incomplete when its record \
times out: `record_pin_source` is an explicit transaction whose cancelled \
future never reaches `tx.commit()`, so the row definitely did not land, and \
an incomplete-and-unmarked set is read as complete and 404s a copy this \
repo would serve"
);
}
#[tokio::test]
async fn test_pin_skipped_when_jwt_empty() {
let client = reqwest::Client::new();
+28 -5
View File
@@ -121,11 +121,26 @@ impl RateLimiter {
true
}
/// Number of keys currently tracked. Tests use it to observe what a sweep
/// reclaimed; there is no production reader.
#[cfg(test)]
pub async fn tracked_keys(&self) -> usize {
self.state.lock().await.len()
/// Non-consuming check: is this key ALREADY at its limit for the current window?
/// Unlike [`check`], it records nothing and never inserts a new key — used to shed
/// expensive preparatory work (e.g. the `/ipfs/{cid}` legacy scan's O(repos) DB
/// preload) BEFORE it runs, without perturbing the per-unit budget the consuming
/// `check` maintains (#173, F3). An unknown key or a disabled limiter is not
/// throttled. Prunes the key's expired timestamps as a side effect (keeps state
/// tidy) but adds none, so it cannot itself fill or grow the map.
pub(crate) async fn is_throttled(&self, key: &str) -> bool {
if self.max_requests == 0 {
return false;
}
let now = Instant::now();
let mut state = self.state.lock().await;
if let Some(window) = state.get_mut(key) {
window
.timestamps
.retain(|t| now.duration_since(*t) < self.window);
return window.timestamps.len() >= self.max_requests;
}
false
}
pub async fn cleanup(&self) {
@@ -137,6 +152,14 @@ impl RateLimiter {
!w.timestamps.is_empty()
});
}
/// Number of distinct keys currently tracked. Test-only introspection so a
/// cross-module test can assert that a sweep actually evicted expired entries
/// and observe what it reclaimed. There is no production reader.
#[cfg(test)]
pub(crate) async fn tracked_keys(&self) -> usize {
self.state.lock().await.len()
}
}
/// Per-source concurrency cap derived from the write-pool size: one resolved client
+326 -7
View File
@@ -8,6 +8,19 @@ use crate::git::repo_store::RepoStore;
use crate::p2p::P2pHandle;
use crate::rate_limit::RateLimiter;
/// HKDF salt for [`AppState::derive_scan_token_key`]. A constant, not a secret: HKDF's
/// salt is a domain qualifier, and the confidentiality of the derived key rests entirely
/// on the node's private seed being the input keying material.
const SCAN_TOKEN_KEY_SALT: &[u8] = b"gitlawb/hkdf-salt/ipfs-scan-token";
/// HKDF `info` for [`AppState::derive_scan_token_key`]: the domain separation AND the
/// rotation handle in one string. Nothing else in the node derives from this label, so
/// the token key is unrelated to the signing key it shares an input with; bumping the
/// trailing version rotates every node's token key on its next boot, which invalidates
/// outstanding continuations (they simply fail to open and the caller restarts at the
/// front) and needs no migration, no config, and no change to this function.
const SCAN_TOKEN_KEY_INFO: &[u8] = b"gitlawb/ipfs-scan-token/v1";
#[derive(Clone, Debug)]
pub struct RefUpdateBroadcast {
pub repo: String,
@@ -66,6 +79,102 @@ pub struct AppState {
/// brake a push flood from a DID farm (one throwaway DID per repo), so the
/// push path throttles on the resolved client IP instead.
pub push_rate_limiter: RateLimiter,
/// Per-client-IP rate limiter for `close_issue`'s pre-lock snapshot path.
/// Distinct from `push_rate_limiter` so a flood of close attempts cannot
/// drain the receive-pack budget for the same source IP.
pub close_issue_rate_limiter: RateLimiter,
/// Per-client-IP ROUTE brake for `GET /ipfs/{cid}`: charged ONCE per request by the
/// `rate_limit_by_ip` middleware (server.rs), never inside the handler. It bounds
/// request RATE (the "requests per hour" contract of `GITLAWB_IPFS_RATE_LIMIT`) on
/// the non-farmable source IP, so an anonymous flood of the public route is capped.
/// The per-probe/per-walk WORK accounting the resolver does WITHIN a request draws
/// from the SEPARATE `ipfs_work_rate_limiter` below — the two cannot share one bucket
/// or a single request that spends a route token and then its own probe token off the
/// same bucket is admitted at the route and falsely shed mid-request (#173 round-10,
/// R6). Keyed by `push_limiter_trust`.
pub ipfs_rate_limiter: RateLimiter,
/// Per-client-IP WORK-budget limiter for the `GET /ipfs/{cid}` resolver's internal
/// fan-out: charged per legacy (NULL-provenance) PROBE (`acquire` + `cat-file`) and
/// per provenance-path WALK, and peeked non-consuming before the O(repos) legacy
/// preload. A legacy CID from the public pins index otherwise lets one request drive
/// O(repos) subprocess spawns and cold Tigris fetches, and repeat requests amplify
/// that across requests with zero limiter contact (INV-10, F3). Charging the work to
/// the non-farmable source IP bounds it. A bucket DISTINCT from the route brake above:
/// one request legitimately spends many work tokens (a full legacy scan is up to
/// `ipfs_max_legacy_probes` probes), so it must not double as the once-per-request
/// route bucket. Capacity is DERIVED from the route limit (`AppState::ipfs_work_budget`,
/// no separate operator knob), floored at the legacy-probe budget so a single default-
/// config deep search never self-throttles mid-scan. Keyed by `push_limiter_trust`.
pub ipfs_work_rate_limiter: RateLimiter,
/// Per-request ceiling on full-history reachability walks the CID resolver
/// may spawn (default `api::ipfs::MAX_HISTORY_WALKS_PER_REQUEST`). A field,
/// not a bare const, so tests can shrink it to exercise the cap cheaply;
/// production keeps the const default.
pub ipfs_max_history_walks: u32,
/// Per-request ceiling on legacy (NULL-provenance) repo probes in the CID
/// resolver's scan fallback (default `api::ipfs::MAX_LEGACY_PROBES_PER_REQUEST`).
/// Bounds the anonymous `acquire` + `cat-file` fan-out across the node (#173,
/// INV-10); a field for the same test-seam reason as `ipfs_max_history_walks`.
pub ipfs_max_legacy_probes: u32,
/// How many repo rows the CID resolver's legacy scan pulls per keyset page
/// (default `api::ipfs::LEGACY_SCAN_PAGE_ROWS`). Bounds the DATABASE-facing half
/// of the same fan-out `ipfs_max_legacy_probes` bounds on the probe side: without
/// it the scan materialized every repo row and every matching visibility rule
/// before spending a single probe (#173, INV-10). A field for the same test-seam
/// reason as the sibling caps.
pub ipfs_legacy_scan_page_rows: usize,
/// Per-request ceiling on how many repo ROWS the CID resolver's legacy scan may
/// fetch (default `api::ipfs::MAX_LEGACY_SCAN_ROWS_PER_REQUEST`, operator-tunable via
/// `GITLAWB_IPFS_MAX_LEGACY_SCAN_ROWS`). `ipfs_max_legacy_probes` bounds the PROBE
/// fan-out but only starts counting once a probe runs, and quarantine plus a
/// root-scope visibility deny both return before a probe or a visit is spent, so an
/// all-denying inventory paged the whole repo table at zero probes (#173 round 13, F2).
/// Truncating here sheds a retryable 503 carrying a sealed continuation token.
pub ipfs_max_legacy_scan_rows: usize,
/// Per-request ceiling on the BYTES of visibility rules the CID resolver's legacy
/// scan may retain (default `api::ipfs::MAX_LEGACY_SCAN_RULE_BYTES_PER_REQUEST`). The
/// row ceiling above bounds the row count but not the memory each row drags in: the
/// pager keeps every fetched page's rules for the whole request, and neither the
/// number of rules per repo nor the length of a rule's reader list is capped, so a
/// rule COUNT would be the wrong unit. Enforced by the rules QUERY
/// (`Db::list_visibility_rules_for_repos_bounded`) rather than by a sum taken once the
/// page has landed, so the oversized page is never materialized at all. Deliberately
/// NOT an operator knob (it is a
/// memory guard, not a reach tradeoff); a field only for the same test-seam reason as
/// the sibling caps.
pub ipfs_max_legacy_scan_rule_bytes: usize,
/// Key sealing the legacy scan's continuation tokens (INV-13), derived from the
/// node's persistent identity by [`AppState::derive_scan_token_key`].
///
/// The token is minted from a FETCHED row on a scan that served nothing, so by
/// construction that row is a private or quarantined repo the caller may not read:
/// its `created_at` and its `id` (which carries the owner's DID) are withheld
/// fields. The token is therefore AEAD-SEALED, never signed plaintext and never
/// base64-of-plaintext, since integrity is not confidentiality.
///
/// DERIVED, not random per boot. This reverses an earlier revision of this design,
/// which argued that derivation "would make old tokens valid across restarts for no
/// benefit and tie a throwaway transport secret to a long-lived signing key." Both
/// halves were wrong. Surviving a restart IS the benefit: the token is the ONLY way
/// a caller resumes a ladder, an unopenable one is treated as absent, and a caller
/// treated as absent silently restarts at the front of the scan. A node whose
/// inventory needs several ladder steps and which deploys more often than a caller
/// can climb therefore keeps that caller from ever reaching a holder, which
/// contradicts the reach bound the README states. And the tie to the signing key is
/// what the HKDF domain separation removes: the derived key is a one-way function of
/// the seed under an `info` string nothing else uses, so it is not the signing key
/// and cannot be worked back into one.
///
/// Persisting a random key instead would need a schema change for a value the node
/// already has on disk, so derivation is also the smaller mechanism. Rotation is a
/// bump of the version in [`SCAN_TOKEN_KEY_INFO`].
pub ipfs_scan_token_key: Arc<[u8; 32]>,
/// Hard ceiling on the byte size of an object `GET /ipfs/{cid}` will buffer and
/// serve (default `api::ipfs::MAX_SERVED_OBJECT_BYTES`). The serve reads via a
/// blocking `git cat-file` and buffers the whole object; without a bound a large
/// public blob could exhaust memory or block a runtime worker (#173, F6, INV-10).
/// A field for the same test-seam reason as the sibling caps.
pub ipfs_max_served_object_bytes: u64,
/// Which forwarded header (if any) the edge is trusted to set, for
/// resolving the push limiter's client-IP key. See `GITLAWB_TRUSTED_PROXY`.
/// Node-wide; also keys the two peer-sync limiters below.
@@ -214,13 +323,6 @@ pub struct AppState {
/// (`with_default_max_keys`, reject-before-insert) so a source-key farm cannot grow
/// it (INV-15).
pub git_ipfs_walk_per_caller: crate::rate_limit::PerCallerConcurrency,
/// Per-client-IP rate limiter for `GET /ipfs/{cid}`. The route is publicly
/// reachable and each request can drive a full-history git walk, so it carries a
/// per-IP flood brake in addition to the concurrency cap above — a rate limit
/// bounds request *rate*, the semaphore bounds concurrent slow holds (different
/// axes). Keyed on the resolved client IP via `push_limiter_trust`. Layered on the
/// `/ipfs` route via `rate_limit_by_ip`.
pub ipfs_rate_limiter: RateLimiter,
/// The `git` executable the served-git withheld-blob walk spawns. Production is
/// `"git"` (resolved via PATH); injectable so a fake `git` can drive the walk's
/// process-group teardown in handler tests without mutating the process-global
@@ -235,6 +337,22 @@ impl AppState {
self.shutdown_tx.subscribe()
}
/// Sweep expired entries from every per-IP/DID rate limiter. Driven by the
/// periodic cleanup task so a bounded limiter's key map sheds stale entries
/// instead of sitting near its cap until an inline capacity sweep reclaims
/// them. Every limiter on the state is swept here; adding a new limiter means
/// adding it to this list.
pub(crate) async fn sweep_rate_limiters(&self) {
self.rate_limiter.cleanup().await;
self.create_ip_rate_limiter.cleanup().await;
self.push_rate_limiter.cleanup().await;
self.close_issue_rate_limiter.cleanup().await;
self.ipfs_rate_limiter.cleanup().await;
self.ipfs_work_rate_limiter.cleanup().await;
self.sync_trigger_rate_limiter.cleanup().await;
self.peer_write_rate_limiter.cleanup().await;
}
/// Trigger graceful shutdown. Idempotent — calling more than once
/// has no effect. Returns `true` if this call was the one that
/// flipped the signal.
@@ -255,6 +373,130 @@ impl AppState {
pub fn is_shutting_down(&self) -> bool {
*self.shutdown_tx.borrow()
}
/// Legacy-probe budget wired from the `GITLAWB_IPFS_MAX_LEGACY_PROBES` operator
/// knob. The knob seeds `ipfs_max_legacy_probes` at construction so it controls the
/// per-request legacy (NULL-provenance) probe fan-out it advertises. It deliberately
/// does NOT feed the history-walk ceiling: that is governed by
/// `ipfs_max_repos_walked` under a `MAX_PIN_SOURCES + 1` floor, because a value
/// below the floor truncates a provenanced request with a full source set into a
/// false 503. The knob is `usize`, the field `u32`; the range cap (1_048_576) keeps
/// the cast lossless.
pub(crate) fn ipfs_legacy_probe_budget(config: &crate::config::Config) -> u32 {
config.ipfs_max_legacy_probes as u32
}
/// Legacy-scan ROW budget wired from the `GITLAWB_IPFS_MAX_LEGACY_SCAN_ROWS` knob,
/// the same helper shape as the probe budget above so the knob cannot become a
/// silent no-op if a struct literal drifts back to the bare constant.
pub(crate) fn ipfs_legacy_scan_row_budget(config: &crate::config::Config) -> usize {
config.ipfs_max_legacy_scan_rows
}
/// The key sealing legacy-scan continuation tokens (INV-13), DERIVED from the node's
/// persistent identity rather than minted per boot.
///
/// HKDF-SHA256 over the node's Ed25519 seed, the same key material
/// `load_or_create_keypair` reads back from its PKCS#8 PEM on every boot, so the
/// derived key is byte-identical after a restart or a rolling deploy.
///
/// Two properties the derivation has to carry, both load-bearing:
///
/// * DOMAIN SEPARATION. The `info` string below is unique to this use, so the
/// derived key is not the signing seed and is not any other secret derived from
/// it. HKDF is one-way, so a leaked token key yields nothing about the signing
/// key and cannot be turned against a signature.
/// * A VERSION component, carried in the same `info` string
/// ([`SCAN_TOKEN_KEY_INFO`]) alongside a fixed [`SCAN_TOKEN_KEY_SALT`]. Bumping
/// the version rotates every token key on the next boot without touching the
/// identity, the token format, or any caller, so rotation is a constant change
/// rather than a fork of this derivation.
pub(crate) fn derive_scan_token_key(keypair: &Keypair) -> [u8; 32] {
use hmac::{Hmac, Mac};
type HmacSha256 = Hmac<sha2::Sha256>;
let seed = keypair.to_seed();
// HKDF-Extract: PRK = HMAC(salt, ikm).
let mut extract =
HmacSha256::new_from_slice(SCAN_TOKEN_KEY_SALT).expect("HMAC takes a key of any size");
extract.update(seed.as_slice());
let prk = extract.finalize().into_bytes();
// HKDF-Expand, one block: T(1) = HMAC(PRK, info || 0x01). One 32-byte output
// needs exactly one block of SHA-256, so there is no counter loop to get wrong.
let mut expand =
HmacSha256::new_from_slice(prk.as_slice()).expect("HMAC takes a key of any size");
expand.update(SCAN_TOKEN_KEY_INFO);
expand.update(&[0x01]);
expand.finalize().into_bytes().into()
}
/// Work-budget capacity for [`ipfs_work_rate_limiter`](Self#structfield.ipfs_work_rate_limiter)
/// (R6, KTD6), DERIVED from the route limit rather than a new operator knob. The route
/// limiter (`ipfs_rate_limiter`) charges once per request; this separate bucket absorbs
/// the resolver's per-probe/per-walk work charges so both the route "requests per hour"
/// contract and the amplification bound hold. Floor: at least one complete COMBINED
/// resolution per window, the provenance phase's walks plus a full legacy search
/// (probes plus page tolls), so a single default-config resolution cannot
/// self-throttle part-way and recreate the admit-then-429 for a legitimate caller.
/// `GITLAWB_IPFS_RATE_LIMIT=0`
/// disables the route brake and this derived bucket alike (a 0-capacity limiter admits
/// everything).
///
/// The floor carries a WALK term (#173 round 15, F2). The provenance visibility walk
/// in `gate_and_serve` debits this SAME bucket (its `!legacy_scan` charge), once per
/// path-scoped source, before the legacy fallback runs at all, and the walk cap is
/// charged per phase. A floor counting only the search therefore under-sizes the
/// window by up to that cap exactly when the floor binds (a route limit set below
/// it): the provenance phase spends from the budget the floor reserved for the
/// search, the fallback 429s short of its configured reach, and the retry re-pays the
/// same provenance charges. The term is
/// `min(MAX_HISTORY_WALKS_PER_REQUEST, ipfs_max_repos_walked)` because that is what
/// the resolver's own `walk_cap` in `gate_and_serve` evaluates to. The two
/// expressions are separate, not one shared value: this one reads the CONSTANT
/// `crate::api::ipfs::MAX_HISTORY_WALKS_PER_REQUEST`, while `walk_cap` reads the
/// `AppState` seam `state.ipfs_max_history_walks`. They agree only because every
/// construction seeds that seam from that constant: the production one in `main.rs`,
/// and the two test ones in `auth`'s test module and `test_support`. Nothing
/// mechanically ties the two expressions beyond this comment and its counterpart at
/// `walk_cap` (no shared helper, no type, no assertion outside the one fixture that
/// pins the seam as a precondition), so the two `min()`s have to be moved together
/// by hand, and a construction that seeded the seam from anything else would size
/// the floor for a cap the resolver does not enforce. It reads the
/// CONSTANT and the CONFIG knob for the same reason the page term below reads the
/// constant page size: `ipfs_max_history_walks` is an `AppState` test seam, and
/// sizing a production floor from a seam would inflate the budget by whatever a test
/// chose.
///
/// The PROBE term is the LEGACY-PROBE knob, not `ipfs_max_repos_walked`. Those were
/// one field before the walk cap and the probe budget were split apart, and sizing
/// the probe term off the walk cap would silently read 64 instead of 256. That is a
/// statement about which knob sizes the PROBE term; it is not a claim that the walk
/// cap has no place in the floor, since the walk term above is added to this one
/// rather than substituted for it.
///
/// The floor also carries the scan's PAGE toll (#173 round 13, F2): every page the
/// legacy scan buys is charged to this same bucket, so a deep scan spends
/// `ceil(ipfs_max_legacy_scan_rows / LEGACY_SCAN_PAGE_ROWS)` tokens on pages on top
/// of its probes. Leaving those out would 429 an honest caller part-way down their
/// own continuation-token ladder, which is the F6 admit-then-429 shape in a new
/// place. The page term uses the CONSTANT page size, not `AppState`'s field: the
/// field is a test seam that shrinks pages to make paging observable, and sizing a
/// production floor from it would inflate the budget by whatever a test chose.
pub(crate) fn ipfs_work_budget(config: &crate::config::Config) -> usize {
if config.ipfs_rate_limit == 0 {
return 0;
}
let pages = config
.ipfs_max_legacy_scan_rows
.div_ceil(crate::api::ipfs::LEGACY_SCAN_PAGE_ROWS);
let walks = std::cmp::min(
crate::api::ipfs::MAX_HISTORY_WALKS_PER_REQUEST as usize,
config.ipfs_max_repos_walked,
);
config
.ipfs_rate_limit
.max(config.ipfs_max_legacy_probes + pages + walks)
}
}
/// Bounds the OUTSTANDING post-push encryption-task set by per-repo coalescing
@@ -1226,3 +1468,80 @@ mod repo_write_lease_tests {
}
}
}
#[cfg(test)]
mod scan_token_key_tests {
use super::AppState;
use gitlawb_core::identity::Keypair;
use gitlawb_core::scan_token::{open_scan_token, seal_scan_token, ScanPosition};
const CID: &str = "bafkreiscantokenkeyfixtureaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
fn pos() -> ScanPosition {
ScanPosition {
created_at_key: "2020-01-01T12:00:00+00:00".to_string(),
id: "did:key:z6MkScanTokenOwner/repo".to_string(),
// 40 hex: the production shape, since the node's own repos are sha1.
sha256_hex: "a1b2c3d4e5f60718293a4b5c6d7e8f9012345678".to_string(),
}
}
/// The RESTART case. A token minted before a restart must still open after it, or a
/// caller laddering a deep inventory is silently returned to the front of the scan on
/// every rolling deploy and can never reach a holder buried past one window.
///
/// The second key is derived from the identity RELOADED THROUGH ITS ON-DISK PEM,
/// which is exactly what `load_or_create_keypair` does on boot, so this exercises the
/// real restart path rather than a clone of the in-memory keypair.
#[test]
fn scan_token_key_survives_a_restart_of_the_same_identity() {
let kp = Keypair::generate();
let pem = kp.to_pem().expect("the identity serializes");
let reloaded = Keypair::from_pem(&pem).expect("the identity reloads");
let before = AppState::derive_scan_token_key(&kp);
let after = AppState::derive_scan_token_key(&reloaded);
let token = seal_scan_token(&before, CID, &pos(), i64::MAX - 1).expect("seal");
assert_eq!(
open_scan_token(&after, CID, &token, 0),
Some(pos()),
"a continuation minted before a restart must open after it: the node's \
identity is the same, so the derived sealing key must be too"
);
}
/// The must-not: a DIFFERENT node identity must derive a DIFFERENT key, so a token is
/// no more portable between nodes than it was when the key was random per boot.
#[test]
fn scan_token_key_does_not_open_under_a_different_identity() {
let mine = Keypair::generate();
let theirs = Keypair::generate();
let token = seal_scan_token(
&AppState::derive_scan_token_key(&mine),
CID,
&pos(),
i64::MAX - 1,
)
.expect("seal");
assert_eq!(
open_scan_token(&AppState::derive_scan_token_key(&theirs), CID, &token, 0),
None,
"a continuation sealed by one node must not open under another node's identity"
);
}
/// Domain separation, executed rather than asserted in prose: the derived token key
/// must not be the signing seed itself. Compromising a token key must not hand an
/// attacker the material that signs.
#[test]
fn scan_token_key_is_not_the_signing_seed() {
let kp = Keypair::generate();
assert_ne!(
AppState::derive_scan_token_key(&kp),
*kp.to_seed(),
"the token key must be a DERIVED secret, never the Ed25519 signing seed"
);
}
}
File diff suppressed because it is too large. Load diff
+25
View File
@@ -437,6 +437,31 @@ mod tests {
);
}
// #135 T1: a Mode-B rule on `/secret/**` must DENY the withheld directory's
// OWN path `/secret` (the `path == prefix` arm), not just strict descendants —
// otherwise get_by_cid's tree gate would serve the /secret tree object and leak
// its children. Pins parity with get_tree, which denies the /secret path.
#[test]
fn subtree_rule_denies_the_withheld_directory_itself() {
let reader = "did:key:z6MkReader";
let rules = [rule("/secret/**", VisibilityMode::B, &[reader])];
assert_eq!(
visibility_check(&rules, true, OWNER, None, "/secret"),
Decision::Deny,
"anon denied at the withheld directory's OWN path /secret"
);
assert_eq!(
visibility_check(&rules, true, OWNER, None, "/public"),
Decision::Allow,
"anon allowed at a sibling path outside the withheld subtree"
);
assert_eq!(
visibility_check(&rules, true, OWNER, Some(reader), "/secret"),
Decision::Allow,
"listed reader allowed at the withheld directory (caller-aware)"
);
}
// #153 regression: cross-method DID must still be denied even when the
// trailing segment collides with a bare owner key.
#[test]
+116 -31
View File
@@ -29,6 +29,7 @@ fn inv22_concurrency_gates_present_and_not_bypassed() {
let repos = src("api/repos.rs");
let smart_http = src("git/smart_http.rs");
let vis = src("git/visibility_pack.rs");
let ipfs = src("api/ipfs.rs");
// U1 / P1-a: run_bounded_git stands the watchdog down only after confirming the
// child actually terminated (WNOWAIT), not on the raw stdout-drain EOF — otherwise
@@ -68,6 +69,57 @@ fn inv22_concurrency_gates_present_and_not_bypassed() {
withheld_recipients_gated, which acquires git_encrypt_semaphore"
);
// U1 / R2 (#173 round-10): the path-scoped filtered-pack serve must thread the
// caller's AdmissionGuard through BOTH git stages so read + per-caller admission is
// held until the pack-objects group is reaped on disconnect, closing the cap bypass
// the plain upload_pack path already fixed. Two load-bearing markers: rev-list hands
// the disarmed guard back (its tuple return type), and build_filtered_pack forwards
// that guard into the pack-objects stage (the `admission` arg after the
// "pack-objects" label). Reverting either — dropping the guard between stages, or
// passing `None` to pack-objects — trips this.
assert!(
smart_http.contains("Result<(Vec<String>, Option<AdmissionGuard>)>")
&& smart_http.contains("\"pack-objects\",\n admission,"),
"U1/R2 gate missing: build_filtered_pack must thread the AdmissionGuard through \
rev-list -> pack-objects so the permits are held until the pack-objects group \
is reaped on disconnect (the path-scoped half of #174 P1-a)"
);
// U2 / R1: a cancelled or timed-out `GET /ipfs/{cid}` must release admission only
// after the blocking work it admitted has finished, not the instant the handler
// future drops (the /ipfs half of #174 P1-a). The mechanism is the shared
// `Arc<WalkAdmission>`: both walk permits are moved into it once per request and a
// clone rides every `spawn_blocking`, so the permits release when the LAST holder
// drops. Reverting to handler-local permits trips this; the per-site clones are
// bound separately by `inv22_ipfs_walk_admission_reaches_every_blocking_site`.
//
// This previously required the permits to be owned by a detached `tokio::spawn`
// running the whole pipeline. That closed the same bypass but kept an abandoned
// request's full legacy scan running against a held slot, so the merge of #173 and
// #174 kept the Arc and dropped the detached task.
assert!(
ipfs.contains("struct WalkAdmission")
&& ipfs.contains("let admission = std::sync::Arc::new(WalkAdmission {"),
"U2/R1 gate missing: get_by_cid must move both /ipfs admission permits into a \
shared Arc<WalkAdmission> whose clones ride the blocking work, so admission is \
released only once that work completes (the /ipfs half of #174 P1-a)"
);
// U2 / KTD2 (#173 round-10): the probe/read children on the /ipfs path must be the
// duration-bounded twins (process-group teardown via run_bounded_git), not the bare
// `store::object_type` / `read_object_content` (or an unbounded `cat-file -s`) a tokio
// timeout cannot cancel — otherwise a wedged cat-file lingers and pins the held
// admission past the deadline. Reverting any twin call site back to a bare read trips
// this.
assert!(
ipfs.contains("object_type_bounded(")
&& ipfs.contains("object_size_bounded(")
&& ipfs.contains("read_object_content_bounded("),
"U2/KTD2 gate missing: the /ipfs probe+read must call the run_bounded_git-backed \
*_bounded twins so a wedged cat-file is reaped at the deadline, not left to pin \
the held /ipfs walk admission"
);
// P1-e non-bypass tripwire: the bounded recipients walk is spawn_blocking'd nowhere
// but inside withheld_recipients_gated. A second call site (count > 1) is a new
// detached git walk that skips the admission gate — exactly the class U5 closed.
@@ -95,8 +147,12 @@ fn inv22_concurrency_gates_present_and_not_bypassed() {
// recovery copies are absent until an unrelated later push. Scan only the
// production half of the file — the u5 tests in its `mod tests` also name the
// drain call, and matching them would make this check vacuous.
// Split at the TEST MODULE, not at the first `#[cfg(test)]`. api/repos.rs carries
// test-only items (the drain fault seam, the task entry point) ABOVE the production
// code these gates scan for, so splitting on the attribute truncated the production
// half above every line being checked and the gate went vacuously blind.
let repos_production = repos
.split("#[cfg(test)]")
.split("\nmod tests {")
.next()
.expect("split always yields a first chunk");
assert!(
@@ -169,7 +225,8 @@ fn f4_release_keeps_conn_owned_until_unlock_resolves() {
let repo_store = src("git/repo_store.rs");
let rel_start = repo_store
.find("pub async fn release(mut self")
.find("pub async fn release(self")
.or_else(|| repo_store.find("pub async fn release(mut self"))
.expect("F4 gate: repo_store.rs no longer defines RepoWriteGuard::release");
let rel_end = repo_store[rel_start..]
.find("impl Drop for RepoWriteGuard")
@@ -201,10 +258,10 @@ fn f4_release_keeps_conn_owned_until_unlock_resolves() {
);
}
/// F6/KTD-5 (initial IPFS metadata queries deadline-wrapped): `get_by_cid` acquires
/// the scarce walk permits (RAII, held for the whole request) BEFORE its two initial
/// metadata queries, and the per-repo loop's first budget gate runs only later. So
/// both `list_all_repos` and `list_visibility_rules_for_repos` must be clamped to the
/// F6/KTD-5 (IPFS metadata queries deadline-wrapped): `get_by_cid` acquires
/// the scarce walk permits (RAII, held for the whole request) BEFORE its metadata
/// queries, and the per-repo loop's first budget gate runs only later. So
/// both `list_repos_page_for_scan` and `list_visibility_rules_for_repos` must be clamped to the
/// remaining request budget — otherwise a query blocked in Postgres pins the walk slot
/// for the whole stall, past the budget. This scans the PRODUCTION half of `api/ipfs.rs`
/// (the `mod tests` half names the same calls in its own harness and would make the
@@ -214,31 +271,44 @@ fn f4_release_keeps_conn_owned_until_unlock_resolves() {
#[test]
fn f6_ipfs_metadata_queries_are_deadline_wrapped() {
let ipfs = src("api/ipfs.rs");
// Split at the TEST MODULE, not at the first `#[cfg(test)]`: the handler carries
// in-line `#[cfg(test)]` query counters, so splitting on the attribute cut the
// production half off above the calls this guard exists to check and the scan went
// vacuously quiet (found 0 of each rather than failing on a missing wrapper).
let production = ipfs
.split("#[cfg(test)]")
.split("\nmod tests {")
.next()
.expect("split always yields a first chunk");
for call in [".list_all_repos()", ".list_visibility_rules_for_repos("] {
assert_eq!(
production.matches(call).count(),
1,
"F6 guard stale: `{call}` must appear exactly once in the production half \
of api/ipfs.rs (the deadline-wrapped handler call) — update this guard"
);
let idx = production
.find(call)
.unwrap_or_else(|| panic!("F6 gate: api/ipfs.rs no longer calls `{call}`"));
// The wrapper opens a few lines above the call (match tokio::time::timeout( ...
// remaining budget ..., <query> )); a 240-char lookback covers it without
// reaching the previous statement.
let window = &production[idx.saturating_sub(240)..idx];
// EVERY occurrence must be wrapped, not just the first. The handler now reaches
// these queries from two places (the provenance fast path, one repo at a time, and
// the legacy scan's preload), and an exact-count check would have to be relaxed
// every time a call site is added, which is how a guard quietly stops covering the
// site that matters. Requiring all of them scales with the code instead.
for call in [
".list_repos_page_for_scan(",
".list_visibility_rules_for_repos(",
".get_repo_by_id(",
".is_repo_quarantined(",
] {
let occurrences = production.matches(call).count();
assert!(
window.contains("tokio::time::timeout("),
"F6 gate missing: `{call}` must be wrapped in tokio::time::timeout(...) \
clamped to the remaining request budget. An unwrapped await pins the held \
walk permit for the whole DB stall, past GITLAWB_IPFS_REQUEST_BUDGET_SECS."
occurrences >= 1,
"F6 guard stale: api/ipfs.rs no longer calls `{call}` — update this guard"
);
for (n, (idx, _)) in production.match_indices(call).enumerate() {
// The wrapper opens a few lines above the call (match tokio::time::timeout(
// ... remaining budget ..., <query> )); a 240-char lookback covers it
// without reaching the previous statement.
let window = &production[idx.saturating_sub(240)..idx];
assert!(
window.contains("tokio::time::timeout("),
"F6 gate missing: occurrence {n} of `{call}` (of {occurrences}) is not \
wrapped in tokio::time::timeout(...) clamped to the remaining request \
budget. An unwrapped await pins the held walk permit for the whole DB \
stall, past GITLAWB_IPFS_REQUEST_BUDGET_SECS."
);
}
}
}
@@ -261,8 +331,12 @@ fn f6_ipfs_metadata_queries_are_deadline_wrapped() {
#[test]
fn f2_pinata_enqueues_refs_not_retained_object_lists() {
let repos = src("api/repos.rs");
// Split at the TEST MODULE, not at the first `#[cfg(test)]`. api/repos.rs carries
// test-only items (the drain fault seam, the task entry point) ABOVE the production
// code these gates scan for, so splitting on the attribute truncated the production
// half above every line being checked and the gate went vacuously blind.
let production = repos
.split("#[cfg(test)]")
.split("\nmod tests {")
.next()
.expect("split always yields a first chunk");
@@ -321,8 +395,12 @@ fn f2_pinata_enqueues_refs_not_retained_object_lists() {
fn f3_second_writer_leased_until_reap() {
let repos = src("api/repos.rs");
let smart_http = src("git/smart_http.rs");
// Split at the TEST MODULE, not at the first `#[cfg(test)]`. api/repos.rs carries
// test-only items (the drain fault seam, the task entry point) ABOVE the production
// code these gates scan for, so splitting on the attribute truncated the production
// half above every line being checked and the gate went vacuously blind.
let repos_production = repos
.split("#[cfg(test)]")
.split("\nmod tests {")
.next()
.expect("split always yields a first chunk");
@@ -372,7 +450,7 @@ fn f3_second_writer_leased_until_reap() {
/// exists: it binds all three sites, and any blocking site added to this loop
/// later, without reversing that deliberate independence.
///
/// MUTATION (RED): delete any one `Arc::clone(&admission)` binding, or drop the
/// MUTATION (RED): delete any one `Arc::clone(ctx.admission)` binding, or drop the
/// clone from inside its closure, and the count falls below three.
#[test]
fn inv22_ipfs_walk_admission_reaches_every_blocking_site() {
@@ -386,7 +464,10 @@ fn inv22_ipfs_walk_admission_reaches_every_blocking_site() {
);
// Every blocking site in the scan takes its own clone...
let clones = ipfs.matches("Arc::clone(&admission)").count();
// `Arc::clone(ctx.admission)` in the gate, `Arc::clone(&admission)` if a site is
// ever added back in the handler body itself; count both spellings.
let clones = ipfs.matches("Arc::clone(ctx.admission)").count()
+ ipfs.matches("Arc::clone(&admission)").count();
assert!(
clones >= 3,
"U1 gate bypassed: expected an admission clone for each of the three \
@@ -443,8 +524,12 @@ fn inv22_ipfs_walk_admission_reaches_every_blocking_site() {
fn inv22_replication_tail_spawns_at_the_durability_boundary() {
let repos = src("api/repos.rs");
// Production half only — the tests below name these identifiers too.
// Split at the TEST MODULE, not at the first `#[cfg(test)]`. api/repos.rs carries
// test-only items (the drain fault seam, the task entry point) ABOVE the production
// code these gates scan for, so splitting on the attribute truncated the production
// half above every line being checked and the gate went vacuously blind.
let production = repos
.split("#[cfg(test)]")
.split("\nmod tests {")
.next()
.expect("split always yields a first chunk");
@@ -460,7 +545,7 @@ fn inv22_replication_tail_spawns_at_the_durability_boundary() {
.find("tokio::spawn(post_receive_replication_tail(")
.expect("U5 gate missing: the replication tail must be spawned by git_receive_pack");
let release = production
.find("guard.release(push_succeeded)")
.find(".release(push_succeeded)")
.expect("U5 gate stale: release must consume the same success flag as the tail gate");
let touch = production
.find("state.db.touch_repo(")
+1 -1
View File
@@ -11,7 +11,7 @@ name = "gl"
path = "src/main.rs"
[dependencies]
gitlawb-core = { path = "../gitlawb-core" }
gitlawb-core = { path = "../gitlawb-core", features = ["redirect"] }
icaptcha-client = { path = "../icaptcha-client" }
base64 = { workspace = true }
tokio = { workspace = true }
+259 -8
View File
@@ -12,9 +12,58 @@ use serde::Deserialize;
use std::path::Path;
use std::process::Command;
use crate::http::NodeClient;
use crate::http::{sanitize_node_msg, NodeClient};
use crate::identity::load_keypair_from_dir;
/// Report why one candidate blob could not be recovered, in the register of the
/// recovery loop's other warnings.
///
/// Both halves are gateway-supplied text on their way to a terminal: the oid comes
/// out of a manifest fetched from a caller-chosen Arweave gateway, and a transport
/// error carries the URL the same manifest chose. So the whole line is defanged and
/// length-capped, like every other node/gateway string this crate prints.
///
/// Under `cfg(test)` the line is also mirrored into a per-thread buffer, so a test
/// can assert the loop actually says what it skipped rather than only that it
/// returned nothing.
fn warn_skip(oid: &str, why: &str) {
emit_warning(&format!(
"warning: could not fetch encrypted blob {oid}: {why}; skipping"
));
}
/// The same report for a manifest rather than a blob. The manifest is one step
/// earlier in the same recovery: losing it silently means every blob it would have
/// named is missing with no reason given anywhere.
fn warn_skip_manifest(id: &str, why: &str) {
emit_warning(&format!(
"warning: could not fetch blob manifest {id}: {why}; skipping"
));
}
/// Sanitize a warning and write it, once.
///
/// The write goes to [`warn_sink`], which is stderr in a normal build and the tests'
/// per-thread mirror under `cfg(test)`. That indirection is the point: the mirror
/// used to sit BESIDE an `eprintln!`, so deleting the user-visible write left every
/// assertion on the mirror green and the shipped behaviour could be removed with the
/// tests unchanged. There is one write now, and the tests observe it.
fn emit_warning(line: &str) {
use std::io::Write;
let line = sanitize_node_msg(line);
let _ = writeln!(warn_sink(), "{line}");
}
#[cfg(not(test))]
fn warn_sink() -> impl std::io::Write {
std::io::stderr()
}
#[cfg(test)]
fn warn_sink() -> impl std::io::Write {
tests::WarnMirror
}
#[derive(Args)]
pub struct CloneArgs {
/// Repo to clone: gitlawb://<owner_did>/<name> or <owner>/<name>.
@@ -312,10 +361,25 @@ async fn recover_encrypted_blobs(
.await
{
Ok(r) if r.status().is_success() => r,
_ => continue,
// The node path has the same silent exit the gateway path had: a 403, a
// 404, or a dead connection all reached the caller as "blob not
// recoverable" with no reason attached. One unreachable blob still must
// not end the recovery of the rest, so it warns and moves on.
Ok(r) => {
warn_skip(oid, &format!("node returned {}", r.status()));
continue;
}
Err(e) => {
warn_skip(oid, &format!("node request failed: {e}"));
continue;
}
};
let Ok(envelope) = env_resp.bytes().await else {
continue;
let envelope = match env_resp.bytes().await {
Ok(b) => b,
Err(e) => {
warn_skip(oid, &format!("reading the envelope failed: {e}"));
continue;
}
};
let plaintext = match open_blob(&envelope, keypair) {
Ok(p) => p,
@@ -642,7 +706,14 @@ async fn recover_from_arweave(
for r in refs {
let m = match client.get(format!("{ag}/{}", r.id)).send().await {
Ok(resp) if resp.status().is_success() => resp,
_ => continue,
Ok(resp) => {
warn_skip_manifest(&r.id, &format!("gateway returned {}", resp.status()));
continue;
}
Err(e) => {
warn_skip_manifest(&r.id, &format!("gateway request failed: {e}"));
continue;
}
};
if let Ok(parsed) = m.json::<Manifest>().await {
manifests.push((parsed, r.height));
@@ -684,10 +755,31 @@ async fn recover_from_arweave(
}
let env_resp = match client.get(format!("{ig}/ipfs/{cid}")).send().await {
Ok(r) if r.status().is_success() => r,
_ => continue,
// Every other outcome used to leave through a bare `continue`, so a
// gateway that answered 503, 404, or nothing at all was reported to the
// caller as "blob not recoverable" with no reason attached. This is the
// second in-repo client of GET /ipfs/{cid} and it has no resume ladder,
// so saying what happened is all the recourse there is. It still skips to
// the next candidate either way: one unreachable blob must not end the
// recovery of the rest.
Ok(r) => {
warn_skip(&oid, &format!("gateway returned {}", r.status()));
continue;
}
Err(e) => {
warn_skip(&oid, &format!("gateway request failed: {e}"));
continue;
}
};
let Ok(envelope) = env_resp.bytes().await else {
continue;
// A body that dies part-way through is the same mid-read failure the capped
// read exists to stop rendering as silence, and it sits INSIDE the loop whose
// status arms were already fixed.
let envelope = match env_resp.bytes().await {
Ok(b) => b,
Err(e) => {
warn_skip(&oid, &format!("reading the envelope failed: {e}"));
continue;
}
};
// open_blob succeeds only if this caller is a recipient: this is the
// authorization gate (no node, no DID check needed).
@@ -801,9 +893,42 @@ pub async fn run(args: CloneArgs) -> Result<()> {
#[cfg(test)]
mod tests {
use super::*;
use std::cell::RefCell;
use std::process::Command;
use tempfile::TempDir;
thread_local! {
/// Test-visible copy of the stderr warnings `warn_skip` emits.
/// `#[tokio::test]` runs the future on the test's own thread, so a
/// thread-local is enough.
static WARNINGS: RefCell<String> = const { RefCell::new(String::new()) };
}
/// The `cfg(test)` warning sink. `emit_warning` writes here instead of to
/// stderr, so the assertions below observe the shipped write rather than a copy
/// made beside it: delete that write and they go red.
pub(super) struct WarnMirror;
impl std::io::Write for WarnMirror {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
let text = String::from_utf8_lossy(buf).into_owned();
WARNINGS.with(|w| w.borrow_mut().push_str(&text));
Ok(buf.len())
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
fn reset_warnings() {
WARNINGS.with(|w| w.borrow_mut().clear());
}
fn warnings() -> String {
WARNINGS.with(|w| w.borrow().clone())
}
fn g(args: &[&str], dir: &Path) {
assert!(Command::new("git")
.args(args)
@@ -1255,6 +1380,132 @@ mod tests {
assert_eq!(a.get("o1").map(String::as_str), Some("cidTS"));
}
/// A gateway that refuses one blob must SAY so, and must not take the rest of the
/// recovery down with it.
///
/// The IPFS fetch used to leave through a bare `_ => continue` on every
/// non-success, so a 503 (the truncation status a tuned node makes more likely),
/// a 404, and a dead connection all reached the caller as the same silent "blob
/// not recoverable". Two withheld blobs here: the gateway answers 503 for the
/// first and serves the second. The recovered path proves the loop continued; the
/// warning proves the skip was reported and names both the object and the status.
#[tokio::test]
async fn recover_from_arweave_reports_a_refusing_gateway_and_continues() {
use gitlawb_core::encrypt::seal_blob;
use gitlawb_core::identity::Keypair;
reset_warnings();
let (td, url) = bare_remote(&[
("public/a.txt", b"pub\n"),
("secret/b.txt", b"SECRET B\n"),
("secret/c.txt", b"SECRET C\n"),
]);
let dest = td.path().join("dest");
let bare = url.strip_prefix("file://").unwrap();
assert!(Command::new("git")
.args(["-C", bare, "config", "uploadpack.allowFilter", "true"])
.status()
.unwrap()
.success());
setup_partial_clone(&dest, &url, &["/secret/**".to_string()], &[], None).unwrap();
let oid_of = |path: &str| {
let out = Command::new("git")
.args([
"-C",
dest.to_str().unwrap(),
"rev-parse",
&format!("HEAD:{path}"),
])
.output()
.unwrap();
String::from_utf8_lossy(&out.stdout).trim().to_string()
};
let refused_oid = oid_of("secret/b.txt");
let served_oid = oid_of("secret/c.txt");
// Origin death, so recovery has to go through the gateways.
std::fs::remove_dir_all(bare).unwrap();
let reader = Keypair::generate();
let envelope = seal_blob(b"SECRET C\n", &[reader.verifying_key()]).unwrap();
let refused_cid = "cidrefused";
let served_cid = "cidserved";
let mut server = mockito::Server::new_async().await;
let _gql = server
.mock("POST", "/graphql")
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"data":{"transactions":{"edges":[{"node":{"id":"TX1"}}]}}}"#)
.create_async()
.await;
let manifest_body = serde_json::json!({
"timestamp": "2026-06-11T00:00:00Z",
"blobs": [
{ "oid": refused_oid, "cid": refused_cid },
{ "oid": served_oid, "cid": served_cid },
],
})
.to_string();
let _tx = server
.mock("GET", "/TX1")
.with_status(200)
.with_header("content-type", "application/json")
.with_body(manifest_body)
.create_async()
.await;
let refusal = server
.mock("GET", format!("/ipfs/{refused_cid}").as_str())
.with_status(503)
.with_header("content-type", "application/json")
.with_body(r#"{"error":"search_incomplete","message":"scan truncated"}"#)
.expect(1)
.create_async()
.await;
let ok = server
.mock("GET", format!("/ipfs/{served_cid}").as_str())
.with_status(200)
.with_body(envelope)
.expect(1)
.create_async()
.await;
let paths = recover_from_arweave(
&server.url(),
&server.url(),
"alice",
"myrepo",
&dest,
&reader,
)
.await
.unwrap();
refusal.assert_async().await;
ok.assert_async().await;
assert_eq!(
paths,
vec!["secret/c.txt".to_string()],
"a refused candidate must not stop the loop reaching the next one"
);
let warnings = warnings();
assert!(
warnings.contains(&refused_oid),
"the skip must name the object it could not fetch, got: {warnings}"
);
assert!(
warnings.contains("503"),
"the skip must name the gateway's status rather than failing silently, \
got: {warnings}"
);
assert!(
!warnings.contains(&served_oid),
"the blob that was served must not be reported as skipped, got: {warnings}"
);
}
/// Read-path end-to-end over a mocked Arweave + IPFS gateway: discover the
/// manifest via GraphQL, fetch it, fetch the envelope, decrypt with the
/// caller's key, and install the previously-withheld blob.
+682 -1
View File
@@ -15,6 +15,47 @@ use icaptcha_client::IcaptchaCfg;
/// (absorbs proof expiry / first-seen replay).
const MAX_ICAPTCHA_RETRIES: usize = 2;
/// Total request timeout: from the start of connecting through the end of the
/// response body, so it bounds a slow download and not just a slow handshake.
const TOTAL_REQUEST_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
/// Follow a redirect only when it stays on the origin that issued it AND re-issues the
/// identical request-target, and only for as long as the chain bound allows.
///
/// The decision itself is [`gitlawb_core::redirect::may_follow`], shared with
/// `git-remote-gitlawb` so the two signing clients cannot drift apart on it. Refusal
/// is `stop`, not `error`: the 3xx comes back as an ordinary response and each caller
/// reports it through the status path it already has.
///
/// `Policy::custom` replaces reqwest's built-in limit, so the chain bound is restated
/// here. Same-origin redirects can cycle, and this is what stops a node answering 302
/// to itself from being followed indefinitely. It is not what makes the request
/// finite: `.timeout(...)` on the same builder is a TOTAL request timeout covering the
/// whole chain, so without this bound the worst case is a 30 second spin, not an
/// endless one. The bound is what keeps that spin from costing the node a request per
/// round trip for the full 30 seconds.
///
/// `>` and not `>=`: reqwest pushes the redirecting URL onto `previous` before
/// consulting the policy, so on the first redirect `previous.len()` is already 1, and
/// `>=` would permit `MAX_REDIRECTS - 1` follows. `Policy::limited(max)` refuses at
/// `previous.len() > max`, and matching it is the point of reusing its value.
fn same_origin_redirect(attempt: reqwest::redirect::Attempt<'_>) -> reqwest::redirect::Action {
let Some(previous) = attempt.previous().last() else {
// No previous URL to compare against. Unreachable through reqwest, which
// pushes the redirecting URL before consulting the policy, but the safe
// reading of "cannot prove same-origin" is to refuse.
return attempt.stop();
};
if attempt.previous().len() > gitlawb_core::redirect::MAX_REDIRECTS {
return attempt.stop();
}
if gitlawb_core::redirect::may_follow(previous, attempt.url()) {
attempt.follow()
} else {
attempt.stop()
}
}
pub struct NodeClient {
inner: reqwest::Client,
pub node_url: String,
@@ -23,8 +64,20 @@ pub struct NodeClient {
impl NodeClient {
pub fn new(node_url: impl Into<String>, keypair: Option<Keypair>) -> Self {
Self::with_timeout(node_url, keypair, TOTAL_REQUEST_TIMEOUT)
}
/// `new` with the total request timeout as a parameter, so a test can drive the
/// timeout's behaviour without waiting out the shipped value. `new` supplies the
/// shipped one, which is what makes the scaled-down test cover the real client.
fn with_timeout(
node_url: impl Into<String>,
keypair: Option<Keypair>,
timeout: std::time::Duration,
) -> Self {
let inner = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(30))
.timeout(timeout)
.redirect(reqwest::redirect::Policy::custom(same_origin_redirect))
.user_agent(format!("gl/{} gitlawb-cli", env!("CARGO_PKG_VERSION")))
.build()
.expect("failed to build HTTP client");
@@ -203,6 +256,79 @@ async fn obtain_proof(cfg: IcaptchaCfg) -> Result<String> {
.context("iCaptcha solver task panicked")?
}
/// What a capped body read produced. `text` is the bytes that arrived; the two flags
/// say why the read stopped where it did, which a caller that CLASSIFIES on the body
/// cannot work out from the text alone.
pub(crate) struct CappedBody {
/// The bytes read, lossily decoded.
pub(crate) text: String,
/// The cap cut the body short.
pub(crate) truncated: bool,
/// A chunk read FAILED part-way through, so the body is not merely short, it is
/// unfinished and the node may have had more to say.
pub(crate) read_failed: bool,
}
/// Read at most `cap` bytes of a response body. Bounds the allocation from a
/// hostile or broken node returning a huge error body — the display is capped
/// separately, but the read itself must not be unbounded (INV-6, read half).
///
/// `truncated` reports whether the cap cut the body short. A caller that CLASSIFIES
/// on the body needs it: a cut body fails JSON parse, and a parse failure is
/// indistinguishable from a node that sent no code at all, so without this flag an
/// oversized body silently picks a different arm.
///
/// `read_failed` reports the other way a body can end early. A mid-body read error
/// used to leave through the same exit as a clean end of stream, so a 500 whose body
/// died in transit surfaced as an empty message and the caller was told
/// `node returned 500: ` with nothing after the colon. That is a report of what the
/// node said, and the node never got to say it.
pub(crate) async fn read_body_capped(mut resp: reqwest::Response, cap: usize) -> CappedBody {
let mut buf: Vec<u8> = Vec::new();
let mut truncated = false;
let mut read_failed = false;
while buf.len() < cap {
match resp.chunk().await {
Ok(Some(chunk)) => {
let take = (cap - buf.len()).min(chunk.len());
buf.extend_from_slice(&chunk[..take]);
if take < chunk.len() {
truncated = true;
break; // hit the cap mid-chunk
}
}
Ok(None) => break, // clean end of body
Err(_) => {
read_failed = true;
break;
}
}
}
// A body that lands exactly on the cap may or may not have more behind it;
// report it as cut, since the classification that follows cannot tell either.
if buf.len() >= cap {
truncated = true;
}
CappedBody {
text: String::from_utf8_lossy(&buf).into_owned(),
truncated,
read_failed,
}
}
/// Strip terminal-dangerous characters from (and cap the length of) a
/// node-supplied error string before surfacing it. The node a caller talks to
/// could be hostile and embed escape sequences in its error body; those must not
/// reach the terminal verbatim (INV-6). We drop the C0/C1 control bytes (which
/// defangs ANSI/OSC escapes) AND the Unicode bidi/format controls (which
/// `char::is_control` does not cover — they can reorder the displayed line).
pub(crate) fn sanitize_node_msg(s: &str) -> String {
s.chars()
.filter(|c| !c.is_control() && !gitlawb_core::sanitize::is_bidi_format(*c))
.take(200)
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
@@ -587,4 +713,559 @@ mod tests {
ic.challenge.assert();
ic.answer.assert();
}
// ── redirect policy ─────────────────────────────────────────────────
/// The signed headers must not survive a redirect off the node's origin.
///
/// reqwest strips only `Authorization`, `Cookie`, `Proxy-Authorization` and
/// `WWW-Authenticate` across hosts, so `Signature` and `Signature-Input` used to
/// ride a 302 straight to whatever origin the node named. The signature binds
/// `@method`, `@path` and `content-digest` and nothing about the authority, so the
/// receiving host holds a credential that reads path-scoped objects as the caller
/// at any node until the clock-skew window closes.
///
/// Two mockito servers are two ports on one host, which is exactly the boundary
/// this policy draws (and the one reqwest's own header stripping draws). The
/// second server answers everything and expects nothing: a followed redirect
/// fails the expectation whether or not the signature came with it. MUTATION
/// (RED): drop the `.redirect(...)` line and the second server is hit.
#[tokio::test]
async fn cross_origin_redirect_is_not_followed() {
let mut elsewhere = Server::new_async().await;
let never = elsewhere
.mock("GET", mockito::Matcher::Any)
.with_status(200)
.with_body("bytes from the redirect target")
.expect(0)
.create_async()
.await;
let signature_seen = elsewhere
.mock("GET", mockito::Matcher::Any)
.match_header("signature", mockito::Matcher::Any)
.with_status(200)
.expect(0)
.create_async()
.await;
let mut node = Server::new_async().await;
let bounce = node
.mock("GET", "/api/v1/thing")
.with_status(302)
.with_header("location", &format!("{}/api/v1/thing", elsewhere.url()))
.expect(1)
.create_async()
.await;
let client = NodeClient::new(node.url(), Some(test_keypair()));
let resp = client.get_signed("/api/v1/thing").await.unwrap();
assert_eq!(
resp.status(),
302,
"a refused redirect stops rather than errors, so the caller sees the 3xx \
and reports it through the status path it already has"
);
let body = resp.text().await.unwrap();
assert!(
!body.contains("bytes from the redirect target"),
"the redirect target's bytes must never reach the caller, got: {body}"
);
bounce.assert_async().await;
never.assert_async().await;
signature_seen.assert_async().await;
}
/// A node redirecting to itself is same-origin, so the origin predicate follows it
/// every time and only the chain bound ends the loop. Deleting the bound left the
/// whole suite green, because nothing here had ever built a cycle.
///
/// It has a second job now. A redirect back to the identical path and query is the
/// only same-origin hop the predicate still follows, so the eleven hits below are
/// also this crate's executed proof that such a hop IS followed. The old positive
/// fixture drove a trailing-slash rewrite, which the request-target rule refuses,
/// and an http-to-https upgrade cannot be mocked over mockito's plain http.
///
/// The route answers 301 pointing back at itself. Bounded, the handler is hit
/// once for the original request plus `MAX_REDIRECTS` follows and the call returns
/// the 301 (a refused redirect stops rather than errors). Unbounded, it runs until
/// the client's total request timeout cuts it off, which is a 30 second spin at the
/// shipped value and a request per round trip for the node.
///
/// MUTATION (RED): delete the `previous().len()` check.
#[tokio::test]
async fn a_self_redirect_stops_at_the_chain_bound() {
let mut node = Server::new_async().await;
let hits = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
let h = hits.clone();
let loop_route = node
.mock("GET", "/api/v1/loop")
.with_status(301)
.with_header("location", "/api/v1/loop")
.with_body_from_request(move |_req| {
h.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
Vec::new()
})
.expect_at_least(1)
.create_async()
.await;
let client = NodeClient::with_timeout(
node.url(),
Some(test_keypair()),
std::time::Duration::from_secs(5),
);
let resp = client
.get_signed("/api/v1/loop")
.await
.expect("the bound must end the chain, not the timeout");
assert_eq!(
resp.status(),
301,
"the chain ends by refusing the next hop, so the last 3xx is what comes back"
);
assert_eq!(
hits.load(std::sync::atomic::Ordering::SeqCst),
gitlawb_core::redirect::MAX_REDIRECTS + 1,
"one original request plus MAX_REDIRECTS follows, matching what \
Policy::limited(MAX_REDIRECTS) would have permitted"
);
loop_route.assert_async().await;
}
/// A same-origin hop that REWRITES the request-target is refused, even though the
/// origin never changes.
///
/// The signature binds `@path` as the literal path-and-query the client signed, and
/// the node rebuilds it from the URI it received. A `/api/v1/thing` to
/// `/api/v1/thing/` bounce therefore presents a signature over a target the node
/// never saw, and every signed read behind such a proxy 401s. Refusing the hop
/// hands the caller the 3xx that names what happened instead.
///
/// The target mock expects zero hits and is asserted: mockito only checks
/// `.expect(N)` when `.assert()` runs, so an unbound or unasserted mock passes
/// vacuously.
#[tokio::test]
async fn same_origin_path_changing_redirect_is_refused() {
let mut node = Server::new_async().await;
let bounce = node
.mock("GET", "/api/v1/thing")
.with_status(301)
.with_header("location", "/api/v1/thing/")
.expect(1)
.create_async()
.await;
let target = node
.mock("GET", "/api/v1/thing/")
.with_status(200)
.with_body("bytes from the rewritten target")
.expect(0)
.create_async()
.await;
let client = NodeClient::new(node.url(), Some(test_keypair()));
let resp = client.get_signed("/api/v1/thing").await.unwrap();
assert_eq!(
resp.status(),
301,
"a refused redirect stops rather than errors, so the caller sees the 3xx \
and reports it through the status path it already has"
);
let body = resp.text().await.unwrap();
assert!(
!body.contains("bytes from the rewritten target"),
"the rewritten target's bytes must never reach the caller, got: {body}"
);
bounce.assert_async().await;
target.assert_async().await;
}
// ── the node's own verification, run over what the client actually sent ──
/// What the verifying mock made of a request that reached it.
///
/// The empty slot (`None`) is its own state and means the mock was never hit at
/// all, which is what the refusal test asserts. It must stay distinguishable from
/// [`Verdict::WrongIdentity`], because "nobody verified anything" and "something
/// verified against the wrong key" are opposite findings.
///
/// The payloads are read through `Debug` in the assertion messages and nowhere
/// else, which the dead-code pass does not count; they carry the detail that makes
/// a failure legible, so they stay.
#[derive(Debug)]
#[allow(dead_code)]
enum Verdict {
/// The chain accepted the signature AND the key it resolved is the test's DID.
Accepted,
/// The chain refused it. Carries the error so a failure reads as the actual
/// rejection rather than a bare hit count.
Rejected(String),
/// The chain accepted a signature made by somebody else. A key resolved from
/// the parsed `key_id` is read out of the artifact under verification, so an
/// accept on it alone proves consistency, never authenticity.
WrongIdentity { expected: String, got: String },
}
/// The node's `require_signature` verification, over a request this crate did not
/// build: parse the headers, recompute the content-digest from the body, rebuild
/// the signing string over `@method`/`@path`/`content-digest`, Ed25519-verify.
/// Returns the DID the signature resolved to, so a caller can pin the identity.
///
/// A hand-copy of its twin in `crates/git-remote-gitlawb/src/main.rs`, which gl
/// cannot import (`git-remote-gitlawb` is a binary crate and this is its test
/// module). Keep the two textually identical apart from the mockito seam around
/// them, so an edit to one is visibly an edit to both.
///
/// The production verifier both copies mirror is `crate::auth::require_signature` in
/// `crates/gitlawb-node/src/auth/mod.rs`. This is a re-implementation, not a call, so
/// an edit to that middleware has to land here too: otherwise the copies drift and
/// this test keeps passing against a rule the node has stopped applying.
///
/// It asserts internally, which is deliberate but constrains its callers: inside
/// `with_body_from_request` those assertions fire on the server thread and reach
/// the client as a transport error, not as a recorded verdict. So the identity
/// check lives in the caller as a [`Verdict`] variant, never as an assert in here.
fn node_verifies(
method: &str,
path_and_query: &str,
body: &[u8],
sig_input: &str,
sig_header: &str,
content_digest: &str,
) -> anyhow::Result<String> {
use gitlawb_core::http_sig::{build_signing_string, compute_content_digest, HttpSignature};
use gitlawb_core::identity::verify;
use std::collections::HashMap;
let sig = HttpSignature::parse(sig_input, sig_header)?;
sig.check_created()?;
assert!(
sig.missing_components().is_empty(),
"signature must cover all required components"
);
assert_eq!(sig.alg, "ed25519");
assert_eq!(
content_digest,
compute_content_digest(body),
"content-digest must match the body"
);
let vk = sig.key_id.to_verifying_key()?;
let mut values = HashMap::new();
values.insert("@method".to_string(), method.to_uppercase());
values.insert("@path".to_string(), path_and_query.to_string());
values.insert("content-digest".to_string(), content_digest.to_string());
let sig_params_value = sig_input.strip_prefix("sig1=").unwrap_or(sig_input);
let components: Vec<&str> = sig.components.iter().map(String::as_str).collect();
let signing_string = build_signing_string(&components, sig_params_value, &values)?;
let sig_array: [u8; 64] = sig.signature_bytes.as_slice().try_into()?;
verify(&vk, signing_string.as_bytes(), &sig_array)?;
Ok(sig.key_id.to_string())
}
/// Pull a header value off a received mockito request, or explain which one the
/// client failed to send.
fn received_header(req: &mockito::Request, name: &str) -> String {
req.header(name)
.first()
.unwrap_or_else(|| panic!("the client sent no {name} header"))
.to_str()
.unwrap()
.to_string()
}
/// Run [`node_verifies`] over a GET that arrived at the mock and record what the
/// node would have made of it, pinned to `expected_did`.
fn record_get_verdict(
req: &mockito::Request,
expected_did: &str,
slot: &std::sync::Arc<std::sync::Mutex<Option<Verdict>>>,
) {
let verdict = match node_verifies(
"GET",
req.path_and_query(),
b"",
&received_header(req, "signature-input"),
&received_header(req, "signature"),
&received_header(req, "content-digest"),
) {
Ok(did) if did == expected_did => Verdict::Accepted,
Ok(did) => Verdict::WrongIdentity {
expected: expected_did.to_string(),
got: did,
},
Err(e) => Verdict::Rejected(e.to_string()),
};
*slot.lock().unwrap() = Some(verdict);
}
/// The finding's repro, now a guard: a rewritten same-origin target must never
/// receive the signature, and the proof is the node's own verification, not a hit
/// count.
///
/// The target mock runs the full `require_signature` chain over the request it
/// receives. Post-fix the hop is refused, so the slot stays empty. Pre-fix the hop
/// is followed and the slot records the Ed25519 rejection of a signature made over
/// `/api/v1/thing` and presented at `/api/v1/thing/`, which is the 401 an operator
/// behind such a proxy actually sees. The verdict is asserted first, so a failure
/// speaks about verification rather than about reachability.
///
/// Its paired positive control is
/// `a_direct_signed_get_verifies_under_the_node_verifier`: without it, an empty
/// slot would be satisfied just as well by a harness that can never record
/// anything.
#[tokio::test]
async fn a_rewritten_target_never_receives_the_signature() {
let kp = test_keypair();
let expected_did = kp.did().to_string();
let slot = std::sync::Arc::new(std::sync::Mutex::new(None::<Verdict>));
let mut node = Server::new_async().await;
let bounce = node
.mock("GET", "/api/v1/thing")
.with_status(301)
.with_header("location", "/api/v1/thing/")
.expect(1)
.create_async()
.await;
let recorder = slot.clone();
let did_for_target = expected_did.clone();
let target = node
.mock("GET", "/api/v1/thing/")
.with_status(200)
.with_body_from_request(move |req| {
record_get_verdict(req, &did_for_target, &recorder);
Vec::new()
})
.expect(0)
.create_async()
.await;
let client = NodeClient::new(node.url(), Some(kp));
let resp = client.get_signed("/api/v1/thing").await.unwrap();
let verdict = slot.lock().unwrap().take();
assert!(
verdict.is_none(),
"the node's own verifier must never see this request: the signature covers \
/api/v1/thing and the rewritten target is /api/v1/thing/, so what arrives \
there is a stale request-target; recorded verdict: {verdict:?}"
);
assert_eq!(
resp.status(),
301,
"the caller sees the 3xx, not the rewritten target's answer"
);
bounce.assert_async().await;
target.assert_async().await;
}
/// The positive control for the test above, and the proof that the client signs
/// the query it sends.
///
/// A direct signed GET, no redirect anywhere, through the same verifying mock. The
/// verdict must be `Accepted`, which is what makes the refusal test's empty slot
/// attributable to the refusal rather than to a harness that cannot record. The
/// path carries a query, so a client that signed the bare path would land here as
/// `Rejected`.
#[tokio::test]
async fn a_direct_signed_get_verifies_under_the_node_verifier() {
let kp = test_keypair();
let expected_did = kp.did().to_string();
let slot = std::sync::Arc::new(std::sync::Mutex::new(None::<Verdict>));
let mut node = Server::new_async().await;
let recorder = slot.clone();
let did_for_target = expected_did.clone();
let route = node
.mock("GET", "/api/v1/thing?x=1")
.with_status(200)
.with_body_from_request(move |req| {
record_get_verdict(req, &did_for_target, &recorder);
Vec::new()
})
.expect(1)
.create_async()
.await;
let client = NodeClient::new(node.url(), Some(kp));
let resp = client.get_signed("/api/v1/thing?x=1").await.unwrap();
assert_eq!(resp.status(), 200);
let verdict = slot.lock().unwrap().take();
assert!(
matches!(verdict, Some(Verdict::Accepted)),
"a direct signed GET must verify under the node's own chain and resolve to \
{expected_did}, or the refusal test's empty slot proves nothing; recorded \
verdict: {verdict:?}"
);
route.assert_async().await;
}
// ── read_body_capped ────────────────────────────────────────────────
/// A body whose read FAILS mid-stream must be distinguishable from a body that
/// ended. Both used to leave through the same `_ => break`, so a 500 whose body
/// died in transit produced an empty message and the caller was told
/// `node returned 500: ` with nothing after the colon.
///
/// The fixture is a raw listener that promises 64 bytes in `Content-Length`,
/// writes 5, and closes. mockito cannot express that: it always completes the
/// response it advertises. MUTATION (RED): restore the single `_ => break` arm
/// (or hard-code `read_failed: false`) and the flag reads false.
#[tokio::test]
async fn read_body_capped_flags_a_mid_body_read_failure() {
let addr = spawn_short_body_listener().await;
let resp = reqwest::get(format!("http://{addr}/truncated"))
.await
.expect("headers arrive before the body is cut");
let body = read_body_capped(resp, 8192).await;
assert!(
body.read_failed,
"a body cut off mid-stream must be reported as a failed read, not as a \
body that ended: got {:?}",
body.text
);
assert!(
!body.truncated,
"the cap did not cut this one; 5 bytes are nowhere near 8 KiB"
);
}
/// The must-not half: a body that ends cleanly must NOT be flagged, or the flag
/// means nothing and every terminal starts claiming the node went quiet.
#[tokio::test]
async fn read_body_capped_does_not_flag_a_clean_body() {
let mut server = Server::new_async().await;
let _m = server
.mock("GET", "/ok")
.with_status(500)
.with_body("node said this")
.create_async()
.await;
let resp = reqwest::get(format!("{}/ok", server.url())).await.unwrap();
let body = read_body_capped(resp, 8192).await;
assert_eq!(body.text, "node said this");
assert!(!body.read_failed, "a complete body is not a failed read");
assert!(!body.truncated, "a complete body is not a truncated one");
}
/// Answer one request with headers promising more body than gets written, then
/// close the connection. Returns the listener's address.
async fn spawn_short_body_listener() -> std::net::SocketAddr {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
let (mut sock, _) = listener.accept().await.unwrap();
let mut scratch = [0u8; 1024];
let _ = sock.read(&mut scratch).await;
let _ = sock
.write_all(b"HTTP/1.1 500 Internal Server Error\r\nContent-Length: 64\r\n\r\nshort")
.await;
let _ = sock.flush().await;
// Drop closes the socket with 59 of the promised bytes never sent.
});
addr
}
// ── total request timeout ───────────────────────────────────────────
/// The client's timeout is a TOTAL request timeout, so it bounds a download and
/// not just the handshake. `gl ipfs get`'s documentation leans on exactly that:
/// the wall-clock deadline covers the search and deliberately stops at the
/// response headers, and this timeout is the only thing left bounding the body.
///
/// Driven at 250ms through `with_timeout`, the seam `new` itself calls with
/// `TOTAL_REQUEST_TIMEOUT`, because a test at the shipped 30s has no place in this
/// suite. What that costs is the value; what it proves is the SHAPE, which is the
/// part in doubt: that the deadline keeps running once the headers have landed. A
/// timeout that covered only the handshake would let this request hang until the
/// listener gives up.
#[tokio::test]
async fn total_timeout_cuts_off_a_body_that_outruns_it() {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
let (mut sock, _) = listener.accept().await.unwrap();
let mut scratch = [0u8; 1024];
let _ = sock.read(&mut scratch).await;
// Headers land immediately, then the body stalls indefinitely.
let _ = sock
.write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 1024\r\n\r\nfirst")
.await;
let _ = sock.flush().await;
tokio::time::sleep(std::time::Duration::from_secs(30)).await;
});
let client = NodeClient::with_timeout(
format!("http://{addr}"),
None,
std::time::Duration::from_millis(250),
);
let started = std::time::Instant::now();
let resp = client.get("/slow").await.expect("headers arrive promptly");
assert_eq!(
resp.status(),
200,
"the stall is in the body, not the status"
);
let err = resp
.bytes()
.await
.expect_err("a body still arriving past the total timeout must be cut off");
let elapsed = started.elapsed();
assert!(
err.is_timeout(),
"the body read must end in a timeout, got: {err}"
);
assert!(
elapsed < std::time::Duration::from_secs(5),
"the timeout must fire on its own schedule, not wait out the listener; \
took {elapsed:?}"
);
}
#[test]
fn shipped_client_uses_the_documented_total_timeout() {
// The scaled-down test above proves the shape at 250ms. This pins the value
// `new` actually ships, so the two together cover the documented behaviour.
assert_eq!(TOTAL_REQUEST_TIMEOUT, std::time::Duration::from_secs(30));
}
#[test]
fn sanitize_strips_controls_bidi_and_caps_length() {
// C0 (ESC/BEL) and the Cf bidi override (U+202E) are both removed; the
// printable text survives. (Note: a stripped ESC leaves any following
// "[31m" as inert literal text — that is the point, so the input here
// avoids that residue to keep the expectation unambiguous.)
let out = sanitize_node_msg("a\u{1b}\u{07}b\u{202e}c");
assert!(
!out.chars().any(|c| c.is_control()),
"control char leaked: {out:?}"
);
assert!(
!out.contains('\u{202e}'),
"RLO bidi override leaked: {out:?}"
);
assert_eq!(out, "abc");
// Length is capped at 200 chars regardless of input size.
let long = "x".repeat(250);
assert_eq!(sanitize_node_msg(&long).chars().count(), 200);
}
#[test]
fn sanitize_preserves_legitimate_and_rtl_text() {
// Must not over-strip: a plain word, a genuine RTL SCRIPT letter (Arabic
// U+0627, category Lo — NOT a format char), and ZWJ (U+200D, a legitimate
// Cf char, e.g. emoji sequences) all survive. Guards the shared predicate
// against being widened into a blanket Cf stripper.
let out = sanitize_node_msg("ok \u{0627}\u{200D}b");
assert_eq!(out, "ok \u{0627}\u{200D}b");
}
}
+2111 -22
View File
File diff suppressed because it is too large. Load diff
+406 -12
View File
@@ -7,7 +7,7 @@ use clap::{Args, Subcommand};
use serde_json::Value;
use std::path::PathBuf;
use crate::http::NodeClient;
use crate::http::{read_body_capped, sanitize_node_msg, NodeClient};
use crate::identity::load_keypair_from_dir;
#[derive(Args)]
@@ -97,6 +97,74 @@ async fn cmd_list(node: String) -> Result<()> {
Ok(())
}
/// The warning text for a local peer-add reply, or `None` when the node
/// accepted it.
///
/// Split out so the refusal path is assertable. The node now answers 403 when
/// an unproven caller tries to repoint an existing peer's `http_url`, so a
/// reply that arrived and refused must not reach the success line.
fn local_add_refusal(status: reqwest::StatusCode, body: &Value) -> Option<String> {
if status.is_success() {
return None;
}
// The message is the node's, not ours, so it is defanged before it reaches
// the terminal. Only the message: the status and the prose are ours.
let msg = sanitize_node_msg(body["message"].as_str().unwrap_or("unknown error"));
Some(format!(
"warning: local peer list not updated ({status}): {msg}"
))
}
/// The error text for a remote announce reply, or `None` when the peer accepted
/// it.
///
/// Split out so the ordering is assertable: the status has to reach the caller
/// even when the body does not parse. That case is reachable rather than
/// theoretical, because the read above is capped, so a peer answering with a
/// body past the cap leaves truncated JSON behind, and `peer_url` is whatever
/// the caller passed, so a non-JSON error page is just as easy. Parsing first
/// and failing on the parse would swallow the status the user needs.
fn remote_announce_failure(status: reqwest::StatusCode, raw: &str) -> Option<String> {
if status.is_success() {
return None;
}
// Falls back to the body itself when there is no `message` field, which is
// what a truncated or non-JSON reply leaves. Sanitized either way: it is the
// peer's text and it is headed for the terminal.
let parsed: Value = serde_json::from_str(raw).unwrap_or(Value::Null);
let msg = sanitize_node_msg(parsed["message"].as_str().unwrap_or(raw));
Some(format!("announce failed ({status}): {msg}"))
}
/// Which line the local peer-add prints, and whether it is a warning.
///
/// Split out because the SELECTION is what the fix is: `local_add_refusal` was
/// already unit-tested six ways, and replacing this whole match with an
/// unconditional "Added to local peer list." still left 312 tests green. The
/// helper was pinned; the wiring was not, which is the shape
/// `unit-test-on-helper-does-not-prove-handler-wiring` names. With the choice
/// extracted, the call site is a two-line dispatch carrying no logic.
fn local_add_report(status: reqwest::StatusCode, body: &Value) -> (bool, String) {
match local_add_refusal(status, body) {
Some(warning) => (true, warning),
None => (false, " Added to local peer list.".to_string()),
}
}
/// The block printed after a peer accepts our announce. Split out so the
/// defanging is assertable: the DID and URL are the remote peer's own strings,
/// and `peer_url` is caller-chosen, so this is the least trusted body the
/// command handles. Only the display is sanitized; the values sent on to our
/// local node stay verbatim, since that is data rather than terminal output and
/// the node runs its own validation on them.
fn announced_peer_summary(their_did: &str, their_url: &str, peer_count: u64) -> String {
format!(
"Announced to peer node:\n DID: {}\n URL: {}\n Their peers: {peer_count}\n",
sanitize_node_msg(their_did),
sanitize_node_msg(their_url),
)
}
async fn cmd_add(peer_url: String, node: String, dir: Option<PathBuf>) -> Result<()> {
let keypair = load_keypair_from_dir(dir.as_deref())?;
let my_did = keypair.did().to_string();
@@ -127,21 +195,26 @@ async fn cmd_add(peer_url: String, node: String, dir: Option<PathBuf>) -> Result
.await
.context("failed to connect to peer")?;
let status = resp.status();
let result: Value = resp.json().await.context("invalid JSON response")?;
// `peer_url` is fully caller-chosen, so this body is the least trusted one
// in the command: bound the read, and defang the message before it reaches
// the terminal through the error return. An announce reply is a DID, a URL
// and a count, so 8 KiB is well past what the shape needs.
let raw = read_body_capped(resp, 8 * 1024).await.text;
if !status.is_success() {
let msg = result["message"].as_str().unwrap_or("unknown error");
anyhow::bail!("announce failed ({status}): {msg}");
if let Some(failure) = remote_announce_failure(status, &raw) {
anyhow::bail!("{failure}");
}
let result: Value = serde_json::from_str(&raw).context("invalid JSON response")?;
let their_did = result["node_did"].as_str().unwrap_or("?");
let their_url = result["node_url"].as_str().unwrap_or("?");
let peer_count = result["peer_count"].as_u64().unwrap_or(0);
println!("Announced to peer node:");
println!(" DID: {their_did}");
println!(" URL: {their_url}");
println!(" Their peers: {peer_count}");
print!(
"{}",
announced_peer_summary(their_did, their_url, peer_count)
);
// Also add their info to our local node's peer list
// (the peer's /announce response includes their did + url)
@@ -150,9 +223,31 @@ async fn cmd_add(peer_url: String, node: String, dir: Option<PathBuf>) -> Result
"did": their_did,
"http_url": their_url,
}))?;
// This requires the local node to be running; ignore errors here
let _ = local_client.post("/api/v1/peers/announce", &add_body).await;
println!(" Added to local peer list.");
// This requires the local node to be running, so a transport failure
// stays a warning rather than failing the command. A reply that
// ARRIVED and refused is different: the announce route answers 403
// when an unproven caller tries to repoint an existing peer, and
// printing the success line over that reports a completed add that
// never happened. Same status handling as the remote announce above,
// except best effort.
match local_client.post("/api/v1/peers/announce", &add_body).await {
Ok(resp) => {
let status = resp.status();
// Same bound as the remote announce above: a compromised local
// node (or a MITM on plain http) must not force an unbounded
// read. A body that does not parse stays `Null`, which still
// routes a non-success status to the warning.
let raw = read_body_capped(resp, 8 * 1024).await.text;
let result: Value = serde_json::from_str(&raw).unwrap_or(Value::Null);
let (is_warning, line) = local_add_report(status, &result);
if is_warning {
eprintln!("{line}");
} else {
println!("{line}");
}
}
Err(e) => eprintln!("warning: local peer list not updated: {e}"),
}
}
Ok(())
@@ -210,3 +305,302 @@ async fn cmd_resolve(did: String, node: String) -> Result<()> {
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::{
announced_peer_summary, cmd_add, local_add_refusal, local_add_report,
remote_announce_failure,
};
use reqwest::StatusCode;
use serde_json::json;
/// The node's 403 for repointing an existing peer must surface, not print
/// as a completed add. Without the status check the command prints "Added
/// to local peer list." over a refusal.
///
/// The fixture body is the real wire shape, not an invented one: the
/// message is the Display of `PeerWriteDenied::UnprovenRepoint`, declared
/// in `crates/gitlawb-node/src/db/mod.rs`. The executable detector for a
/// wording change is the node-side pin
/// `announce_unsigned_repoint_is_a_403_denial` in
/// `crates/gitlawb-node/src/api/peers.rs`, which asserts this same
/// substring against a response driven through the real router, so
/// rewording the Display turns that test red and the grep for the old
/// phrase lands here.
///
/// The assertion bites rather than echoing its own input because the
/// substring is reachable only through the helper's `body["message"]`
/// extraction; the fixture's `error` field carries a different string, so
/// reading `body["error"]` instead fails this test.
#[test]
fn a_refused_local_add_warns_with_the_node_reason() {
let warning = local_add_refusal(
StatusCode::FORBIDDEN,
&json!({
"error": "forbidden",
"message": "unproven announce cannot change an existing peer's http_url: did:key:z6MkuMqUm4i228K9qXidJ57zqSWAcQLgrcbMxB8RKVLuqitj"
}),
)
.expect("a refusal must not render as success");
assert!(warning.contains("403"), "must name the status: {warning}");
assert!(
warning.contains("unproven announce cannot change an existing peer"),
"must carry the node's own reason: {warning}"
);
}
/// A refusal with no message body still warns rather than passing silently.
#[test]
fn a_refusal_without_a_message_still_warns() {
let warning = local_add_refusal(StatusCode::BAD_REQUEST, &serde_json::Value::Null)
.expect("a refusal must not render as success");
assert!(warning.contains("400"), "must name the status: {warning}");
}
/// The accepted case stays quiet, so the success line still prints.
#[test]
fn an_accepted_local_add_produces_no_warning() {
assert!(local_add_refusal(StatusCode::OK, &json!({ "peer_count": 3 })).is_none());
}
/// The warning is printed straight to a terminal, so a node-supplied message
/// carrying an OSC title-rewrite plus a C0 escape must reach it defanged.
#[test]
fn a_refusal_message_cannot_carry_terminal_escapes() {
let warning = local_add_refusal(
StatusCode::FORBIDDEN,
&json!({ "message": "\u{1b}]2;pwned\u{7}legit-looking text" }),
)
.expect("a refusal must not render as success");
assert!(
!warning.chars().any(|c| c.is_control()),
"control char leaked to the terminal: {warning:?}"
);
assert!(
warning.contains("legit-looking text"),
"printable text dropped: {warning:?}"
);
}
/// Bidi format chars are not `char::is_control`, yet they reorder the
/// displayed line, so a refusal must not carry them either.
#[test]
fn a_refusal_message_cannot_carry_bidi_overrides() {
let warning = local_add_refusal(
StatusCode::FORBIDDEN,
&json!({ "message": "peer \u{202e}refused\u{202c} here" }),
)
.expect("a refusal must not render as success");
assert!(
!warning.chars().any(gitlawb_core::sanitize::is_bidi_format),
"bidi format char leaked to the terminal: {warning:?}"
);
}
/// A hostile node must not be able to dump an arbitrary amount of text into
/// the user's scrollback through the warning line.
#[test]
fn a_refusal_message_is_length_bounded() {
let warning = local_add_refusal(
StatusCode::FORBIDDEN,
&json!({ "message": "x".repeat(5000) }),
)
.expect("a refusal must not render as success");
assert!(
warning.chars().count() < 500,
"warning not bounded: {} chars",
warning.chars().count()
);
}
/// The status is the part the user can act on, so it must survive a body
/// that does not parse. The read is capped, so a peer answering past the cap
/// leaves exactly this: a valid status and truncated JSON.
#[test]
fn a_remote_announce_failure_keeps_the_status_when_the_body_does_not_parse() {
let truncated = r#"{"error":"service_unavailable","message":"backend is do"#;
let failure = remote_announce_failure(StatusCode::SERVICE_UNAVAILABLE, truncated)
.expect("a non-success status must produce a failure");
assert!(
failure.contains("503"),
"the status must survive an unparseable body: {failure:?}"
);
}
/// A parseable error body still reports the peer's own message.
#[test]
fn a_remote_announce_failure_carries_the_peers_message() {
let failure = remote_announce_failure(
StatusCode::FORBIDDEN,
r#"{"error":"forbidden","message":"unproven announce cannot change an existing peer"}"#,
)
.expect("a non-success status must produce a failure");
assert!(
failure.contains("403")
&& failure.contains("unproven announce cannot change an existing peer"),
"status and reason must both survive: {failure:?}"
);
}
/// The fallback path prints the raw body, so it is defanged like every other
/// peer-supplied string headed for the terminal.
#[test]
fn a_remote_announce_failure_defangs_an_unparseable_body() {
let failure = remote_announce_failure(
StatusCode::BAD_GATEWAY,
"\u{1b}]2;pwned\u{7}<html>gateway error</html>",
)
.expect("a non-success status must produce a failure");
assert!(
!failure.chars().any(|c| c.is_control()),
"control char leaked to the terminal: {failure:?}"
);
assert!(failure.contains("502"), "status dropped: {failure:?}");
assert!(
failure.contains("gateway error"),
"the body is all the user gets when there is no message field: {failure:?}"
);
}
/// The unit tests above bind the helper. This one binds the CALL SITE: the
/// failure check has to run before the parse, or a body past the read cap
/// turns a 503 into "invalid JSON response" and the status the user needs is
/// gone. Reordering the two statements in `cmd_add` compiles and passes every
/// other test in this file, so only driving the command catches it.
#[tokio::test]
async fn a_peer_failure_past_the_read_cap_still_reports_its_status() {
let mut peer = mockito::Server::new_async().await;
let mut local = mockito::Server::new_async().await;
let dir = tempfile::TempDir::new().unwrap();
let kp = gitlawb_core::identity::Keypair::generate();
std::fs::write(
dir.path().join("identity.pem"),
kp.to_pem().unwrap().as_bytes(),
)
.unwrap();
let _local_info = local
.mock("GET", "/")
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"public_url":"https://me.example"}"#)
.create_async()
.await;
// Valid JSON, but past the 8 KiB read cap, so what survives the read is
// truncated and unparseable. This is the shape the cap itself creates.
let oversized = format!(r#"{{"message":"{}"}}"#, "x".repeat(16 * 1024));
let _peer_refusal = peer
.mock("POST", "/api/v1/peers/announce")
.with_status(503)
.with_header("content-type", "application/json")
.with_body(oversized)
.create_async()
.await;
let err = cmd_add(peer.url(), local.url(), Some(dir.path().to_path_buf()))
.await
.expect_err("a 503 from the peer must fail the command");
let rendered = format!("{err:#}");
assert!(
rendered.contains("503"),
"the status must survive a body past the read cap: {rendered:?}"
);
assert!(
!rendered.contains("invalid JSON response"),
"the parse failure must not stand in for the status: {rendered:?}"
);
}
/// An accepted announce produces no error, so the caller proceeds to parse.
#[test]
fn an_accepted_remote_announce_produces_no_failure() {
assert!(remote_announce_failure(StatusCode::OK, r#"{"peer_count":3}"#).is_none());
}
/// The selection, not the helper. A refusal must never yield the success
/// line, and the success line must never carry the warning prefix. This is
/// what stayed green when the entire match was replaced with an
/// unconditional "Added to local peer list."
#[test]
fn a_local_add_refusal_never_selects_the_success_line() {
let (is_warning, line) = local_add_report(
StatusCode::FORBIDDEN,
&json!({ "message": "unproven announce cannot change an existing peer's http_url: did:key:zAbc" }),
);
assert!(is_warning, "a 403 must select the warning stream: {line:?}");
assert!(
!line.contains("Added to local peer list"),
"a refusal rendered as the success line: {line:?}"
);
assert!(
line.contains("unproven announce cannot change an existing peer"),
"the node's reason must reach the user: {line:?}"
);
}
/// The other direction, so the fix cannot be satisfied by always warning.
#[test]
fn an_accepted_local_add_selects_the_success_line() {
let (is_warning, line) = local_add_report(StatusCode::OK, &json!({ "peer_count": 3 }));
assert!(!is_warning, "a 200 must not select the warning stream");
assert!(
line.contains("Added to local peer list"),
"the success line must still print: {line:?}"
);
}
/// The accepted-announce block prints the remote peer's own DID and URL, so
/// a hostile peer answering 200 must not get escapes onto the terminal that
/// way either. The refusal path is not the only sink in this command.
#[test]
fn an_accepted_announce_summary_cannot_carry_terminal_escapes() {
let summary = announced_peer_summary(
"did:key:z6Mk\u{1b}]2;pwned\u{7}abc",
"https://peer.example\u{202e}moc.live//:sptth",
3,
);
assert!(
!summary.chars().any(|c| c.is_control() && c != '\n'),
"control char leaked to the terminal: {summary:?}"
);
assert!(
!summary.chars().any(gitlawb_core::sanitize::is_bidi_format),
"bidi format char leaked to the terminal: {summary:?}"
);
assert!(
summary.contains("did:key:z6Mk") && summary.contains("Their peers: 3"),
"printable content dropped: {summary:?}"
);
}
/// The counterweight to the stripping tests: a legitimate message with RTL
/// script letters and a ZWJ must survive intact, so the sanitizer is not
/// proven by over-stripping.
#[test]
fn a_legitimate_refusal_message_survives_intact() {
let warning = local_add_refusal(
StatusCode::FORBIDDEN,
&json!({ "message": "peer refused \u{0627}\u{200D}b" }),
)
.expect("a refusal must not render as success");
assert!(
warning.contains("peer refused \u{0627}\u{200D}b"),
"legitimate text mangled: {warning:?}"
);
}
}
+3 -67
View File
@@ -2,7 +2,7 @@ use anyhow::{Context, Result};
use clap::{Args, Subcommand};
use std::path::PathBuf;
use crate::http::NodeClient;
use crate::http::{read_body_capped, sanitize_node_msg, NodeClient};
use crate::identity::load_keypair_from_dir;
#[derive(Args)]
@@ -46,7 +46,7 @@ pub async fn run(args: SyncArgs) -> Result<()> {
if !status.is_success() {
// Bound the read: a hostile or broken node must not force an
// unbounded allocation just to surface a denial (INV-6, read half).
let raw = read_body_capped(resp, 8 * 1024).await;
let raw = read_body_capped(resp, 8 * 1024).await.text;
let msg = serde_json::from_str::<serde_json::Value>(&raw)
.ok()
.and_then(|v| {
@@ -104,39 +104,6 @@ fn trigger_counts(resp: &serde_json::Value) -> (u64, u64) {
)
}
/// Read at most `cap` bytes of a response body. Bounds the allocation from a
/// hostile or broken node returning a huge error body — the display is capped
/// separately, but the read itself must not be unbounded (INV-6, read half).
async fn read_body_capped(mut resp: reqwest::Response, cap: usize) -> String {
let mut buf: Vec<u8> = Vec::new();
while buf.len() < cap {
match resp.chunk().await {
Ok(Some(chunk)) => {
let take = (cap - buf.len()).min(chunk.len());
buf.extend_from_slice(&chunk[..take]);
if take < chunk.len() {
break; // hit the cap mid-chunk
}
}
_ => break, // end of body or read error — return what we have
}
}
String::from_utf8_lossy(&buf).into_owned()
}
/// Strip terminal-dangerous characters from (and cap the length of) a
/// node-supplied error string before surfacing it. The node a caller talks to
/// could be hostile and embed escape sequences in its error body; those must not
/// reach the terminal verbatim (INV-6). We drop the C0/C1 control bytes (which
/// defangs ANSI/OSC escapes) AND the Unicode bidi/format controls (which
/// `char::is_control` does not cover — they can reorder the displayed line).
fn sanitize_node_msg(s: &str) -> String {
s.chars()
.filter(|c| !c.is_control() && !gitlawb_core::sanitize::is_bidi_format(*c))
.take(200)
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
@@ -260,37 +227,6 @@ mod tests {
run(args).await.unwrap();
}
#[test]
fn sanitize_strips_controls_bidi_and_caps_length() {
// C0 (ESC/BEL) and the Cf bidi override (U+202E) are both removed; the
// printable text survives. (Note: a stripped ESC leaves any following
// "[31m" as inert literal text — that is the point, so the input here
// avoids that residue to keep the expectation unambiguous.)
let out = sanitize_node_msg("a\u{1b}\u{07}b\u{202e}c");
assert!(
!out.chars().any(|c| c.is_control()),
"control char leaked: {out:?}"
);
assert!(
!out.contains('\u{202e}'),
"RLO bidi override leaked: {out:?}"
);
assert_eq!(out, "abc");
// Length is capped at 200 chars regardless of input size.
let long = "x".repeat(250);
assert_eq!(sanitize_node_msg(&long).chars().count(), 200);
}
#[test]
fn sanitize_preserves_legitimate_and_rtl_text() {
// Must not over-strip: a plain word, a genuine RTL SCRIPT letter (Arabic
// U+0627, category Lo — NOT a format char), and ZWJ (U+200D, a legitimate
// Cf char, e.g. emoji sequences) all survive. Guards the shared predicate
// against being widened into a blanket Cf stripper.
let out = sanitize_node_msg("ok \u{0627}\u{200D}b");
assert_eq!(out, "ok \u{0627}\u{200D}b");
}
#[tokio::test]
async fn trigger_handles_oversized_error_body_without_unbounded_output() {
// A hostile/broken node returns a 2 MB error body. The command must still
@@ -325,7 +261,7 @@ mod tests {
.create_async()
.await;
let resp = reqwest::get(format!("{}/big", server.url())).await.unwrap();
let out = read_body_capped(resp, 8192).await;
let out = read_body_capped(resp, 8192).await.text;
assert!(out.len() <= 8192, "read not bounded: {} bytes", out.len());
assert!(!out.is_empty(), "expected some body");
}
+334 -14
View File
@@ -1,11 +1,11 @@
//! `gl whoami` — print current identity and optional node registration info.
use anyhow::Result;
use anyhow::{bail, Result};
use clap::Args;
use serde_json::{json, Value};
use std::path::PathBuf;
use crate::http::NodeClient;
use crate::http::{read_body_capped, sanitize_node_msg, NodeClient};
use crate::identity::load_keypair_from_dir;
#[derive(Args)]
@@ -22,6 +22,10 @@ pub struct WhoamiArgs {
}
pub async fn run(args: WhoamiArgs) -> Result<()> {
run_to_writer(args, &mut std::io::stdout()).await
}
pub(crate) async fn run_to_writer(args: WhoamiArgs, w: &mut impl std::io::Write) -> Result<()> {
let keypair = load_keypair_from_dir(args.dir.as_deref())?;
let did = keypair.did().to_string();
let short = did.split(':').next_back().unwrap_or(&did).to_string();
@@ -41,7 +45,7 @@ pub async fn run(args: WhoamiArgs) -> Result<()> {
if let Some(caps) = info["capabilities"].as_array() {
capabilities = caps
.iter()
.filter_map(|c| c.as_str().map(String::from))
.filter_map(|c| c.as_str().map(sanitize_node_msg))
.collect();
}
// Try to get repo count
@@ -51,11 +55,35 @@ pub async fn run(args: WhoamiArgs) -> Result<()> {
}
}
}
Ok(_) => {
Ok(resp) if resp.status().as_u16() == 404 => {
registered = Some(false);
}
Err(_) => {
registered = Some(false);
Ok(resp) => {
let status = resp.status();
let raw = read_body_capped(resp, 8 * 1024).await.text;
let msg = serde_json::from_str::<Value>(&raw)
.ok()
.and_then(|v| {
let non_empty = |m: Option<&Value>| {
m.and_then(|m| m.as_str())
.map(String::from)
.filter(|s| !s.is_empty())
};
non_empty(v.get("message")).or_else(|| non_empty(v.get("error")))
})
.unwrap_or(raw);
bail!(
"agent lookup failed ({status}): {}",
sanitize_node_msg(&msg)
);
}
Err(e) => {
let detail: String = e
.chain()
.map(|e| sanitize_node_msg(&e.to_string()))
.collect::<Vec<_>>()
.join(": ");
bail!("agent lookup failed: {detail}");
}
}
}
@@ -77,21 +105,21 @@ pub async fn run(args: WhoamiArgs) -> Result<()> {
if let Some(rc) = repo_count {
out["repos"] = json!(rc);
}
println!("{}", serde_json::to_string_pretty(&out)?);
writeln!(w, "{}", serde_json::to_string_pretty(&out)?)?;
} else {
println!("DID: {did}");
println!("Short: {short}");
writeln!(w, "DID: {did}")?;
writeln!(w, "Short: {short}")?;
if let Some(reg) = registered {
println!("Registered: {}", if reg { "yes" } else { "no" });
writeln!(w, "Registered: {}", if reg { "yes" } else { "no" })?;
}
if let Some(ts) = trust_score {
println!("Trust: {ts:.2}");
writeln!(w, "Trust: {ts:.2}")?;
}
if !capabilities.is_empty() {
println!("Caps: {}", capabilities.join(", "));
writeln!(w, "Caps: {}", capabilities.join(", "))?;
}
if let Some(rc) = repo_count {
println!("Repos: {rc}");
writeln!(w, "Repos: {rc}")?;
}
}
Ok(())
@@ -192,7 +220,236 @@ mod tests {
node: Some(server.url()),
json: false,
};
run(args).await.unwrap();
let mut out = Vec::new();
run_to_writer(args, &mut out).await.unwrap();
let out = String::from_utf8(out).unwrap();
assert!(out.contains("Registered: no"), "unexpected output: {out}");
let args = WhoamiArgs {
dir: Some(dir.path().to_path_buf()),
node: Some(server.url()),
json: true,
};
let mut out = Vec::new();
run_to_writer(args, &mut out).await.unwrap();
let out = String::from_utf8(out).unwrap();
assert!(
out.contains("\"registered\": false"),
"unexpected output: {out}"
);
}
#[tokio::test]
async fn test_whoami_with_node_forbidden() {
let dir = TempDir::new().unwrap();
let kp = gitlawb_core::identity::Keypair::generate();
let pem = kp.to_pem().unwrap();
std::fs::write(dir.path().join("identity.pem"), pem.as_bytes()).unwrap();
let did = kp.did().to_string();
let mut server = mockito::Server::new_async().await;
let _agent = server
.mock("GET", format!("/api/v1/agents/{did}").as_str())
.with_status(403)
.with_header("content-type", "application/json")
.with_body(r#"{"message":"forbidden"}"#)
.create_async()
.await;
let args = WhoamiArgs {
dir: Some(dir.path().to_path_buf()),
node: Some(server.url()),
json: false,
};
let err = run(args).await.unwrap_err();
let msg = format!("{err:?}");
assert!(msg.contains("403"), "expected 403 error, got: {msg}");
assert!(
msg.contains("forbidden"),
"expected 'forbidden' in error, got: {msg}"
);
}
#[tokio::test]
async fn test_whoami_with_node_server_error() {
let dir = TempDir::new().unwrap();
let kp = gitlawb_core::identity::Keypair::generate();
let pem = kp.to_pem().unwrap();
std::fs::write(dir.path().join("identity.pem"), pem.as_bytes()).unwrap();
let did = kp.did().to_string();
let mut server = mockito::Server::new_async().await;
let _agent = server
.mock("GET", format!("/api/v1/agents/{did}").as_str())
.with_status(500)
.with_header("content-type", "application/json")
.with_body(r#"{"message":"internal error"}"#)
.create_async()
.await;
let args = WhoamiArgs {
dir: Some(dir.path().to_path_buf()),
node: Some(server.url()),
json: false,
};
let err = run(args).await.unwrap_err();
let msg = format!("{err:?}");
assert!(msg.contains("500"), "expected 500 error, got: {msg}");
assert!(
msg.contains("internal error"),
"expected 'internal error' in error, got: {msg}"
);
}
#[tokio::test]
async fn test_whoami_server_error_unusable_message_uses_error() {
let dir = TempDir::new().unwrap();
let kp = gitlawb_core::identity::Keypair::generate();
let pem = kp.to_pem().unwrap();
std::fs::write(dir.path().join("identity.pem"), pem.as_bytes()).unwrap();
let did = kp.did().to_string();
let mut server = mockito::Server::new_async().await;
let _agent = server
.mock("GET", format!("/api/v1/agents/{did}").as_str())
.with_status(500)
.with_header("content-type", "application/json")
.with_body(r#"{"message":"","error":"boom"}"#)
.create_async()
.await;
let args = WhoamiArgs {
dir: Some(dir.path().to_path_buf()),
node: Some(server.url()),
json: false,
};
let err = run(args).await.unwrap_err();
let msg = format!("{err:?}");
assert!(
msg.contains("boom"),
"expected 'boom' from error field, got: {msg}"
);
}
#[tokio::test]
async fn test_whoami_with_node_transport_error() {
let dir = TempDir::new().unwrap();
let kp = gitlawb_core::identity::Keypair::generate();
let pem = kp.to_pem().unwrap();
std::fs::write(dir.path().join("identity.pem"), pem.as_bytes()).unwrap();
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let node = format!("http://{}", listener.local_addr().unwrap());
drop(listener);
let args = WhoamiArgs {
dir: Some(dir.path().to_path_buf()),
node: Some(node),
json: false,
};
let err = run(args).await.unwrap_err();
let msg = format!("{err:?}");
assert!(
msg.contains("agent lookup failed"),
"expected transport error, got: {msg}"
);
}
#[tokio::test]
async fn test_whoami_transport_sanitizes_control_chars_in_node_url() {
let dir = TempDir::new().unwrap();
let kp = gitlawb_core::identity::Keypair::generate();
let pem = kp.to_pem().unwrap();
std::fs::write(dir.path().join("identity.pem"), pem.as_bytes()).unwrap();
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let port = listener.local_addr().unwrap().port();
drop(listener);
let node = format!("http://127.0.0.1:{port}/\u{1b}]0;PWNED\u{7} \u{202e}gnitirw");
let args = WhoamiArgs {
dir: Some(dir.path().to_path_buf()),
node: Some(node),
json: false,
};
let err = run(args).await.unwrap_err();
let msg = format!("{err:?}");
assert!(
msg.contains("agent lookup failed"),
"expected transport error, got: {msg}"
);
assert!(!msg.contains('\u{1b}'), "ESC leaked into output: {msg:?}");
assert!(!msg.contains('\u{7}'), "BEL leaked into output: {msg:?}");
assert!(!msg.contains('\u{202e}'), "RLO leaked into output: {msg:?}");
}
#[tokio::test]
async fn test_whoami_server_error_body_display_bounded() {
let dir = TempDir::new().unwrap();
let kp = gitlawb_core::identity::Keypair::generate();
let pem = kp.to_pem().unwrap();
std::fs::write(dir.path().join("identity.pem"), pem.as_bytes()).unwrap();
let did = kp.did().to_string();
let mut server = mockito::Server::new_async().await;
let _agent = server
.mock("GET", format!("/api/v1/agents/{did}").as_str())
.with_status(502)
.with_header("content-type", "application/json")
.with_body("x".repeat(100_000))
.create_async()
.await;
let args = WhoamiArgs {
dir: Some(dir.path().to_path_buf()),
node: Some(server.url()),
json: false,
};
let err = run(args).await.unwrap_err();
let msg = format!("{err:?}");
assert!(
msg.contains("502"),
"expected 502 error with bounded body, got: {msg}"
);
let display = format!("{err}");
assert!(
display.len() < 1000,
"error message too long ({} bytes) — body was not capped",
display.len()
);
}
#[tokio::test]
async fn test_whoami_server_error_sanitizes_controls() {
let dir = TempDir::new().unwrap();
let kp = gitlawb_core::identity::Keypair::generate();
let pem = kp.to_pem().unwrap();
std::fs::write(dir.path().join("identity.pem"), pem.as_bytes()).unwrap();
let did = kp.did().to_string();
let mut server = mockito::Server::new_async().await;
let _agent = server
.mock("GET", format!("/api/v1/agents/{did}").as_str())
.with_status(500)
.with_header("content-type", "application/json")
.with_body("{\"message\":\"\\u001b[31mowned\\u0007\\u202eevil\"}")
.create_async()
.await;
let args = WhoamiArgs {
dir: Some(dir.path().to_path_buf()),
node: Some(server.url()),
json: false,
};
let err = run(args).await.unwrap_err();
let msg = format!("{err:?}");
assert!(
msg.contains("owned"),
"expected sanitized error body, got: {msg}"
);
assert!(!msg.contains('\u{1b}'), "ESC control char leaked: {msg}");
assert!(!msg.contains('\u{07}'), "BEL control char leaked: {msg}");
assert!(!msg.contains('\u{202e}'), "RTL override leaked: {msg}");
}
#[tokio::test]
@@ -230,4 +487,67 @@ mod tests {
};
run(args).await.unwrap();
}
#[tokio::test]
async fn test_whoami_sanitizes_hostile_capabilities() {
let dir = TempDir::new().unwrap();
let kp = gitlawb_core::identity::Keypair::generate();
let pem = kp.to_pem().unwrap();
std::fs::write(dir.path().join("identity.pem"), pem.as_bytes()).unwrap();
let did = kp.did().to_string();
let short = did.split(':').next_back().unwrap().to_string();
let mut server = mockito::Server::new_async().await;
let _agent = server
.mock("GET", format!("/api/v1/agents/{did}").as_str())
.with_status(200)
.with_header("content-type", "application/json")
.with_body(
"{\"trust_score\":0.5,\"capabilities\":[\"\\u001b]0;PWNED\\u0007repo:write\",\"\\u202egnitirw-tfel\"]}",
)
.create_async()
.await;
let _repos = server
.mock(
"GET",
mockito::Matcher::Regex(format!(r"^/api/v1/repos\?owner={short}")),
)
.with_status(200)
.with_header("content-type", "application/json")
.with_body("[]")
.create_async()
.await;
// Human mode: no control bytes or bidi overrides reach the terminal
let mut buf = Vec::new();
let args = WhoamiArgs {
dir: Some(dir.path().to_path_buf()),
node: Some(server.url()),
json: false,
};
run_to_writer(args, &mut buf).await.unwrap();
let out = String::from_utf8(buf).unwrap();
assert!(!out.contains('\u{1b}'), "ESC leaked in human mode: {out:?}");
assert!(!out.contains('\u{07}'), "BEL leaked in human mode: {out:?}");
assert!(
!out.contains('\u{202e}'),
"RLO leaked in human mode: {out:?}"
);
assert!(out.contains("repo:write"), "benign text missing: {out:?}");
assert!(out.contains("tfel"), "reversed text missing: {out:?}");
// JSON mode: serde escapes C0 but passes bidi — ensure no U+202E
let mut buf = Vec::new();
let args = WhoamiArgs {
dir: Some(dir.path().to_path_buf()),
node: Some(server.url()),
json: true,
};
run_to_writer(args, &mut buf).await.unwrap();
let out = String::from_utf8(buf).unwrap();
assert!(
!out.contains('\u{202e}'),
"RLO leaked in JSON mode: {out:?}"
);
}
}
+4
View File
@@ -44,6 +44,10 @@ services:
GITLAWB_BOOTSTRAP_PEERS: ${GITLAWB_BOOTSTRAP_PEERS:-}
GITLAWB_BOOTSTRAP_DISABLE_SEEDS: ${GITLAWB_BOOTSTRAP_DISABLE_SEEDS:-false}
GITLAWB_REQUIRE_SIGNED_PEER_WRITES: ${GITLAWB_REQUIRE_SIGNED_PEER_WRITES:-false}
# Owner-only push. The default matches the node's own, so the container path
# behaves like a source build; forwarding it is what makes the documented
# rolling-upgrade opt-out reachable for an operator running compose.
GITLAWB_ENFORCE_OWNER_PUSH: ${GITLAWB_ENFORCE_OWNER_PUSH:-true}
GITLAWB_P2P_BOOTSTRAP: ${GITLAWB_P2P_BOOTSTRAP:-}
GITLAWB_PUBLIC_READ: ${GITLAWB_PUBLIC_READ:-true}
GITLAWB_MAX_PACK_BYTES: ${GITLAWB_MAX_PACK_BYTES:-2147483648}
+2 -2
View File
@@ -38,8 +38,8 @@ Owner focus: node reliability.
Owner focus: protocol/auth.
- Implement repo write authorization: repo owner checks, protected branches, UCAN capability checks, and clear delegation semantics.
- Implement private-read enforcement or remove private-repo affordances until it exists.
- Close default-open repo write authorization and wire trusted UCAN delegation into repository permissions.
- Close the remaining visibility gaps: restrict or document task, pin, and anchor metadata routes, and publish an irreversible-publication policy.
- Add UCAN revocation or blocklisting, with an emergency compromised-key runbook.
- Add mutation-aware GraphQL auth before GraphQL becomes a public write API surface.
- Harden peer registration and outbound peer calls against SSRF and peer-list poisoning.
+6 -1
View File
@@ -3,6 +3,11 @@
Date: 2026-05-28
Repo state inspected: `main` tracking `origin/main`, starting at `b12c6bc feat: per-DID rate limiting on creation endpoints (10/hour) (#13)`.
> **Historical snapshot.** This audit describes the repository at the commit above. Its
> statements about incomplete UCAN chain validation and absent per-repository read
> enforcement have been superseded; see [`SECURITY.md`](../SECURITY.md) for the current
> security posture.
## Commands run
```sh
@@ -88,7 +93,7 @@ Fixed or staged in this pass:
Live-network blockers to prioritize:
- GraphQL POST is still open for compatibility; GraphQL mutations should get mutation-aware auth before it becomes a public write API surface.
- Push authorization is still not capability-complete. A valid DID signature is authentication, not authorization; unprotected repo branches do not yet enforce owner/UCAN capability checks.
- Push authorization is still not capability-complete. A valid DID signature is authentication, not authorization. Owner checks are now enforced on every branch, protected or not (`GITLAWB_ENFORCE_OWNER_PUSH`, on by default); what remains is that a UCAN `git/push` capability is not yet honored, so a delegated or CI key cannot push.
- UCAN chain validation is incomplete and UCAN revocation/blocklisting is not implemented as an operator feature.
- Private repository reads are not enforced. `is_public` and `GITLAWB_PUBLIC_READ` exist, but per-repository private-read behavior is not wired.
- Peer URLs are self-asserted by DIDs. Signatures prove control of the DID key when present, not ownership/safety of the announced URL.
+35 -20
View File
@@ -141,31 +141,43 @@ During the cooldown your node still earns rewards if it keeps heartbeating.
---
## Hardening: owner-only push
## Owner-only push
By default the node authenticates every `git-receive-pack` push (a valid RFC 9421
`did:key` signature) but does **not** check that the pusher owns the repo, except
on branches that are explicitly protected. Because `did:key` is self-certifying,
any party can generate a key, derive its DID, sign, and push to an unprotected
branch — authentication is not authorization.
The node requires the authenticated pusher to be the repo owner on **every**
branch. A push whose authenticated DID is not the repo owner is rejected with
HTTP 403 before any ref update is applied. The owner is matched in both the full
`did:key:z6Mk…` form and its bare `z6Mk…` suffix.
To require the authenticated pusher to be the repo owner on **every** branch, set:
This is on by default, and the default is the point. The node authenticates every
`git-receive-pack` push with a valid RFC 9421 `did:key` signature, but `did:key`
is self-certifying: any party can generate a key, derive its DID and sign.
Authentication is not authorization, so without this gate every signed caller can
push to every repository — private ones included — on any branch that is not
explicitly protected.
### Turning it off
```bash
GITLAWB_ENFORCE_OWNER_PUSH=true
GITLAWB_ENFORCE_OWNER_PUSH=false
```
- **Default `false`** — preserves current behavior so live nodes are unaffected by
an upgrade. Turn it on once you're ready for owner-only writes.
- **When `true`** — a push whose authenticated DID is not the repo owner is
rejected (HTTP 403) before any ref update is applied. The owner is matched in
both the full `did:key:z6Mk…` form and its bare `z6Mk…` suffix.
- **Caution: this blocks every non-owner pusher, including your own delegated and
CI agents.** Push authorization is owner-only today — a UCAN `git/push`
capability is verified but not yet honored for authorization, so delegated keys
cannot push while this is on. Don't enable it until every identity that pushes
to your repos is the owner, or you'll lock out your own automation. Scoped
collaborator / UCAN-delegated push rights are a planned follow-up.
Both the environment variable and `--enforce-owner-push false` disable it; the
bare `--enforce-owner-push` flag still means `true`.
Do this only for a rolling upgrade whose pushers are not yet the repo owner, and
treat it as temporary: while it is off, your node accepts a push to any repository
from anyone who can generate a keypair.
### Caution: delegated and CI keys are non-owners
Push authorization is owner-only today. A UCAN `git/push` capability is verified
but **not yet honored for authorization**, so a delegated key cannot push while
this gate is on, even when it holds a valid capability for the repo.
If your automation pushes under its own DID rather than the owner's, it will start
getting 403s. Either have it push as the owner, or set
`GITLAWB_ENFORCE_OWNER_PUSH=false` until scoped collaborator / UCAN-delegated push
rights land — that work is what removes this trade-off.
---
@@ -177,7 +189,8 @@ GITLAWB_ENFORCE_OWNER_PUSH=true
| Missed heartbeats | After 3 days without one, you're excluded from rewards until you beat again |
| Operator key | Dedicated wallet, small ETH balance, not your main treasury |
| Monitoring | Watch `lastHeartbeat` on-chain; alert if > 22h since last beat |
| Public URL | Must resolve and serve `/health` — peers will ping it |
| Public URL | Must resolve and serve `/health` for liveness and DB-aware `/ready` for peer readiness |
| Rolling upgrade (advisory-lock algorithm) | The SHA-256 advisory-lock key swap (replacing `DefaultHasher`) re-keys every repo in the first release that carries it. During a shared-Postgres rolling upgrade, old nodes compute a different `i64` key than new nodes for the same repo, so PostgreSQL treats them as independent locks and cross-machine write-exclusion is lost. **Drain in-flight writes** or cut over through a **single node** before bringing new-binary nodes online. |
---
@@ -187,6 +200,8 @@ GITLAWB_ENFORCE_OWNER_PUSH=true
**Node refuses to start with "strict-mode operator check failed"** — either `gl node register` first, or unset `GITLAWB_OPERATOR_STRICT_MODE`.
**Node refuses to start with "GITLAWB_DB_MAX_CONNECTIONS (20) must be at least max_concurrent_git_pushes (32) + 8 headroom"**: raise `GITLAWB_DB_MAX_CONNECTIONS` to at least the push cap plus 8 (40 with the default cap of 32; 48 is the shipped default and the recommended value), or lower `GITLAWB_MAX_CONCURRENT_GIT_PUSHES`. This bites a node upgraded in place that still sets the old pool size of 20. The check is deliberate: each concurrent push pins one pooled connection for its whole receive-pack, so a pool that does not clear the push cap lets a burst of slow pushes starve every other database path.
**Rewards are 0 after a week** — run `gl node onchain-status`. If `currentlyActive: false`, check your heartbeat loop (node logs for `operator heartbeat sent`).
**Want to rotate operator wallet** — requires unstake → re-register with new wallet. No in-place rotation in v1.
+60 -11
View File
@@ -91,13 +91,39 @@ User-data only runs at first boot, so upgrades go through SSM:
$(terraform output -raw upgrade_command)
```
This runs `docker compose pull && docker compose up -d` on the instance.
This reinstalls the rendered `/opt/gitlawb/compose.yaml`, then runs
`docker compose pull && docker compose up -d` on the instance.
**Run `terraform apply` before the upgrade command.** The command is an SSM
document managed by Terraform, and it embeds the compose file rendered from your
current variables. An instance that has not had a fresh `apply` still holds the
previous document and will reinstall the previous compose file.
- With `image_tag = "latest"` (default) that picks up the newest release.
- With a **pinned tag**, first edit the tag in `/opt/gitlawb/compose.yaml` on
the instance (via SSM session), then run the upgrade command — and keep
`image_tag` in terraform.tfvars in sync so a future instance replacement
boots the same version.
- With a **pinned tag**, set `image_tag` in `terraform.tfvars` and `apply` — the
rendered compose carries it, so the upgrade installs the right version.
- **`compose.yaml` is Terraform-owned and is overwritten on every upgrade.**
Per-instance settings belong in `/opt/gitlawb/.env`, which is never touched.
- **The upgrade reconciles the service set, not just the image.** The rendering is
conditional — `postgres` exists only when `db_host` is empty, `caddy` only when
`domain_name` is set — and the command runs `--remove-orphans`. So setting
`use_rds`, or clearing `domain_name`, and then upgrading will **stop and remove**
the local postgres or the TLS terminator. That is the intended outcome of those
variables, but it happens on the upgrade rather than on `apply`. Data survives
either way: both bind-mount under `/mnt/data`.
### Why the upgrade rewrites the compose file
`aws_instance` deliberately ignores `user_data` drift, so an instance keeps the
compose file it was created with. Compose passes only the variables named in a
service's `environment:` block into the container, so **a node setting added to
the template never reaches an existing instance** — writing it to
`/opt/gitlawb/.env` and restarting leaves the node on its built-in default, with
no error to indicate it. `pull && up -d` alone cannot fix that, because the file
on disk is the authoritative one.
Reinstalling the rendering first makes the upgrade a real migration. It is
idempotent: on an already-current instance it writes identical bytes.
Replace the instance itself (OS/AMI/instance-type changes) with
`terraform apply -replace=aws_instance.node` — the data volume reattaches and
@@ -105,12 +131,24 @@ Replace the instance itself (OS/AMI/instance-type changes) with
## Changing configuration
User-data only runs at first boot, and the instance ignores `user_data` drift
(`ignore_changes`), so editing terraform.tfvars values that feed the bootstrap
(`bootstrap_peers`, `public_url`, integrations, `image_tag`) does **not**
affect a running instance on `terraform apply`. To roll out such changes,
either edit `/opt/gitlawb/.env` on the instance (SSM session, then
`docker compose up -d`), or replace the instance:
`terraform apply` alone never changes a running instance: user-data runs once and
the instance ignores `user_data` drift. How a change rolls out depends on which
of the two files carries it.
**Rendered into `compose.yaml`** — `image_tag`, `gitlawb_port`, `metrics_port`,
`domain_name`, `db_host`, the `icaptcha_*` values. `apply`, then run
`upgrade_command`: it installs the new rendering and restarts.
```sh
terraform apply
$(terraform output -raw upgrade_command)
```
**Written into `/opt/gitlawb/.env` at first boot** — `public_url`,
`bootstrap_peers`, `auto_sync`, `max_pack_bytes`, and the integration secrets.
Editing these in `terraform.tfvars` does **not** reach a running instance, because
nothing rewrites `.env` after the first boot. Either edit `/opt/gitlawb/.env` on
the instance and `docker compose up -d`, or replace it:
```sh
terraform apply -replace=aws_instance.node
@@ -118,6 +156,17 @@ terraform apply -replace=aws_instance.node
The data volume reattaches; repos, postgres data, and the identity key survive.
**A node setting the installed `compose.yaml` does not already name** —
`GITLAWB_ENFORCE_OWNER_PUSH` on an instance created before it was added, for
example. Editing `.env` is **not enough**: compose only passes through the
variables its `environment:` block lists, so the value is read from `.env` and
then dropped. Run `apply` plus `upgrade_command` first, which installs a compose
file that names the key; after that, `.env` governs it like any other.
That last case is the one worth remembering: `.env` reaches the container only for
keys the *installed* compose interpolates, and the upgrade is what makes a newly
added key one of them.
## Remote state (optional)
Local state is the default. To move state to S3: create a versioned bucket,
+3
View File
@@ -72,6 +72,9 @@ services:
GITLAWB_BOOTSTRAP_PEERS: $${GITLAWB_BOOTSTRAP_PEERS}
GITLAWB_AUTO_SYNC: $${GITLAWB_AUTO_SYNC}
GITLAWB_MAX_PACK_BYTES: $${GITLAWB_MAX_PACK_BYTES}
# Owner-only push. Matches the node's own default; forwarding it is what makes
# the documented rolling-upgrade opt-out reachable on the container path.
GITLAWB_ENFORCE_OWNER_PUSH: $${GITLAWB_ENFORCE_OWNER_PUSH:-true}
# On-chain PoS (optional — empty unless set in terraform.tfvars)
GITLAWB_CHAIN_RPC_URL: $${GITLAWB_CHAIN_RPC_URL}
GITLAWB_CONTRACT_NODE_STAKING: $${GITLAWB_CONTRACT_NODE_STAKING}
+62 -7
View File
@@ -350,10 +350,16 @@ resource "aws_instance" "node" {
# ami: new AL2023 releases shouldn't churn the instance; replace
# deliberately for OS upgrades.
# user_data: only runs at first boot, so re-rendering it on a live
# instance is a pointless stop/start. Config changes that feed user-data
# (bootstrap peers, integrations, image tag) require a deliberate
# `terraform apply -replace=aws_instance.node` — see README "Changing
# configuration".
# instance is a pointless stop/start. How a config change reaches a running
# instance depends on which file carries it:
# - rendered into compose.yaml (image_tag, ports, domain_name, db_host,
# icaptcha_*): `terraform apply` then `upgrade_command`, which
# reinstalls the rendering and restarts.
# - written into /opt/gitlawb/.env at first boot (public_url,
# bootstrap_peers, integrations): nothing rewrites .env afterwards, so
# these need an edit on the instance or
# `terraform apply -replace=aws_instance.node`.
# See README "Changing configuration".
ignore_changes = [ami, user_data]
}
}
@@ -437,13 +443,62 @@ resource "aws_ssm_document" "upgrade" {
content = jsonencode({
schemaVersion = "2.2"
description = "Pull the latest gitlawb node image and restart the compose stack"
description = "Reinstall the rendered compose file, pull the latest image, and restart"
mainSteps = [{
action = "aws:runShellScript"
name = "upgrade"
inputs = {
runCommand = [
"cd /opt/gitlawb && docker compose pull && docker compose up -d --remove-orphans && docker image prune -f"
# Reinstall the CURRENT rendering before restarting.
#
# `aws_instance` ignores user_data drift, so an instance created before a
# template change keeps the compose file it was born with. Compose passes
# only the variables named in its `environment:` block into the container,
# so a node setting added to the template never reaches an existing
# instance — the operator writes it to /opt/gitlawb/.env, restarts, and the
# node silently keeps the built-in default. `docker compose pull && up -d`
# cannot fix that: the authoritative file is the one on disk.
#
# Idempotent: writing the same bytes and restarting is a no-op. This does
# overwrite hand-edits to compose.yaml, which is intended — the file is
# Terraform-owned; per-instance settings belong in /opt/gitlawb/.env.
#
# One shell step, under flock, writing through mktemp. Two overlapping
# executions — a double run, a retry while the first is still pulling, two
# operators — otherwise share one fixed temp path: the second truncates the
# file the first is about to install, and once the first renames, the
# second's open descriptor keeps writing into the live compose.yaml inode.
# Both outcomes stay valid YAML, so the `up -d --remove-orphans` on the next
# line accepts a blended or truncated service set and deletes whatever fell
# off — on a live node, the local postgres or the TLS terminator. The lock
# also covers pull and up, so a second run cannot restart against a file the
# first is mid-install.
runCommand = [<<-UPGRADE
set -euo pipefail
install -d -m 0755 /opt/gitlawb
exec 9>/opt/gitlawb/.upgrade.lock
if ! flock -n 9; then
echo "another upgrade is already running on this instance" >&2
exit 1
fi
tmp="$(mktemp /opt/gitlawb/compose.yaml.XXXXXX)"
trap 'rm -f "$tmp"' EXIT
# base64 rather than a nested heredoc: the rendered compose contains
# `$${VAR}` passthroughs that must reach the file unexpanded, and a
# heredoc inside an indented Terraform heredoc depends on the dedent
# landing the delimiter at column 0. Decoding a single line has neither
# failure mode, and no delimiter can collide with the content.
echo '${base64encode(local.compose_yaml)}' | base64 -d > "$tmp"
chmod 0644 "$tmp"
mv "$tmp" /opt/gitlawb/compose.yaml
trap - EXIT
cd /opt/gitlawb
docker compose pull
docker compose up -d --remove-orphans
docker image prune -f
UPGRADE
]
}
}]
@@ -19,6 +19,11 @@ class Config {
var operatorPrivateKey: String = ""
var tigrisBucket: String = ""
var autoSync: Bool = false
/// Owner-only push. Persisted as a setting rather than read from a hand-edited
/// `.env`, because `writeEnvFile()` regenerates that file on every start and
/// would discard the edit. Matches the node's own default; set false only for a
/// rolling upgrade whose pushers are not yet the repo owner.
var enforceOwnerPush: Bool = true
var repoPathString: String = ""
// MARK: - Paths
@@ -76,6 +81,7 @@ class Config {
"operatorPrivateKey": operatorPrivateKey,
"tigrisBucket": tigrisBucket,
"autoSync": autoSync,
"enforceOwnerPush": enforceOwnerPush,
"repoPath": repoPathString,
]
@@ -105,6 +111,7 @@ class Config {
operatorPrivateKey = dict["operatorPrivateKey"] as? String ?? operatorPrivateKey
tigrisBucket = dict["tigrisBucket"] as? String ?? tigrisBucket
autoSync = dict["autoSync"] as? Bool ?? autoSync
enforceOwnerPush = dict["enforceOwnerPush"] as? Bool ?? enforceOwnerPush
repoPathString = dict["repoPath"] as? String ?? repoPathString
}
@@ -121,6 +128,7 @@ class Config {
lines.append("GITLAWB_PORT=\(httpPort)")
lines.append("GITLAWB_P2P_PORT=\(p2pPort)")
lines.append("GITLAWB_AUTO_SYNC=\(autoSync)")
lines.append("GITLAWB_ENFORCE_OWNER_PUSH=\(enforceOwnerPush)")
if !chainRpcURL.isEmpty {
lines.append("GITLAWB_CHAIN_RPC_URL=\(chainRpcURL)")
@@ -294,6 +294,10 @@ services:
GITLAWB_HOST: 0.0.0.0
GITLAWB_PUBLIC_URL: ${GITLAWB_PUBLIC_URL:-http://localhost:7545}
GITLAWB_P2P_PORT: 7546
# Owner-only push. Must match the bundled Resources/docker-compose.yml: this
# generated file — not the bundled resource — is what a packaged app runs, so
# omitting it here leaves the documented opt-out unreachable for those users.
GITLAWB_ENFORCE_OWNER_PUSH: ${GITLAWB_ENFORCE_OWNER_PUSH:-true}
GITLAWB_CHAIN_RPC_URL: ${GITLAWB_CHAIN_RPC_URL:-}
GITLAWB_CONTRACT_NODE_STAKING: ${GITLAWB_CONTRACT_NODE_STAKING:-}
GITLAWB_OPERATOR_PRIVATE_KEY: ${GITLAWB_OPERATOR_PRIVATE_KEY:-}
@@ -33,6 +33,9 @@ services:
GITLAWB_PORT: ${GITLAWB_PORT:-7545}
GITLAWB_PUBLIC_URL: ${GITLAWB_PUBLIC_URL:-http://localhost:7545}
GITLAWB_P2P_PORT: ${GITLAWB_P2P_PORT:-7546}
# Owner-only push. Matches the node's own default; forwarding it is what makes
# the documented rolling-upgrade opt-out reachable on the container path.
GITLAWB_ENFORCE_OWNER_PUSH: ${GITLAWB_ENFORCE_OWNER_PUSH:-true}
# Sync
GITLAWB_AUTO_SYNC: ${GITLAWB_AUTO_SYNC:-false}
# On-chain PoS (optional — leave unset for local/dev)
@@ -28,6 +28,7 @@ struct SettingsView: View {
@State private var operatorPrivateKey: String = Config.shared.operatorPrivateKey
@State private var tigrisBucket: String = Config.shared.tigrisBucket
@State private var autoSync: Bool = Config.shared.autoSync
@State private var enforceOwnerPush: Bool = Config.shared.enforceOwnerPush
var body: some View {
VStack(alignment: .leading, spacing: 12) {
@@ -53,6 +54,15 @@ struct SettingsView: View {
.font(.caption)
.foregroundColor(.secondary)
}
VStack(alignment: .leading, spacing: 2) {
Toggle("Require repo owner to push", isOn: $enforceOwnerPush)
Text("A signature proves who is pushing, not that they own the repo. "
+ "Turn this off only during a rolling upgrade whose pushers are "
+ "not yet the owner — while off, anyone who can sign may push to "
+ "any repo on this node.")
.font(.caption)
.foregroundColor(.secondary)
}
Divider()
@@ -96,6 +106,7 @@ struct SettingsView: View {
Config.shared.operatorPrivateKey = operatorPrivateKey
Config.shared.tigrisBucket = tigrisBucket
Config.shared.autoSync = autoSync
Config.shared.enforceOwnerPush = enforceOwnerPush
Config.shared.persist()
Config.shared.writeEnvFile()
+2 -1
View File
@@ -12,7 +12,8 @@
{ "type": "generic", "path": "crates/gitlawb-node/Cargo.toml" },
{ "type": "generic", "path": "crates/gl/Cargo.toml" },
{ "type": "generic", "path": "crates/git-remote-gitlawb/Cargo.toml" },
{ "type": "generic", "path": "crates/gitlawb-attest/Cargo.toml" }
{ "type": "generic", "path": "crates/gitlawb-attest/Cargo.toml" },
{ "type": "generic", "path": "crates/icaptcha-client/Cargo.toml" }
]
}
}
+117
View File
@@ -0,0 +1,117 @@
#!/usr/bin/env bash
#
# Release-versioning coverage gate for the workspace crates.
#
# release-please owns every crate's version. It rewrites a version line only
# when BOTH halves hold: the line carries an `# x-release-please-version`
# annotation, and the crate's manifest is listed under `extra-files` in
# release-please-config.json. A crate missing either half is silently skipped
# at release time and freezes at whatever version it was created with, which
# is exactly what happened to icaptcha-client (added as a workspace member
# after the config was last written, so it sat at 0.4.0 while the rest of the
# workspace walked to 0.7.1 across four releases).
#
# Exhaustive-by-construction: the required set is DERIVED from the workspace
# member list via `cargo metadata`, never from a hand-maintained list here. A
# newly added crate reds CI whether or not anyone remembered this gate exists,
# which is the property a hand list cannot have.
#
# Hard-fail directions, all three release-silent:
# 1. a workspace member absent from extra-files
# 2. a workspace member whose version line lacks the annotation
# 3. an extra-files entry pointing at a manifest that no longer exists
#
# Runnable from anywhere in the repo.
set -euo pipefail
ROOT="$(git rev-parse --show-toplevel)"
CONFIG="$ROOT/release-please-config.json"
if [ ! -f "$CONFIG" ]; then
echo "ERROR: release-please config not found at $CONFIG" >&2
exit 1
fi
# Resolving can refresh Cargo.lock as a side effect, so snapshot and restore it:
# this check must never leave the working tree dirty when run locally. Same
# reasoning as scripts/check-gitlawb-core-deps.sh.
lock_backup="$(mktemp)"
cp "$ROOT/Cargo.lock" "$lock_backup"
restore_lock() { cp "$lock_backup" "$ROOT/Cargo.lock"; rm -f "$lock_backup"; }
trap restore_lock EXIT
# Workspace members as repo-relative manifest paths, one per line. This is the
# derived required set: whatever cargo considers a member must be release-managed.
members="$(
cargo metadata --no-deps --format-version 1 --manifest-path "$ROOT/Cargo.toml" \
| jq -r '.packages[].manifest_path' \
| sed "s#^$ROOT/##" \
| sort -u
)"
# Manifest paths release-please is configured to rewrite.
listed="$(jq -r '.packages["."]["extra-files"][] | .path' "$CONFIG" | sort -u)"
unlisted="$(comm -23 <(printf '%s\n' "$members") <(printf '%s\n' "$listed"))"
orphaned="$(comm -13 <(printf '%s\n' "$members") <(printf '%s\n' "$listed"))"
# Second half of the invariant: being listed does nothing without the marker,
# because the generic updater keys on the annotation to find the line.
#
# Scoped to the [package] table on purpose. A manifest-wide match is satisfied by
# an annotated `version` under any other table, so a crate could leave the version
# cargo actually reads unannotated, pass this gate, and still freeze at release
# time while release-please rewrote the decoy line instead.
unmarked=""
while IFS= read -r manifest; do
[ -n "$manifest" ] || continue
if ! awk '
/^[[:space:]]*\[/ { in_package = ($0 ~ /^[[:space:]]*\[package\][[:space:]]*$/); next }
in_package && /^version = ".*" # x-release-please-version/ { found = 1 }
END { exit(found ? 0 : 1) }
' "$ROOT/$manifest"; then
unmarked="$unmarked$manifest"$'\n'
fi
done <<< "$members"
failed=0
if [ -n "$unlisted" ]; then
{
echo "ERROR: workspace members missing from extra-files in release-please-config.json:"
printf ' %s\n' $unlisted
echo
echo "release-please will never bump these crates, so they freeze at their"
echo "current version while the rest of the workspace moves. Add each one as"
echo ' { "type": "generic", "path": "<manifest path>" }'
} >&2
failed=1
fi
if [ -n "$unmarked" ]; then
{
echo "ERROR: workspace members whose version line lacks the release-please annotation:"
printf ' %s\n' $unmarked
echo
echo 'The generic updater finds the line by its trailing `# x-release-please-version`'
echo "comment. Without it the crate is listed but never rewritten."
} >&2
failed=1
fi
if [ -n "$orphaned" ]; then
{
echo "ERROR: extra-files entries pointing at manifests that are not workspace members:"
printf ' %s\n' $orphaned
echo
echo "release-please fails the release when an extra-file path does not exist."
echo "Drop the stale entry, or restore the crate to the workspace."
} >&2
failed=1
fi
if [ "$failed" -ne 0 ]; then
exit 1
fi
echo "release versioning coverage: OK ($(printf '%s\n' "$members" | wc -l | tr -d ' ') workspace members, all annotated and listed)."