Compare commits

...
Author SHA1 Message Date
beardthelion ad0fdd57c9 fix(git-remote): normalize the shim's accepted socket to blocking (#192)
start_shim sets the listener non-blocking so its accept loop can poll the
stop flag. On unix accept() hands back a blocking socket, so that never
showed; on Windows the accepted socket inherits the listener's FIONBIO
state and std does not reset it, so every connection arrived non-blocking.

handle_conn reads with read_line(..).unwrap_or(0), which turns the
resulting WouldBlock into "peer closed" and returns without writing a
response, dropping the connection and aborting the client mid-request. The
shim accepts as soon as the handshake completes, strictly before the
request bytes arrive, so the window was open on every connection and
closed only because the 5ms accept poll usually lagged the client. That is
what made real_git_withheld_shaped_first_post fail on windows-latest with
os error 10053 and zero POSTs recorded.

Normalizing at handle_conn entry also restores the 30s read timeout on the
next line, which a non-blocking socket silently made meaningless.

The regression test forces the Windows socket shape explicitly, so it is
load-bearing on every platform rather than only the one that exhibits the
bug, and it establishes the accept-before-request interleaving with a
handshake rather than a sleep: a timing-based version cannot flake red but
can flake green under load, which would make the guard vacuous.
2026-08-09 23:24:23 -05:00
beardthelion a53c151879 chore: sync Cargo.lock after rebase onto main 2026-08-09 22:39:31 -05:00
beardthelion 01971f7e11 test(git-remote): make the Windows Job Object mandatory in run_bounded
Resolves jatmn's #192 P2. run_bounded's clean-exit teardown only terminated
the job when one had been established; when assign_job returned None it did
nothing, so a git-remote-gitlawb helper descendant that outlived the `git
fetch` leader while holding the inherited stdout/stderr pipe write-ends stalled
the reader joins on the helper's ~300s HTTP timeout, past the bound run_bounded
promises. The timeout path's reap_tree (taskkill /T) fallback cannot rescue the
clean path: once the leader has cleanly exited it no longer roots the
descendant in the parent-PID tree, so taskkill /T cannot reach the orphan.

The Job Object is the only reliable Windows teardown, so make it mandatory:
assign_job now returns JobHandle and panics per-step if the job cannot be
established, and both teardown paths unconditionally terminate it. Nested
unnamed jobs with KILL_ON_JOB_CLOSE are standard on the shipped
x86_64-pc-windows-msvc / CI runners, so this costs no coverage; it converts a
silent ~300s hang into a loud, actionable failure and strengthens the bound
(KILL_ON_JOB_CLOSE kills the descendant at latest when JobHandle drops).
reap_tree, whose only Windows use was the removed None arm, is narrowed to
cfg(all(not(unix), not(windows))).

The change is entirely cfg(windows). Verified by cross-compiling the tests for
x86_64-pc-windows-gnu (compiles clean, clippy -D warnings clean, no dead_code
from the reap_tree narrowing) and by the unchanged Linux git-remote-gitlawb
suite (green). The Windows runtime teardown is reasoned-not-run here; CI's
Windows job is the runtime gate.
2026-08-09 22:39:31 -05:00
beardthelion 659ee00d7d test(git-remote): tear down the fetch tree with a Job Object on Windows (#192)
run_bounded's clean-exit descendant reap was Unix-only: on Windows a
git fetch leader that exits cleanly leaves the git-remote-gitlawb helper
blocked in its ~300s HTTP request holding the inherited pipes, and
taskkill /T can't reach a descendant once the leader is gone, so the
reader joins blew past the promised bound. The child is now assigned to
a Job Object at spawn (the analog of the Unix process_group(0)), and the
job is terminated on both the timeout and clean-exit paths, which kills
the whole tree regardless of leader liveness. KILL_ON_JOB_CLOSE is the
safety net. The clean-exit regression is now cross-platform so it
exercises this on a future Windows CI lane (#228).

Unix tests unchanged and green; the Windows path is compile-verified for
x86_64-pc-windows-msvc (runtime coverage lands with #228's CI lane).
2026-08-09 22:39:31 -05:00
beardthelion 63cb52b253 test(git-remote): bind the withheld-path test to the expected ACK/NAK rejection
The test accepted any nonzero git exit (with posts>=1) as the recorded #191
withheld rejection, so a truncated response or a later HTTP failure after the
first POST could pass green. Assert the specific `expected ACK/NAK` signature so
only that break passes, and pin LC_ALL=C so the diagnostic stays the untranslated
English string on a git build with l10n installed. (#192 F6)
2026-08-09 22:39:31 -05:00
beardthelion 5a5d4e1b43 fix(node): count fetch completion from the response outcome, not the request
The gitlawb_fetches_total gate keyed on whether the request carried `done`,
which both undercounts a no-done completion (a client can finish a flush-
terminated have round with `ACK <oid> ready` + pack and no `done` -> counted 0)
and, on the filtered path, counts a rejected response per POST (the pre-#191
two-POST negotiation the client refuses was counted twice).

Derive completion from the response instead. upload_pack / upload_pack_excluding
now return whether they served a pack; response_served_pack detects the PACK
magic in both side-band-64k and raw framing. The plain path counts when the
response delivered a pack (catches no-done, skips negotiation-only rounds so an
N-round fetch counts once); the filtered path stays gated on the finalizing
`done` round until #191 makes that negotiation valid. Also tighten
request_finalizes_fetch to reject 0003 framing and trailing bytes after the
`done` pkt-line. (#192 F1/F2 + CodeRabbit repos.rs:761)
2026-08-09 22:39:31 -05:00
beardthelion c45ec9605d fix(node): make metrics::init race-safe with Once
init() did an unsynchronized REGISTRY.get() check then OnceLock::set(...).
expect(...) for each metric, so two concurrent callers both passed the check
and the loser panicked at 'set INFO once'. Two tests call init(), so the
parallel test runner could hit this. Guard the body with std::sync::Once so it
runs exactly once. Verified: a 32-thread barrier stress-test panics before the
fix and passes after (run in isolation; not committed since the process-global
registry makes the race non-deterministic in the shared-binary suite). (#192)
2026-08-09 22:21:00 -05:00
beardthelion 8c04043fb5 fix(git-remote): reject malformed pkt-line length headers instead of flushing
A non-UTF-8 or non-hex 4-byte length prefix was collapsed to 0000 via
unwrap_or, so a garbled header was handled exactly like a flush -- masking
corrupted/desynchronized protocol input and letting accumulated have lines
go out as a synthesized flush round. Propagate the parse error instead;
a real 0000 still parses as a valid flush. (#192)
2026-08-09 22:21:00 -05:00
beardthelion bf4d8cd9d4 test(git-remote): bound the fetch reader joins when the leader exits before a pipe-holder
run_bounded only tore down the process group on the deadline path. When the git
leader exits on its own but a descendant (the git-remote-gitlawb helper mid-HTTP
request) outlives it holding the inherited stdout/stderr write-ends, the reader
joins blocked on the helper's ~300s timeout, past the deadline run_bounded
promises. Close the group on the clean-exit path too, guarded by a kill(-pgid, 0)
liveness probe so a recycled pid is never signalled (the process-group ID stays
reserved while any member is alive). Adds a leader-exits-first fixture and a
regression test.

Verified by execution: with the clean-path reap removed the new test blocks ~10s
on the descendant (RED); with the reap it returns promptly.
2026-08-09 22:21:00 -05:00
beardthelion 1c690c125c fix(node): count a completed fetch once per fetch, not per stateless-RPC POST
git_upload_pack recorded a fetch and observed a pack size on every successful
POST. In HTTP stateless-RPC the plain serve path streams a pack only on the
finalizing `done` round, so an N-round incremental fetch was reported as N
completed fetches. The filtered path (upload_pack_excluding) instead serves a
self-contained full pack on the single POST that reaches it, with no `done`, so
gating purely on `done` would drop those private (path-scoped) fetches to zero.

Count once on the POST that actually completes a fetch: the `done` round on the
plain path (request_finalizes_fetch parses pkt-lines and fails closed on a
malformed body), or the one filtered-serve POST (served_filtered_pack).
should_count_fetch captures the decision; a completed fetch is exactly one such
POST, so a multi-round negotiation is never double-counted.

The observe_pack_size call still measures the request body, not the served pack;
that mislabel predates this change and is left as a follow-up.

Verified by execution: a three-round plain fetch and a no-`done` filtered serve
each record exactly one completion; reverting the gate to count per POST makes
the plain fetch record 3, and breaking the decision to AND drops both to 0.
2026-08-09 22:21:00 -05:00
beardthelion 22ec66a846 test(git-remote): make the run_bounded harness portable to the shipped windows target (#192)
The large-output and descendants tests re-invoke the test binary as their
fixtures instead of spawning sh/seq, so they run on x86_64-pc-windows-msvc;
the non-unix timeout path tears down the whole fetch tree with taskkill /T
before joining the drainers instead of killing only the leader. Also folds
the receive-pack POST block into post_pack_round (review nitpick), removing
the duplicated status-check/forward logic.
2026-08-09 22:21:00 -05:00
beardthelion 4201ce4d58 test(git-remote): reap the whole fetch process group and make the withheld guard load-bearing (#192)
Address jatmn's two P2 review findings on real_git_fetch.rs.

F1 (INV-22): run_bounded killed only the top-level `git fetch` on the
timeout path, so the git-remote-gitlawb helper descendant kept the
inherited stdout/stderr pipe write-ends open and the reader joins blocked
on the helper's ~300s HTTP timeout instead of the 30s deadline. Spawn the
child with process_group(0) and, on timeout, tear the whole group down
(SIGTERM, ESRCH poll, SIGKILL escalation, bounded cap, then reap) before
joining, mirroring gitlawb-node/src/git/smart_http.rs. libc is added as a
cfg(unix) dev-dep; non-unix keeps the child.kill() fallback so the
always-on test still compiles. New regression run_bounded_reaps_descendants_holding_the_pipe:
a backgrounded sleep inherits the pipe and outlives a 1s-deadline leader;
reverting to a leader-only kill turns it RED (elapsed ~10s).

F2 (INV-21): the non-success branch of real_git_withheld_shaped_first_post
accepted any `git fetch` failure and only printed the POST count, so a
failure before the first POST (broken advertisement, helper lookup,
connection) passed as the expected withheld rejection. Assert posts >= 1
before accepting the nonzero exit. Forcing a pre-POST failure (0 POSTs)
turns the assertion RED.
2026-08-09 22:21:00 -05:00
beardthelion 1cb930cbd5 fix(git-remote): abort fetch on EOF without POSTing; harden the real-git test harness (#192)
F1: a nonempty have-batch terminated by EOF means git aborted mid-negotiation, but
the fetch loop folded RoundEnd::Eof into the Done arm, appended a synthetic
0009done, and POSTed it — so a cancelled private fetch still issued an authorized,
signed upload-pack request and made the node generate and stream a full pack to a
client that had gone away. Split the arm: only a real done pkt finalizes; a
nonempty EOF terminates without a POST. The nonempty_eof_batch_posts_as_final_round
test that locked in the wrong behavior is flipped to assert no POST.

F2: fetch_with_helper piped stdout+stderr but only polled try_wait, so a fetch
emitting more than an OS pipe buffer (~64 KiB) blocked git before it exited and the
harness reported a false 30s deadlock. Drain both streams on reader threads while
enforcing the deadline, so the deadline measures a real protocol hang only.

F3: the integration repos were removed only on the success path (predictable
pid/counter names), so a timeout/panic leaked real git repos and could poison a
later run. Root them in a RAII tempfile::TempDir that drops on unwind.

RED->GREEN: nonempty_eof_batch_aborts_without_posting (pre-fix POSTs a synthetic
done → the aborted-negotiation POST fails against the single-response server; fixed
makes one request, no POST). repos_are_cleaned_up_on_unwind (temp dir gone after a
panic unwinds past the guard). Full git-remote-gitlawb suite 42 unit + 3 integration
green, fmt + clippy clean.
2026-08-09 22:21:00 -05:00
beardthelion c4609d464a test(git-remote): close branch-coverage gaps in the fetch loop (#117)
Adds executed coverage for the negotiation branches the first pass left
unrun: an invalid pkt-line length (rejected, not underflowed), a non-EOF
read error (propagated, not swallowed as end-of-stream), an immediately
empty upload-pack request (skips the POST), a nonempty have-batch
terminated by EOF (POSTed as a final done round), and an empty
receive-pack body (skips the POST). Each was mutation-checked to fail
without its guard, so the coverage is load-bearing rather than vacuous.
2026-08-09 22:21:00 -05:00
beardthelion 68ba49267c fix(git-remote): skip content-free flush rounds in the fetch loop (#117)
A code-review pass found the negotiation loop would POST once per bare `0000`
flush that carried no haves, so a malformed `wants + N*0000 + EOF` stream
amplified into N signed POSTs. Real git never sends a content-free mid-negotiation
flush and the peer is upstream of the local git process, so this was not
reachable in practice, but it left the loop unbounded per input. Skip a round
that carries no new haves (an empty flush) and only POST rounds with have content
or a terminator, which also makes the loop provably bounded by git's finite have
set. The fresh-clone done-with-no-haves round still POSTs.
2026-08-09 22:21:00 -05:00
beardthelion 89c0ad597f test(git-remote): real-git integration proof of the multi-round fetch loop (#117)
A committed integration test drives a real `git fetch` through the built helper
against a real `git upload-pack --stateless-rpc`, so the stateful-to-stateless
ACK bridging is proven by execution rather than reasoned. The main.rs mock tests
cannot falsify it: a pre-scripted Cursor never reacts to the server's ACKs. An
in-test shim replicates the node's v0 serving; the fetch is forced to at least
two negotiation rounds (a fixture that resolved in one round fails the test) and
the resulting object graph is verified.

The second scenario drives the withheld-blob shape (a full pack on the first
POST, as upload_pack_excluding does) through real git and records the outcome:
real git rejects a pack where it expected an ACK continuation ("expected
ACK/NAK, got ..."), so a multi-round fetch of a withheld repo does not complete.
The helper forwards and terminates cleanly rather than hanging, so this is a
node-side concern, not a helper defect, left as a follow-up per the plan's
Withheld-Path Decision. The test guards the helper's non-hang behavior and
surfaces the break if the assumption ever changes.
2026-08-09 22:21:00 -05:00
beardthelion a64b14150d fix(git-remote): drive multi-round fetch as a v0 stateless-RPC client loop (#117)
git-remote-gitlawb advertised `connect`, so git spoke the stateful native
protocol, but handle_connect collapsed the exchange into one GET plus one POST.
A multi-round fetch (more than ~32 overlapping commits) deadlocked: git sent a
flush-terminated have-batch and blocked for an ACK/NAK the helper never sent,
while the helper blocked reading for a `done` git never sent.

Turn Phase 2 into a per-round stateless-RPC client loop. read_upload_pack_round
returns one round at a time (at a flush, done, or EOF) instead of buffering past
flushes waiting for `done`; negotiate_upload_pack captures the wants once and
POSTs a self-contained request per flush-terminated batch: wants, one flush,
every have accumulated so far, and exactly one terminator. Each ACK/NAK is
streamed back to git so it advances, until the done round returns the pack. The
node's `git upload-pack --stateless-rpc` keeps no state between POSTs, so wants
and all prior haves are re-sent every round and no intermediate flush survives
into a body (which would truncate the negotiation server-side).

Signing is preserved per round: every POST carries the Phase-1 decision (signed
after the 404 escalation for a private repo, anonymous for a public one), and a
mid-negotiation denial surfaces through the sanitized error path rather than
reading as an empty or successful fetch. The receive-pack path is unchanged.

Covers the deadlock repro, multi-round accumulation on the wire, per-round
signing for private and public fetches, mid-negotiation denial surfacing, the
withheld-shaped forwarding path, and single-round non-regression.
2026-08-09 22:21:00 -05:00
Kevin Codex c10ccf1f05 Merge pull request #255 from Ayush7614/fix/250-opaque-graphql-db-errors
fix(node): opaque GraphQL DB error messages (#250)
2026-08-10 10:24:15 +08:00
Kevin Codex 7b365b0dc3 Merge pull request #249 from MikeTomlin19/agent/harden-release-tag-239
fix(ci): reject unsafe release dispatch tags
2026-08-10 10:22:10 +08:00
Kevin Codex 218d6f169a Merge pull request #214 from Gravirei/fix/issue-211-icaptcha-mock-consumed-guard-gaps
test(icaptcha): harden mock-consumed guard with proxy observer and blackhole tripwire
2026-08-10 10:20:24 +08:00
Kevin Codex 817698a0d9 Merge pull request #209 from Gitlawb/docs/agents-md
docs(agents): add AGENTS.md carrying the authorization rules PRs most often break
2026-08-10 10:16:53 +08:00
Kevin Codex e969dca718 Merge pull request #245 from Gitlawb/test/icaptcha-empty-key-filter
test(icaptcha-client): bind new()'s empty-API-key filter (#216)
2026-08-10 10:12:53 +08:00
Kevin Codex 1989fcca11 Merge pull request #202 from euxaristia/fix/pr-triage-async-test-detection
fix(ci): recognize namespaced test attributes in pr-triage's needs-tests check
2026-08-10 10:04:53 +08:00
Kevin Codex 5981257be5 Merge pull request #174 from Gitlawb/fix/served-git-concurrency-cap
feat(node,git): cap concurrent served git ops with a 503 load-shed (#62)
2026-08-10 10:00:06 +08:00
beardthelion 91ff1b7237 docs(node): correct the read-pool comment and list the served-git concurrency knobs
The git_read_semaphore doc claimed it bounds both info/refs advertisements,
but the receive-pack advertisement draws from git_push_advert_semaphore
(api/repos.rs), so an operator sizing the read pool for both would undercount
their read capacity.

README's settings table also omitted the served-git concurrency env vars that
.env.example already documents: MAX_CONCURRENT_GIT_OPS, MAX_CONCURRENT_GIT_PUSHES,
MAX_CONCURRENT_READS_PER_CALLER, and MAX_CONCURRENT_PIN_TASKS.
2026-08-06 23:27:03 -05:00
beardthelion dac29e550c docs(env): stop claiming the pin pool bounds retained memory
GITLAWB_MAX_CONCURRENT_PIN_TASKS caps how many pin loops run at once, not how
much object-id list memory is retained. On the local IPFS path the caller
materializes the list before pin_new_objects_gated acquires, so a parked loop
still holds its full list and the parked count is capped only per repo by
EncryptInflight. config.rs and the pin_new_objects_gated docstring already say
this; the env template still said the knob bounds the memory.
2026-08-06 21:54:31 -05:00
Gravirei dc6b19ddc4 test(icaptcha): tag the observer flush by source port, not by counter tick
The drain barrier sampled target = count + 1 before connecting the flush and
waited for any next increment. A leak accepted after that sample and before
the flush was counted satisfied the wait; the test then subtracted the flush
as if it were that leak, so leak_count could be 0 with a real leak present
(false green).

Tag the flush connection instead: the accept loop records each accepted
connection's source port, and drain waits until the flush's own source port
is recorded (the flush is never counted). Because accept() is FIFO, when the
flush is recorded every connection that completed its handshake before it is
recorded too, so the post-drain count still covers all egress in flight.
Verified with a probe that reproduces the reported scenario (leak accepted
before the flush is now counted) before removing it.
2026-08-06 11:36:03 +06:00
Gravirei 59378ea4e3 test(icaptcha): use [::1] for the blackhole tripwire fixture
The tripwire bound a hard-coded 127.0.0.2 alias, which macOS does not route
by default (only 127.0.0.1), so the test could not run on apple-darwin
targets.  Switch the fixture to [::1] (IPv6 loopback), which is present by
default on Linux, macOS, and Windows and is NOT covered by NO_PROXY
(127.0.0.1, localhost).

The negative control still holds: hyper-util's proxy matcher has no implicit
loopback bypass — a host is only proxied-around when literally listed in
NO_PROXY (verified in the pinned hyper-util source).  Both the positive
control (direct reach of [::1]) and negative control (proxy intercept) still
pass.
2026-08-05 14:31:53 +06:00
Gravirei f741c72103 test(icaptcha): address round-3 review findings on guard and tripwire
- Tripwire: fail loudly (expect) when 127.0.0.2 cannot be bound, instead of
  printing SKIP and returning green with zero coverage of the negative
  control.  Linux routes 127/8 as loopback; macOS needs an explicit alias,
  and the message says so.
- Observer: restore a bounded drain/ack barrier (Mutex + Condvar).  The
  detached accept thread signals after each accept; the test opens a flush
  connection and blocks until it is acknowledged.  accept() is FIFO, so every
  leak connection that completed its handshake before the flush is counted
  before the flush.  Leaks that start their handshake after the drain remain
  the irreducible window, now documented as such.
- support: document that arm_blackhole / disarm_proxy_env mutate process-wide
  env vars and are safe only because each consuming binary has exactly one
  test; a future second test must be serialized against them.
2026-08-05 14:16:40 +06:00
Gravirei 366120e081 test(icaptcha): remove unbounded stabilise polling from mock-consumed observer
The stabilise() polling loop cannot prove no more connections are coming,
only that none arrived in the last 100ms. A carefully timed background
leak could schedule its connection after stabilise stabilises and the test
would pass despite a live egress.  Replace with a simple counter read
before and after obtain_proof.  Connections accepted during the synchronous
call are counted; a leak whose TCP handshake completes after the counter
is read is outside the assertion window — this is now documented as a
known limitation of the single-threaded test harness.

Also narrow the fire-and-forget claim: the observer catches leaks that a
closed-port blackhole would miss (because it counts connections regardless
of error propagation), but the timing race inherent in a detached accept
thread means the guarantee is best-effort for post-return handshakes.
2026-08-05 14:06:53 +06:00
Gravirei 68a68a30f9 test(icaptcha): fix tripwire fixture read deadlock 2026-08-05 14:06:53 +06:00
Gravirei a1b96813ba test(icaptcha): address review findings on mock-consumed guard (#211) 2026-08-05 14:06:53 +06:00
Gravirei 419c2bd230 test(icaptcha): harden mock-consumed guard with proxy observer and blackhole tripwire (#211)
Replace the closed-port proxy with a ProxyObserver (TCP listener with
acknowledged accept-loop flushing) so fire-and-forget leaks whose error
is discarded are caught and the count is synchronized before assertion.
Add a blackhole tripwire asserting obtain_proof fails against 127.0.0.2
with the proxy armed, narrowed to a connect-level error to prevent
vacuous passes. Share arm_blackhole across both test files via a
tests/support module.

Document the observer's coverage scope: it only witnesses leaks on the
single exercised happy path; error/retry branches and non-proxy
transports are outside its guarantee.
2026-08-05 14:06:53 +06:00
mike dfe52dc8dd test(ci): pin release resolver checkouts 2026-08-05 01:22:10 -04:00
mike bc0895299c test: snapshot resolver whitespace 2026-08-05 01:22:10 -04:00
mike fa5cd65031 test: pin release resolver steps 2026-08-05 01:22:10 -04:00
mike 4c1d7814d4 test(ci): bind resolver job mutation 2026-08-05 01:22:10 -04:00
mike af5a3888a8 test(ci): harden release workflow guard 2026-08-05 01:22:10 -04:00
mike ab653e9b94 test(ci): stabilize release bypass fixture 2026-08-05 01:22:10 -04:00
mike fe07564837 test(ci): harden release tag drift guards 2026-08-05 01:22:10 -04:00
mike eac0cf7bca test(ci): enforce release tag invariants 2026-08-05 01:22:10 -04:00
mike 62513c8555 test(ci): cover remaining invalid release tags 2026-08-05 01:22:10 -04:00
mike 1b65f70403 fix(ci): add tested release tag resolver 2026-08-05 01:22:10 -04:00
mike 0b990d687f fix(ci): reject unsafe release dispatch tags 2026-08-05 01:22:10 -04:00
beardthelion 5b54814929 fix(node): stop reporting an unreadable object store as readable, and bound the pin permit
Two review findings from the last round.

The object-store readability probe swallowed EACCES on objects/pack exactly
like the benign NotFound of a loose-only store, and never checked the loose
fan-out at all. A present object behind an unreadable directory therefore
looked absent: git cat-file printed a clean missing, the probe said the store
was fine, and /ipfs/{cid} answered 404 for content that exists. The same bool
drives classify_store_fault, so an unreadable store also came back as a
terminal 500 rather than a retryable 503. Only NotFound stays benign now, and
the probe additionally opens the object's own fan-out path. That check is
scoped to the oid (validated as 40 or 64 hex before any path component is
derived) rather than sweeping all 256 directories, so an unrelated broken
directory cannot veto every lookup in the repo, and the caller pays one open
instead of a directory drain it does not need. An unreadable directory now
fails closed: absence cannot be certified through something we cannot read, so
a missing object there sheds a retryable 503 instead of a 404.

Both pin loops held the shared pin permit across a synchronous, unbounded
git read, so a wedged filesystem pinned a slot in a pool that defers rather
than sheds, and blocked a runtime worker while it did. The read now runs
through the bounded, process-group-reaped helper under spawn_blocking, against
the batch deadline, and the Pinata loop gets the batch budget it never had.
The budget gate is shared between the two loops so they cannot drift, and it
declines to start a read it cannot pay for rather than spawning a child only
to reap it.

A fault stops the batch only when the object store is genuinely unreadable,
where every remaining object would fail the same way. An object-scoped fault
skips that object and continues, because one corrupt object should not cost a
healthy batch, and the recovery path re-offers the same list and would break at
the same place forever.
2026-08-04 17:44:59 -05:00
beardthelion 8ccf2b12e4 docs(agents): drop retired ungated exceptions; fix forged-cert test rule
Jatmn's #209 round: known_ungated is empty on main and the three listed
handlers are gated, so the guide must not teach them as exceptions. Restore
the signature/certificate section with a trusted positive case and a forged
rejection case instead of blessing a forged "valid" drive.
2026-08-04 14:18:26 -05:00
beardthelion c8a6941941 docs(agents): correct three inaccurate claims flagged in review
Scope the denial-test guidance to each route's actual authorization
rule instead of a blanket non-owner assertion, since several
documented routes (star_repo, replica actions, bounty actions,
author-based closing) intentionally authorize non-owners.

Name the known_ungated exceptions (list_webhooks, list_replicas,
list_protected_branches) instead of implying full repo-scoped read
coverage.

Describe the listed jobs as workflow checks that run on every PR
rather than merge-blocking CI, matching the live main branch ruleset
(one required review, no required status checks).
2026-08-04 14:17:41 -05:00
beardthelion 6c9d109c1b docs(agents): add AGENTS.md carrying the authorization rules PRs most often break
Encode the contributor-facing gate rules (the three owner-gate forms in
use and the by-design non-owner exceptions, path-scoped reads and the
listings contract, denial semantics, deny-test expectations, migration
versioning, client denial surfacing, MSRV and the exact CI gate set) in
a root AGENTS.md that coding agents load automatically, and point
CONTRIBUTING.md at it. Enforcement labels state only what is on main
today.
2026-08-04 14:17:40 -05:00
Ayush7614 726ba3361b fix(node): fail-closed GraphQL AppError mapping (#255)
Whitelist eight safe AppError variants; opaque Git and anything else.
Add a source-scrape guard for map_err sites, a real sqlx context chain
test, and a tasks(limit) ceiling pin at 200.
2026-08-04 12:00:07 -05:00
euxaristia 13f6948b3e fix(ci): use anchored regex for pr-triage needs-tests detection
Drop the head-file fetch and Rust lexer. Match test attributes on
added lines only, with horizontal-whitespace anchoring, covering bare
#[test], #[cfg(test)], namespaced forms (#[tokio::test(...)]), optional
leading ::, and raw-identifier spellings. needs-tests is advisory, so
the simpler matcher is enough.
2026-08-04 12:00:07 -05:00
Ayush7614 5077883ad1 fix(node): surface safe AppError variants in GraphQL (#255)
Keep Db/Internal opaque; pass through NotFound/BadRequest/etc. and document
the tasks limit clamp in the schema.
2026-08-04 12:00:07 -05:00
euxaristia 8319c20660 fix(ci): make bounded scan inconclusive, bound blob size, handle no-newline marker, recognize raw-ident test
Addresses jatmn's review: an exhausted 15-file scan or an oversized/
unreadable blob no longer asserts "no test found" (needs-tests was
being applied on unverified evidence), blobs over 5MB are skipped
before decoding, the "\ No newline at end of file" patch marker no
longer miscounts as an added context line, and #[r#test] /
#[tokio::r#test] raw-identifier attributes are now recognized.
2026-08-04 12:00:07 -05:00
Ayush7614 030c5bd3a4 fix(node): address #255 review (clippy, sqlx-only opaque)
Use function-pointer map_err to clear redundant_closure. Opaque only
when a sqlx::Error is in the anyhow chain so claim/finish business
messages survive. Clamp GraphQL tasks(limit); add mutation coverage.
2026-08-04 12:00:07 -05:00
euxaristia fa51d428cb fix(ci): stop bounded scan from aborting triage, use blob API, tighten matching
Addresses jatmn's second round of review on pr-triage's needs-tests check:
hitting the per-PR fetch cap now logs and stops scanning instead of
throwing (which was skipping label reconciliation entirely), file content
comes from the Git blob API instead of the Contents API to avoid the
encoding: none gap for 1-100MB files, the added-line check now requires a
non-whitespace attribute character (not just added whitespace) on the
added line, stripped literals leave a placeholder so they can't glue
adjacent tokens into a false match, and the namespaced-test regex now
accepts a leading :: for absolute-path attributes.
2026-08-04 12:00:07 -05:00
Ayush7614 6b1e1b229b fix(node): opaque GraphQL DB error messages (#250)
Route query and mutation DB failures through a shared helper that
logs the cause chain server-side and returns a generic client
message, so unauthenticated POST /graphql cannot leak sqlx text.
2026-08-04 12:00:07 -05:00
euxaristia 24952b16be fix(ci): handle nesting, multiline attributes, precise terminal matching, and bound Contents API scans 2026-08-04 12:00:07 -05:00
euxaristia 51f72ee3be fix(ci): lex the full head file so state survives across diff hunks
A diff hunk carries no lexical context from lines outside it, so scanning
patch text alone could misclassify an added line that sits inside a comment
or raw string opened earlier in the file (or outside the hunk's context
window). Fetch the head-version file content via the contents API and lex it
in file order, checking only the lines the diff actually added.
2026-08-04 12:00:07 -05:00
euxaristia e458cc2d84 fix(ci): robustly strip comments/strings and match parameterized tests 2026-08-04 12:00:07 -05:00
euxaristia 474b42cfc5 fix(ci): update pr-triage needs-tests regex to anchor attributes and cover parametrized tokio::test 2026-08-04 12:00:07 -05:00
euxaristia 6452e996f8 fix(ci): recognize namespaced test attributes in pr-triage's needs-tests check
The needs-tests detection regex only matched a literal #[test] or #[cfg(test)
immediately after #[, so it missed namespaced test attributes like
#[tokio::test] or #[async_std::test]. Any PR whose only new tests were async
(the common pattern in this codebase, e.g. crates/gl/src/agent.rs) got falsely
labeled needs-tests even when tests were added in the same commit.

Reproduced against PR #198: it added two #[tokio::test]-only tests in the same
commit as the source change, and the triage bot still commented needs-tests.

Widens the regex to also match any attribute ending in ::test], verified
against representative patches including PR #198's actual diff.

Closes #201
2026-08-04 12:00:07 -05:00
beardthelion 753a13b093 test(node): widen the leader-reap test deadline past the child's start race
The `test (beta)` lane failed on 17f9eda reading `leader.pid`, NotFound, while
`test (stable)` passed the same commit. The test armed a 100ms watchdog deadline
and the fake git records its pid as its first statement, so a runner slow enough
to delay `/bin/sh` past 100ms gets the whole process group torn down before the
pid is ever written.

Reproduced by delaying the fake git's first statement by 300ms, which yields the
identical NotFound panic at the same line. With the deadline at 2s the same
injection passes at 0.3s, 1.0s and 1.8s of start delay, so the race window moves
from 100ms to 2s.

The property under test is unchanged: a pre-fix build stands the watchdog down on
drain and blocks on `child.wait()` for the child's full `sleep 30`, which still
trips the 10s recv budget. Confirmed by injecting that exact defect, which goes
red on the budget assertion rather than on some other failure.
2026-08-04 11:31:05 -05:00
beardthelion 730432d0cd test(icaptcha-client): bind new()'s empty-API-key filter (#216)
IcaptchaCfg::new normalizes an empty GITLAWB_ICAPTCHA_API_KEY to None
via .filter(|s| !s.is_empty()) before storing api_key. No test drove
the empty-string value through new(), so a regression would attach an
empty bearer token to the challenge and answer requests and ship green.

The new test drives an empty key through new() on a trusted operator
origin and asserts it normalizes to None, with a contrast arm showing
a real key still attaches. Mutation check: removing the filter fails
this test and no other.
2026-08-03 09:05:14 -05:00
beardthelion 17f9edad47 Merge remote-tracking branch 'origin/main' into fix/served-git-concurrency-cap 2026-08-03 00:20:03 -05:00
beardthelion eb697364bf docs(node): correct the ceiling margin, it is 5.85x not an order of magnitude
Both doc comments describing GIT_SERVICE_TIMEOUT_SECS_MAX said the 100-year
ceiling sits "an order of magnitude" under the ~584-year limit of a
u64-nanosecond Instant. The real margin is 3,153,600,000 seconds against
18,446,744,074, a factor of about 5.85. An order of magnitude means ten.

Worth noting how it spread, since the fix is trivial and the pattern is not.
The phrase was written once on the constant, then copied into the
ipfs_request_budget_secs doc by the commit that was at that moment correcting
a different false claim in the same file. Nobody re-ran the division at either
site, because quoting an existing sentence feels like citation rather than
assertion.

The direction of the error is the part that matters: a vague "well under"
would have been true and would have invited a reader to check. The
precise-sounding "order of magnitude" was false and discouraged checking.

Comments only; the diff carries no executable line, checked by filtering it
for non-comment changes. fmt and clippy clean under the CI form.
2026-07-31 02:09:32 -05:00
beardthelion bc826df2ec fix(review): correct the budget-knob claims and mirror the range in operator docs
Review round on the three preceding commits.

The knob's doc said a value past the ceiling "aborts every /ipfs/{cid}
request in a release build". That is wrong, and three reviewers plus an
outside model landed on it independently: the ceiling is 100 years, the
Instant addition does not actually overflow until roughly 584 years on a
u64-nanosecond Instant, so values in between are representable and are
rejected as a deliberate safety margin, not because they panic. Reworded
to say that, since a confidently wrong comment is worse than the vague
one it replaced.

Also on the knob:
- GIT_SERVICE_TIMEOUT_SECS_MAX has two consumers now, so its own doc says
  so, and notes this knob derives only the Instant addition and not the
  lease-steal multiply.
- The "clears every disable setting" phrasing overclaimed: u32::MAX sits
  above the ceiling. Now names the documented sentinels instead.

Operator-facing docs: the sibling knob's ceiling commit updated both
README.md and .env.example in the same change, and this one had not. Both
now carry the accepted range, matching that precedent.

House style: five comment lines added by the preceding commits used em
dashes, which this project does not allow. Four were carried over from the
text being edited and one was copied from the sibling test. All replaced.
Checked mechanically over the added lines rather than by eye.

Two comments restated an argument in full immediately after pointing at
the canonical site for it. Trimmed to the pointer plus the local delta.

Scoping correction worth recording: the two preceding commits claimed "no
behavior change". That holds for repos.rs, state.rs, main.rs and
test_support.rs, but a config.rs field doc IS clap --help output, so the
pin-knob doc edit changes operator-visible help text. No runtime behavior
changes either way.

Verification: full suite 733 passed, fmt clean, clippy clean under the CI
form. The range guard was separately proven load-bearing in both
directions by mutation: reverting to range(1..) goes RED on the rejection
assertions, and tightening the ceiling to 500_000_000 goes RED on the
disable-value assertions.

Known residual, not fixed here: the bound is parse-side only, so a Config
built by direct construction rather than through clap can still hold an
unrepresentable value.
2026-07-31 01:24:46 -05:00
beardthelion 050b6d2832 docs(node): state what the permit-less load-shed peek actually delivers
The three handler peeks said they shed "before any DB/disk work", which
reads as an admission bound on the DB window. It is not one. The peek
holds no permit, and permits are only held from the authoritative acquire
below, after the visibility and rate gates, so a burst arriving while
permits are free all proceeds into the DB and none of it sheds. What the
peek actually gives is a cheap 503 for one request once the pool has
ALREADY filled, sparing that request's DB work.

The gate ordering that makes it a peek is deliberate and stays: a denied
or rate-limited request must consume no slot, and one source must not
hold global slots through the DB/visibility window. Reordering to bound
the window would undo that. So this corrects the claims and changes no
behavior; bounding the DB window would need an admission mechanism the
peek is not.

test_support.rs's receive-pack docstring additionally called the peek
"the permit" and "the first statement". Both were wrong at this head: it
is a non-holding peek, and the authoritative write-pool acquire sits
after the per-repo lease. Corrected along with the shed claim. The
"remove the block and this goes red" sentences stay, since those tests
drive an already-saturated pool, which is the case the peek does handle.

Sweep over "before any DB", "before DB", "pre-DB". Left alone with
reason: api/ipfs.rs:142 describes a HELD walk-admission acquire, so its
claim is true; graphql/mutation.rs:201 and test_support.rs:528 are about
auth ordering, unrelated; the repos.rs test docstrings at 4194/4213/4556/
4618/4682 describe test mechanics against a saturated pool, which is the
case the peek delivers.

No behavior change: the diff is comments and docstrings only, checked by
filtering the diff for non-comment lines.
2026-07-31 00:59:51 -05:00
beardthelion f91b224bab docs(node): state what the pin-admission pool actually bounds
The pin pool's docs claimed it bounded "how many MB-scale lists are held
at once across all repos". It does not. On the local IPFS path the caller
materializes the object list before pin_new_objects_gated acquires, so a
task parked waiting for a permit is still holding its full list, and how
many tasks park is capped only per repo by EncryptInflight. Across
distinct repos that retained memory is unbounded, which is the gap a
reviewer raised against this claim.

Correct the claim at each site rather than implying the pool closes an
exposure it cannot see. The pool does bound concurrent pin loops, and
therefore lists held while being pinned; that part was always true and is
kept. The residual is now named where each claim lives, including on the
config knob, so an operator does not read the knob as a memory cap it
cannot deliver.

Bounding the retention is a real change to the capture shape (the Pinata
twin already avoids it by acquiring before it derives) and is not
attempted here.

Sites corrected: pin_new_objects_gated's docstring and its F2b empty-list
comment, the empty-list test's docstring, AppState::pin_semaphore and
AppState::encrypt_inflight field docs, the max_concurrent_pin_tasks knob
doc, and the construction comment in main.

Sweep over pin_semaphore, F6, and "held at once". Left alone with reason:
ipfs_pin.rs 16/31/174-200/518 describe the loop that genuinely holds the
permit, so their claims are about hold duration and are true; repos.rs
1029-1036 and 2276-2294 describe the Pinata path, which acquires before
materializing, so its "at most pin_semaphore permits' worth exist at once"
holds; repos.rs 5898 claims concurrent loops are bounded, which is the
true concurrency claim; the F6 hits in api/ipfs.rs are that file's own
review numbering for walk-permit ordering, unrelated to this pool;
constructions and test fixtures carry no claim.

No behavior change: the diff is comments and docstrings only, checked by
filtering the diff for non-comment lines.
2026-07-31 00:57:53 -05:00
beardthelion cf7fe46df2 fix(node): bound GITLAWB_IPFS_REQUEST_BUDGET_SECS by representability
ipfs_request_budget_secs parsed range(1..), so every positive u64 was
accepted, but get_by_cid derives the request deadline as
Instant::now() + Duration::from_secs(that). That addition is an explicit
overflow check rather than a debug-only one, so an oversized value aborts
every /ipfs/{cid} request in a release build instead of setting a very
long budget. The route is anon-reachable, so an operator typo is felt
publicly.

Reuse GIT_SERVICE_TIMEOUT_SECS_MAX, the ceiling the sibling knob already
carries: the bound is representability, not a policy view of a sane
budget, so the large "set it very large to disable" values operators were
told to use still parse. Reject rather than clamp, matching the other
bounds in this config.

Execution evidence:
- RED before the change: the new test failed with "3153600001 is past the
  representable ceiling and must be rejected at parse time" while its
  disable-value assertions passed, so the test is not vacuous in either
  direction.
- GREEN after: config::tests 13 passed, including the sibling
  git_service_timeout regression test.

Sweep for the same defect class, re-run with a widened shape because the
original grep missed a site. Passes: Instant::now() sites; from_secs()
fed by config fields; config fields reaching a Duration through an
arithmetic expression; config fields reaching a tokio timer constructor.
The arithmetic and timer passes each return exactly one hit,
heartbeat_interval_hours (config.rs 150, main.rs 916, operator.rs 161),
which has no range at all and multiplies by 3600 before the Duration.
That knob is a separate defect on a path this branch does not touch, so
it is recorded for a follow-up rather than widened into here. Every other
Instant addition derives from the already-bounded git_service_timeout_secs
or from a constant.
2026-07-31 00:56:04 -05:00
beardthelion 33063baf5a fix(node): set the git_service_timeout ceiling by representability, not tidiness (#174)
A second-model pass on the previous two commits caught an upgrade footgun in the
first and stale prose in the second.

The 365-day ceiling was tighter than the defect required. The help text has
always told operators to set this very large to disable the bound, and clap
accepted every positive u64, so values like 999999999 (~31 years) are working
production "off" settings that neither overflow `* 2 + 60` nor exceed an Instant
deadline. A node running one of those would have failed to start after this
upgrade, over a value that was never the defect. The ceiling moves to 100 years,
which clears every such setting and stays an order of magnitude under the
~584-year limit of a u64-nanosecond Instant, the tightest representation on any
platform we build for. Every value that worked before still parses; only the
ones that would have panicked are rejected. The test now pins the pre-existing
disable values alongside the rejections so the distinction cannot quietly
regress.

Clamp-and-warn was the suggested remedy and is not what landed: it would keep
accepting a value the node cannot honor and silently substitute a different one,
where the range says what is representable and fails fast, matching how every
other bound in this file behaves.

The lease-bound comment and its test still described the pre-ceiling world. Both
now say what is true, and the test deliberately keeps u64::MAX rather than moving
to the ceiling as suggested: Config is reachable by direct construction, and at
the ceiling the arithmetic does not overflow, so a test pinned there would pass
with the saturation removed and prove nothing. Re-confirmed load-bearing after
the rewording.
2026-07-30 18:07:16 -05:00
beardthelion b231d32703 fix(node): bound git_service_timeout_secs to a range its deadlines can represent (#174)
Tracing the lease-bound overflow to its other consumers turned up a worse
instance of the same defect. This PR routed git_service_timeout_secs into
build_filtered_pack and blob_paths, and both build a deadline as
`Instant::now() + Duration::from_secs(..)`. That addition panics on overflow in
RELEASE as well as debug, so a large-but-accepted value crashes the serve path
rather than merely disabling a timeout.

Hardening each arithmetic site as it is found would leave the next one open, so
the bound goes where the value enters: clap now rejects anything above 365 days
at parse time, which keeps every derived duration in range at once and fails
fast at boot instead of at the first path-scoped clone. A year is many orders of
magnitude above any real clone or push, so it remains the practical way to
disable the bound.

The test pins both derivations at the maximum and rejects the value one past it
and u64::MAX. RED when the upper bound is dropped from the value_parser.
2026-07-30 17:28:36 -05:00
beardthelion 6032fd2d95 fix(node): saturate the derived lease steal bound instead of overflowing it (#174)
GITLAWB_GIT_SERVICE_TIMEOUT_SECS accepts every positive u64, so deriving the
per-repo write-lease steal bound as `* 2 + 60` panics every push in a debug
build across the top of the accepted range, and in release wraps to a short
Duration that would let a waiter steal a live push's lease.

Saturating keeps the knob's documented meaning: a timeout that large means the
service bound is effectively off, so the steal bound should be effectively
infinite rather than tiny.

The regression drives git_receive_pack rather than the arithmetic, so the
call-site wiring is what is under test. RED before at repos.rs:1733 with
"attempt to multiply with overflow", GREEN after; re-confirmed load-bearing by
reinstating the unchecked expression.
2026-07-30 17:28:25 -05:00
beardthelion feefaf4f9e fix(node): dispose the write-guard connection when the Drop unlock errors (#174)
Review finding from jatmn, round nine. RepoWriteGuard::release closes the
connection when pg_advisory_unlock returns Err (F3b): the await resolved, so
the session is alive and still holds the lock, and returning that connection
to the pool hands the next caller a lock nobody tracks. The Drop backstop
carried the same hazard and only logged.

Both arms are fixed.

The spawned detached unlock now closes the connection through
close_conn_bounded on the error path, instead of letting the async block end
and return it to the pool.

The off-runtime arm was worse than the finding described. It dropped an
already-taken connection with no unlock attempted at all, and this turned out
to panic rather than leak: PoolConnection::drop calls rt::spawn for its
return-to-pool task, whose no-runtime fallback panics with "this
functionality requires a Tokio context". So any off-runtime guard drop
panicked in a destructor today. It now detaches, which gives up the pool slot
and yields a plain PgConnection whose drop closes the socket, ending the
session that holds the lock. Drop cannot await, so detach is the whole
disposal available here. Confirmed against the vendored sqlx 0.8.6 source
rather than assumed, and min_connections is 0 everywhere in this repo.

Three tests, modeled on the release F3b cases, on a pool with the idle reaper
disabled so a reaped session cannot pass the assertion for us. Observed RED
first:

  write_guard_dropped_off_runtime_disposes_the_connection ... FAILED
    panicked at sqlx-core-0.8.6/src/pool/connection.rs:208:
    this functionality requires a Tokio context
    then: dropping a write guard off a Tokio runtime must not panic
  write_guard_drop_with_failing_unlock_does_not_return_the_connection ... FAILED
    timed out waiting for the connection whose detached unlock errored to be
    closed rather than returned to the pool still holding the session lock
  write_guard_drop_with_successful_unlock_keeps_the_connection ... ok

The success case is green on both sides on purpose: it is the must-not case,
so it only ever catches a widening of disposal onto the healthy path.

GREEN after, and all 12 guard tests pass, so the release coverage did not
regress. Both new guards mutation-proved load-bearing, each red on its own
message, 2/2.

Full suite 729 passed, 0 failed. fmt and clippy clean.
2026-07-30 13:41:11 -05:00
beardthelion fad18bc923 docs(node): stop the per-source cap comments claiming a bound the keying lacks (#174)
Both new comments said saturating the pool takes ~8 distinct source IPs,
each also rate-limited. Neither half holds for an IPv6 caller with a routed
/64: the sub-cap and the rate limiter both key on rate_limit::client_key,
which with TrustedProxy::None falls through to the full peer address with no
prefix folding, so one caller has 2^64 keys. max_keys does not compensate,
since keys are freed on permit drop and live keys never approach the 100,000
default.

The comments now say what the cap actually bounds, concurrent slots per
resolved client key, and name the residual.

The keying fix stays deferred as a design call, and that deferral is
defensible: full-IP keying predates this PR, so these caps help IPv4 and
single-address callers without regressing anything. Landing a comment that
overstates the guarantee is not defensible, in a round that corrects a
weaker overclaim one screen away.

The per_source_push_cap docstring added earlier in this round repeated the
same claim, so it is corrected here rather than shipping fresh.

Comments only, no behaviour change. check, fmt clean.
2026-07-30 13:41:11 -05:00
beardthelion 1501bd1bfd docs(node): drop a backstop claim with no backstop behind it (#174)
The pinata_object_list_for_refs docstring said a visibility tightening that
lands after tail-start is covered by the reconciliation sweep. No such sweep
exists; state.rs says so directly, that a dropped job would be lost forever
precisely because there is none.

The claim is dropped and nothing replaces it. An earlier draft wanted to
substitute a next-push claim, which no site establishes either, and swapping
one unbacked claim for another is the same defect.

Four pre-existing sites make the same claim (repos.rs:30, push_delta.rs at 8,
189 and 281 with warns at 349 and 358). They predate this PR and are left for
a follow-up rather than widening a nine-round diff.
2026-07-30 13:41:11 -05:00
beardthelion 6ef2effe77 test(node): bind the per-source cap test to the production derivation (#174)
advert_per_caller_cap_sizing_is_never_zero opened with a local closure
re-implementing (pushes / 8).max(1), so it bound nothing. Production derives
that expression twice, and the second site was added after the test was
written. per_source_push_cap now lives in rate_limit.rs next to
PerCallerConcurrency, the type the value feeds, and both main.rs sites and
the test call it.

The finding was confirmed by execution before the fix. Deleting .max(1) from
both production sites with the closure still in place:

  injected (2 sites), re-running ... still green
    VACUOUS  the guard's own defect was injected and nothing failed
    0/1 load-bearing

And after, deleting .max(1) from the helper:

  injected (1 site), re-running ... red, on the expected message
    LOAD-BEARING  matched 'advert cap must be >= 1 for pushes=1'
    1/1 load-bearing

The first attempt at that second proof came back RED-WRONG-REASON against
'minimum write pool must derive cap 1, not 0': the loop's assertion for
pushes=1 fires first and aborts before that line is reached. Same property,
earlier assertion.

No mutation residue. Test green, fmt and clippy clean.
2026-07-30 13:41:11 -05:00
beardthelion d9f8f414eb test(node): bind the repo-key boundary test to the separator it names (#174)
separator_prevents_the_owner_name_boundary_collision asserted that
("a","b_c") and ("a_b","c") produce different keys. Those differ under any
join at all, bare included (ab_c against a_bc), so deleting the separator
from repo_identity_key left the test green. It could not fail for the reason
it named.

The inputs are now ("a","bc") against ("ab","c"), both abc under a bare
join. The test docstring said the separator was \0; the code joins with /.
Corrected. The docstring on repo_identity_key itself carried the same wrong
collision example, so it is corrected too.

Proven by mutation rather than argued, joining with no separator:

  old inputs, injected, re-running ... still green
    VACUOUS  the guard's own defect was injected and nothing failed
    0/1 load-bearing

  new inputs, injected, re-running ... red, on the expected message
    LOAD-BEARING  matched 'owner/name boundary must not be ambiguous'
    1/1 load-bearing

Separator restored, no mutation residue. All 6 repo_identity_key_tests pass.
2026-07-30 13:41:11 -05:00
beardthelion 35f71e515d fix(node): charge the /ipfs walk budget for blocking-queue delay (#174)
The walk's deadline clamp was computed on the async side, before
spawn_blocking was called, then used inside the closure. Time spent queued
for a blocking thread went uncharged, so the walk ran with a budget larger
than the request's true remainder. The clamp is now derived inside the
closure at task start.

This is the sibling of the upload-pack site fixed in 28a6ca4, and unlike
that one it sits on the anonymously reachable route.

A queue delay that eats the whole remainder saturates the clamp to zero and
the bounded walk fails closed through the existing taint arm, no verdict,
which is the safe direction.

Testing gap, recorded at the site rather than papered over: the queue-delay
path is reasoned, not executed. Observing it needs a runtime with the
blocking pool pinned and parked, and the sqlx::test harness gives no seam
for that. 28a6ca4 shipped with the same gap for the same reason, verified
against its diff and message rather than assumed. The tests in this file
saturate the admission semaphores, never the blocking pool, so there was
nothing to reuse. An assertion on the arithmetic would only restate the
implementation.

cargo test ipfs::: 24 passed, 0 failed.
2026-07-30 13:41:11 -05:00
beardthelion 03dde22e98 fix(node): bound the IPFS pin batch and let a slow healthy upload finish (#174)
pin_new_objects runs under a pin_semaphore permit and that pool defers rather
than sheds. Both sinks in ipfs_pin.rs built a bare reqwest::Client::new(),
which carries no timeout, so a silent endpoint parked the pool indefinitely.
Both now use the shared no-redirect client from build_http_client() through a
module-local OnceLock, rather than the hand-rolled twin its docstring forbids.

The client alone does not bound the hold: without a ceiling the loop is O(N)
with N chosen by the pusher. A 120s batch budget is taken once at loop start
and gated at the top of every iteration, so no object's work begins with zero
budget left. The remainder is also passed to each add as a per-request
timeout, which is what lets one large healthy upload run past the client's 10s
default without letting the batch run forever. RequestBuilder::timeout
replaces the client value for that request only and leaks nothing to other
callers, verified by execution against the pinned reqwest.

An earlier version of this commit classified a dead endpoint by
reqwest::Error::is_timeout() || is_connect() and broke the batch. That was
wrong in both directions and is gone. It missed the ordinary dead-peer shapes,
since a peer closing mid-request yields IncompleteMessage and a peer RST
mid-upload yields ConnectionReset, both observed with is_timeout and is_connect
false, so a load-balancer idle timeout evaded it entirely. And it fired on a
slow but alive endpoint, because .timeout() is a total deadline covering body
upload, so a healthy multi-megabyte add was read as a dead endpoint and
abandoned the rest of the push. A wall-clock bound needs neither judgement.

The docstring now states what is bounded and what is not, rather than implying
the hold is closed. Not bounded: the git read, since store::read_object shells
two bare Command::output() calls with no timeout or process-group reaping and
this loop does not run it under spawn_blocking; the DB round-trips; and the
pool itself, since the Pinata replication task holds the same semaphore across
a full git re-derivation with no deadline. Truncation is stated plainly: a
batch stopped at the deadline leaves the rest unpinned and nothing sweeps them
up, so recovery is opportunistic through a later full-scan push.

Tests written first, both RED by assertion rather than by a harness timeout:

  pin_new_objects_stops_the_batch_at_its_deadline ... FAILED
    the batch must stop partway, not pin all five and not stall on the
    first: pinned 5
  pin_new_objects_does_not_abandon_the_batch_on_a_slow_but_alive_endpoint ... FAILED
    a slow but progressing endpoint must pin both objects: an upload past
    the client's 10s default is not a dead endpoint
    left: 1, right: 2

Both guards mutation-proved load-bearing against present-but-wrong mutants,
each red on its own message: the loop-top break widened to continue, and the
per-request timeout replaced with None.

Also bounds encrypted_pin.rs's pin_git_object call, where a hang wedged a
repo's EncryptInflight key. Its Drop backstop covers unwind, not hang.

Full suite 730 passed, 0 failed. fmt and clippy clean.
2026-07-30 13:41:11 -05:00
beardthelion a261f7d289 Merge branch 'main' into fix/served-git-concurrency-cap
Brings the branch onto post-#243 main so cargo metadata --locked passes
against the committed tree.
2026-07-30 10:25:07 -05:00
beardthelion afa65e0a59 fix(node): skip an /ipfs absence re-probe the budget cannot pay for (#62)
The absent-CID path disambiguates a clean `missing` by re-running the probe,
but the re-probe took the same deadline with no check that any budget was left.
A first probe that nearly exhausted the budget still spawned a second child
that could only be reaped, and the watchdog's SIGTERM grace plus SIGKILL settle
carried the whole call to roughly twice the budget. Measured on
object_type_bounded with a 1s budget and a probe that burns 0.9s of it: two
spawns and 2021ms elapsed.

That is the same ~2x occupancy shape this PR exists to remove, on a different
handler, and it is worse here than on the served-git path: /ipfs/{cid} carries
only optional_signature, and an absent CID drives this branch once per repo, so
an unauthenticated caller spraying well-formed absent CIDs pays the overshoot
per repo. It also degraded an honest absence into a retryable 503.

Re-probe only when the remaining budget covers what the first probe actually
took, since the re-probe runs the same command. Below that, taint to Transient
rather than spawn: an inconclusive disambiguation is not an absence verdict, so
it must not become a false 404 either. The same 1s case is now one spawn and
finishes inside the budget.

Both introduced by this PR (d39fc8d, the F5 disambiguation); neither
object_type_bounded nor the re-probe exists at the merge base, so this is not a
pre-existing condition. The companion test pins that an ample budget still
re-probes, so the check cannot silently disable the disambiguation it guards.
2026-07-29 22:56:47 -05:00
beardthelion b89dec4ce6 fix(node): charge blocking-queue delay against the shared upload-pack deadline (#62)
The shared deadline computed the walk's budget on the async side, before the
task was queued on the blocking pool. withheld_blob_oids_bounded starts its own
clock when it actually runs, so a budget measured at queue time handed the walk
a full budget from whenever the pool got to it: the read permit was held for
queue delay PLUS the budget, and only the serve phase saw the outer deadline
already spent.

Computing the remainder inside the closure charges that delay against the shared
deadline, which is what makes the ~1x bound in the docs true rather than
approximate. Not a regression from the previous commit (the old code passed a
constant git_timeout, so the delay went uncharged there too, and the combined
hold was queue delay plus two full budgets), but the claim this PR now prints in
README and .env.example should be exact.

The two shared-deadline tests still pass and still fail on revert; they cover
the budget arithmetic, not the queue-saturation path. Reproducing that needs a
runtime with the blocking pool pinned and parked, which the sqlx::test harness
does not give a seam for, so the queue-delay path is reasoned, not executed.
2026-07-29 22:11:43 -05:00
beardthelion 16f4d31900 docs: normalize README punctuation and record the two admission doc changes (#62)
Two table rows needed correcting for the changes in this PR: the
GITLAWB_GIT_SERVICE_TIMEOUT_SECS row now says the classification walk and the
pack serve share one deadline on the path-scoped upload-pack path, so the value
bounds them combined; and the GITLAWB_IPFS_REQUEST_BUDGET_SECS row no longer
claims the probe and content-read subprocesses are unbounded, while keeping the
object-store-readability sweep named as a live residual.

The other thirteen changed lines are punctuation only, no wording changes: em
dashes replaced with colons, commas, or sentence breaks, and one pair of curly
quotes straightened. The repo's prose-style hook validates a whole file rather
than a diff, so these pre-existing characters were rejecting every edit to
README.md, including the two rows above.
2026-07-29 21:48:49 -05:00
beardthelion 9bfe7d5d30 docs(node): correct stale admission-bound comments and document derived caps (#62)
Four prose corrections on this PR's surfaces, no behavior change.

The receive-pack advertisement no longer draws from the write pool, it draws
from a dedicated advert pool, but five comments still described the old
shape: the info/refs per-source block, the cap construction in main.rs, the
State field doc, and a test doc plus its assertion message. The state.rs and
main.rs ones mattered most because they contradicted the operator text being
written in this same commit.

The /ipfs probe and content-read subprocesses each run under their own
deadline now (the lesser of git_service_timeout_secs and the remaining
budget, reaped by process-group teardown), but four surfaces still claimed
they had no duration bound past the pre-start budget check. The correction
stays scoped rather than declaring the route safe: the probe's
object_store_readable check is a synchronous filesystem sweep with nothing to
reap, so a wedged filesystem can still hold the walk slot past the deadline,
and that residual is now stated where the old warning used to be. A grep for
the full phrase misses two of these surfaces, since one wraps across doc-comment
lines and one says "duration clamp" instead, so the sweep uses the fragment.

Both push-side per-source concurrency caps are derived as
max_concurrent_git_pushes / 8 with a floor of 1 and have no environment
variable, so neither was documented anywhere an operator reads. The write-side
one is load-bearing for push availability: it is acquired before the global
write permit, and owner enforcement defaults off, so without it one host
minting disposable did:key identities could monopolize the write pool. They
are described in .env.example beside the read cap rather than in README's
table, which is keyed by variable name.

The hung-walk assertion in visibility_pack printed {err}, showing only the top
anyhow context and dropping the underlying io error. A beta-lane failure
surfaced as "failed to spawn git for-each-ref" with no errno, which made it
undiagnosable. {err:#} prints the chain. The underlying flake is still
undiagnosed: it did not reproduce in 123 local runs and the stable lane passed
on the same commit, so this only improves the next occurrence.
2026-07-29 21:44:53 -05:00
beardthelion 1219c21469 fix(node): share one deadline across the withheld walk and the pack serve (#62)
The path-scoped git-upload-pack branch spent git_service_timeout_secs twice:
once on the withheld-blob classification walk, then a second full budget on
the serve. One clone could hold a read-pool permit for ~2x the configured
budget, which is the occupancy hole this PR otherwise exists to close.

Thread one Instant deadline through both phases, mirroring
fail_closed_full_scan_objects and build_filtered_pack: the walk runs against
the remainder and the serve gets what the walk left, computed after the
blocking task's await so it cannot pick up a fresh budget. A walk that
consumes the whole budget saturates the serve to zero, which surfaces as
GitServiceTimeout -> 504 rather than running unbounded. Under-serving one
slow clone is the safe direction versus over-holding the pool.

Both serve arms are covered because a fix that reached only the plain arm
would leave the ~2x hold reachable by any clone of a repo that actually
withholds a blob. Each test was observed RED before the change (status 200,
the serve completing on a fresh budget) and GREEN after, and reverting the
filtered arm alone flips only its own test, so neither guard is a proxy for
the other.

The knob's meaning changed for this path, so .env.example now says the value
bounds the two phases combined rather than each separately.
2026-07-29 21:44:28 -05:00
beardthelion fed460cf4b docs(node): record the Tigris ordering residual the tail reorder introduces
Spawning the replication tail above guard.release() also moves it above the
Tigris upload, so a ref can be announced while the shared durable copy is still
the old one, and a disconnect there cancels the upload while the detached tail
still pins and announces.

Accepted rather than fixed. Upload-then-announce was never guaranteed: release
tolerates a failed upload by design, warning and continuing to the unlock, so an
announce over a stale Tigris copy was already reachable before this reorder, and
it self-heals through acquire_fresh's local fallback plus the next push. The
alternative, detaching release and the tail together, would return 200 before
the durable copy lands, which changes the client contract more than the window
it closes.
2026-07-29 14:31:51 -05:00
beardthelion 9308b9cc27 test(node): record why the handler path cannot probe the stranded-lock case
A handler-level "the next acquire_write succeeds" check does not discriminate:
it still passes with the RepoWriteGuard::Drop backstop disabled. Measured with
the backstop off, the lock is held at the drop and reads free about two seconds
later from a session outside the pool. #[sqlx::test] builds its pool with
idle_timeout(1s), so the guard's connection is reaped once the handler future
drops it, the session ends, and postgres frees the lock with no help from the
code under test. acquire_write retries for far longer than that window, so it
waits the release out and reports success either way.

write_guard_release_cancelled_mid_unlock_frees_the_lock is the real proof: it
probes from a connection held out of the pool, 400ms after the drop, inside that
window rather than past it.
2026-07-29 13:46:33 -05:00
beardthelion 3ced62fab0 fix(node): spawn the replication tail above guard.release() (#174)
The post-receive tail was spawned after `guard.release(...)`, which on a
successful push awaits the Tigris upload and then the advisory unlock while
the handler future is still tied to the client connection. The pack has
already landed on disk by then, so a disconnect in that window left a durable
push whose pins, recovery copy and announce were never spawned: the dropped
tail F2 closed, one step earlier in the handler.

Spawn it above `release`, gated on an explicit `receive_result.is_ok()`. The
`?` below the spawn can no longer be what keeps a rejected push from pinning
and announcing a half-applied repo, so the success check is now a named flag
that `release` consumes too. The tail is read-only on the repo directory and
holds neither the lease nor the advisory lock, so overlapping it with the
upload waits on nothing the handler still owns.

Tests: a handler-level disconnect driven through a parked `release` (new
store-level test seam), plus the must-not direction, a failed receive-pack
that spawns no tail. The U5 source gate now binds the new boundary: spawn
inside `if push_succeeded`, above `guard.release(push_succeeded)`.
2026-07-29 12:37:39 -05:00
beardthelion 7e5e6aabd0 fix(node): keep the repo identity key printable
The key is logged as the `repo` field on the lease waiter-cap shed and the
steal-bound warning (state.rs), and it was joined with a NUL. Those are the two
messages an operator reads to find a contended repo, and a NUL-hostile log sink
truncates the field at the separator, rendering two different repos' warnings
identically. Before this PR that field was a UUID or a mirror id, both printable,
so the NUL was a regression this change introduced.

Join with '/' instead. It cannot occur in the owner slug by construction - the
slug is owner_did.replace([':','/'], "_") - and it mirrors the shape of the disk
path the key exists to reproduce. A test asserts the key carries no control
characters so this cannot regress silently.

Rotating the U2 lease test's row to a UUID id is not incidental: the mirror path
that seeds it uses 'owner_short/name' as the row id, which now coincides with the
identity key and would leave the test unable to tell the two keys apart. The
rotated id is also the real shape of the bug. Re-verified RED against the reverted
call site after the change.
2026-07-29 06:32:40 -05:00
beardthelion 7fa12bebff style: rustfmt the repo_identity_key tests 2026-07-29 03:04:58 -05:00
beardthelion ade53cbf47 fix(node): spawn the replication tail at the durability boundary
The post-receive replication tail was spawned at the end of git_receive_pack, after
touch_repo, the trust-score update, per-ref certificate issuance and webhook fan-out.
A client or proxy disconnect anywhere in that window dropped the tail, and with it
this push's pins, recovery copy, and announcements - even though the push was already
durable on disk.

Spawn it immediately after the receive_result ? instead. That is the first point where
the push is known to have landed: guard.release() a few lines up runs on failure too,
so anchoring there would spawn a tail for a push git rejected, pinning and announcing
a half-applied repo - the state release() deliberately refuses to upload to Tigris,
and one a hostile pusher can produce on demand by aborting a pack mid-transfer.

RefUpdate gains Clone so the tail takes the parsed updates while the certificate and
webhook loops keep iterating their own copy.

This makes the tail run concurrently with certificate issuance rather than after it,
so a ref can be announced before its signed certificate exists. Accepted, and stated
here rather than left for a reviewer to find: cert issuance already fails open (errors
are logged and skipped) and the gossip event carries cert_id: None regardless, so no
announce consumer reads a certificate out of it.

Evidence is an ordering gate in inv22_gates.rs rather than a cancellation-race test,
matching the F3 gate's instrument: forcing a disconnect to land inside the cert/webhook
window is timing-dependent and would be flaky, while the ordering is precisely what
changed. The gate is mutation-verified in both directions - moving the spawn back below
the webhooks fails the upper bound, hoisting it above the ? fails the lower one.
2026-07-29 03:04:30 -05:00
beardthelion 19291dfd64 fix(node): hold the encrypt-scan permit in its closure and run info/refs under git_bin
Two single-site wiring fixes from the same review round.

withheld_recipients_gated held its git_encrypt_semaphore permit in the async frame
and then awaited spawn_blocking, so dropping the future released the permit while the
uncancellable walk kept running. Every other post-receive scan helper moves the permit
into the closure - replication_withheld_set says so in as many words - so this aligns
the last site with the F4 contract. The regression drops the future mid-walk and
asserts the pool stays at zero while the git child is still alive.

git_info_refs passed a literal "git" to smart_http::info_refs while upload-pack and
receive-pack both pass state.git_bin, so no fake-git harness could drive the
advertisement path. The regression points git_bin at a fake emitting a distinctive
advertisement and asserts it reaches the response body; before the fix the body came
back from real system git (agent=git/2.43.0).
2026-07-29 02:55:20 -05:00
beardthelion 7d32a79e96 fix(node): hold /ipfs walk admission through the blocking walk, not the handler future
The global and per-source walk permits were handler locals, so a client disconnect
released them the moment the future dropped while the uncancellable spawn_blocking
probe/walk/read kept running against a slot the pool now considered free. A disconnect
flood could admit replacements while prior blocking git children still occupied
threads and PIDs, weakening GITLAWB_MAX_CONCURRENT_IPFS_WALKS and
GITLAWB_IPFS_WALK_PER_SOURCE on a surface any unauthenticated caller can reach.

Hold the pair in an Arc and clone it into each of the three blocking closures. The
permits release when the last clone drops, which collapses both failure directions
into one case: a disconnect leaves the abandoned closure holding the slot, and a
panicking closure leaves the handler holding it. No loop arm changes and the
JoinError arms keep today's taint-and-continue.

A move-and-return shuttle was implemented and rejected. It is correct only if every
arm that continues the loop re-binds the returned value, which the compiler cannot
enforce - the absent-object arm Ok(Ok(None)) => continue, taken on nearly every
iteration of a random-CID scan, drops it - and it forces a second decision about the
panic arms whose least-bad answer would let one broken repo shed every other owner's
/ipfs lookups.

The regression strengthens the existing admission test: it now asserts the slot is
still held after the future is dropped and before the child is killed, with a
child-still-alive precondition so it cannot pass vacuously. Only the walk site runs
under state.git_bin and can be pinned that way; the probe and read deliberately shell
to real git, so a source-scan gate in inv22_gates.rs binds all three clone sites
instead. Removing any one clone turns that gate red.
2026-07-29 02:46:51 -05:00
beardthelion 99449cf00b fix(node): key the write lease and encrypt coalescing on the stable disk identity
repo_write_leases and encrypt_inflight keyed on record.id, but RepoStore::local_path
and the pg advisory lock key on the sanitized owner slug plus repo name, and the bare
repo on disk is reused when a repo is deleted and recreated under the same slug. The
row id rotates across that recreate, so both in-process serializers stopped serializing
exactly where the shared objects/ directory still needed them: a disconnected push's
reaper could still be tearing down while a new push took a different key and proceeded.

Add repo_identity_key(owner_did, repo_name) and use it at both sites. Three details are
load-bearing and carry tests: the sanitization is repo_disk_path's replace([':','/'],"_")
rather than normalize_owner_key, which folds a did🔑 prefix and would disagree with
the disk path; the separator is a NUL, without which owner 'a' + name 'b_c' and owner
'a_b' + name 'c' collide; and both call sites pass record.owner_did / record.name rather
than the request path segments, since get_repo normalizes DID aliases and the raw
segments would mint two keys for one directory.

The sanitization is deliberately not injective. Two did:web spellings can fold to one
slug, which would misroute encrypt coalescing if it were reachable, but repos.disk_path
is NOT NULL UNIQUE and holds exactly this derivation, so two such rows cannot coexist.
A test pins that fold in place: making the key injective is what would let two owners
sharing one objects/ directory push concurrently.
2026-07-29 02:39:09 -05:00
beardthelion ea84dc5927 fix(node): write coalesced-drain pin metadata under the re-fetched repo id
resolve_drain_object_list already re-fetches the repo row by owner/name and
deliberately reads visibility rules under record.id, because a delete+recreate
under the same slug gives the row a new id while the bare repo on disk is
reused. The encrypt write a few lines later still took ctx.repo_id, frozen when
the task was spawned, so recovery-pin metadata for a coalesced drain landed
against the deleted row and was unreachable from the live one.

Return the re-fetched id from the resolver and pass it explicitly to
pin_and_encrypt_objects. The parameter is explicit rather than read from ctx so
the two callers stay honest about which id they mean: the snapshot iteration
passes its own spawn-time id, the drain passes the fresh one.

The Arweave anchor is unaffected - EncryptedManifest carries no repo id and
builds its slug from owner and name, both stable across the rotation.
2026-07-29 02:16:04 -05:00
beardthelion 799759f1fc fix(node): sweep the /ipfs rate limiter and document the lease waiter cap
The 300s cleanup task swept the DID, create-IP, push, sync-trigger and
peer-write limiters but not ipfs_rate_limiter, so its expired keys sat
until the map hit max_keys and the inline capacity sweep reclaimed them.
Drive the sweep off AppState in a named sweep_rate_limiters so a limiter
added to the state and forgotten here fails a test instead of leaking
keys, and cover all six with a regression that goes red when any one is
dropped from the sweep.

GITLAWB_REPO_LEASE_MAX_WAITERS was also missing from .env.example and the
README settings table, leaving operators without a way to discover or
tune the cap behind the "repo is busy with another push" 503.
2026-07-28 16:01:30 -05:00
beardthelion c8e8de2949 fix(node): stop a failed walk from taking a pin permit, and bound the close on an errored unlock
Three findings from the review round.

Moving the Pinata gate to the rules-only predicate was necessary, because a
coalesced push has no walk of its own to consult. But it also meant a push whose
OWN walk failed now acquired a global pin permit and ran a second full
re-derivation that failed the same way. Measured on a repo whose rev-list always
fails: twelve git children and three walk attempts, against four and one before.
With eight pin slots that defer rather than shed, eight such pushes stall pins for
every repo on the node. The gate now excludes a non-coalesced push whose own walk
failed, leaving the coalesced path on the rules-only predicate it needs, and the
pin helper returns before taking a permit for an empty object list.

The connection close on the failing-unlock path had no deadline, and release is
awaited inline while the write lease and both admission permits are still held.
The condition that produces that branch is a connection whose last statement
errored, and a blackholed socket to Postgres is one way to get there, so the await
could pin three admission resources for a TCP timeout. It now runs under a five
second bound; on expiry the future is dropped, which closes the socket and ends
the session anyway, which is what frees the lock. What is tested is the deadline.
That sqlx's close is the thing that stalls in production is reasoned from its
source, not reproduced, and the test says so.

The two tests for that path waited a fixed four hundred milliseconds before
asserting. Replacing the sleep with polling was not enough on its own: the test
pool sets a one second idle timeout, so a connection returned to the pool is
reaped shortly after and the session ends without any help from the code under
test. Both tests passed under mutation once the sleep was long enough to reach
that reaper, which means a straightforward flake fix would have quietly gutted
them. They now run against a pool with no idle reaper, so release is the only
thing that can end the session, and both mutations go red again.

Adds tokio's test-util as a dev-dependency for paused-time assertions. Production
dependencies are unchanged and the build stays --locked clean.
2026-07-28 09:14:54 -05:00
beardthelion de325f9f46 fix(node): bound lease waiters per repo, and stop spending cross-repo budget on a per-repo wait
The previous attempt moved the per-source write permit above the lease park to
bound buffered pack bodies. Two reviewers demonstrated that it denies far more
than the source doing the parking. A source parked on one repo's lease was shed
on repos with no contention at all, and because the trusted-proxy setting
defaults to trusting nothing and keying on the socket peer, every pusher behind a
proxy, NAT, or CI runner pool resolves to one key: four parked pushes shed every
push from every pusher to every repo for up to twenty-one minutes. That is the
node-wide shed the permit was moved below the lease to prevent in the first
place, reintroduced at an eighth of the threshold.

The permit goes back below the lease. What is actually unbounded is the waiter
set, so that is what is bounded now, per repo, with a shed past the cap.

The cap counts live waiters, never the entry refcount. The refcount includes the
holder, so capping it would let a lease whose owner died without unwinding pin a
slot forever, which is the permanent wedge the steal timeout exists to survive.
The count is claimed in the same critical section as the refcount and held by a
guard scoped to the cancellable wait, so a cancelled or shed waiter gives it back.
An uncontended acquire takes the fast path and spends no waiter budget; that path
does not barge a queued waiter, which was checked over two thousand rounds rather
than assumed.

Two tests now guard the defect that was just removed: a source parked on one repo
is still served on another, and a parked push does not shed an unrelated pusher
sharing its resolved key. Restoring the old ordering turns both red. Capping on
the refcount instead of live waiters turns the steal-backstop test red. Holding
the waiter slot past the wait turns the fast-path test red.

What this bounds is the parked count, not the parked bytes. At a two gigabyte
pack limit the byte figure stays large, and the total across repos, whose
dominant term is the buffering the signature middleware does before any admission
point at all, is tracked in #263. Past the cap, same-repo concurrency sheds
instead of queueing; that is the trade the knob buys.
2026-07-28 08:01:51 -05:00
beardthelion 1b1a9bab70 test(node): observe the write-cap shed as state, not as a wall-clock deadline
The two F1 tests asserted "the capped source shed rather than parked" by racing a
ten second timeout, and drove the holding push in ten millisecond slices. Under
`cargo test --workspace` that is a race against the scheduler rather than a test:
a reviewer saw two of five runs go red, and under load the pair stretched from
1.8s to 19.2s while the fake git holding the lease only lived for 10s, past which
every assertion in both tests is meaningless.

Shed versus park is a question of which path ran, so both are now read as state.
A shed is `Err(Overloaded)`. A park is a second reference on the lease entry,
exposed by a test-only `refs_for`; `acquire` takes that reference synchronously
before the cancellable wait, so a waiter is visible the instant it reaches the
lease, and it stays visible because the steal bound is 1260s. The tests spawn
their pushes and poll until one of the two mutually exclusive states appears, so
a starved machine iterates more rather than failing.

Every wall-clock assertion is gone. The worst of them was a sleep followed by
`!is_finished()`: a runtime that had not yet polled that task satisfied it
vacuously, which means the constant-key mutation could have gone green under
load, and the discriminator is the whole reason that test exists.

All three mutations still go red, now in seconds rather than by timeout expiry:
the per-caller acquire below the lease, the global permit above it, and the
per-caller key replaced with a constant.

Also fixes a third flake found on the way: the acquire-deadline test's follow-up
push inherited the deliberately short two second acquire timeout, which has to
cover a real advisory-lock round trip, so its `Overloaded` was indistinguishable
from the drained-pool `Overloaded` the assertion was actually testing for.

The sqlx `databases_pkey` collisions seen alongside this are machine residue, not
a leak from these tests: sqlx derives the test database name from a hash of the
test path and does cleanup-then-insert without a lock, so any two concurrent test
binaries running the same path collide. The orphan count here held at 53 across
every run.
2026-07-28 02:40:37 -05:00
beardthelion eb0460ae81 Merge remote-tracking branch 'origin/main' into fix/served-git-concurrency-cap 2026-07-28 00:21:00 -05:00
beardthelion d93f78400c fix(node): coalesce a repeat push before it parks on the encrypt pool, not after
The post-push tail resolved the withheld set and materialized the object list
before it ever consulted the per-repo coalescer, so rapid pushes to one repo each
parked on the encrypt semaphore and repeated the scan before discovering they
should have coalesced. The gate now runs first, keyed on the cheap
listable-at-root predicate that `replication_withheld_set` already computes from
rules alone before it touches the semaphore.

Three things made this less mechanical than it looks, all of them ways a naive
move would have shipped a regression.

The withheld set had a second consumer. It also gated the Pinata pin and the
branch-to-CID upsert, so a coalesced push that skipped the walk would have pinned
nothing and recorded no CID, while a test asserting merely that the announce
spawn ran would have stayed green. That gate moves to the announce predicate too.

Coalescing must not return from the tail. The announce spawn sits below the
coalescer in the same task, and returning early would drop gossip, the GraphQL
broadcast, Arweave anchoring and peer notify, which are per-push and
non-idempotent. A flag skips the encrypt section and falls through.

The publish paths are gated on announce and iterate the ref updates, not the
object list, so handing the tail the cheap predicate would have made a repo whose
walk is failing start gossiping and anchoring, losing the fail-closed property.
`pinata_object_list_for_refs` already recomputes the real set inside the pin
permit, so it now returns that verdict alongside the list and the tail uses it.
No additional walk is introduced.

Moving the gate earlier also opens a window where a push can coalesce while the
admitted one is still walking. If that walk then fails, the admitted task now
still spawns the drain with an empty snapshot so those tips are consumed, rather
than dropping the guard and discarding them. Note a released guard and a dropped
guard are indistinguishable by the in-flight map, since Drop clears the key
either way, so that case is asserted on drained work rather than on the map being
empty.

Each of the four is pinned by a mutation: walking anyway, returning early,
leaving the Pinata gate on the withheld set, and dropping instead of draining
each turn exactly one test red.
2026-07-28 00:05:04 -05:00
beardthelion 72fe5cb3bf fix(node): end the session when the advisory unlock errors, instead of pooling a locked connection
`release` discarded the unlock's result and set `released` regardless, so an
unlock that errored while its session stayed alive (a statement timeout, an admin
cancel) returned a connection to the pool still holding the repo's advisory lock,
with the Drop backstop already disarmed by that flag.

The error is now captured. On failure the connection is taken and closed, which
ends the session and makes Postgres drop the lock server-side, and only then is
the guard marked released, because the lock went with the session rather than
with the unlock. `close()` rather than `detach()` since we are already in an async
fn and can wait for the teardown deterministically.

Disposal is the only mechanism on this path. It cannot coexist with an armed Drop
backstop: disposal needs the connection taken, and Drop returns early when there
is none, while leaving the connection in place so Drop can use it just puts the
locked connection back in the pool when that spawned task ends. The Drop doc now
says so rather than claiming to cover a case it cannot.

The cancellation path is untouched. It was already correct, since `released` is
set after the await, and its test still passes unmodified.

The new tests assert against `pg_locks` from a connection held out of the pool
before the acquire, never by calling `acquire_write` a second time. That shortcut
would pass whether or not the fix is present, because the pool can hand back the
same connection and session advisory locks are re-entrant. The unlock error is
injected by leaving the pinned connection in an aborted transaction, which errors
the next statement while keeping the session and its lock alive, and the test
asserts that precondition before releasing.

Not covered: that the failure path still skips the Tigris upload. The test store
builds with no Tigris client, so the upload site is unobservable here without a
new seam, and that block is untouched by this change.
2026-07-27 23:02:15 -05:00
beardthelion 5b46796b36 fix(node): take the per-source write permit before parking on the repo lease
A push buffers its whole pack before the handler runs, then parks on the per-repo
write lease for up to git_service_timeout_secs * 2 + 60, which is 1260s at
defaults. Both admission permits were taken after that park, so nothing accounted
for waiters: any number of requests could sit on one contended repo, each holding
a fully buffered body, invisible to both semaphores.

The per-source sub-cap now runs before the park. It already exists, is sized at a
quarter of the push pool, and is keyed on the resolved source IP, so parked bodies
are bounded at four per source without a new knob or a new cap to tune.

The global permit deliberately stays below the lease. Moving it up is what
f02c5e1 fixed: a lease-blocked waiter holding a global slot sheds pushes to every
other repo node-wide. A per-source key does not have that shape, since exhausting
it denies only the source doing the exhausting, so the two permits belong on
opposite sides of the park and the comment there now says why.

The load-bearing test is not that a capped source sheds, which any cap satisfies,
including the global one this rejects. It is that a different source still parks
and is served while the first is at its cap. Keying the permit on a constant
leaves the shed test green and turns only that one red. Moving the global permit
above the lease turns the f02c5e1 guard red.

Bounded per source, not globally. A distributed set of sources can still each hold
their four, and the unbounded-total term is the pre-auth buffering in #263 rather
than the lease.
2026-07-27 22:45:48 -05:00
beardthelion 6afa236e65 docs(node): correct three lease/replication comments from code review (#174)
- steal_after: the reclaim is NOT a guarantee that only a leaked lease is
  reclaimed. A waiter's timeout starts at acquire(), not at the FIFO head, so a
  same-repo backlog whose cumulative wait exceeds steal_after can steal while an
  earlier waiter still writes; correctness rests on the retained pg advisory
  lock (which serializes the stealer at acquire_write), not on the bound.
- pinata_object_list_for_refs: recomputes from the tail-start rules snapshot,
  not a fresh read at pin-worker time, so a rule tightened AFTER tail-start is
  not reflected (matches the old retained-list behavior; reconciliation sweep is
  the backstop). Removed the overstated 'rule tightened since the push is honored'.
- RepoWriteGuard::Drop: note that on runtime shutdown the detached unlock task
  may be dropped before it polls, bounded by pool teardown / connection close.
2026-07-20 12:02:46 -05:00
beardthelion e1ab240c7a fix(node): don't 500 an /ipfs miss when a transient taint co-occurs (#174 F5 review)
The deterministic-fault terminal 500 was checked before the transient
truncated_by 503, unconditionally. When one repo is corrupt (deterministic)
while a DIFFERENT repo is transiently skipped in the same scan, the requested
CID may live in the transient repo -- a retryable 503 is correct, but the
handler returned a terminal 500 and a conformant client won't retry, hiding
retrievable content until the unrelated corrupt repo is repaired.

Gate the 500 on truncated_by.is_empty(): it fires only when a deterministic
fault is the SOLE reason nothing served; a co-occurring transient taint falls
through to the retryable 503. A pure deterministic fault still terminally 500s
an absent lookup by design. Adds a co-occurrence regression test.
2026-07-20 11:54:09 -05:00
beardthelion e383047d71 fix(node): take the write permits after the per-repo lease, not before (#174 F3 review)
Code review found a DoS the F3 lease introduced: git_receive_pack acquired the
global and per-source write permits before block-waiting on the per-repo lease,
so a second same-repo push that parked on the lease pinned a scarce global
write-pool slot (1 of 32) for up to steal_after while sending zero bytes. A few
hostile sources stacking same-repo pushes could hold every slot on idle
lease-waiters and shed 503 on every push to every other repo node-wide.

Move both permit acquisitions to after the lease acquire so a lease-blocked
waiter pins no write-pool slot, and keep a cheap non-holding availability peek
before the DB lookup so a push flood on a saturated pool still sheds 503 without
touching Postgres (the authoritative held permit is taken after the lease).
Adds a blocked-waiter-holds-no-permit regression test.
2026-07-20 11:54:09 -05:00
beardthelion 9f4a8c7d2f fix(node): serialize a disconnected push's repo until its group is reaped (#174 F3)
On client disconnect, RepoWriteGuard::Drop released the per-repo advisory
lock immediately (it holds no child/pgid), while KillGroupOnDrop's detached
reaper was still tearing down the git process group for ~4s. A second same-node
push could then acquire the repo and race the first's still-writing objects/
dir and Tigris upload.

Add an in-process per-repo write lease (block-and-wait, keyed like the advisory
lock) that SUPPLEMENTS the retained cluster-wide PG lock. The lease rides into
the disconnect reaper via a clone on the write-path AdmissionGuard (None on all
read paths) and a second clone held across guard.release() for the clean-path
Tigris upload, so it frees only after the group is reaped. A bounded-wait steal
reclaims a leaked lease (block-and-wait has no coalesce degradation). The lease
is always taken before the PG lock and released after, so the two serializers
cannot invert. Deterministic fake-git race test + inv22 gate row.
2026-07-20 10:58:37 -05:00
beardthelion b897e27cf2 fix(node): bound post-receive Pinata replication memory (#174 F2)
The detached Pinata task moved each push's full object-id list into the
closure and parked it on pin_semaphore.acquire_owned(); a tokio semaphore
bounds active holders, not the waiter queue, so a slow Pinata backend let N
queued pushes each retain an MB-scale OID list (unbounded memory/tasks).

Capture only the small ref-update tuples and re-derive the object set via the
existing reaped/deadline-bounded git helpers AFTER the pin permit is acquired,
so retained memory is O(ref tuples), not O(object lists). Per-ref effects are
unchanged (still one spawn per push, no coalescing or shedding, so no dropped
announcements). Adds an inv22 gate row forbidding a retained object list across
the acquire, plus re-derivation-equivalence and reaping tests.
2026-07-20 10:19:50 -05:00
beardthelion e866584784 fix(node): clamp the initial /ipfs metadata queries to the request budget (#174 F6)
list_all_repos and list_visibility_rules_for_repos ran as bare awaits while
holding the scarce IPFS walk permits, so a query blocked in Postgres pinned the
slots past GITLAWB_IPFS_REQUEST_BUDGET_SECS and shed later requests as 503s.
Wrap both in the remaining request-deadline budget; on timeout the RAII permits
drop and the handler sheds a retryable budget 503. Fails closed: a timeout on
the visibility-rules query denies the request rather than serving an unfiltered
listing.

Appends an inv22 gate row asserting both queries stay deadline-wrapped.
2026-07-20 10:00:40 -05:00
beardthelion 48dbb75ac6 fix(node): classify /ipfs object probe with a version-stable signal (#174 F5)
Probe existence via `git cat-file --batch-check` (structured `<oid> missing`
on exit 0) instead of matching English `cat-file -t` prose, so a git wording
change cannot flip a miss into an error or vice-versa. A bad-config/corrupt
repo on a readable store is now a terminal, non-retryable 500 (not a false 404,
and not a retryable 503 that would fan out cat-file retries); a transient
unreadable store stays 503. Error bodies are opaque (no raw git stderr).

Layers onto the existing object_store_readable + re-probe disambiguation.
2026-07-20 09:36:09 -05:00
beardthelion 671cee184f fix(node): make RepoWriteGuard::release cancellation-safe (#174 F4)
Await pg_advisory_unlock while the pooled connection is still owned by
self; take it and set released only after the await resolves. A cancel
mid-unlock now leaves the Drop backstop armed (conn still Some), so the
session-level advisory lock is no longer leaked onto a pooled connection.

Adds a cancel-mid-unlock regression test (via a test-only pre-unlock gate)
and an append-only inv22 gate row asserting the take/released orderings.
2026-07-20 09:14:01 -05:00
beardthelion cc9615961f fix(node): bound concurrent pin-loop memory; document the per-repo task cap (#174 F6)
EncryptInflight bounds the outstanding post-push encryption-task set to one per
repo, but its state.rs doc implied a global cap it does not deliver, and each pin
loop (ipfs_pin + pinata pin_new_objects) holds a full per-push object-id list
while walking it — so N distinct authenticated first-pushes could hold N MB-scale
lists at once, unbounded.

Add a global pin_semaphore (GITLAWB_MAX_CONCURRENT_PIN_TASKS, default 8) and gate
BOTH pin loops through it: they DEFER when the pool is full, never drop a pin.
Tighten the state.rs comment to state the bound is PER-REPO, with the cross-repo
residual (throttled by auth + rate limits, memory bounded by the pin permit)
called out honestly.

Also correct .env.example GITLAWB_DB_MAX_CONNECTIONS (20 -> 48): with the #174 F1
boot validation, 20 is below max_concurrent_git_pushes(32)+headroom(8) and would
fail startup — closing the INV-24 doc gap from the F1 change.

Tests: pin_new_objects_gated_defers_when_pin_pool_exhausted (RED without the
permit acquire); max_concurrent_pin_tasks range/default.
2026-07-18 11:13:32 -05:00
beardthelion e489968fde fix(node): disambiguate absence from an unreadable object store in the /ipfs probe (#174 F5)
object_type_bounded treated any bare `fatal:` cat-file exit as absence. A packed
object whose pack/idx is transiently unreadable (permissions, or a mid-repack
race) emits `could not get object info` — byte-identical to a genuine miss — so a
present object could return Ok(None) and the /ipfs scan a definitive 404 instead
of the intended retryable 503. The stderr cannot separate the two (the CID is
pre-validated, so that string is also the only legitimate absence path), so the
disambiguation must be out of band.

On the collided fatal, probe object_store_readable (objects/ listable, every
pack/idx openable). Unreadable -> Err (taint -> 503). Readable -> re-probe once;
still absent -> Ok(None), now present -> the type. This narrows but does not close
the concurrent-repack TOCTOU (the check samples a different instant), stated
honestly rather than claimed deterministic.

Test: object_type_bounded_unreadable_pack_is_error_not_absence packs a real repo,
chmod 000 the pack, and asserts Err (root-guarded). RED with the readability check
disabled: the present blob returned Ok(None).
2026-07-18 10:57:16 -05:00
beardthelion c866ccb5c8 fix(node): share one whole-scan deadline across both full-scan phases (#174 F4)
fail_closed_full_scan_objects ran phase 1 (replicable_blob_set_bounded) then
phase 2 (all_blob_oids) with a FRESH Instant::now() + timeout for phase 2, under
one held scan permit. A large-but-successful phase 1 plus a full phase 2 held the
permit ~2x the configured budget instead of the one whole-scan budget.

Compute one deadline before phase 1 and share it: phase 2 takes the shared
Instant directly, phase 1 takes the remaining-until-deadline. Total occupancy is
now ~1x. The completeness cost is honest: a repo whose phase 1 nears the budget
under-pins this push (fail closed) rather than over-holding, so size the budget so
both phases normally fit.

Test: full_scan_shares_one_deadline_across_both_phases drives a slow-but-
successful phase 1 that eats the budget; the shared deadline reaps phase 2 and the
scan fails closed (empty). RED before: phase 2's fresh budget completed and kept
the non-blob candidate.
2026-07-18 10:49:18 -05:00
beardthelion fab61e4188 fix(node): bound the /ipfs cat-file probe and content read (#174 F3)
The GET /ipfs/{cid} object-type probe and content read ran bare synchronous
Command::output() on the async worker, budget-checked only before they started.
A hung or corrupt object store therefore pinned a Tokio runtime worker and both
held IPFS admission permits indefinitely, so enough requests exhausted the route
despite GITLAWB_IPFS_REQUEST_BUDGET_SECS. The walk beside them already used
spawn_blocking + the reaped bounded runner; these two stages were left bare.

Extract run_bounded_git_raw (returns ExitStatus/stdout/stderr so callers can
classify exit codes; run_bounded_git stays a thin bail-on-nonzero wrapper for the
walk callers) and add store::object_type_bounded / read_object_content_bounded on
it, preserving object_type's Ok(None)-on-absence vs Err classification so the
serve path's 404-vs-503 semantics are unchanged. Run both /ipfs stages in
spawn_blocking under a deadline clamped to min(git timeout, remaining budget),
with the IPFS permits held across the awaited spawn_blocking.

Test: get_by_cid_hung_probe_is_reaped_and_sheds_503 hangs git on a FIFO alternates
with no feeder; the reaped probe sheds a 503 in bounded time (GREEN), while a
neutralized deadline blocks the handler forever (RED). The existing probe-error /
corrupt-repo tests still assert the absence-vs-error classification at the handler.
2026-07-18 10:36:16 -05:00
beardthelion 6df972d294 fix(node): own the post-receive replication tail in a detached task (#174 F2)
The post-receive replication tail (withheld/candidate/full-scan resolution, then
the encrypt+recovery spawn and the announce spawn) ran inline in the request
future and parked on git_encrypt_semaphore before the durable try_begin gate. A
client/proxy disconnect while parked dropped the future and silently lost this
push's pins, recovery copy, and announcements, with no reconciliation sweep to
recover them (state.rs documented this exact residual).

Move the whole tail into an independently owned tokio task and return the git
response immediately, so a disconnect can no longer drop the parked work. Each
push owns its own tail, including its own always-spawned announce, so the
announce is never brought under the per-repo encrypt coalescing (which would
drop a coalesced push's per-ref announcements). Update state.rs to retire the
disconnect-loses-work residual.

Test: receive_pack_landed_push_returns_without_parking_on_scan_pool asserts the
handler returns 200 while the scan pool is held (RED with the tail inline: the
request future parks and the timeout fires; GREEN detached). The burst
serialization test is retimed to poll for scan completion since the withheld
walk is now detached too.
2026-07-18 10:14:27 -05:00
beardthelion 1b3a5f6064 fix(node): make write-acquire cancellation-safe and connection-affine (#174 F1)
acquire_write took the pg advisory lock, then awaited Tigris, then built the
guard. An outer tokio::time::timeout firing during the Tigris await dropped the
future with the lock held and no guard, so no unlock ran and later pushes to the
repo spun 60s. release() also unlocked on an arbitrary pooled connection, not the
one that locked (session advisory locks are connection-affine), so even the
normal release was unreliable.

Pin a PoolConnection in RepoWriteGuard, build the guard before the lock query,
run the lock loop through it, unlock on that same connection in release(), and
add a Drop backstop that spawns a detached affine unlock (Handle::try_current so
an off-runtime drop logs instead of panicking). Two distinct-session tests
(drop-without-release, affine release) go RED without the fix, GREEN with it.

Pinning a connection per active write is a DB-pool DoS unless the pool clears
the concurrent-write cap: the shipped default (20) was below
max_concurrent_git_pushes (32). Raise the db_max_connections default to 48 and
add Config::validate() to reject db_max_connections < max_concurrent_git_pushes
+ 8 headroom at boot; wire it into startup.
2026-07-18 09:51:10 -05:00
beardthelion 2eaf8b1086 fix(review): state the probe/read stages' unbounded-hang residual honestly; net the acquire-taint found path
The budget-knob docs claimed the overshoot was bounded by one unclamped
probe; an unclamped subprocess is not a bound, so the README,
.env.example, config help, and module comments now say plainly that a
hung git probe holds the walk slot for the hang's duration. New
regression: an acquire-tainting ghost row ahead of a healthy public copy
must not stop the scan; the later found short-circuit still serves 200.
2026-07-17 20:42:48 -05:00
beardthelion ec36e7f2d7 fix(review): close the round's review findings on the #174 fix series
object_type now distinguishes repo-level git failures (not-a-repository,
error:-prefixed corruption) from genuine object absence, so a corrupt
repo taints the /ipfs scan into the retryable 503 instead of vouching
for a definitive 404; the drain reads visibility rules by the freshly
fetched record id, closing a latent fail-open on id divergence. The four
budget gates collapse into one stage-labelled helper, the three scan
sites share state::acquire_scan_permit (inv22 tripwire repointed and
sever-verified), and the Tigris stall tests use a local accept-and-park
listener instead of a non-routable address.
2026-07-17 20:16:04 -05:00
beardthelion 79910fc169 fix(node): requeue coalesced post-push work via an in-lock pending slot and loop-drain (#174)
try_begin now merges a losing push's tip pairs into the in-flight key's
pending slot in the same critical section as the presence check, bounded
at 1024 pairs with a full-scan marker on overflow (force_full_scan
threads to the resolver; empty tips never encode the marker). The task
loop-drains: finish_or_take_pending either hands back the batch with the
key retained, or removes the key and disarms the guard atomically, so a
successor task's key survives the guard drop. Drains re-fetch rules
fresh and replay the tail pipeline under the U4 helper gates only; the
loop holds no task-level pool permit, so a hot repo cannot deadlock the
pool at size 1.
2026-07-17 19:17:17 -05:00
beardthelion 2259a248ce fix(node): admit the post-receive git scans to the encrypt pool, defer-not-shed (#174)
replication_withheld_set's walk arm, resolve_candidates_for_push's git
stages, and fail_closed_full_scan_objects each take a
git_encrypt_semaphore permit before their spawn_blocking and carry it
inside the closure, so a push burst holds at most pool-many concurrent
scans and a started scan finishes with its permit. Deletion-only pushes
and repos without path-scoped rules never park. Contention defers the
landed push's tail (logged with queue_wait_ms), never errors it; the
acquire sites document the accepted disconnect-during-park residual.
2026-07-17 18:37:40 -05:00
beardthelion 00618603ad fix(node): bound an admitted /ipfs request's total lifetime with a shared budget (#174)
GITLAWB_IPFS_REQUEST_BUDGET_SECS (default 600) is captured as one
deadline at handler entry: every acquire and expensive walk is clamped
to the remaining budget, every stage checks remaining before starting,
and exhaustion taints the scan into the retryable 503. A walk in flight
is never aborted at the tokio layer (that would free the walk permit
while the blocking thread still runs); its clamped git deadline plus
group teardown ends it. Docs state the residual overshoot: the in-flight
stage's kill/reap slack plus the unclamped probe subprocesses.
2026-07-17 18:03:49 -05:00
beardthelion d3ead53aee fix(node): make /ipfs scan verdicts honest: walk-scoped cap, visit ceiling, 503 on truncation (#174)
A request-scoped taint list replaces the silent folds: any repo skipped
without a verdict (acquire timeout, probe or read failure, walk failure,
cap or ceiling exhaustion) turns the terminal into a retryable 503 with
Retry-After, and 404 is reserved for a scan in which every candidate
reached a verdict. The walk cap now counts only expensive allowed-set
walks and skip-continues on exhaustion so a plain public copy past the
cap still serves; a new GITLAWB_IPFS_MAX_REPO_VISITS knob (default 1024)
bounds the acquire+probe cost class and stops the scan, documented with
its worst-case object-store fetch count.
2026-07-17 17:39:03 -05:00
beardthelion bda9cb102e fix(node): hold filtered-pack admission through disconnect teardown (#174)
drive_git_child hands the AdmissionGuard back on success so
upload_pack_excluding can thread it across the rev-list and pack-objects
stages; on timeout or error it is dropped post-reap, and on future-drop
it rides the detached reaper as before. The upload-pack handler builds
the guard once ahead of the withheld/plain branch, so a client
disconnect mid-filtered-serve no longer frees the read-pool and
per-source permits while the git process group is still being reaped.
2026-07-17 17:05:28 -05:00
beardthelion cdc62a703a fix(node): reject unsupported git service before the read slot; document the new admission knobs (#174)
P2-1: git_info_refs treated every service other than git-receive-pack as a read op
and did the DB/visibility/Tigris work under a read permit, validating the service
string only downstream in smart_http. So an unauthenticated `?service=anything` to
a public repo consumed a read slot + storage work before rejection. Validate the
service is exactly git-upload-pack or git-receive-pack immediately after parsing,
returning 400 before the pre-DB shed and the permit. Handler-layer regression:
with the read pool exhausted, `?service=git-explode` returns 400 (validated first),
not 503 — removing the validation makes it 503 (RED).

U5/INV-24 docs: document the new knobs from U2/U3 in .env.example and the README
env-var table — GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS, GITLAWB_MAX_CONCURRENT_IPFS_WALKS,
GITLAWB_IPFS_WALK_PER_SOURCE, GITLAWB_IPFS_MAX_REPOS_WALKED, GITLAWB_IPFS_RATE_LIMIT.
Each is already enforced on the path it names (U2/U3 tests), so the docs match the
implemented routing.
2026-07-15 13:16:40 -05:00
beardthelion 98ebb47216 fix(node): bound the post-push encryption task set by per-repo coalescing without dropping recovery copies (#174)
After a successful path-scoped push the handler spawns a DETACHED task that runs
pin_new_objects then withheld_recipients_gated, which acquire_owned().await's on
git_encrypt_semaphore — it DEFERS (blocks) when the pool is full. The semaphore
caps active walks but nothing capped how many detached tasks were spawned and
PARKED on that await: N rapid path-scoped pushes spawned N tasks, each holding
cloned object lists/rules/paths/keys — an unbounded parked-waiter set (P2-2).

2a54c15 deliberately chose defer-not-shed because dropping the walk loses the
withheld-blob recovery copy and there is no reconciliation sweep to rebuild it.
So bound the OUTSTANDING-TASK set by per-repo coalescing rather than shedding:
EncryptInflight (a bounded Arc<Mutex<HashSet<repo_id>>>) + try_begin — before the
spawn, if a task for the repo is already in-flight skip the duplicate (the pending
walk covers the newer objects); otherwise spawn and move an RAII
EncryptInflightGuard into the task that removes the key on drop (completion, error,
or panic-unwind). This bounds the set to <=1 pending task per repo and never drops
work; withheld_recipients_gated's defer is unchanged.

The second detached post-push spawn (Pinata replication, also runs pin_new_objects)
is scoped out with rationale: it parks on no semaphore (no unbounded waiter set)
and does per-push per-ref work (branch->CID, gossip, subscription, Arweave,
peer-notify) keyed to this push's ref_updates — coalescing it would DROP a later
push's announcements, a correctness regression.

Regressions (mutation-verified RED->GREEN): the outstanding set is bounded to 1
per repo under saturation (never-coalesce -> 32 tasks RED); a coalesced repo's key
is released when its task ends so it is reprocessed, never permanently skipped
(guard-drop no-op -> repo locked out, recovery copy lost forever, RED — the exact
durability regression the fix prevents); distinct repos each admit; cold-set first
push admits; plus an inv22_gates structural tripwire that fails if the handler gate
is removed.
2026-07-15 13:10:04 -05:00
beardthelion 7c23a66da9 fix(node): gate the /ipfs walk with bounded concurrency admission + per-source cap + route rate limit (#174)
GET /ipfs/{cid} (get_by_cid, publicly reachable via optional_signature) ran
allowed_blob_set_for_caller_bounded in spawn_blocking inside a per-repo loop with
NO global/per-source admission and NO route rate limit — the surface had zero
concurrency or rate control. A permissionless caller could fan out concurrent
full-history git walks (each up to the walk timeout) exhausting blocking-pool
threads and PIDs outside every served-git pool (P1-3).

Acquire a global git_ipfs_walk permit (try_acquire_owned, shed 503) plus a
per-source sub-cap (with_default_max_keys, reject-before-insert, keyed via
client_key/TrustedProxy so XFF can't spoof it) ONCE after CID validation, held as
handler locals across the whole request — including every spawn_blocking walk, so
the slot reflects real blocking-thread occupancy (a tokio timeout cannot cancel
spawn_blocking). None key -> global pool only. Cap repos walked per request so one
request can't serialize N full-history walks. Attach rate_limit_by_ip +
IpRateLimiter to the /ipfs route (mirrors the git/create/peer routers; the
extension is required or the middleware is a silent no-op). Visibility/authz
semantics unchanged.

New knobs: GITLAWB_MAX_CONCURRENT_IPFS_WALKS (32), GITLAWB_IPFS_WALK_PER_SOURCE
(4), GITLAWB_IPFS_MAX_REPOS_WALKED (64), GITLAWB_IPFS_RATE_LIMIT (600/hr, 0
disables) — all range-validated and enforced on-path.

Handler-layer regressions (all mutation-verified RED->GREEN): shed-at-capacity
503 (delete acquire -> falls through), per-source cap sheds same source / admits
another, None-key arm sheds on the global pool, map self-bounds reject-before-insert,
the walk permit is held while the spawn_blocking walk runs (available_permits()==0;
dropping it before the loop -> RED), repos-walked cap (remove break -> 2 walks),
and the route IP rate limit actually fires 429 (drop the Extension -> RED no-op).
2026-07-15 12:42:55 -05:00
beardthelion 90bd9a6270 fix(node): bound the served-git storage-acquisition phase with a release-on-expiry deadline (#174)
The served-git admission permit is acquired, then RepoStore::acquire*/advisory-lock
runs before the bounded git runner starts — with no local deadline. acquire/acquire_fresh
await Tigris HEAD/GET, and acquire_write's advisory-lock loop awaits a per-iteration
pg_try_advisory_lock that can block indefinitely on a hung Postgres pool. Since
GITLAWB_GIT_SERVICE_TIMEOUT_SECS only starts once git spawns, a stalled backend pins
the permit and drains the pool until every later request 503s (P1-2).

Wrap each git-path acquire (git_info_refs read/advert, git_upload_pack, git_receive_pack)
and the /ipfs per-repo acquire loop in tokio::time::timeout(git_acquire_timeout_secs, ..).
On expiry the git paths return AppError::Overloaded (503 + Retry-After); the permit is a
handler-local at that point (moved into the AdmissionGuard only after acquire) so the early
return frees the slot. The /ipfs per-repo acquire keeps its fail-closed `continue` on
timeout (never serves an un-acquired repo). No repo_store.rs signature change: the outer
timeout cancels a mid-sleep/mid-fetch_one future, so no in-loop deadline is needed.

New knob GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS (default 30, range 1..), modeled on
git_service_timeout_secs, kept separate because acquisition and git execution are distinct
cost centers.

Handler-layer regression receive_pack_acquire_deadline_sheds_and_releases_permit: holds the
same pg advisory lock acquire_write derives on a second connection so the real loop must
retry, drives git-receive-pack with the deadline at 2s and write pool size 1, and asserts
503 at ~2s + the permit recovers + a follow-up admits once the lock frees. Removing the
timeout wrapper turns it RED (blocks ~31s to the test ceiling on the ~59s advisory loop).
2026-07-15 12:00:02 -05:00
beardthelion d5de423393 fix(node): hold served-git admission until the process group is reaped on the plain spawn paths (#174)
On the plain (non-path-scoped) info_refs / upload-pack / receive-pack paths the
global + per-source admission permits were handler-locals that dropped the
instant the handler future dropped on a client disconnect, while KillGroupOnDrop
reaps the git process group in a DETACHED task (SIGTERM -> grace -> SIGKILL ->
reap). So a disconnect-spammer admitted replacements while prior groups were
still alive, defeating the process cap and the per-source cap during teardown.
be0cdd6 already fixed this for the path-scoped upload-pack walk; this closes the
residual plain paths (P1-a).

Introduce AdmissionGuard, a move-only holder of the permits, threaded through
info_refs/upload_pack/receive_pack/run_git_service into drive_git_child's
KillGroupOnDrop. On disconnect the guard moves into the detached reaper and drops
only after the group is ESRCH-confirmed reaped; on success/timeout disarm()
returns it for the earliest provably-free drop. The rev-list/pack-objects
visibility-walk callers hold no admission and pass None.

Thread git_bin through upload_pack/receive_pack (they hardcoded "git") so the
plain POST path is drivable by an injected fake git, matching the existing
injectable seam on info_refs/run_git_service/upload_pack_excluding.

Handler-layer regression upload_pack_plain_permit_held_through_group_reap_after_disconnect:
isolates the global read pool (size 1, per-source + rate limiter permissive), a
SIGTERM-ignoring descendant keeps the group alive across the reap window, and
asserts available_permits()==0 held on disconnect + a cross-source replacement
sheds 503, then frees after ESRCH. Reverting the guard-into-reaper move (release
immediately) turns it RED at the held-through-reap assertion. Plus a None-key arm.
2026-07-15 12:00:02 -05:00
beardthelion ca7a7074dd test(node): INV-22 completeness guard for the served-git concurrency surface (#174)
A source-scan tripwire that fails when a new site reintroduces the concurrency-cap
class PR #174 fixed, or when one of the five gates (U1-U5) is removed. INV-22: a
permit held per op recovers only if every path is duration-bounded and reaps the
group before releasing admission, and every detached git task carries admission.

Lives in tests/ (a separate crate) on purpose — a guard scanning the file it lives
in would match its own identifier literals and pass vacuously; scanning src/ from
here keeps every check load-bearing. Checks: run_bounded_git confirms child exit via
child_terminated_without_reaping (U1); KillGroupOnDrop launches the reaper via
Handle::try_current (U2); git_receive_pack acquires state.git_write_per_caller (U4);
the encryption walk runs through withheld_recipients_gated over git_encrypt_semaphore
(U5); and the bounded recipients walk is invoked exactly once — only inside the gated
helper — so a new spawn_blocking bypass (count > 1) fails (P1-e non-bypass).

Proven load-bearing: a synthetic second call site turns the count tripwire RED, and
each named gate maps to a fix whose own regression already goes RED on revert.
2026-07-14 20:33:01 -05:00
beardthelion 68043f5844 fix(node): bound concurrent post-push encryption walks with an admission pool (#174)
After a successful path-scoped push the handler released its write permit and
then ran a DETACHED tokio task whose spawn_blocking(withheld_blob_recipients_bounded)
performed another full-history git walk under no admission. N fast completed
pushes spawned N concurrent full-history walks past GITLAWB_MAX_CONCURRENT_GIT_PUSHES
(which bounds only the in-handler phase), and the per-child bounding from 71389e6
caps each walk's duration but not their count (P1-e).

Add git_encrypt_semaphore, a pool of its own (sized from max_concurrent_git_pushes,
no new knob — Q1) so a long background walk never holds a foreground write slot and
a handler holding a write permit can't self-deadlock. Route the walk through a new
withheld_recipients_gated helper that acquires the pool before the walk and DEFERS
(blocks) when full rather than shedding — dropping the walk would lose the
withheld-blob recovery copy, so durability stays fail-closed.

Regression: encrypt_walk_defers_when_pool_exhausted drives the gated helper with the
pool exhausted and asserts the walk blocks and does not run its rev-list, then runs
once a permit frees. RED before (the walk runs regardless of the pool), GREEN after.
The detached push task calls this exact helper.
2026-07-14 20:28:52 -05:00
beardthelion f592378bdc fix(node): cap concurrent receive-pack pushes per source IP (#174)
The authenticated git-receive-pack POST acquired only the global write semaphore
— no per-source sub-cap, unlike the read and anon-advert pools. Owner enforcement
defaults off, so one host minting disposable did:key identities could open
max_concurrent_git_pushes slow POSTs from a single source IP and 503 every other
source's push; the 600/hour push limiter bounds arrival rate, not in-flight
concurrency (P1-d).

Add git_write_per_caller (a PerCallerConcurrency sized like the advert cap,
max_concurrent_git_pushes/8), and acquire it before the global write permit,
keyed on the resolved source IP via read_caller_key — never the signed DID (a DID
farm defeats a DID key). This required adding the PeerAddr + HeaderMap extractors
to git_receive_pack, which it lacked (without them the key is None and the cap is
inert). Neutralized the shared acquire helper's log wording now that it serves
both read and write paths.

Regression: receive_pack_per_source_write_cap_sheds_capped_source_not_others — a
source at its write sub-cap sheds Overloaded/503 (proving the extractors resolve a
key), while a different source is not shed. RED before (the capped source proceeds
past admission to a git error instead of shedding), GREEN after; the existing
receive-pack advert cap test still passes.
2026-07-14 20:20:49 -05:00
beardthelion 0c767d1585 fix(node): hold upload-pack admission through the walk on disconnect (#174)
On the path-scoped upload-pack path the per-source and global read permits were
handler locals, so a client disconnect dropped the handler future and released
both permits while the awaited spawn_blocking withheld-blob walk kept running
(spawn_blocking can't be cancelled). A permitless caller could disconnect-spam to
exceed the read cap while real git work continued (P1-b).

Move both permits into the blocking task and return them out: on success they
flow back so the serve phase keeps them; on a dropped future the returned tuple
(with the permits) is discarded only when the blocking task completes, so
admission tracks the walk's real duration.

Regression: upload_pack_permit_held_through_walk_after_disconnect drives the
handler to mid-walk (fake git hangs on rev-list), disconnects, and asserts the
global read slot stays held (available_permits == 0). RED before (the slot frees
to 1 the instant the future drops), GREEN after. The test roots the repo store at
its own TempDir so it is isolated per run.
2026-07-14 20:09:40 -05:00
beardthelion b900151468 fix(node): reap the served-git process group on disconnect, not just SIGTERM (#174)
On a client disconnect KillGroupOnDrop sent a lone SIGTERM with no escalation or
reap, so a group member that ignores SIGTERM (a wedged pack-objects, a malicious
helper) survived while the handler released its concurrency permit — letting
disconnect-spam accumulate live git processes and defeat the load shed (P1-c).
The timeout path already ran the full teardown.

Make the disconnect teardown as strong as the timeout path: the guard now owns
the child and, on drop, launches a detached reaper that runs the same
TERM -> grace -> SIGKILL -> reap-the-group sequence (reap_group_on_timeout).
Owning the tokio Child in the reaper keeps it the sole reaper of the leader, so
it never races tokio's own orphan reaper (a raw waitpid(-pgid) would).

Regression: run_git_service_sigkills_a_sigterm_ignoring_child_on_disconnect, RED
before (the descendant survives the lone SIGTERM), GREEN after; all existing
teardown/timeout/disarm tests still pass (the three hand-built-guard tests were
updated to the owned-child shape).

Residual: capacity (the handler's semaphore permit) still releases at disconnect
rather than strictly after the reaper confirms the group dead, so accumulation is
now bounded to ~one teardown window (<=4s) rather than eliminated; gating the
permit on the reap needs threading it through the serve handlers and is left for
a follow-up (it also touches the receive-pack permit lifetime U4 changes).
2026-07-14 19:50:41 -05:00
beardthelion dd3672ef0e fix(node): keep the visibility-walk watchdog armed until the child is reaped (#174)
A group member (or the leader itself) that closes stdout before the deadline made
the stdout drain return EOF early, which stood the watchdog down via done_tx
before it could ever fire; child.wait() then blocked on the still-live child,
pinning the walk thread and its read/write permit past
GITLAWB_GIT_SERVICE_TIMEOUT_SECS. This is distinct from the descendant case
already covered: there the leader sleeps until the deadline so the watchdog does
time out; here the drain-EOF races ahead of it.

Stand the watchdog down only once the child has actually terminated, detected
without reaping (waitid + WNOWAIT) so the leader's pid stays unreaped and its
pgid un-recycled until the watchdog finishes its teardown and is joined. Past the
deadline the watchdog still owns the full SIGTERM -> grace -> SIGKILL sequence.

Regression: run_bounded_git_reaps_a_leader_that_closes_stdout_then_hangs, RED
before (blocks on child.wait past the recv budget), GREEN after; the three
existing teardown tests still pass (no recycle-hazard regression).
2026-07-14 19:27:12 -05:00
beardthelion ec201effb6 test(node): retry the ETXTBSY exec race in build_filtered_pack timeout tests (#174)
build_filtered_pack_times_out_a_hung_rev_list and _a_hung_pack_objects
spawn a freshly-written fake git directly, bypassing the fake_git_run_with_pids
retry the rest of the module uses. Under cargo test fork-storm load a concurrent
worker forks while the fake's write fd is still open, so execve returns ETXTBSY
and the raw io::Error surfaces where the test asserts GitServiceTimeout, failing
test (stable) intermittently.

Route both tests through build_filtered_pack_or_exec_race_retry, which retries
ONLY errno 26 (ETXTBSY) and keeps the outer 10s watchdog per attempt, so a
genuinely-missing async timeout bound still trips the watchdog and a wrong error
type still fails at the assertion. Production paths unchanged.
2026-07-13 16:01:49 -05:00
beardthelion 7671df98c7 test(node): execution-cover the delta-path push-scan timeout wiring (#174)
The finding-2 wiring test covered only list_all_objects (the full-scan fallback).
Add delta_path_exec_fns_time_out_on_a_hung_git covering the common push path:
peeled_object_type (cat-file), rev_list_delta (rev-list), and list_all_objects_with_type
(cat-file, which all_blob_oids delegates to). Each must return within the watchdog
budget on a hung git; reverting any to a bare Command::output() blocks past the recv
budget (RED-verified).
2026-07-13 10:55:57 -05:00
beardthelion 1e1058ca29 docs(node): correct concurrency + timeout docs to match the final git routing (#174)
jatmn P2 (INV-24): the config help, .env.example, and README described routing the
handler no longer does.

- The read pool serves upload-pack and the UPLOAD-PACK info/refs advertisement only;
  the anon receive-pack info/refs advertisement draws from its own dedicated pool
  (sized like the write pool but disjoint), not the read pool and not the write pool.
- The write pool is the authenticated push POST only.
- The per-caller read cap is keyed on the resolved source IP, never the DID: a
  signature does not move a caller off it (the docs claimed a per-DID budget, which
  would tell a NATed authenticated client it escapes the shared cap when it does not).
- git_service_timeout_secs now also bounds the push-side pin-candidate discovery
  (rev-list / cat-file), reaped via process-group teardown, alongside the info/refs
  advertisements and the withheld-blob walk.
2026-07-13 10:46:24 -05:00
beardthelion 5aa37a17a0 fix(node): bound and reap push-candidate git children under the write permit (#174)
Route push_delta's candidate-discovery git children (cat-file / rev-list) through the
bounded run_bounded_git (deadline + process-group SIGKILL teardown) instead of bare
Command::output(). A hung scan no longer pins the git-receive-pack write permit past
the deadline or a client disconnect. Threads git_bin + a shared per-scan deadline
(git_service_timeout_secs) through resolve_candidates_for_push and all_blob_oids.
2026-07-13 10:41:31 -05:00
beardthelion 39fab32e91 fix(node): SIGKILL a SIGTERM-ignoring member of the visibility-walk process group (#174)
jatmn/CodeRabbit P1: the walk watchdog stood down as soon as the group LEADER was
reaped. A leader that exits cleanly on SIGTERM while a background member ignores
SIGTERM and closes its inherited pipes let the main drain unblock, the leader be
reaped, and the watchdog skip its SIGKILL escalation — so the handler returned while
the descendant kept running. jatmn's suggested "continue until ESRCH" could not work:
the main thread reaping the leader is what frees the pgid, so continuing to kill(-pgid)
after that races a recycled group.

Restructure the coordination instead: the main thread now defers reaping the leader
until the watchdog thread returns. The watchdog always escalates through an
unconditional group SIGKILL (no leader-reap short-circuit), and because the leader is
still unreaped while every kill(-pgid) fires, the pgid cannot have been recycled. The
grace before SIGKILL is a fixed budget (only paid on the exceptional timeout path).

Regression run_bounded_git_sigkills_a_sigterm_ignoring_descendant_after_leader_exits:
a leader that exits 0 on SIGTERM spawns an sh -c descendant (its own $$, not a ( )
subshell's parent pid) that ignores SIGTERM and closes its pipes; the descendant must
be dead when the runner returns. RED on the old teardown (survives), GREEN now. The
existing leader-ignores-SIGTERM and hung-walk teardown tests still pass.
2026-07-13 10:11:41 -05:00
beardthelion 4f2a36fcb4 fix(node): compile the bounded visibility git runner on non-Unix targets (#174)
jatmn P1: run_bounded_git was gated #[cfg(unix)], but its callers
(assert_all_refs_are_commits, blob_paths) and the module are compiled on all
targets, so a non-Unix build (the continue-on-error Windows release target) could
not resolve the function.

Add a #[cfg(not(unix))] twin with the same signature and result semantics. It bounds
a single child (no process-group teardown, which is Unix-only): threads feed stdin
and drain stderr, the main thread drains stdout, and a watchdog thread kills the
child at the deadline. The child is shared with the watchdog behind a mutex the main
thread does not hold while draining, so the watchdog can always acquire it to kill,
and killing closes stdout to unblock the drain. Best-effort (reaps only the direct
child), which is why the hardened group-aware path stays Unix-only.

Verified the twin type-checks by compiling its (portable-std) body on the native
target; the full non-Unix link is CI's Windows job.
2026-07-13 09:49:33 -05:00
beardthelion c5d19df8d6 fix(node): give the receive-pack advert its own pool so it can't shed authed pushes (#174)
jatmn P1: the anon-reachable GET info/refs?service=git-receive-pack drew from the
same git_write_semaphore as the authenticated git-receive-pack POST. The per-source
advert cap reserves no global slots, so ~8 sources (each within the per-IP push rate
limit) could hold all 32 write permits across acquire_fresh + the advertisement, and
the next authenticated push then 503s at admission. That is an anonymous caller
shedding an authenticated push across the auth boundary (INV-10).

Add a dedicated git_push_advert_semaphore, disjoint from the write pool. Both the
pre-DB early-shed peek and the held acquire for the receive-pack advertisement now
draw from it; the write pool is left exclusively for the POST. An advert flood can at
worst exhaust the advert pool (each source still capped by git_push_advert_per_caller
and the per-IP push rate limiter); the authenticated push pool is untouched.

Test receive_pack_advertisement_draws_from_dedicated_advert_pool proves at the handler
layer that the advert sheds when its own pool is saturated AND survives when the write
pool is saturated (the reservation). Both flip under revert, verified independently for
the pre-DB peek and the held acquire.
2026-07-13 09:44:53 -05:00
beardthelion e4c14ccd01 fix(node): shed the served-git pool before the DB and cap per-source before it (#174)
CodeRabbit review: the held git_permit was acquired at the top of git_info_refs
and git_upload_pack, before the per-source cap, so one source could occupy the
global pool during the DB/visibility window before its sub-cap rejected the
excess. Move the held git_permit to after acquire_read_caller_permit (still
before acquire_fresh/git, so INV-10's bound on the fresh Tigris acquire holds),
and add a cheap pre-DB early shed (available_permits() == 0 -> 503, holds no
permit) so the #62 shed-before-DB property is preserved without holding a permit
across the DB work.

The receive-pack POST is unchanged (owner-only; its top-of-handler git_permit is
deliberate for acquire_write bounding).

Tests: the two #62 shed-before-DB tests now exercise the explicit early check
(load-bearing: disable it and they fall through to the DB); three new
*_per_source_cap_sheds_with_global_capacity tests prove the per-source sub-cap
sheds a capped source even with global capacity free (load-bearing on the caller
cap). Full workspace green.
2026-07-12 14:40:59 -05:00
beardthelion a4beba2eea docs(node): the withheld-walk is bounded by the service timeout, not a fixed const (#174)
Follow-up to threading the configured timeout into the walk: the walk now honors
GITLAWB_GIT_SERVICE_TIMEOUT_SECS on both the serve and replication paths, so the
README no longer says a fixed internal deadline.
2026-07-12 02:12:37 -05:00
beardthelion b2939ad990 test(node): vet the withheld-walk bound end to end; use the configured timeout (#174)
Close the reasoned-not-run gaps from the code review by making the walk's git
binary and timeout injectable, then driving the missing branches with a real
handler and a fake git instead of reasoning about them.

- Add state.git_bin and *_bounded variants of the walk entry points taking
  (git_bin, timeout); the served handlers (upload-pack serve, receive-pack
  replication and full-scan and encrypt-pin, and the ipfs gate) now pass the
  operator-configured GITLAWB_GIT_SERVICE_TIMEOUT_SECS, so the whole walk is bounded
  by the same budget as the other served-git ops rather than a fixed constant. The
  git_bin-less wrappers stay for the real-git classification tests.

Newly vetted by execution (not reasoning):
- receive-pack replication path is bounded: replication_withheld_set with an injected
  hung git returns within the budget and fails closed, so it cannot pin the write
  permit git_receive_pack holds across it.
- a hung withheld-blob walk on the upload-pack POST returns 504 (real handler, real
  repo on disk, injected hung git), proving the GitServiceTimeout -> git_service_app_error
  wiring end to end.
- the watchdog status-gate: a child that exits successfully is not reported as a
  timeout even when the watchdog fired (mutation-checked: drop the guard -> RED).
- SIGKILL escalation: a SIGTERM-ignoring child is still reaped via SIGKILL and the
  group is gone; a truly uninterruptible D-state child (unreapable by any signal) is
  the documented residual, matching the async teardown.
- the advertisement per-source cap sizing never derives 0.

Full gitlawb-node suite 515 green.
2026-07-12 02:12:08 -05:00
beardthelion 6c7bbaa193 fix(review): cap the receive-pack advertisement per source so anon cannot starve the write pool (#174)
U2 moved the receive-pack info/refs advertisement onto git_write_semaphore to keep
an anonymous read flood from starving the push handshake. But the advertisement is
anon-reachable on public repos and holds its write permit across the slow
acquire_fresh Tigris download, and the only per-source brake on it was the push
RATE limiter, not a concurrency cap. So a multi-source flood of receive-pack
advertisements could hold the write pool's slots across those downloads and shed
authenticated pushes (both the advertisement and the owner-gated git-receive-pack
POST draw from the same pool). U2 thus introduced the first anonymous consumer of
the write pool the state doc promised anon could never reach; the plan's residual
note (no worse than the POST) was wrong, because the POST is owner-gated and the
advertisement is not.

Add git_push_advert_per_caller, a per-source concurrency sub-cap on the receive-pack
advertisement keyed on the resolved source IP (the same PerCallerConcurrency
mechanism U1 uses for reads), sized to an eighth of the write pool so a single
source holds at most that share and saturating the pool takes many distinct source
IPs, each also braked by the per-IP push rate limiter. The upload-pack advertisement
keeps its read-pool per-caller cap; the owner-gated POST is unchanged. Correct the
state doc for git_write_semaphore accordingly.

Handler-layer proof: a source at its receive-pack advertisement cap sheds 503 (RED
before the acquisition, 500-not-503), while a different source and the upload-pack
advertisement are unaffected. Full suite 510 green.
2026-07-12 00:56:24 -05:00
beardthelion 39e1f12255 fix(review): close the withheld-walk watchdog reused-pgid race (#174)
Code review found run_bounded_git's watchdog could return a spurious 504 and
signal a recycled process group. The watchdog runs off a wall clock on its own
thread; done_tx.send() only fires after child.wait() reaps the leader, so a walk
that finished within microseconds of the deadline took the watchdog's Timeout
branch, discarded a fully-captured successful result, and returned GitServiceTimeout
(a 504 for a walk that actually completed). Worse, the Timeout branch SIGTERMed
-pgid unconditionally after the leader was reaped, so a recycled pgid could be
signalled, the exact hazard smart_http guards via disarm-after-wait.

Set a reaped AtomicBool the instant the main thread reaps the child; the watchdog
checks it before every kill and stands down if the leader is already reaped. Gate
the timeout verdict on !status.success(), so a child that exited on its own is never
reported as a timeout even if the watchdog fired late. Add the survived-SIGKILL warn
smart_http's reap already emits, for operator visibility on a wedged (D-state) git.

The hung-walk test stays green (a killed child exits by signal, not success, so it
still surfaces GitServiceTimeout and reaps the group) and the 32 real-git
classification tests stay green (a fast walk is never spuriously killed).
2026-07-12 00:36:42 -05:00
beardthelion 87b6d2d498 docs(node): correct the git-service-timeout coverage note (#174)
GITLAWB_GIT_SERVICE_TIMEOUT_SECS bounds the info/refs advertisement too:
smart_http::info_refs drives it through drive_git_child under this timeout, with a
passing test proving the 504. The old note claimed it does not. It also claimed the
withheld-blob path is unbounded; after the blob_paths seam bound (this PR) the walk
is bounded and reaped, by a fixed internal deadline rather than this env var, so the
line now states that precisely instead of overclaiming this setting covers it.
2026-07-12 00:20:07 -05:00
beardthelion 5204502678 fix(node): bound the withheld-blob walk at the shared blob_paths seam (#174)
The withheld-blob classification walk (blob_paths) fanned out blocking git children
with no deadline and no process-group teardown: git for-each-ref, git cat-file, git
rev-list, a git ls-tree per commit, and an uncounted git rev-parse (via
store::head_commit). A hung or pathologically slow child pinned the caller's
served-git permit for the whole hang, and on client disconnect the spawn_blocking
task and its git children ran on, orphaned. blob_paths is the shared core of five
callers: the upload-pack serve path (holds a read permit) AND, inside
git_receive_pack, the post-push replication and encrypt-then-pin walks (hold the
write permit U2 reserves for pushes). So the same unbounded walk could pin either
pool, and leaving the write-side twin unbounded would have made U2's reservation a
claim that does not match behavior.

Bound every git child at the blob_paths spawn seam on the blocking side: each child
runs in its own process group with a watchdog thread that SIGTERMs (then SIGKILLs)
the group on one shared deadline spanning the whole walk, and retains admission
until the group is reaped. This is the blocking-side counterpart of
smart_http::drive_git_child (spawn_blocking cannot be cancelled by an async
timeout). blob_paths stays sync, so all five callers keep their signatures and the
32 classification tests are unchanged; because every caller funnels through
blob_paths, one seam bounds both the serve and replication paths. The previously
unbounded store::head_commit child becomes a bounded git rev-parse inside the walk.
A walk that hits its deadline carries GitServiceTimeout, which the serve handler now
maps to 504 rather than a generic 500.

Proof: a fake git that hangs on rev-list makes blob_paths return GitServiceTimeout
within the watchdog budget (not block on the child) and the recorded process-group
leader is reaped, not orphaned; neutralizing the watchdog kill makes it hang past
the budget (RED). The 32 real-git classification tests stay green through the
refactor, including detached-HEAD, non-standard-ref, and deleted-in-history cases.
2026-07-12 00:20:07 -05:00
beardthelion d0f96684ed fix(node): draw the receive-pack advertisement from the write pool (#174)
git_info_refs acquired git_read_semaphore for BOTH services, so the push handshake
(GET /info/refs?service=git-receive-pack) competed in the global read pool. An
anonymous clone flood could exhaust that pool and shed a legitimate push with 503
during its required advertisement phase, before it ever reached git_write_semaphore
on the POST. The write pool exists precisely so anonymous reads cannot shed an
authenticated push, but only the POST drew from it.

Select the pool by service: the receive-pack advertisement (phase one of a push)
now draws from git_write_semaphore, like the git-receive-pack POST, so a saturated
read pool cannot starve it. The per-IP push_rate_limiter that already brakes the
advertisement stays as the anti-flood control, and the advertisement stays
reader-visible with no new auth requirement. Because the receive-pack branch is now
a write-path op, it no longer consumes a read per-caller slot.

Handler-layer proofs: with the read pool at zero the receive-pack advertisement
survives while the upload-pack advertisement sheds; with the write pool at zero the
receive-pack advertisement sheds while upload-pack is unaffected; and a receive-pack
advertisement from an IP whose read per-caller budget is full still gets through
(mutation-checked, RED when the skip is neutralized).
2026-07-11 23:54:21 -05:00
beardthelion 45408309f8 fix(node): key the per-caller read cap on source IP, not the signed DID (#174)
read_caller_key returned the authenticated DID when a caller signed, dropping the
source-IP key. Public read routes accept any valid did:key via optional_signature
with no admission step, so one host could mint N disposable DIDs and hold
max_concurrent_reads_per_caller slots under each, multiplying its budget N-fold and
filling the global read pool, while the same host unsigned was capped on its IP.

Key the read sub-cap on the resolved source IP for every caller, signed or not,
mirroring the push path's IpRateLimiter which already throttles on source IP for
this exact DID-farm reason. Drops the now-unused caller_did parameter at both call
sites (git_info_refs, git_upload_pack).

Inverts info_refs_per_caller_cap_keys_on_did_not_ip into
info_refs_per_caller_cap_keys_on_ip_not_did: fill one source IP's slot, then two
requests signed under different DIDs from that same IP both shed 503 (farm
defeated), while a signed request from a different IP keeps its own budget. RED on
the DID-keyed tree, GREEN after.
2026-07-11 23:44:13 -05:00
beardthelion 8816684c71 fix(node,git): bound the rev-list stage under drive_git_child so a hung enumeration can't pin a git permit (#174) 2026-07-11 13:02:24 -05:00
beardthelion 6176fbb697 docs(config): add GITLAWB_MAX_CONCURRENT_GIT_OPS to .env.example (#174)
The read-pool knob was referenced by the push and per-caller entries'
comments but had no example line of its own, so operators couldn't
discover it from the template. Add it with the config default (128).
2026-07-10 16:05:29 -05:00
beardthelion 08fb3a02df fix(node,concurrency): resolve #174 code-review findings
Review of the served-git concurrency cap found no P0/P1; these are the
verified P2 follow-ups.

config: the max_concurrent_git_ops doc overclaimed that "every capped path
is duration-bounded." The rev-list object enumeration in the withheld-blob
path still runs in an uncancellable spawn_blocking, so a stuck rev-list can
hold its slot until git exits. Scope the guarantee to the streaming stages
and name the residual. Also tighten the fairness claim: the receive-pack
advertisement shares the read pool (a shed advertisement is a cheap retryable
GET); only the push POST is on the isolated write pool.

api/repos: add info_refs_per_caller_cap_keys_on_did_not_ip, the missing
handler proof that a signed caller is keyed by its DID, not its source IP.
Filling the DID slot sheds a request from a free IP; collapsing read_caller_key
to its IP arm turns the assertion green-not-503 (mutation-verified RED).

api/repos: extract acquire_read_caller_permit so both read handlers share one
shed path instead of a duplicated match block.

rate_limit: recover from a poisoned PerCallerConcurrency mutex instead of
panicking. The critical section is pure counter arithmetic and cannot poison
the lock, but a panic there would brick the limiter for every caller.

505 tests pass; clippy -D warnings and fmt clean.

Part of #174.
2026-07-10 13:45:38 -05:00
beardthelion b519251e00 docs(node,config): reconcile the concurrency-cap comments with the fix (#62)
The max_concurrent_git_ops and git_service_timeout_secs doc-comments (and .env.example) described the info/refs and withheld-blob paths as unbounded follow-up gaps. Both are now closed (#174): the comments reflect the read/write pool split, the per-caller sub-cap, and that every capped path is duration-bounded with process-group teardown. Verified the pattern-doc pre-ship checklist: every git_permit / write-permit / per-caller site holds only a timeout+teardown git path, and all three size knobs are range(1..=1_048_576).

Closes the #174 work. No behavior change.
2026-07-10 13:20:18 -05:00
beardthelion 0fdaaef3f7 fix(node,git): bound and reap the withheld-blob pack build (#62)
The filtered-pack path ran the whole rev-list + pack-objects build inside a spawn_blocking, so an outer tokio timeout could not cancel the blocking thread and a client disconnect orphaned the git child while the permit freed (jatmn P1b, the second gap path). Split it: rev-list enumeration stays blocking off the runtime (rev_list_keep), but the streaming pack-objects stage now runs under the shared drive_git_child on the async side, so it is duration-bounded (GitServiceTimeout -> 504) and its process group is reaped on disconnect. build_filtered_pack becomes async and takes a git_bin seam + timeout; upload_pack_excluding threads the git_service_timeout through.

A watchdog-bounded test proves a hung pack-objects times out (rev-list fast, pack-objects hangs). The refactor's happy path is covered by the existing filtered-pack correctness and real-git partial-clone/fetch tests, all still green; disconnect/group-teardown is the shared drive_git_child code proven by the run_git_service tests. 504 tests pass.

Part of #174.
2026-07-10 13:16:41 -05:00
beardthelion 9ca626e415 fix(node,git): bound and reap the info/refs advertisement (#62)
info_refs ran a bare Command::output() with no timeout and no process-group teardown, so a hung git pinned its concurrency slot indefinitely and a client disconnect orphaned the child (jatmn P1b). Extract the timeout + process_group(0) + KillGroupOnDrop core from run_git_service into a shared drive_git_child, and route info_refs through it with an injectable git_bin. A hung advertisement now aborts with GitServiceTimeout (mapped to 504); disconnect reaps the group.

run_git_service's teardown tests all pass through the shared core (proving the group teardown info_refs inherits), the real-git filter tests cover the advertisement happy path, and a new watchdog-bounded test proves a hung advertisement times out. 503 tests pass.

Part of #174.
2026-07-10 13:09:34 -05:00
beardthelion 222fb570a3 fix(node,git): per-caller concurrency sub-cap on the read pool (#62)
Adds PerCallerConcurrency, a bounded-keyed in-flight limiter (distinct from the request-rate RateLimiter) so no single caller monopolizes the served-git read pool. Each caller (per-DID when signed via optional_signature, else per-source-IP via client_key) may hold at most max_concurrent_reads_per_caller concurrent reads; over that it sheds 503. The key map is self-bounding (a key is dropped when its in-flight count hits zero) with a reject-before-insert max_keys backstop so a key farm can't grow it (INV-15). Applied in git_upload_pack and both info/refs advertisements, acquired after the visibility gate so a denied request never consumes a slot (KTD7).

Primitive unit-tested (cap + self-bounding + reject-before-insert) and mutation-checked. Handler-layer SC2: same-caller sheds while a different caller passes, proven on BOTH git_info_refs and git_upload_pack with independent mutation probes; plus a None-key bypass test. Per-source-IP keying is trust-config dependent, documented on the config knob. 502 tests pass.

Part of #174.
2026-07-10 12:50:46 -05:00
beardthelion c97fe3a551 fix(node,git): isolate authed pushes in a dedicated write pool (#62)
git-receive-pack now draws from a separate git_write_semaphore (max_concurrent_git_pushes) instead of the shared pool, so a flood of anonymous reads can no longer shed an authenticated push at admission (jatmn P1a). The shared field is renamed git_read_semaphore and continues to gate upload-pack and both info/refs advertisements. The write permit stays above acquire_write so it precedes the Tigris fresh-acquire (INV-10).

Handler-layer tests: write-pool shed (503), and a cross-boundary proof that an exhausted read pool does NOT shed a push; both mutation-checked (routing receive-pack back to the read pool flips each RED). 497 tests pass.

Part of #174.
2026-07-10 12:29:30 -05:00
beardthelion 2c6cef6ad0 feat(node,config): add write-pool and per-caller read-cap knobs (#62)
Add max_concurrent_git_pushes (default 32) and max_concurrent_reads_per_caller (default 16), both clap range(1..=1_048_576) so an oversized value is a clean CLI error, not a Semaphore::new boot panic. The per-caller knob documents that per-source-IP keying is only as granular as GITLAWB_TRUSTED_PROXY. Wiring lands in the following commits; these are the config surface for the #174 concurrency-fairness fix.

Resolves jatmn P1a/P1b groundwork on #174.
2026-07-10 12:18:07 -05:00
beardthelion ba82ae785e feat(node,git): cap concurrent served git ops with a 503 load-shed (#62)
PR3 of the #62 served-git hardening stack (timeout #165 and teardown
wiring #150 are merged). A bounded semaphore caps how many upload-pack /
receive-pack / info-refs operations run at once; past the cap a request
is shed with a clean 503 + Retry-After before spawning another git
subprocess, instead of exhausting the PID/thread table. A permit is
acquired at the top of each of the three handlers and held for the whole
op, releasing on return.

The cap is a portable backstop: the compose pids_limit is absent on Fly,
whose 500-connection cap is a different axis. Size --max-concurrent-git-ops
(GITLAWB_MAX_CONCURRENT_GIT_OPS, default 128) below the process budget.
Range 1..=1_048_576 so 0 (shed everything) and an oversized value that
would panic tokio's Semaphore at boot are clean CLI errors.

Known gap, tracked separately: info/refs and the withheld-blob
(upload_pack_excluding) path are not duration-bounded and do not reap
their git child on client disconnect, so a hung git on those two paths
holds its slot until it exits and live git can briefly exceed the cap.
The main pack path (run_git_service) tears its group down on drop.

Tests: Overloaded maps to 503 + Retry-After; the config knob defaults
and rejects out-of-range; git_permit sheds at capacity and releases; and
each of the three endpoints sheds with 503 when the semaphore is
exhausted (load-bearing: drop the permit line and the endpoint test goes
red).
2026-07-10 12:10:30 -05:00
43 changed files with 22118 additions and 783 deletions

No files matched your search

+127 -4
View File
@@ -24,8 +24,11 @@ DATABASE_URL=postgresql://gitlawb:changeme@localhost:5432/gitlawb
# ── Database pool & startup resilience ────────────────────────────────────
# Maximum connections in the PostgreSQL pool. A cap, not a floor —
# connections open lazily. Size against the DB server's max_connections,
# remembering admin tooling opens its own pool.
GITLAWB_DB_MAX_CONNECTIONS=20
# remembering admin tooling opens its own pool. Each concurrent write pins one
# connection for its whole duration (the connection-affine advisory lock), so the
# node REJECTS at boot any value below GITLAWB_MAX_CONCURRENT_GIT_PUSHES + 8
# headroom — keep this comfortably above that (default 48 for pushes 32).
GITLAWB_DB_MAX_CONNECTIONS=48
# Seconds a request waits for a pool connection before failing with 503.
GITLAWB_DB_ACQUIRE_TIMEOUT_SECS=5
# Upper bound on each startup connect+migrate attempt, in seconds. Keep it
@@ -109,15 +112,135 @@ GITLAWB_MAX_PACK_BYTES=2147483648
# Max seconds a served git upload-pack / receive-pack (clone / push) may run
# before it is aborted with a 504. Bounds a hung git that would otherwise pin a
# worker and, on push, the repo write lock. Does NOT cover the info/refs
# advertisement or the withheld-blob path, which remain unbounded. Default 600.
# worker and, on push, the repo write lock. Also bounds both info/refs
# advertisements, the withheld-blob pack build, and the push-side candidate
# discovery (rev-list / cat-file), all reaped via process-group teardown (#174).
# On the path-scoped upload-pack path the withheld-blob classification walk and
# the pack serve share ONE deadline, so this value bounds their COMBINED duration
# rather than giving each stage a full budget: a walk that consumes it leaves the
# serve nothing and the clone gets a 504. Serving large path-scoped repos may
# need a higher value here than when each stage was budgeted separately.
# Must be 1..=3153600000 (100 years): the node derives deadlines from this value,
# and a larger one cannot be represented. Default 600.
GITLAWB_GIT_SERVICE_TIMEOUT_SECS=600
# Max seconds the storage-ACQUISITION phase of a served git op may run before the
# request is shed with a 503, separate from the git-run timeout above. A
# concurrency permit is taken before this phase and GITLAWB_GIT_SERVICE_TIMEOUT_SECS
# only starts once git spawns, so without this a stalled backend (a hung Tigris
# HEAD/GET, or a hung pg advisory-lock iteration on push) pins the permit and drains
# the pool until every later request 503s. On expiry the permit is released
# (fail-closed). Kept separate because acquisition and git execution are distinct
# cost centers. Must be positive; set very large to effectively disable. Default 30.
GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS=30
# Max concurrent git READ ops (upload-pack + the upload-pack info/refs
# advertisement) served at once, a global pool separate from the push pool below.
# The anon receive-pack info/refs advertisement has its OWN pool (see below), not
# this one. Over-cap sheds a clean 503 + Retry-After. Anonymous reads draw from
# here, so pair it with GITLAWB_MAX_CONCURRENT_READS_PER_CALLER (below) so one
# caller cannot monopolize the pool. Default 128.
GITLAWB_MAX_CONCURRENT_GIT_OPS=128
# Max concurrent git-receive-pack (push) POST operations, in a pool separate from
# the read pool (GITLAWB_MAX_CONCURRENT_GIT_OPS) so anonymous reads cannot shed an
# authenticated push at admission. The anon receive-pack info/refs advertisement
# runs in a SEPARATE pool of the same size (disjoint from this one), so an
# advertisement flood cannot shed a push either. Over-cap sheds a 503 +
# Retry-After. Default 32.
GITLAWB_MAX_CONCURRENT_GIT_PUSHES=32
# Max pushes PARKED at once waiting for the per-repo write lease of the SAME repo.
# A parked push holds its already-buffered pack body in memory, so this bounds that
# memory for a hot repo. Past the cap the newest push sheds a clean 503 +
# Retry-After ("repo is busy with another push") instead of joining the queue; a
# push to a DIFFERENT repo is never affected. The lease holder is not counted.
# Raising GITLAWB_MAX_CONCURRENT_GIT_PUSHES does not raise this; set it explicitly.
# Default 8.
GITLAWB_REPO_LEASE_MAX_WAITERS=8
# Max concurrent post-push pin loops (IPFS + Pinata pin_new_objects) across all
# repos. The per-repo encrypt-task set caps the pin-task COUNT; this caps how many
# of those loops RUN CONCURRENTLY, and so how much MB-scale object-id list memory is
# held WHILE BEING PINNED. It does not bound retained memory in general: on the local
# IPFS path the caller builds its list before this pool is entered, so a loop parked
# waiting for a permit still holds its full list, and how many park is capped only
# per repo. Do not size memory from this knob alone. A loop DEFERS (waits) when the
# pool is full, never drops a pin. Default 8.
GITLAWB_MAX_CONCURRENT_PIN_TASKS=8
# Max concurrent read ops (upload-pack + the upload-pack info/refs advertisement)
# a single caller may hold, so one caller cannot monopolize the read pool. Keyed
# on the resolved SOURCE IP, never the DID: a signature does not move a caller off
# this cap. The source-IP key is only as granular as GITLAWB_TRUSTED_PROXY below:
# left unset, a node behind an edge/NAT keys all callers on the edge IP and this
# collapses to one global cap. Set GITLAWB_TRUSTED_PROXY for per-client keying; a
# high-fanout caller (CI behind one NAT) then needs the operator to raise this.
# Default 16.
GITLAWB_MAX_CONCURRENT_READS_PER_CALLER=16
# Two further per-source concurrency caps exist on the PUSH side but have NO
# environment variable of their own: both are DERIVED as
# GITLAWB_MAX_CONCURRENT_GIT_PUSHES / 8, with a floor of 1. They are not settable
# independently, so raising the push pool raises both proportionally, and the
# default push pool of 32 gives each a cap of 4.
# * The anon receive-pack info/refs advertisement cap: one source IP may hold at
# most this many slots in the dedicated advert pool, so saturating that pool
# takes ~8 distinct source IPs.
# * The authenticated receive-pack POST cap: one source IP may hold at most this
# many WRITE-pool slots. This one is load-bearing for push availability, since
# it is acquired before the global write permit: without it, one host minting
# disposable did:key identities could open enough slow pushes to monopolize the
# write pool and 503 every other source (owner enforcement defaults off, and
# the push rate limiter caps arrival rate, not in-flight concurrency).
# Keyed on the resolved source IP, never the DID, so a DID farm does not defeat
# them; keying granularity follows GITLAWB_TRUSTED_PROXY like the read cap above.
# ── Push rate limiting (git-receive-pack flood brake) ─────────────────────
# Max receive-pack requests (info/refs advertisement + push POST) per client
# IP per hour. 0 disables. Default 600.
GITLAWB_PUSH_RATE_LIMIT=600
# ── /ipfs/{cid} visibility-walk admission (#174) ──────────────────────────
# GET /ipfs/{cid} runs a per-repo full-history git walk in a blocking thread to
# decide whether the caller may read a path-scoped blob. It is publicly reachable,
# so it is bounded to keep a permissionless caller from fanning out unbounded
# concurrent walks and exhausting blocking-pool threads + PIDs.
# Max concurrent /ipfs walks across all callers (a pool of its own, disjoint from
# the served-git pools). Over-cap sheds a 503. Default 32.
GITLAWB_MAX_CONCURRENT_IPFS_WALKS=32
# Max concurrent /ipfs walks a single SOURCE IP may hold (keyed like the git
# per-caller caps via GITLAWB_TRUSTED_PROXY; reject-before-insert bounded map).
# Default 4.
GITLAWB_IPFS_WALK_PER_SOURCE=4
# Max EXPENSIVE path-scope visibility walks per single /ipfs request (only a
# blob in a path-scoped repo costs a full-history walk). Over-cap repos are
# skipped without a verdict and the scan continues; if the object is then found
# nowhere the request sheds a retryable 503 instead of a false 404. Default 64.
GITLAWB_IPFS_MAX_REPOS_WALKED=64
# Ceiling on repos one /ipfs request may VISIT past the visibility gate. Each
# visit costs a repo acquire — on a Tigris cache miss a full archive download,
# so this is also the worst-case object-store fetch count per request — plus a
# cat-file probe. On exhaustion the scan stops and sheds a retryable 503.
# Default 1024.
GITLAWB_IPFS_MAX_REPO_VISITS=1024
# Absolute wall-clock budget for one admitted /ipfs request's acquire+walk
# lifetime (all stages of the whole scan). Per-stage clamps bound the acquire
# and walk stages to the remaining budget, and no stage starts once it is
# exhausted (the scan then sheds a retryable 503). The object-type probe and
# content-read cat-file subprocesses are budget-checked before starting AND each
# run under their own deadline (the lesser of GITLAWB_GIT_SERVICE_TIMEOUT_SECS and
# the remaining budget), reaped via process-group teardown, so a hung cat-file
# cannot hold the request's walk slot past it. One hang path is still unbounded:
# the probe's object-store readability check is a plain filesystem sweep with
# nothing to reap, so a wedged filesystem can hold the slot past the deadline.
# Must be 1..=3153600000 (100 years): the node derives an Instant deadline from
# this value, and a larger one cannot be represented. Default 600.
GITLAWB_IPFS_REQUEST_BUDGET_SECS=600
# Max /ipfs/{cid} requests per client IP per hour (route flood brake, distinct
# from the concurrency caps above). 0 disables. Default 600.
GITLAWB_IPFS_RATE_LIMIT=600
# ── Creation rate limiting (repo/agent/issue/PR flood brake) ──────────────
# Max creation requests (POST /api/v1/repos, /api/register, fork, issues,
# pulls) per client IP per hour, in addition to the per-DID limit. The per-DID
+13
View File
@@ -55,6 +55,19 @@ jobs:
- name: cargo clippy
run: cargo clippy --locked --workspace --all-targets -- -D warnings
release-tag-validation:
name: release tag validation
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Check out repository
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- name: Test release tag resolver
run: scripts/test-resolve-release-tag.sh
test:
name: test (${{ matrix.toolchain }})
runs-on: ubuntu-latest
+7 -1
View File
@@ -70,9 +70,15 @@ jobs:
const changedRust = names.some(n => n.startsWith("crates/") && n.endsWith(".rs"));
// A test can live in a tests/ dir, a *_test.rs file, or (the common Rust
// pattern) an inline #[test] / #[cfg(test)] block added inside the source file.
// Async tests use a namespaced attribute (#[tokio::test], #[async_std::test],
// possibly with args like #[tokio::test(flavor = ...)]) rather than the bare
// #[test]. Match only on added lines, with the attribute at the start of the
// line after horizontal whitespace, covering bare, cfg(test), namespaced,
// optional leading ::, and raw-identifier (r#test) forms. No head-file fetch
// or lexer: needs-tests is advisory, not a merge gate.
const addsInlineTest = files.some(f =>
f.filename.endsWith(".rs") && f.patch &&
/^\+.*#\[(test\]|cfg\(test\))/m.test(f.patch));
/^\+[ \t]*#\[[ \t]*(?:cfg[ \t]*\([ \t]*test[ \t]*\)|(?:::[ \t]*)?(?:[\w-]+[ \t]*::[ \t]*)*(?:r#)?test\b)/m.test(f.patch));
const touchedTests =
names.some(n => n.includes("/tests/") || n.endsWith("_test.rs")) || addsInlineTest;
if (changedRust && !touchedTests) want.add("needs-tests");
+19 -21
View File
@@ -216,6 +216,11 @@ jobs:
contents: read
packages: write
steps:
- name: Check out workflow scripts
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- name: Resolve release tag
id: rel
env:
@@ -223,13 +228,7 @@ jobs:
RELEASE_TAG: ${{ needs.release-please.outputs.tag_name }}
run: |
set -euo pipefail
TAG="${DISPATCH_TAG:-$RELEASE_TAG}"
case "$TAG" in
v[0-9]*) ;;
*) echo "::error::'$TAG' does not look like a release tag (vX.Y.Z)"; exit 1 ;;
esac
echo "tag=$TAG" >> "$GITHUB_OUTPUT"
echo "version=${TAG#v}" >> "$GITHUB_OUTPUT"
scripts/resolve-release-tag.sh "${DISPATCH_TAG:-$RELEASE_TAG}"
# ghcr requires a lowercase repository path, and unlike metadata-action,
# buildx's `--output name=` does no lowercasing — a mixed-case owner
# makes the digest push fail with "invalid reference format".
@@ -311,8 +310,14 @@ jobs:
if: ${{ !cancelled() && needs.docker.result == 'success' }}
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Check out workflow scripts
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- name: Resolve release tag
id: rel
env:
@@ -320,13 +325,7 @@ jobs:
RELEASE_TAG: ${{ needs.release-please.outputs.tag_name }}
run: |
set -euo pipefail
TAG="${DISPATCH_TAG:-$RELEASE_TAG}"
case "$TAG" in
v[0-9]*) ;;
*) echo "::error::'$TAG' does not look like a release tag (vX.Y.Z)"; exit 1 ;;
esac
echo "tag=$TAG" >> "$GITHUB_OUTPUT"
echo "version=${TAG#v}" >> "$GITHUB_OUTPUT"
scripts/resolve-release-tag.sh "${DISPATCH_TAG:-$RELEASE_TAG}"
- name: Download digests
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
@@ -542,6 +541,11 @@ jobs:
contents: read
id-token: write # npm trusted publishing (OIDC) + provenance
steps:
- name: Check out workflow scripts
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- name: Resolve release tag
id: rel
env:
@@ -549,13 +553,7 @@ jobs:
RELEASE_TAG: ${{ needs.release-please.outputs.tag_name }}
run: |
set -euo pipefail
TAG="${DISPATCH_TAG:-$RELEASE_TAG}"
case "$TAG" in
v[0-9]*) ;;
*) echo "::error::'$TAG' does not look like a release tag (vX.Y.Z)"; exit 1 ;;
esac
echo "tag=$TAG" >> "$GITHUB_OUTPUT"
echo "version=${TAG#v}" >> "$GITHUB_OUTPUT"
scripts/resolve-release-tag.sh "${DISPATCH_TAG:-$RELEASE_TAG}"
- name: Checkout release tag
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+44
View File
@@ -0,0 +1,44 @@
# Contributor guide for coding agents
This file carries the rules a patch author cannot recover from the code alone, for humans and coding agents alike. For build, test, and PR basics, read `CONTRIBUTING.md` first.
## Authorization invariants (the rules PRs most often break)
Every repo-scoped endpoint must bind the caller to an authorization decision before serving or mutating anything. The `known_ungated` allowlist in `authz_guard::every_repo_scoped_handler_is_gated` is empty on main: `list_webhooks`, `list_replicas`, and `list_protected_branches` are gated (`list_webhooks` with read-visibility then owner; the other two with `authorize_repo_read`). Do not re-open those surfaces as exceptions.
- Owner-only mutations (visibility, webhooks, protected branches, merges) are gated against the repo owner. Three gate forms are in use and all are recognized: webhooks and merges call `require_repo_owner` (`crates/gitlawb-node/src/api/mod.rs`), visibility uses its module-local `require_owner`, and protected branches compare inline with `did_matches`. Use the form the surrounding module already uses.
- Not every write is owner-only, and that is by design: `star_repo`/`unstar_repo` bind to the signer (they still require repo read access), `register_replica`/`unregister_replica` bind to the replica's own DID, the bounty actions (claim, submit, approve, cancel, dispute) have their own multi-party rules, and closing a PR or issue is owner-or-author. Do not add owner gates to these.
- Content-serving reads (blobs, trees, raw files) gate through `authorize_repo_read` with the specific path being served, so a withheld subtree is denied even on an otherwise-public repo; repo-level reads (listings) pass `"/"` by the helper's own contract. The source scans below cannot check this argument's value, so review will.
- A route that reaches repo data through a global id, with no repo in its path, or through an extractor shape the scans do not recognize still needs the same gate; the scans cannot see those.
- Read-surface denials must not reveal existence: a caller who may not read a repo (or a withheld subtree) gets the same 404 as a missing repo, never a 403. (Owner-gated mutations currently return 403 after the repo lookup; do not widen that shape to read paths.)
- Two tests in module `authz_guard` (`crates/gitlawb-node/src/api/mod.rs`) scan handler sources and fail on a repo-scoped handler without a recognized gate: `every_in_scope_mutation_has_its_gate` and `every_repo_scoped_handler_is_gated`. If they fail on your handler, add the gate. The second test's `known_ungated` allowlist is empty; never add an entry to it without an open issue and review.
## Adding or changing routes
- A new gated handler needs tests proving its route-specific gate fires: exercise an unauthorized authenticated caller and an anonymous caller where applicable, assert the exact denial status each way, and assert the body leaks nothing. The non-owner tests in `crates/gitlawb-node/src/test_support.rs` show the shape for owner-gated routes.
- A new mutation handler also gets a row in `every_in_scope_mutation_has_its_gate` naming its gate type; the per-handler row is the point of that test.
- A table-driven deny suite covering every deny-bearing route is in review (#194, #195). If it is present in your checkout, add a probe row there for any new deny-bearing route.
## Verifying signatures and certificates
- Derive the verifying key from something outside the artifact you are checking: the node's own DID, a pinned or configured value, a peers lookup. A key read from a field of the object being verified (`Did::from_str(&cert.node_did)`) only proves the artifact is self-consistent, and identities here are permissionless, so anyone can mint a keypair, name it in the artifact, self-sign, and be told it is valid. A mismatch against that anchor has to fail the result, not log a warning.
- A handler whose correct answer is "valid" needs a positive case driven by a trusted, independently anchored artifact, and a rejection case for a well-formed, fully populated forged artifact. Invalid-only suites miss an implementation that rejects everything; a forged artifact that still returns valid is the regression that matters.
- The fields a signature covers are a versioned format. Adding one invalidates every artifact already signed, so give the payload a version, keep a path that verifies the older form, and add a test that verifies an artifact signed before your change.
## Removing a serving path
When a fix removes a path that used to serve content, the same PR must add a test asserting the removed path now denies. The deletion is what turns that test green; without it the path can quietly come back.
## Database migrations
Schema changes are code-defined in `MIGRATIONS` (`crates/gitlawb-node/src/db/mod.rs`). Always append a new versioned entry; never edit an entry that has merged (operators deploy from main). Test runs build the schema from scratch, so editing an applied version stays green in CI while breaking every existing deployment.
## Client behavior (`gl`, `git-remote-gitlawb`)
When a node denies a request, surface the denial to the user. Never render a denial as an empty list or a silent success; that turns an authorization boundary into invisible data loss.
## Toolchain and CI reality
- MSRV is Rust 1.91 (`rust-version` in the workspace `Cargo.toml`). The MSRV CI job runs `cargo check --workspace --all-targets` on exactly 1.91, so check that an API you rely on is stable there; the full test suite runs on stable.
- The PR workflow runs these jobs on every PR: `cargo fmt --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo test --workspace` (against Postgres), a release build, `cargo audit`, the MSRV check, and a Docker build. The `main` branch ruleset currently requires one approving review but does not require these checks to pass before merge; a failing job should block review approval in practice, but nothing enforces that mechanically today.
- Conventional commit titles (`feat:`, `fix:`, `docs:`, ...) are required by convention and drive releases, but no CI job checks them; a bad title fails review, not the pipeline.
+1
View File
@@ -54,6 +54,7 @@ Smart contracts live in a separate repo: [github.com/Gitlawb/contracts](https://
3. **No breaking changes without discussion.** Open an issue first for protocol-level changes.
4. **Conventional commits.** Use `feat:`, `fix:`, `docs:`, `refactor:`, `test:`, `chore:`. Releases are automated by [release-please](https://github.com/googleapis/release-please) — your commit prefixes drive the next version bump.
5. **Format and lint.** Run `cargo fmt --all` and `cargo clippy --workspace --all-targets -- -D warnings` before submitting. CI will reject anything that fails these.
6. **Security-surface rules.** [`AGENTS.md`](AGENTS.md) carries the authorization invariants PRs most often break; it is written for coding agents and humans alike, so read it before touching endpoints, visibility, or migrations.
## What gets merged, what gets closed
Generated
+3
View File
@@ -3378,10 +3378,13 @@ version = "0.7.1"
dependencies = [
"anyhow",
"gitlawb-core",
"libc",
"mockito",
"reqwest",
"tempfile",
"tracing",
"tracing-subscriber",
"windows-sys 0.61.2",
]
[[package]]
+25 -13
View File
@@ -4,7 +4,7 @@
Gitlawb Node is the open-source node software behind the Gitlawb network. It lets anyone run a self-hosted node, publish repositories under a DID, sign writes with Ed25519 HTTP signatures, replicate git activity across peers, and move toward a resilient app-delivery network where code and build assets can be served closer to users.
Gitlawb is not trying to be only “another git host.” The long-term direction is:
Gitlawb is not trying to be only "another git host." The long-term direction is:
```txt
Decentralized GitHub
@@ -34,12 +34,12 @@ This is a Rust workspace with four crates:
Most git hosting today depends on a small number of centralized platforms. Gitlawb Nodes are designed for a different model:
- **Own your identity** — every user, agent, and node is an Ed25519 keypair represented as `did:key:z6Mk...`.
- **Signed writes by default** — write requests use RFC 9421 HTTP Signatures instead of passwords.
- **Git-native transport** — repositories are still real git repositories served over smart HTTP.
- **Agent-native workflows** — the `gl` CLI and MCP server expose repo, issue, task, PR, and UCAN flows to AI agents.
- **Peer-aware delivery** — nodes can announce, discover, gossip, and sync with each other.
- **App CDN direction** — the network can evolve from decentralized code storage into code + asset + app delivery.
- **Own your identity**: every user, agent, and node is an Ed25519 keypair represented as `did:key:z6Mk...`.
- **Signed writes by default**: write requests use RFC 9421 HTTP Signatures instead of passwords.
- **Git-native transport**: repositories are still real git repositories served over smart HTTP.
- **Agent-native workflows**: the `gl` CLI and MCP server expose repo, issue, task, PR, and UCAN flows to AI agents.
- **Peer-aware delivery**: nodes can announce, discover, gossip, and sync with each other.
- **App CDN direction**: the network can evolve from decentralized code storage into code + asset + app delivery.
---
@@ -189,14 +189,14 @@ For public-network use, make sure `GITLAWB_NODE` points to the node you want. Th
Public nodes (e.g. `node.gitlawb.com`) require two things on writes:
1. **RFC 9421 HTTP Signatures** — every write is signed by your identity key. `gl`
1. **RFC 9421 HTTP Signatures**: every write is signed by your identity key. `gl`
and the `git-remote-gitlawb` helper do this automatically. An old/unsigned CLI
fails with `401 not_an_agent`; `gl` will tell you to upgrade and register.
2. **An iCaptcha proof** on the spam-gated writes (**repo create, fork, register**).
`gl` solves this for you: on the node's `403 icaptcha_proof_required` it reads the
`x-icaptcha-url` / `x-icaptcha-level` hints, requests a challenge, solves it
locally (arithmetic / algebra / sequence), and **retries the same signed request**
with the `x-icaptcha-proof` header — no manual steps, no env vars.
with the `x-icaptcha-proof` header. No manual steps, no env vars.
```bash
gl identity new # create did:key identity
@@ -215,14 +215,14 @@ Notes:
- **Proofs are short-lived (~5 min TTL) and single-use.** If one expires between
solving and use, the client transparently solves a fresh one and retries.
- **What needs what:** create / fork / register are signed **and** iCaptcha-gated;
`git push` is **signed-only** (owner signature is the gate — no per-push challenge);
`git push` is **signed-only** (owner signature is the gate, no per-push challenge);
reads (clone / fetch / `repo info`) need no proof. A non-existent repo returns a
clear `404`, never a placeholder.
- **API-key iCaptcha deployments:** set `GITLAWB_ICAPTCHA_URL` to your iCaptcha
origin and `GITLAWB_ICAPTCHA_API_KEY` to its key. The client only talks to an
`https` origin whose host is allowlisted (that URL or the public default), and
sends the bearer token **only** to your configured origin — never to a URL a
node advertises — so a hostile node can't capture the key or redirect the solve.
sends the bearer token **only** to your configured origin, never to a URL a
node advertises, so a hostile node can't capture the key or redirect the solve.
---
@@ -343,7 +343,19 @@ Important node settings:
| `GITLAWB_REQUIRE_SIGNED_PEER_WRITES` | Require signed peer announce/sync writes. |
| `GITLAWB_AUTO_SYNC` | Enable automatic sync from known peers. |
| `GITLAWB_MAX_PACK_BYTES` | Max git pack body size for smart-HTTP routes. |
| `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack/receive-pack may run before it is aborted (504). Default 600. Does not bound `info/refs` or the withheld-blob path. |
| `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack, receive-pack, or `info/refs` advertisement may run before it is aborted (504). Default 600. Also bounds the withheld-blob classification walk (on both the upload-pack serve and receive-pack replication paths) and the push-side pin-candidate discovery (`rev-list` / `cat-file`), each reaped via process-group teardown at the deadline. On the path-scoped upload-pack path the classification walk and the pack serve share ONE deadline, so this value bounds their combined duration rather than granting each stage a full budget: a walk that consumes it leaves the serve nothing and the clone gets a 504. Serving large path-scoped repos may therefore need a higher value than they did when each stage was budgeted separately. Accepted range is 1 to 3153600000 (100 years), since the node derives deadlines from this value and a larger one cannot be represented. |
| `GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS` | Max seconds the storage-acquisition phase (Tigris HEAD/GET, push advisory-lock) of a served git op may run before the request is shed with a 503, separate from the git-run timeout. The concurrency permit is released on expiry so a stalled backend cannot pin the pool. Default 30. |
| `GITLAWB_MAX_CONCURRENT_GIT_OPS` | Max concurrent served git READ ops (upload-pack and its `info/refs` advertisement) across all callers; over-cap sheds a 503 + Retry-After. Anonymous reads draw from this pool, so pair it with `GITLAWB_MAX_CONCURRENT_READS_PER_CALLER`. Pushes and the receive-pack advertisement have their own pools, so a read flood cannot shed an authenticated push. Default 128. |
| `GITLAWB_MAX_CONCURRENT_GIT_PUSHES` | Max concurrent `git-receive-pack` POST operations, in a pool separate from the read pool. The anon receive-pack `info/refs` advertisement runs in a third pool of the same size, disjoint from both, so an advertisement flood cannot shed a push either. Two per-source push caps are derived from this value (`/8`, floor 1) and have no env var of their own. Over-cap sheds a 503 + Retry-After. Default 32. |
| `GITLAWB_MAX_CONCURRENT_READS_PER_CALLER` | Max concurrent read ops a single caller may hold, so one caller cannot monopolize the read pool. Keyed on the resolved source IP, never the DID, and only as granular as `GITLAWB_TRUSTED_PROXY`: left unset, a node behind an edge or NAT keys every caller on the edge IP and this collapses to one global cap. Default 16. |
| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | Max post-push pin loops (IPFS + Pinata) running concurrently across all repos. This caps how many loops RUN at once, not how much object-id list memory the node retains: on the local IPFS path a loop parked waiting for a permit still holds its full list. Do not size memory from this knob alone. A loop over cap waits, never drops a pin. Default 8. |
| `GITLAWB_REPO_LEASE_MAX_WAITERS` | Max pushes parked at once waiting for the same repo's write lease. Each waiter pins its buffered pack body, so this bounds that memory for a hot repo; past the cap the newest push sheds a 503 + Retry-After instead of queueing. Pushes to other repos are unaffected, and the lease holder is not counted. Default 8. |
| `GITLAWB_MAX_CONCURRENT_IPFS_WALKS` | Max concurrent `GET /ipfs/{cid}` visibility walks across all callers (own pool, disjoint from the served-git pools); over-cap sheds 503. Default 32. |
| `GITLAWB_IPFS_WALK_PER_SOURCE` | Max concurrent `/ipfs` walks a single source IP may hold. Default 4. |
| `GITLAWB_IPFS_MAX_REPOS_WALKED` | Max expensive path-scope visibility walks per `/ipfs/{cid}` request; over-cap repos are skipped and the scan continues, shedding a retryable 503 (not a false 404) if the object is then found nowhere. Default 64. |
| `GITLAWB_IPFS_MAX_REPO_VISITS` | Ceiling on repos one `/ipfs/{cid}` request may visit (acquire + probe) past the visibility gate. Also the worst-case per-request Tigris fetch count. On exhaustion the scan stops with a retryable 503. Default 1024. |
| `GITLAWB_IPFS_REQUEST_BUDGET_SECS` | Absolute wall-clock budget for one admitted `/ipfs/{cid}` request's acquire+walk lifetime. Per-stage clamps bound the acquire and walk stages to the remaining budget, and no stage starts once it is exhausted; the scan then stops with a retryable 503. The object-type probe and content-read `cat-file` subprocesses are budget-checked before starting and each also run under their own deadline (the lesser of `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` and the remaining budget), reaped via process-group teardown, so a hung `cat-file` cannot hold the request's walk slot past it. One hang path is still unbounded: the probe's object-store readability check is a plain filesystem sweep with nothing to reap, so a wedged filesystem can hold the slot past the deadline. Default 600. Accepted range is 1 to 3153600000 (100 years), since the node derives a deadline from this value and a larger one cannot be represented. |
| `GITLAWB_IPFS_RATE_LIMIT` | Max `/ipfs/{cid}` requests per client IP per hour (route flood brake). 0 disables. Default 600. |
| `GITLAWB_TIGRIS_BUCKET` | Optional S3/Tigris shared repo storage bucket. |
| `GITLAWB_PINATA_JWT` | Optional Pinata/IPFS warm-storage pinning. |
| `GITLAWB_IRYS_URL` | Optional Irys/Arweave permanent anchoring. |
+17
View File
@@ -19,3 +19,20 @@ tracing-subscriber = { workspace = true }
[dev-dependencies]
mockito = "1"
tempfile = "3"
[target.'cfg(unix)'.dev-dependencies]
libc = "0.2"
[target.'cfg(windows)'.dev-dependencies]
# Job Object bindings for the real_git_fetch harness: the Windows analog of the
# unix process-group teardown, so the whole `git fetch` tree can be killed
# together on both the timeout and clean-exit paths (INV-22). Win32_Security is
# required by CreateJobObjectW, and Win32_System_Threading by
# JOBOBJECT_EXTENDED_LIMIT_INFORMATION (it embeds IO_COUNTERS).
windows-sys = { version = "0.61", features = [
"Win32_Foundation",
"Win32_Security",
"Win32_System_JobObjects",
"Win32_System_Threading",
] }
+832 -62
View File
@@ -276,27 +276,32 @@ fn handle_connect<R: Read>(
// ── Phase 2: pack exchange (POST /<service>) ──────────────────────────────
//
// The two services behave differently with their write pipe:
// The two services frame Phase 2 differently:
//
// git-upload-pack (clone/fetch):
// Client sends pkt-line want/have negotiation ending with "done\n",
// but does NOT close its write pipe — it waits for the pack response.
// We must detect the terminal "done\n" pkt-line to know when to POST.
// Git speaks the stateful native protocol over `connect`: it sends its
// wants, a flush, then have-batches each terminated by a flush, and blocks
// for the server's ACK/NAK after every flush before it sends more haves or
// the terminal "done\n". The node serves `git upload-pack --stateless-rpc`,
// which keeps no state between POSTs, so we run a per-round loop: POST a
// self-contained request once per flush-terminated batch and stream each
// response back to git, until the "done" round returns the pack. Collapsing
// this into one POST deadlocks a multi-round fetch (#117).
//
// git-receive-pack (push):
// Client sends ref-update commands + complete PACK blob, then closes
// its write pipe. read_to_end is safe and correct here.
// Git sends ref-update commands + the complete PACK blob, then closes its
// write pipe. read_to_end is safe and correct here, a single POST.
let request_body = if service == "git-upload-pack" {
read_upload_pack_request(stdin).context("reading upload-pack request")?
} else {
let mut buf = Vec::new();
stdin
.read_to_end(&mut buf)
.context("reading receive-pack request")?;
buf
};
let post_url = format!("{}/{}", repo_base, service);
if service == "git-upload-pack" {
return negotiate_upload_pack(&client, &post_url, service, signing_key, stdin, &mut stdout);
}
let mut request_body = Vec::new();
stdin
.read_to_end(&mut request_body)
.context("reading receive-pack request")?;
tracing::debug!("pack request: {} bytes from git", request_body.len());
if request_body.is_empty() {
@@ -305,32 +310,14 @@ fn handle_connect<R: Read>(
return Ok(());
}
let post_url = format!("{}/{}", repo_base, service);
tracing::debug!("POST {post_url} ({} bytes)", request_body.len());
let req = build_pack_post_request(&client, &post_url, service, &request_body, signing_key);
// Attach the body after signing so the pack bytes are moved, not cloned —
// packs can be large and the clone doubled peak memory on push.
let pack_resp = req
.body(request_body)
.send()
.with_context(|| format!("POST {post_url}"))?;
if !pack_resp.status().is_success() {
let status = pack_resp.status();
let body = read_error_body(pack_resp);
let path = format!("/{service}");
bail!("{}", http_error_message("POST", &path, status, &body, None));
}
let pack_bytes = pack_resp.bytes().context("reading pack response")?;
tracing::debug!("pack response: {} bytes from node", pack_bytes.len());
stdout.write_all(&pack_bytes)?;
stdout.flush()?;
Ok(())
post_pack_round(
&client,
&post_url,
service,
signing_key,
request_body,
&mut stdout,
)
}
// ── Smart-protocol request builders ───────────────────────────────────────────
@@ -424,33 +411,55 @@ fn parse_gitlawb_url(url: &str) -> Result<(String, String, String)> {
Ok((did_string.to_string(), short_owner, repo_name))
}
/// Read a complete git-upload-pack request from the pkt-line stream.
/// How a single upload-pack negotiation round ended.
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
enum RoundEnd {
/// A flush pkt (`0000`) ended the round; more negotiation may follow.
Flush,
/// The terminal `done\n` pkt ended the round; the pack follows.
Done,
/// The stream ended (git closed its write pipe).
Eof,
}
/// Read ONE round of git's upload-pack request from the pkt-line stream.
///
/// For upload-pack, git sends its want/have negotiation ending with the pkt-line
/// `"done\n"` but does NOT close its write pipe afterwards — it waits for the
/// server's pack response. We detect the terminal "done\n" and stop reading.
///
/// We also handle the flush-only case (`"0000"`) that git sends when it already
/// has everything it needs (up-to-date clone).
fn read_upload_pack_request<R: Read>(stdin: &mut R) -> Result<Vec<u8>> {
let mut buf = Vec::new();
/// Git speaks the stateful native protocol over the `connect` capability: it
/// sends its `want` lines, a flush, then `have` batches each terminated by a
/// flush, and blocks for the server's ACK/NAK after every flush before sending
/// more haves or the terminal `done\n`. Each call returns one such round: the
/// pkt-lines up to (but not including) the terminating flush or `done`, plus
/// which terminator ended it. `negotiate_upload_pack` reassembles these into
/// self-contained stateless-RPC POST bodies. Returning at the flush, instead of
/// buffering past it waiting for `done`, is what lets a multi-round fetch make
/// progress rather than deadlock (#117).
fn read_upload_pack_round<R: Read>(stdin: &mut R) -> Result<(Vec<u8>, RoundEnd)> {
let mut content = Vec::new();
loop {
// Read the 4-byte hex pkt-line length prefix
// Read the 4-byte hex pkt-line length prefix.
let mut len_bytes = [0u8; 4];
match stdin.read_exact(&mut len_bytes) {
Ok(_) => {}
Err(e) if e.kind() == io::ErrorKind::UnexpectedEof => break,
Err(e) if e.kind() == io::ErrorKind::UnexpectedEof => {
return Ok((content, RoundEnd::Eof));
}
Err(e) => return Err(e.into()),
}
buf.extend_from_slice(&len_bytes);
let len_hex = std::str::from_utf8(&len_bytes).unwrap_or("0000");
let pkt_len = usize::from_str_radix(len_hex, 16).unwrap_or(0);
// A malformed 4-byte length header must NOT be silently collapsed to a
// 0000 flush (#192 F5): that masks corrupted/desynchronized protocol input
// and can emit accumulated have lines as a synthesized flush round.
let Ok(len_hex) = std::str::from_utf8(&len_bytes) else {
bail!("malformed pkt-line length prefix: non-UTF-8 bytes {len_bytes:02x?}");
};
let Ok(pkt_len) = usize::from_str_radix(len_hex, 16) else {
bail!("malformed pkt-line length prefix: non-hex {len_hex:?}");
};
if pkt_len == 0 {
// Flush pkt "0000" — keep buffering (more pkt-lines may follow)
continue;
// Flush pkt "0000": this round is complete.
return Ok((content, RoundEnd::Flush));
}
if pkt_len < 4 {
@@ -462,16 +471,149 @@ fn read_upload_pack_request<R: Read>(stdin: &mut R) -> Result<Vec<u8>> {
stdin
.read_exact(&mut data)
.context("reading pkt-line data")?;
buf.extend_from_slice(&data);
// "done\n" signals the end of the want/have negotiation
// "done\n" ends the negotiation; the pack follows. The terminator is
// reported out-of-band and re-emitted by the caller, so it is not
// appended to `content`.
if data == b"done\n" {
tracing::debug!("upload-pack: got 'done', request complete");
break;
tracing::debug!("upload-pack: got 'done', round complete");
return Ok((content, RoundEnd::Done));
}
content.extend_from_slice(&len_bytes);
content.extend_from_slice(&data);
}
}
/// POST one self-contained stateless-RPC request body and stream the response to
/// `stdout`. Signs the request when `signing_key` is present, so every round of a
/// private-repo fetch carries the caller's signature (the node visibility-gates
/// each POST at "/"). A non-2xx status surfaces through the sanitized error path
/// rather than being rendered as an empty or successful fetch (INV-6/INV-8).
fn post_pack_round(
client: &reqwest::blocking::Client,
post_url: &str,
service: &str,
signing_key: Option<&Keypair>,
body: Vec<u8>,
stdout: &mut impl Write,
) -> Result<()> {
tracing::debug!("POST {post_url} ({} bytes)", body.len());
let req = build_pack_post_request(client, post_url, service, &body, signing_key);
// Attach the body after signing so the bytes are moved, not cloned.
let resp = req
.body(body)
.send()
.with_context(|| format!("POST {post_url}"))?;
if !resp.status().is_success() {
let status = resp.status();
let err_body = read_error_body(resp);
let path = format!("/{service}");
bail!(
"{}",
http_error_message("POST", &path, status, &err_body, None)
);
}
Ok(buf)
let bytes = resp.bytes().context("reading pack response")?;
tracing::debug!("pack response: {} bytes from node", bytes.len());
stdout.write_all(&bytes)?;
stdout.flush()?;
Ok(())
}
/// Drive a git-upload-pack fetch as a v0 smart-HTTP stateless-RPC client.
///
/// Git (over `connect`) speaks the stateful native protocol; the node serves
/// `git upload-pack --stateless-rpc`, which keeps no state between POSTs. Bridge
/// the two: capture the want block once, then for each flush-terminated have
/// batch POST a self-contained request (the wants plus every have accumulated so
/// far, then exactly one terminator) and stream the ACK/NAK back to git so it can
/// continue. The `done` round returns the pack. Every POST re-sends the wants and
/// all prior haves because the server is stateless; leaving an intermediate flush
/// in the body would truncate the negotiation, so each body carries exactly one
/// terminator.
fn negotiate_upload_pack<R: Read>(
client: &reqwest::blocking::Client,
post_url: &str,
service: &str,
signing_key: Option<&Keypair>,
stdin: &mut R,
stdout: &mut impl Write,
) -> Result<()> {
// Opening round: the want block, a bare `done`, or an empty/flush-only
// request from an up-to-date or aborted fetch.
let (wants, wend) = read_upload_pack_round(stdin)?;
match wend {
// Up-to-date with an empty request, truncated, or aborted before a
// terminator: nothing safe to POST.
RoundEnd::Eof => return Ok(()),
// A `done` with no preceding want-section flush: the bare-`done` request
// the single-round tests feed. Forward it verbatim as one POST.
RoundEnd::Done => {
let mut body = wants;
body.extend_from_slice(b"0009done\n");
return post_pack_round(client, post_url, service, signing_key, body, stdout);
}
// Normal: the want section ended with its flush; negotiate the haves.
// (An empty want block here is a flush-only opener and falls through to
// the loop, which POSTs nothing and returns at EOF.)
RoundEnd::Flush => {}
}
// The node is stateless between POSTs, so re-send the wants and every have so
// far in each round.
let mut acc_haves: Vec<u8> = Vec::new();
loop {
let (batch, bend) = read_upload_pack_round(stdin)?;
// Only POST when a round carries new haves or a terminator. A round with no
// new haves needs no request: an empty EOF ends the fetch, and a bare flush
// (git never sends a content-free flush mid-negotiation, but a malformed
// stream could) is skipped so a run of empty flushes cannot amplify into one
// signed POST each. A `done` round with no haves still POSTs (the fresh-clone
// case), handled by the terminator match below.
match (batch.is_empty(), bend) {
(true, RoundEnd::Eof) => return Ok(()),
(true, RoundEnd::Flush) => continue,
_ => {}
}
acc_haves.extend_from_slice(&batch);
// Compose a self-contained stateless-RPC request: wants + one flush + all
// accumulated haves + exactly one terminator. No intermediate flush
// survives into the body, or the stateless server treats the first flush
// after the haves as end-of-request and truncates the negotiation.
let mut body = wants.clone();
body.extend_from_slice(b"0000");
body.extend_from_slice(&acc_haves);
let final_round = match bend {
RoundEnd::Flush => {
body.extend_from_slice(b"0000");
false
}
// Only a real `done` pkt finalizes the stateless request.
RoundEnd::Done => {
body.extend_from_slice(b"0009done\n");
true
}
// A nonempty batch terminated by EOF (no flush, no done) means git
// ABORTED mid-negotiation (#192, F1). Terminate WITHOUT POSTing:
// synthesizing a `done` here would make the node generate and stream a
// full pack — signed, for a private fetch — to a client that has gone
// away. The accumulated haves are discarded; there is no one to serve.
RoundEnd::Eof => return Ok(()),
};
post_pack_round(client, post_url, service, signing_key, body, stdout)?;
if final_round {
return Ok(());
}
}
}
/// Strip the HTTP smart-protocol service announcement from a GET /info/refs response.
@@ -1100,6 +1242,35 @@ mod tests {
"a tampered @path must fail verification"
);
}
// #117: a multi-round POST body (wants + flush + accumulated haves + done)
// signs and verifies over its EXACT bytes. Each per-round POST a private
// fetch emits is a different accumulated body, and build_pack_post_request
// signs the same bytes it sends, so every round is independently accepted
// by the node verifier, not just the single-round `done` body.
let mut acc = Vec::new();
acc.extend_from_slice(&pkt(&format!(
"want {} multi_ack_detailed side-band-64k\n",
"a".repeat(40)
)));
acc.extend_from_slice(b"0000");
acc.extend_from_slice(&pkt(&format!("have {}\n", "1".repeat(40))));
acc.extend_from_slice(&pkt(&format!("have {}\n", "2".repeat(40))));
acc.extend_from_slice(&pkt("done\n"));
let post_url = "http://node.example/zOwner/myrepo/git-upload-pack";
let req = build_pack_post_request(&client, post_url, "git-upload-pack", &acc, Some(&kp))
.body(acc.clone())
.build()
.unwrap();
node_verifies(
"POST",
&path_and_query(&req),
&acc,
&header(&req, "signature-input"),
&header(&req, "signature"),
&header(&req, "content-digest"),
)
.expect("a multi-round accumulated POST body must verify under the node's verifier");
}
#[test]
@@ -1399,4 +1570,603 @@ mod tests {
assert!(help.contains("GITLAWB_NODE"));
assert!(help.ends_with('\n'));
}
// ── #117 multi-round fetch negotiation ───────────────────────────────────
/// Encode a git pkt-line: 4-byte hex length (incl. the 4 bytes) + data.
fn pkt(data: &str) -> Vec<u8> {
format!("{:04x}{}", data.len() + 4, data).into_bytes()
}
/// A realistic capability-bearing opening want line (git puts its capability
/// list on the first want), so the multi-round tests exercise the want shape
/// real git actually sends, not a bare `want <sha>`.
fn want_line(sha: &str) -> Vec<u8> {
pkt(&format!(
"want {sha} multi_ack_detailed side-band-64k ofs-delta agent=git/2.43\n"
))
}
/// A reader that yields `seed`, then BLOCKS until `gate` fires. Models git
/// holding the upload-pack pipe open after a have-batch flush, waiting for the
/// server's ACK/NAK before it sends more. A real pipe blocks here; an in-memory
/// Cursor would EOF and hide the bug, which is why the pre-#117 tests missed it.
struct BlockAfterSeed {
seed: io::Cursor<Vec<u8>>,
gate: std::sync::mpsc::Receiver<()>,
}
impl Read for BlockAfterSeed {
fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {
let n = self.seed.read(buf)?;
if n > 0 {
return Ok(n);
}
let _ = self.gate.recv(); // block like a live pipe; unblock -> EOF
Ok(0)
}
}
/// #117 primitive contract (matrix item 1): `read_upload_pack_round` returns at
/// a flush-terminated batch instead of buffering past it waiting for `done`.
/// The pre-#117 `read_upload_pack_request` blocked here (the deadlock). A
/// blocking reader proves the return under live-pipe semantics: if the reader
/// buffered past the flush it would block on the gate and the test would time out.
#[test]
fn read_upload_pack_round_returns_on_flush_without_done() {
let mut seed = Vec::new();
seed.extend_from_slice(&want_line(&"a".repeat(40)));
seed.extend_from_slice(b"0000"); // end of wants
for i in 0..32u32 {
seed.extend_from_slice(&pkt(&format!("have {:040x}\n", i)));
}
seed.extend_from_slice(b"0000"); // have-batch flush; git awaits ACK, sends NO done
let (tx, rx) = std::sync::mpsc::channel::<()>();
let mut reader = BlockAfterSeed {
seed: io::Cursor::new(seed),
gate: rx,
};
let (done_tx, done_rx) = std::sync::mpsc::channel::<(Vec<u8>, RoundEnd)>();
let handle = std::thread::spawn(move || {
// Opening round (wants) returns at the first flush; the have batch
// returns at the second flush. Neither may block for a `done`.
let _wants = read_upload_pack_round(&mut reader).unwrap();
let haves = read_upload_pack_round(&mut reader).unwrap();
let _ = done_tx.send(haves);
});
let got = done_rx.recv_timeout(std::time::Duration::from_secs(2));
let _ = tx.send(()); // unblock the reader so the thread can exit
let _ = handle.join();
let (batch, end) = got.expect(
"read_upload_pack_round blocked past a flush-terminated have-batch \
waiting for `done` (multi-round fetch deadlock #117)",
);
assert_eq!(end, RoundEnd::Flush, "a flush terminates the round");
assert!(
batch.windows(4).any(|w| w == b"have"),
"the have batch content is returned to the caller"
);
}
/// #117 core (matrix item 2): a multi-round negotiation issues MORE THAN ONE
/// POST, and each POST body is a self-contained stateless-RPC request: wants +
/// one flush + every have accumulated so far + exactly one terminator. The
/// round-2 body is asserted byte-for-byte, which pins both accumulation (round
/// 1's have is present) and the one-terminator invariant (no intermediate flush
/// survives). Pre-#117 this was a single POST.
#[test]
fn multi_round_fetch_posts_each_round_with_accumulated_wants_and_haves() {
let want_sha = "a".repeat(40);
let have1 = "1".repeat(40);
let have2 = "2".repeat(40);
let (base_url, server) = serve_http(vec![
TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-upload-pack",
status: "200 OK",
body: "0000",
},
// Round 1 (flush-terminated): an ACK/NAK continuation, no pack yet.
TestResponse {
request_line: "POST /zOwner/myrepo/git-upload-pack",
status: "200 OK",
body: "0008NAK\n",
},
// Round 2 (done): final response plus the (fake) pack.
TestResponse {
request_line: "POST /zOwner/myrepo/git-upload-pack",
status: "200 OK",
body: "0008NAK\nPACKfake",
},
]);
let repo_base = format!("{base_url}/zOwner/myrepo");
let wants = want_line(&want_sha);
let mut stdin_bytes = Vec::new();
stdin_bytes.extend_from_slice(&wants);
stdin_bytes.extend_from_slice(b"0000"); // end of wants
stdin_bytes.extend_from_slice(&pkt(&format!("have {have1}\n")));
stdin_bytes.extend_from_slice(b"0000"); // round-1 flush: git awaits ACK
stdin_bytes.extend_from_slice(&pkt(&format!("have {have2}\n")));
stdin_bytes.extend_from_slice(&pkt("done\n")); // round-2 terminator
let mut stdin = io::Cursor::new(stdin_bytes);
handle_connect(&repo_base, "git-upload-pack", None, &mut stdin)
.expect("multi-round fetch should complete");
let requests = server.join().unwrap();
assert_eq!(
requests.len(),
3,
"GET + two POSTs, one per negotiation round"
);
assert!(
request_body(&requests[1])
.windows(want_sha.len())
.any(|w| w == want_sha.as_bytes()),
"round-1 POST must carry the wants"
);
// Round-2 body: wants + one flush + BOTH haves + done, with no flush
// between the haves (the accumulation + one-terminator invariant).
let mut expected = Vec::new();
expected.extend_from_slice(&wants);
expected.extend_from_slice(b"0000");
expected.extend_from_slice(&pkt(&format!("have {have1}\n")));
expected.extend_from_slice(&pkt(&format!("have {have2}\n")));
expected.extend_from_slice(&pkt("done\n"));
assert_eq!(
request_body(&requests[2]),
expected,
"round-2 POST body must be wants + flush + all accumulated haves + one done terminator"
);
}
/// Extract the HTTP request body (bytes after the header/body separator) from a
/// captured request string. pkt-lines are ASCII, so the lossy round-trip is exact.
fn request_body(request: &str) -> Vec<u8> {
match request.split_once("\r\n\r\n") {
Some((_, body)) => body.as_bytes().to_vec(),
None => Vec::new(),
}
}
/// U4 / INV-8 / INV-12: a private-repo multi-round fetch escalates on the 404
/// exactly once, then EVERY per-round POST carries the signature. Must-not: no
/// round goes out unsigned (an unsigned round 2 would 404 the owner's own fetch).
#[test]
fn private_multi_round_signs_every_post() {
let kp = Keypair::generate();
let (base_url, server) = serve_http(vec![
TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-upload-pack",
status: "404 Not Found",
body: r#"{"message":"not found"}"#,
},
TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-upload-pack",
status: "200 OK",
body: "0000",
},
TestResponse {
request_line: "POST /zOwner/myrepo/git-upload-pack",
status: "200 OK",
body: "0008NAK\n",
},
TestResponse {
request_line: "POST /zOwner/myrepo/git-upload-pack",
status: "200 OK",
body: "0008NAK\nPACKfake",
},
]);
let repo_base = format!("{base_url}/zOwner/myrepo");
let mut stdin = io::Cursor::new(two_round_stdin());
handle_connect(&repo_base, "git-upload-pack", Some(&kp), &mut stdin)
.expect("private multi-round fetch should complete");
let requests = server.join().unwrap();
assert_eq!(
requests.len(),
4,
"anon GET, signed GET retry, then two signed POSTs"
);
assert!(
!requests[0].to_lowercase().contains("signature-input"),
"first GET is anonymous"
);
assert!(
requests[1].to_lowercase().contains("signature-input"),
"GET retry is signed"
);
assert!(
requests[2].to_lowercase().contains("signature-input"),
"round-1 POST is signed"
);
assert!(
requests[3].to_lowercase().contains("signature-input"),
"round-2 POST is signed: no per-round POST may go out unsigned for a private repo"
);
}
/// U4: a public multi-round fetch stays anonymous on EVERY round even with a
/// keypair present. Must-not: no round signs (no DID disclosure on a public fetch).
#[test]
fn public_multi_round_stays_anonymous_every_post() {
let kp = Keypair::generate();
let (base_url, server) = serve_http(vec![
TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-upload-pack",
status: "200 OK",
body: "0000",
},
TestResponse {
request_line: "POST /zOwner/myrepo/git-upload-pack",
status: "200 OK",
body: "0008NAK\n",
},
TestResponse {
request_line: "POST /zOwner/myrepo/git-upload-pack",
status: "200 OK",
body: "0008NAK\nPACKfake",
},
]);
let repo_base = format!("{base_url}/zOwner/myrepo");
let mut stdin = io::Cursor::new(two_round_stdin());
handle_connect(&repo_base, "git-upload-pack", Some(&kp), &mut stdin)
.expect("public multi-round fetch should complete");
let requests = server.join().unwrap();
assert_eq!(requests.len(), 3, "GET + two POSTs, no retry");
for (i, r) in requests.iter().enumerate() {
assert!(
!r.to_lowercase().contains("signature-input"),
"request {i} must stay anonymous on a public fetch"
);
}
}
/// U4 / INV-8 / INV-12: a denial mid-negotiation (round-2 POST 404) surfaces as
/// an error, not a silent empty/successful fetch, and does NOT re-trigger the
/// Phase-1 signed-retry escalation (that is a GET-only, Phase-1-only behavior).
#[test]
fn mid_negotiation_post_denial_surfaces_without_reescalation() {
let (base_url, server) = serve_http(vec![
TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-upload-pack",
status: "200 OK",
body: "0000",
},
TestResponse {
request_line: "POST /zOwner/myrepo/git-upload-pack",
status: "200 OK",
body: "0008NAK\n",
},
TestResponse {
request_line: "POST /zOwner/myrepo/git-upload-pack",
status: "404 Not Found",
body: r#"{"message":"repository is no longer readable"}"#,
},
]);
let repo_base = format!("{base_url}/zOwner/myrepo");
let mut stdin = io::Cursor::new(two_round_stdin());
let err = handle_connect(&repo_base, "git-upload-pack", None, &mut stdin).unwrap_err();
let message = err.to_string();
assert!(
message.contains("POST /git-upload-pack returned 404 Not Found"),
"the mid-negotiation denial must surface, not read as an empty/successful fetch"
);
assert!(message.contains("repository is no longer readable"));
let requests = server.join().unwrap();
assert_eq!(
requests.len(),
3,
"GET + two POSTs, then bail: a POST denial must not re-run the Phase-1 escalation"
);
}
/// Two-round upload-pack request: wants, flush, have1, flush (round 1), have2,
/// done (round 2). Shared by the U4 signing/denial tests.
fn two_round_stdin() -> Vec<u8> {
let mut v = Vec::new();
v.extend_from_slice(&want_line(&"a".repeat(40)));
v.extend_from_slice(b"0000");
v.extend_from_slice(&pkt(&format!("have {}\n", "1".repeat(40))));
v.extend_from_slice(b"0000");
v.extend_from_slice(&pkt(&format!("have {}\n", "2".repeat(40))));
v.extend_from_slice(&pkt("done\n"));
v
}
/// U5 (matrix item 5, plumbing only): the node's withheld-blob path
/// (`upload_pack_excluding`) ignores negotiation and answers the first POST
/// with NAK plus a full self-contained pack. The helper must forward that
/// response and terminate without hanging. Whether REAL git accepts a pack
/// where it expected an ACK continuation is U7's real-git scenario, not this
/// mock (a Cursor never reacts to the response).
#[test]
fn withheld_shaped_response_is_forwarded_without_hanging() {
let (base_url, server) = serve_http(vec![
TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-upload-pack",
status: "200 OK",
body: "0000",
},
// Withheld shape: NAK + full pack on the FIRST POST, negotiation ignored.
TestResponse {
request_line: "POST /zOwner/myrepo/git-upload-pack",
status: "200 OK",
body: "0008NAK\nPACKfullselfcontainedpack",
},
]);
let repo_base = format!("{base_url}/zOwner/myrepo");
// A single flush-terminated round then EOF: the helper POSTs once, forwards
// the NAK+pack, and terminates at EOF rather than looping forever.
let mut stdin_bytes = Vec::new();
stdin_bytes.extend_from_slice(&want_line(&"a".repeat(40)));
stdin_bytes.extend_from_slice(b"0000");
stdin_bytes.extend_from_slice(&pkt(&format!("have {}\n", "1".repeat(40))));
stdin_bytes.extend_from_slice(b"0000");
let mut stdin = io::Cursor::new(stdin_bytes);
handle_connect(&repo_base, "git-upload-pack", None, &mut stdin)
.expect("withheld-shaped fetch should forward the pack and terminate");
let requests = server.join().unwrap();
assert_eq!(
requests.len(),
2,
"GET + one POST; the forwarded pack does not hang the helper"
);
}
/// U6 (matrix item 6): a fresh clone (wants, flush, done) issues exactly ONE POST.
#[test]
fn fresh_clone_issues_single_post() {
let want_sha = "a".repeat(40);
let (base_url, server) = serve_http(vec![
TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-upload-pack",
status: "200 OK",
body: "0000",
},
TestResponse {
request_line: "POST /zOwner/myrepo/git-upload-pack",
status: "200 OK",
body: "0008NAK\nPACKfake",
},
]);
let repo_base = format!("{base_url}/zOwner/myrepo");
let mut stdin_bytes = Vec::new();
stdin_bytes.extend_from_slice(&want_line(&want_sha));
stdin_bytes.extend_from_slice(b"0000");
stdin_bytes.extend_from_slice(&pkt("done\n"));
let mut stdin = io::Cursor::new(stdin_bytes);
handle_connect(&repo_base, "git-upload-pack", None, &mut stdin)
.expect("clone should complete");
let requests = server.join().unwrap();
assert_eq!(requests.len(), 2, "fresh clone is exactly one POST");
assert!(request_body(&requests[1])
.windows(want_sha.len())
.any(|w| w == want_sha.as_bytes()));
}
/// U6 (matrix item 6): an up-to-date / flush-only opener issues ZERO POSTs and
/// completes without entering an ACK wait.
#[test]
fn flush_only_opener_skips_post() {
let (base_url, server) = serve_http(vec![TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-upload-pack",
status: "200 OK",
body: "0000",
}]);
let repo_base = format!("{base_url}/zOwner/myrepo");
let mut stdin = io::Cursor::new(b"0000".to_vec()); // flush only, nothing to fetch
handle_connect(&repo_base, "git-upload-pack", None, &mut stdin)
.expect("up-to-date fetch should complete");
let requests = server.join().unwrap();
assert_eq!(requests.len(), 1, "flush-only opener issues no POST");
}
/// U6 (matrix item 6): a single-shot fetch git resolved in one round (wants,
/// flush, haves, done, no intermediate flush) issues exactly ONE POST. Must-not:
/// the fix must not split a single-shot negotiation into multiple POSTs.
#[test]
fn single_shot_fetch_is_one_post() {
let (base_url, server) = serve_http(vec![
TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-upload-pack",
status: "200 OK",
body: "0000",
},
TestResponse {
request_line: "POST /zOwner/myrepo/git-upload-pack",
status: "200 OK",
body: "0008NAK\nPACKfake",
},
]);
let repo_base = format!("{base_url}/zOwner/myrepo");
let mut stdin_bytes = Vec::new();
stdin_bytes.extend_from_slice(&want_line(&"a".repeat(40)));
stdin_bytes.extend_from_slice(b"0000");
stdin_bytes.extend_from_slice(&pkt(&format!("have {}\n", "1".repeat(40))));
stdin_bytes.extend_from_slice(&pkt(&format!("have {}\n", "2".repeat(40))));
stdin_bytes.extend_from_slice(&pkt("done\n")); // done with no intermediate flush
let mut stdin = io::Cursor::new(stdin_bytes);
handle_connect(&repo_base, "git-upload-pack", None, &mut stdin)
.expect("fetch should complete");
let requests = server.join().unwrap();
assert_eq!(
requests.len(),
2,
"a single-shot negotiation must be exactly one POST, not split"
);
}
/// Hardening: content-free flush pkts between the wants and the first haves do
/// not each fire a POST. Real git never sends a bare mid-negotiation flush, but
/// a malformed stream must not amplify into one signed POST per empty flush; the
/// loop skips them and POSTs only the round that carries haves + done.
#[test]
fn repeated_bare_flushes_do_not_amplify_posts() {
let (base_url, server) = serve_http(vec![
TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-upload-pack",
status: "200 OK",
body: "0000",
},
TestResponse {
request_line: "POST /zOwner/myrepo/git-upload-pack",
status: "200 OK",
body: "0008NAK\nPACKfake",
},
]);
let repo_base = format!("{base_url}/zOwner/myrepo");
let mut stdin_bytes = Vec::new();
stdin_bytes.extend_from_slice(&want_line(&"a".repeat(40)));
stdin_bytes.extend_from_slice(b"0000");
stdin_bytes.extend_from_slice(b"0000"); // bare flush, no haves
stdin_bytes.extend_from_slice(b"0000"); // bare flush, no haves
stdin_bytes.extend_from_slice(&pkt(&format!("have {}\n", "1".repeat(40))));
stdin_bytes.extend_from_slice(&pkt("done\n"));
let mut stdin = io::Cursor::new(stdin_bytes);
handle_connect(&repo_base, "git-upload-pack", None, &mut stdin)
.expect("fetch with stray flushes should complete");
let requests = server.join().unwrap();
assert_eq!(
requests.len(),
2,
"content-free flushes must not each produce a POST"
);
}
// ── Branch-coverage closure: exercise the remaining changed arms ─────────
/// A reader that fails with a non-EOF io error, to exercise the error
/// propagation arm of `read_upload_pack_round` (distinct from a clean EOF).
struct FailingReader;
impl Read for FailingReader {
fn read(&mut self, _buf: &mut [u8]) -> io::Result<usize> {
Err(io::Error::other("simulated read failure"))
}
}
/// G4: a non-EOF read error propagates; it is not swallowed as end-of-stream.
#[test]
fn read_upload_pack_round_propagates_read_error() {
let mut r = FailingReader;
assert!(read_upload_pack_round(&mut r).is_err());
}
/// G1: an invalid pkt-line length (1..=3) is rejected rather than underflowing
/// `pkt_len - 4`.
#[test]
fn read_upload_pack_round_rejects_invalid_pkt_length() {
let mut r = io::Cursor::new(b"0001".to_vec()); // pkt_len 1, < 4
let err = read_upload_pack_round(&mut r).unwrap_err();
assert!(
err.to_string().contains("invalid pkt-line length"),
"unexpected error: {err}"
);
}
/// #192 F5: a non-hex length header (e.g. "zzzz") is a malformed frame and must
/// be rejected, not silently collapsed to a 0000 flush.
#[test]
fn read_upload_pack_round_rejects_non_hex_length() {
let mut r = io::Cursor::new(b"zzzzdone\n".to_vec());
let err = read_upload_pack_round(&mut r).unwrap_err();
assert!(
err.to_string().to_lowercase().contains("pkt-line"),
"unexpected error: {err}"
);
}
/// #192 F5: a non-UTF-8 length header must be rejected, not treated as a flush.
#[test]
fn read_upload_pack_round_rejects_non_utf8_length() {
let mut r = io::Cursor::new(vec![0xffu8, 0xff, 0xff, 0xff]);
let err = read_upload_pack_round(&mut r).unwrap_err();
assert!(
err.to_string().to_lowercase().contains("pkt-line"),
"unexpected error: {err}"
);
}
/// G2: an immediately-empty upload-pack request (EOF at the opening round, with
/// a 200 advertisement) skips the POST entirely.
#[test]
fn upload_pack_empty_request_skips_post() {
let (base_url, server) = serve_http(vec![TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-upload-pack",
status: "200 OK",
body: "0000",
}]);
let repo_base = format!("{base_url}/zOwner/myrepo");
let mut stdin = io::Cursor::new(Vec::<u8>::new()); // immediate EOF
handle_connect(&repo_base, "git-upload-pack", None, &mut stdin)
.expect("empty upload-pack request should skip the POST");
let requests = server.join().unwrap();
assert_eq!(
requests.len(),
1,
"empty request must issue the GET only, no POST"
);
}
/// G3 (#192, F1): a nonempty have-batch terminated by EOF (no flush, no done)
/// means git ABORTED mid-negotiation. The helper must terminate WITHOUT POSTing
/// — synthesizing a `done` and sending it would make the node generate and
/// stream a full pack (signed, for a private fetch) to a client that has gone
/// away. Only a real `done` pkt finalizes the stateless request. RED before the
/// fix (the folded `Done | Eof` arm POSTs a synthetic done → 2 requests), GREEN
/// after (only the info/refs GET, no upload-pack POST).
#[test]
fn nonempty_eof_batch_aborts_without_posting() {
// Only the info/refs GET is served (serve_http accepts exactly N conns). The
// fixed helper makes exactly that one request and stops. The pre-fix code
// additionally POSTs the synthetic done; that POST hits the now-closed
// listener and errors, so handle_connect returns Err and the `.expect` below
// fires — the RED. The fixed code returns Ok with one recorded request.
let (base_url, server) = serve_http(vec![TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-upload-pack",
status: "200 OK",
body: "0000",
}]);
let repo_base = format!("{base_url}/zOwner/myrepo");
let mut stdin_bytes = Vec::new();
stdin_bytes.extend_from_slice(&want_line(&"a".repeat(40)));
stdin_bytes.extend_from_slice(b"0000");
stdin_bytes.extend_from_slice(&pkt(&format!("have {}\n", "1".repeat(40))));
// No trailing flush or done: the stream just ends (EOF) after the have —
// git aborted the negotiation.
let mut stdin = io::Cursor::new(stdin_bytes);
handle_connect(&repo_base, "git-upload-pack", None, &mut stdin)
.expect("an aborted (EOF) negotiation is a clean no-op, not an error");
let requests = server.join().unwrap();
assert_eq!(
requests.len(),
1,
"an aborted EOF must NOT trigger an upload-pack POST (only the info/refs GET)"
);
}
/// G5: an empty receive-pack (push) body skips the POST, unchanged from before.
#[test]
fn receive_pack_empty_body_skips_post() {
let (base_url, server) = serve_http(vec![TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-receive-pack",
status: "200 OK",
body: "0000",
}]);
let repo_base = format!("{base_url}/zOwner/myrepo");
let mut stdin = io::Cursor::new(Vec::<u8>::new()); // empty push
handle_connect(&repo_base, "git-receive-pack", None, &mut stdin)
.expect("empty receive-pack should skip the POST");
let requests = server.join().unwrap();
assert_eq!(
requests.len(),
1,
"empty push must issue the GET only, no POST"
);
}
}
File diff suppressed because it is too large. Load diff
+3
View File
@@ -77,3 +77,6 @@ libp2p-dns = { version = "0.44.0", features = ["tokio"] }
[dev-dependencies]
mockito = "1"
tempfile = "3"
# `test-util` for `#[tokio::test(start_paused = true)]`: lets the deadline tests assert
# on which timer fired instead of on wall clock.
tokio = { workspace = true, features = ["test-util"] }
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+16
View File
@@ -520,6 +520,22 @@ mod tests {
sync_trigger_rate_limiter: RateLimiter::new(60, Duration::from_secs(3600)),
peer_write_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
shutdown_tx: tokio::sync::watch::channel(false).0,
git_read_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_write_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_push_advert_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_encrypt_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
pin_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
encrypt_inflight: crate::state::EncryptInflight::new(),
repo_write_leases: crate::state::RepoWriteLeases::new(8),
git_read_per_caller: crate::rate_limit::PerCallerConcurrency::with_default_max_keys(16),
git_push_advert_per_caller:
crate::rate_limit::PerCallerConcurrency::with_default_max_keys(8),
git_write_per_caller: crate::rate_limit::PerCallerConcurrency::with_default_max_keys(8),
git_ipfs_walk_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_ipfs_walk_per_caller:
crate::rate_limit::PerCallerConcurrency::with_default_max_keys(16),
ipfs_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
git_bin: "git".to_string(),
}
}
+709 -8
View File
@@ -1,6 +1,33 @@
use clap::Parser;
use std::path::PathBuf;
/// Upper bound on `git_service_timeout_secs` and `ipfs_request_budget_secs`, in seconds
/// (100 years).
///
/// Two consumers now, so a future tightening moves both. `ipfs_request_budget_secs`
/// derives only the `Instant` addition in `get_by_cid`, not the lease-steal multiply
/// below, but it shares this ceiling because the defect class and the "set it very large
/// to disable" contract are the same.
///
/// The knob is not just stored, it is arithmetic input: the write path derives the
/// per-repo lease steal bound from it (`* 2 + 60`), and #174 routed it into
/// `build_filtered_pack` and `blob_paths`, which each build a deadline as
/// `Instant::now() + Duration::from_secs(this)`. That addition panics on overflow in
/// RELEASE as well as debug, so an unbounded `u64` here turns an operator typo into a
/// serve-path crash rather than a very long timeout. Bounding at parse time keeps every
/// derived duration in range at once, instead of hardening each site as it is found.
///
/// The ceiling is representability, NOT a policy view of a sane timeout, and that
/// distinction is what sets the number. The help text has always told operators to set
/// this very large to disable the bound, so values like `999999999` (~31 years) are
/// working production "off" settings; a tighter, tidier cap would fail those nodes at
/// boot on upgrade over a value that was never the defect. 100 years clears every such
/// setting while staying a factor of about 5.85 under the ~584-year ceiling of a
/// `u64`-nanosecond `Instant`, which is the tightest representation on any platform we
/// build for. Every value that worked before still parses; only the ones that would have
/// panicked are rejected.
pub const GIT_SERVICE_TIMEOUT_SECS_MAX: u64 = 100 * 365 * 24 * 60 * 60;
#[derive(Parser, Debug, Clone)]
#[command(name = "gitlawb-node", about = "gitlawb node daemon", version)]
pub struct Config {
@@ -170,26 +197,54 @@ pub struct Config {
/// Maximum wall-clock time a single served git operation (upload-pack /
/// receive-pack through `run_git_service`) may run before it is aborted and
/// its process group torn down, in seconds. Bounds a git that neither
/// finishes nor disconnects. Must be positive; set it very large to
/// effectively disable the bound. Default: 600s (10 min), generous for large
/// clones. Does not cover the ref advertisement (`info/refs`) or the
/// withheld-blob fetch path (`upload_pack_excluding`, a blocking
/// `spawn_blocking` a tokio timeout cannot cancel); both remain unbounded.
/// finishes nor disconnects. Must be positive and at most
/// [`GIT_SERVICE_TIMEOUT_SECS_MAX`] (100 years, the largest value every derived
/// deadline can represent); setting it very large is still the way to disable the
/// bound. Default: 600s (10 min), generous for large clones. Also bounds the ref
/// advertisement
/// (`info/refs`) and the withheld-blob pack build (`upload_pack_excluding`'s
/// pack-objects stage), which now share the same timeout + process-group
/// teardown (#174).
#[arg(
long,
env = "GITLAWB_GIT_SERVICE_TIMEOUT_SECS",
default_value_t = 600,
value_parser = clap::value_parser!(u64).range(1..)
value_parser = clap::value_parser!(u64).range(1..=GIT_SERVICE_TIMEOUT_SECS_MAX)
)]
pub git_service_timeout_secs: u64,
/// Maximum wall-clock time the storage-acquisition phase of a served git
/// operation may run before the request is shed with a 503, in seconds. This
/// bounds `RepoStore::{acquire,acquire_fresh,acquire_write}` — the Tigris
/// HEAD/GET on a read/advert acquire and the advisory-lock retry loop (incl. a
/// per-iteration `pg_try_advisory_lock` that can block on a hung Postgres pool)
/// on a write acquire. A concurrency permit is taken BEFORE this phase, and
/// `git_service_timeout_secs` only starts once git spawns, so without this the
/// acquire phase is unbounded: a stalled backend pins the permit and drains the
/// pool until every later request 503s. On expiry the permit is released and a
/// bounded 503 + Retry-After is returned (fail-closed). Kept separate from
/// `git_service_timeout_secs` because acquisition and git execution are distinct
/// cost centers — one shared budget would let a slow acquire starve git. Must be
/// positive; set it very large to effectively disable the bound. Default: 30s.
#[arg(
long,
env = "GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS",
default_value_t = 30,
value_parser = clap::value_parser!(u64).range(1..)
)]
pub git_acquire_timeout_secs: u64,
/// Maximum connections in the PostgreSQL pool. This is a cap, not a floor
/// (connections open lazily). Size against the database server's
/// max_connections, remembering admin tooling opens its own pool.
/// max_connections, remembering admin tooling opens its own pool. Each
/// concurrent write pins one pooled connection for its whole duration (the
/// advisory lock in `repo_store::acquire_write` is connection-affine), so this
/// must exceed `max_concurrent_git_pushes` by `DB_POOL_APP_HEADROOM` or slow
/// pushes starve every other DB path — enforced by `Config::validate`.
#[arg(
long,
env = "GITLAWB_DB_MAX_CONNECTIONS",
default_value_t = 20,
default_value_t = 48,
value_parser = clap::value_parser!(u32).range(1..)
)]
pub db_max_connections: u32,
@@ -234,6 +289,256 @@ pub struct Config {
value_parser = clap::value_parser!(u64).range(1..)
)]
pub db_retry_max_secs: u64,
/// Maximum number of served git operations (upload-pack / receive-pack /
/// info-refs) allowed to run concurrently. Beyond this the node sheds the
/// request with a clean 503 + Retry-After instead of spawning another git
/// subprocess and risking PID/thread exhaustion. Portable backstop: the
/// compose `pids_limit` is not present on Fly, whose connection-concurrency
/// cap is a different axis (500 connections each fan out to git +
/// pack-objects + threads). Size below the process budget with headroom.
///
/// This is the READ pool (`git_read_semaphore`): upload-pack and the UPLOAD-PACK
/// `info/refs` advertisement only. The authenticated push POST draws from a
/// separate write pool (`max_concurrent_git_pushes`) that anonymous reads can
/// never reach, and each read caller is additionally bounded by
/// `max_concurrent_reads_per_caller`, so an anonymous flood cannot shed the actual
/// push nor monopolize reads (#174). The anon-reachable RECEIVE-PACK `info/refs`
/// advertisement draws from its OWN dedicated pool (sized like the write pool but
/// disjoint), so an advertisement flood can never occupy a permit the
/// authenticated push POST needs at admission (#174).
///
/// A permit is held for the whole op. Every git subprocess that STREAMS is
/// duration-bounded and reaps its process group on disconnect: upload-pack,
/// receive-pack, and both info/refs advertisements run under
/// `git_service_timeout_secs` with `process_group(0)` teardown, and the
/// withheld-blob (`upload_pack_excluding`) pack-objects stage plus the push-side
/// candidate-discovery children (`rev-list` / `cat-file`) now run under the same
/// bounded runner with process-group teardown, so a stuck git child no longer
/// holds its slot indefinitely (#174 closed the duration/cancellation gaps this
/// comment previously tracked).
///
/// Default: 128. Must be between 1 and 1_048_576; the ceiling keeps the value
/// well under tokio's `Semaphore` permit limit so an oversized value is a
/// clean CLI error rather than a boot-time panic.
#[arg(
long,
env = "GITLAWB_MAX_CONCURRENT_GIT_OPS",
default_value_t = 128,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub max_concurrent_git_ops: usize,
/// Maximum number of concurrent `git-receive-pack` (push) operations. The
/// authenticated push POST draws from this dedicated pool, separate from
/// `max_concurrent_git_ops` (reads), so a flood of anonymous reads cannot shed an
/// authenticated push at admission (#174). The anon-reachable receive-pack
/// `info/refs` advertisement runs in a SEPARATE pool of the same size (derived
/// from this knob), disjoint from this one, so an advertisement flood cannot
/// occupy a POST's slot either (#174). Beyond this a push sheds a clean 503 +
/// Retry-After.
///
/// Default: 32. Must be between 1 and 1_048_576 (the ceiling keeps the value
/// under tokio's `Semaphore` permit limit so an oversized value is a clean CLI
/// error rather than a boot-time panic).
#[arg(
long,
env = "GITLAWB_MAX_CONCURRENT_GIT_PUSHES",
default_value_t = 32,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub max_concurrent_git_pushes: usize,
/// Maximum number of pushes that may be PARKED on one repo's in-process write lease
/// at once. Same-repo pushes are serialized (block-and-wait), and a parked push has
/// already had its entire pack buffered by axum, so an unbounded queue on a contended
/// repo is unbounded buffered memory held for up to `git_service_timeout_secs * 2 +
/// 60` (1260s at defaults). Past this cap the newest push sheds a clean 503 +
/// Retry-After instead of joining the queue.
///
/// The trade: raising it lets more same-repo pushes wait their turn (fewer 503s for a
/// hot repo, more memory pinned by waiters); lowering it sheds sooner. Only pushes to
/// the SAME repo count, and only ones parked right now, so the cap can never deny a
/// push to a different repo. The holder is deliberately not counted: a holder whose
/// cleanup never ran would otherwise pin a slot forever and wedge the repo, which is
/// the failure the `steal_after` reclaim exists to survive.
///
/// Default: 8, a quarter of the default `max_concurrent_git_pushes` (32). Raising the
/// push pool does not raise this; set it explicitly. Must be between 1 and 1_048_576.
#[arg(
long,
env = "GITLAWB_REPO_LEASE_MAX_WAITERS",
default_value_t = 8,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub repo_lease_max_waiters: usize,
/// Max concurrent post-push pin loops (`ipfs_pin` and `pinata`
/// `pin_new_objects`) across all repos. `EncryptInflight` bounds the outstanding
/// pin-task COUNT to one per repo, but each pin loop holds a full per-push
/// object-id list (up to `git_max_pack_bytes` worth of OIDs) while it walks it,
/// so N distinct repos could hold N such lists at once. This caps how many run
/// concurrently (#174 F6). Beyond it a pin loop DEFERS (waits) and never drops,
/// since a dropped pin would lose the object's replication copy.
///
/// It does not cap the memory itself: the local IPFS path builds its list before
/// taking a permit, so tasks parked on this pool still hold theirs, and how many
/// park is capped only per repo. Lowering this knob bounds concurrent pinning, not
/// how much an actor pushing to many repos can retain.
///
/// Default: 8. Must be between 1 and 1_048_576.
#[arg(
long,
env = "GITLAWB_MAX_CONCURRENT_PIN_TASKS",
default_value_t = 8,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub max_concurrent_pin_tasks: usize,
/// Maximum concurrent read operations (`upload-pack` and the upload-pack
/// `info/refs` advertisement) a single caller may hold at once, so one caller
/// cannot monopolize the `max_concurrent_git_ops` read pool (#174). Callers are
/// keyed on the RESOLVED SOURCE IP, never the DID — a signature does not move a
/// caller off this cap, so an authenticated client cannot mint DIDs to escape it.
/// IMPORTANT: the source-IP key is only as granular as `GITLAWB_TRUSTED_PROXY`.
/// Left unset (the default), a node behind an edge/NAT keys all callers on the
/// edge IP, so this cap collapses to a single global cap rather than per-client.
/// Set `GITLAWB_TRUSTED_PROXY` to key on the real client; a high-fanout caller (a
/// CI fleet behind one NAT) then needs the operator to raise this. Over-cap for a
/// caller sheds a clean 503 + Retry-After.
///
/// Default: 16. Must be between 1 and 1_048_576.
#[arg(
long,
env = "GITLAWB_MAX_CONCURRENT_READS_PER_CALLER",
default_value_t = 16,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub max_concurrent_reads_per_caller: usize,
/// Maximum number of concurrent `GET /ipfs/{cid}` requests that may run their
/// visibility walk at once. The publicly-reachable `/ipfs/{cid}` route runs
/// `allowed_blob_set_for_caller_bounded` in `spawn_blocking` — a full-history
/// git walk (up to `git_service_timeout_secs`) — for each candidate repo. It
/// draws from THIS pool, not any served-git pool: a distinct public cost center
/// on a distinct surface, so sharing a git pool would let anonymous /ipfs
/// traffic shed authenticated git ops (the auth-boundary trap). A permit is
/// held for the whole request (across the repo loop) so it reflects real
/// blocking-thread occupancy, not merely the tokio wait. Beyond this the request
/// sheds a clean 503 + Retry-After. Must be between 1 and 1_048_576; the ceiling
/// keeps the value under tokio's `Semaphore` permit limit so an oversized value
/// is a clean CLI error rather than a boot-time panic. Default: 32.
#[arg(
long,
env = "GITLAWB_MAX_CONCURRENT_IPFS_WALKS",
default_value_t = 32,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub max_concurrent_ipfs_walks: usize,
/// Maximum concurrent `/ipfs/{cid}` walk requests a single source may hold at
/// once, so one source cannot monopolize `max_concurrent_ipfs_walks` (#174).
/// Callers are keyed on the RESOLVED SOURCE IP (`client_key`/`GITLAWB_TRUSTED_PROXY`),
/// never the DID — `/ipfs` accepts any `did:key` via `optional_signature` with no
/// admission step, so keying on the DID would let one host mint disposable DIDs to
/// multiply its budget. A request with no resolvable key (no trusted header, no
/// peer) is bounded by the global pool only, never this sub-cap. Over-cap sheds a
/// clean 503 + Retry-After. Must be between 1 and 1_048_576. Default: 4.
#[arg(
long,
env = "GITLAWB_IPFS_WALK_PER_SOURCE",
default_value_t = 4,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub ipfs_walk_per_source: usize,
/// Upper bound on the number of EXPENSIVE visibility walks
/// (`allowed_blob_set_for_caller_bounded`, a full-history git walk in a
/// blocking thread) a single `/ipfs/{cid}` request may run. Only a blob in a
/// path-scoped repo costs a walk, so the cap counts exactly those candidates
/// — cheap probe-only visits are bounded by `ipfs_max_repo_visits` instead
/// (counting them here would starve a plain public copy past the cap out of
/// its 200). On exhaustion the walk-needing repo is skipped WITHOUT a verdict
/// and the scan continues; if the request then finds the object nowhere it
/// sheds a retryable 503 + Retry-After rather than misreport existing content
/// absent with a 404. The handler still short-circuits the moment it serves.
/// Must be between 1 and 1_048_576. Default: 64.
#[arg(
long,
env = "GITLAWB_IPFS_MAX_REPOS_WALKED",
default_value_t = 64,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub ipfs_max_repos_walked: usize,
/// Ceiling on the number of repos a single `/ipfs/{cid}` request may VISIT —
/// pass the repo-level visibility gate into the acquire + `cat-file` probe.
/// Each visit costs a `RepoStore::acquire` (on a Tigris cache miss that is a
/// full repo-archive download from object storage, so the worst-case
/// object-store fetch count for one request equals this ceiling) plus a git
/// probe subprocess. On exhaustion the scan STOPS — unlike
/// `ipfs_max_repos_walked`, which skips just the walk-needing repo, there is
/// no cheaper way to keep scanning — and the request sheds a retryable 503 +
/// Retry-After rather than a false 404. Must be between 1 and 1_048_576.
/// Default: 1024.
#[arg(
long,
env = "GITLAWB_IPFS_MAX_REPO_VISITS",
default_value_t = 1024,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub ipfs_max_repo_visits: usize,
/// Absolute wall-clock budget for one admitted `GET /ipfs/{cid}` request's
/// acquire+walk lifetime, in seconds. `max_concurrent_ipfs_walks` bounds how
/// MANY requests hold walk slots; this bounds how LONG one admitted request
/// may keep its slot. Without it, each repo iteration draws a fresh
/// `git_acquire_timeout_secs` and each expensive walk a fresh
/// `git_service_timeout_secs`, so one request scanning many repos could hold
/// a scarce walk slot for hours. Every stage (acquire, `cat-file` probe,
/// visibility walk, content read) starts only while budget remains, and the
/// acquire wait and walk deadline are clamped to `min(their own timeout,
/// remaining budget)`; a stage is never started with zero remaining. On
/// exhaustion the scan stops without a verdict and the request sheds a
/// retryable 503 + Retry-After rather than a false 404. The clamps bound
/// only the acquire and walk stages (overshoot there is the walk watchdog's
/// SIGTERM grace + SIGKILL settle); the `object_type` /
/// `read_object_content` probe subprocesses are budget-checked before they
/// start AND each run under their own deadline (the lesser of
/// `git_service_timeout_secs` and the remaining budget), reaped by
/// process-group teardown, so a hung `cat-file` cannot hold the request's walk
/// slot past it. Still unbounded: the probe's `object_store_readable` check is a
/// synchronous filesystem sweep with nothing to reap, so a wedged filesystem can
/// hold the slot past the deadline.
/// Must be positive, and no larger than `GIT_SERVICE_TIMEOUT_SECS_MAX`. The ceiling is
/// representability, NOT a policy view of a sane budget: `get_by_cid` derives the
/// request deadline as `Instant::now() + Duration::from_secs(this)`, and that addition
/// is an explicit overflow check rather than a debug-only one, so a value near the top
/// of the `u64` range aborts every `/ipfs/{cid}` request in a release build instead of
/// setting a very long budget. The ceiling sits well below where that starts, about a
/// factor of 5.85 (see the constant's own note), so it is a conservative margin rather than the
/// exact overflow point; rejecting at parse time keeps the unrepresentable values out
/// of every reachable configuration. Setting it very large is still the way to
/// effectively disable the budget, and the documented sentinels (`999999999`,
/// `1000000000`) are well inside the range.
/// Default: 600s (10 min), matching `git_service_timeout_secs` so a single full-length
/// walk still fits.
#[arg(
long,
env = "GITLAWB_IPFS_REQUEST_BUDGET_SECS",
default_value_t = 600,
value_parser = clap::value_parser!(u64).range(1..=GIT_SERVICE_TIMEOUT_SECS_MAX)
)]
pub ipfs_request_budget_secs: u64,
/// Per-client-IP rate limit for `GET /ipfs/{cid}`, in requests per hour. The
/// route is publicly reachable (`optional_signature`) and each request can drive
/// a full-history git walk, so it carries a per-IP flood brake in addition to the
/// concurrency cap above (a rate limit bounds request *rate*, the semaphore
/// bounds concurrent slow holds — different axes). Keyed on the resolved client
/// IP via `GITLAWB_TRUSTED_PROXY`. `0` disables. Default: 600.
#[arg(long, env = "GITLAWB_IPFS_RATE_LIMIT", default_value_t = 600)]
pub ipfs_rate_limit: usize,
}
impl Config {
@@ -250,6 +555,36 @@ impl Config {
}
PathBuf::from(&self.key_path)
}
/// DB connections reserved for everything other than held write-locks: auth
/// lookups, visibility-rule reads, the post-receive tail's own DB writes, and
/// admin tooling. A write pins one pooled connection for its whole duration, so
/// the pool must clear the concurrent-write cap by at least this margin.
pub const DB_POOL_APP_HEADROOM: u32 = 8;
/// Cross-field boot validation. Single-field ranges are enforced by clap; this
/// catches combinations that ship a denial-of-service under otherwise-valid
/// values. Call once at startup and fail fast on `Err`.
pub fn validate(&self) -> Result<(), String> {
// A write pins one pooled connection for its whole duration (the
// connection-affine advisory lock in repo_store::acquire_write), and
// concurrent writes are capped at max_concurrent_git_pushes. If the pool
// does not exceed that cap by DB_POOL_APP_HEADROOM, a burst of slow pushes
// drains every connection and every other DB path 503s. (#174 F1)
let floor = (self.max_concurrent_git_pushes as u64) + (Self::DB_POOL_APP_HEADROOM as u64);
if (self.db_max_connections as u64) < floor {
return Err(format!(
"GITLAWB_DB_MAX_CONNECTIONS ({}) must be at least max_concurrent_git_pushes ({}) \
+ {} headroom = {}: each concurrent write pins one pooled connection for its whole \
duration, so a smaller pool lets a burst of slow pushes starve every other DB path.",
self.db_max_connections,
self.max_concurrent_git_pushes,
Self::DB_POOL_APP_HEADROOM,
floor
));
}
Ok(())
}
}
#[cfg(test)]
@@ -272,4 +607,370 @@ mod tests {
Config::try_parse_from(["gitlawb-node", "--git-service-timeout-secs", "0"]).is_err()
);
}
/// #174 (RED-before/GREEN-after): the upper bound is what keeps every duration
/// derived from this knob in range — the lease steal bound's `* 2 + 60` on the write
/// path, and the `Instant::now() + Duration::from_secs(..)` deadlines in
/// `build_filtered_pack` / `blob_paths` on the serve path, which panic on overflow in
/// release builds too. Checked at parse time so no reachable configuration can carry a
/// value those sites cannot represent.
///
/// The pre-existing "set it very large to disable the bound" settings are asserted
/// alongside the rejections on purpose. The bound exists to exclude unrepresentable
/// values, not to impose a view of a reasonable timeout, so a node that has been
/// running on ~31 years must not start failing at boot on upgrade.
#[test]
fn git_service_timeout_rejects_values_no_derived_duration_can_represent() {
let parse = |secs: u64| {
Config::try_parse_from([
"gitlawb-node",
"--git-service-timeout-secs",
&secs.to_string(),
])
};
// Large "disable the bound" values that predate the ceiling still parse.
for disable in [1_000_000_000, 999_999_999] {
assert_eq!(
parse(disable)
.unwrap_or_else(|e| panic!("{disable} was a working setting: {e}"))
.git_service_timeout_secs,
disable
);
}
// At the ceiling: accepted, and every derived duration still fits.
let at_max =
parse(GIT_SERVICE_TIMEOUT_SECS_MAX).expect("the documented maximum must parse");
assert_eq!(
at_max.git_service_timeout_secs,
GIT_SERVICE_TIMEOUT_SECS_MAX
);
assert!(at_max
.git_service_timeout_secs
.checked_mul(2)
.and_then(|v| v.checked_add(60))
.is_some());
assert!(std::time::Instant::now()
.checked_add(std::time::Duration::from_secs(
at_max.git_service_timeout_secs
))
.is_some());
// Past the ceiling, and the top of the u64 range clap used to accept — the value
// that panics `Instant::now() + Duration::from_secs(..)` outright.
for over in [GIT_SERVICE_TIMEOUT_SECS_MAX + 1, u64::MAX] {
assert!(
parse(over).is_err(),
"{over} is past the representable ceiling and must be rejected at parse time"
);
}
assert!(std::time::Instant::now()
.checked_add(std::time::Duration::from_secs(u64::MAX))
.is_none());
}
#[test]
fn max_concurrent_pin_tasks_defaults_and_rejects_out_of_range() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).max_concurrent_pin_tasks,
8
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-pin-tasks", "2"])
.max_concurrent_pin_tasks,
2
);
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-pin-tasks", "0"]).is_err()
);
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-pin-tasks", "1048577"])
.is_err()
);
}
#[test]
fn max_concurrent_git_ops_defaults_and_rejects_out_of_range() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).max_concurrent_git_ops,
128
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-git-ops", "8"])
.max_concurrent_git_ops,
8
);
// 0 permits would shed every served-git request with a 503; clap must reject it.
assert!(Config::try_parse_from(["gitlawb-node", "--max-concurrent-git-ops", "0"]).is_err());
// Above the ceiling would panic tokio's Semaphore::new at boot (permits >
// usize::MAX >> 3); clap must reject it as a clean CLI error instead.
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-git-ops", "1048577"])
.is_err()
);
// The ceiling itself is accepted.
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-git-ops", "1048576"])
.max_concurrent_git_ops,
1_048_576
);
}
#[test]
fn max_concurrent_git_pushes_defaults_and_rejects_out_of_range() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).max_concurrent_git_pushes,
32
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-git-pushes", "8"])
.max_concurrent_git_pushes,
8
);
// 0 permits would shed every push with a 503; clap must reject it.
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-git-pushes", "0"]).is_err()
);
// Above the ceiling would panic tokio's Semaphore::new at boot; clap rejects it.
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-git-pushes", "1048577"])
.is_err()
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-git-pushes", "1048576"])
.max_concurrent_git_pushes,
1_048_576
);
}
#[test]
fn max_concurrent_ipfs_walks_defaults_and_rejects_out_of_range() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).max_concurrent_ipfs_walks,
32
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-ipfs-walks", "4"])
.max_concurrent_ipfs_walks,
4
);
// 0 permits would shed every /ipfs walk with a 503; clap must reject it.
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-ipfs-walks", "0"]).is_err()
);
// Above the ceiling would panic tokio's Semaphore::new at boot; clap rejects it.
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-ipfs-walks", "1048577"])
.is_err()
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-ipfs-walks", "1048576"])
.max_concurrent_ipfs_walks,
1_048_576
);
}
#[test]
fn ipfs_walk_per_source_defaults_and_rejects_out_of_range() {
assert_eq!(Config::parse_from(["gitlawb-node"]).ipfs_walk_per_source, 4);
assert_eq!(
Config::parse_from(["gitlawb-node", "--ipfs-walk-per-source", "2"])
.ipfs_walk_per_source,
2
);
// 0 would shed every /ipfs walk from a keyed source; clap must reject it.
assert!(Config::try_parse_from(["gitlawb-node", "--ipfs-walk-per-source", "0"]).is_err());
assert!(
Config::try_parse_from(["gitlawb-node", "--ipfs-walk-per-source", "1048577"]).is_err()
);
}
#[test]
fn ipfs_max_repos_walked_defaults_and_rejects_out_of_range() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).ipfs_max_repos_walked,
64
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--ipfs-max-repos-walked", "8"])
.ipfs_max_repos_walked,
8
);
// 0 would walk no repos (serve nothing); clap must reject it.
assert!(Config::try_parse_from(["gitlawb-node", "--ipfs-max-repos-walked", "0"]).is_err());
assert!(
Config::try_parse_from(["gitlawb-node", "--ipfs-max-repos-walked", "1048577"]).is_err()
);
}
#[test]
fn ipfs_max_repo_visits_defaults_and_rejects_out_of_range() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).ipfs_max_repo_visits,
1024
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--ipfs-max-repo-visits", "8"])
.ipfs_max_repo_visits,
8
);
// 0 would visit no repos (serve nothing); clap must reject it.
assert!(Config::try_parse_from(["gitlawb-node", "--ipfs-max-repo-visits", "0"]).is_err());
assert!(
Config::try_parse_from(["gitlawb-node", "--ipfs-max-repo-visits", "1048577"]).is_err()
);
}
#[test]
fn ipfs_request_budget_secs_defaults_to_600_and_rejects_zero() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).ipfs_request_budget_secs,
600
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--ipfs-request-budget-secs", "30"])
.ipfs_request_budget_secs,
30
);
// 0 would expire every /ipfs request at its first stage (unconditional
// 503); clap must reject it.
assert!(
Config::try_parse_from(["gitlawb-node", "--ipfs-request-budget-secs", "0"]).is_err()
);
}
/// #174 (RED-before/GREEN-after): the upper bound is what keeps the deadline derived
/// from this knob in range. `get_by_cid` builds the request budget as
/// `Instant::now() + Duration::from_secs(this)` (api/ipfs.rs), and that addition is an
/// explicit overflow check rather than a debug-only one, so an oversized value aborts
/// every `/ipfs/{cid}` request in a release build instead of setting a very long budget.
/// The route is anon-reachable, so the failure is operator-triggered but publicly felt.
/// Checked at parse time so no reachable configuration can carry a value the deadline
/// cannot represent.
///
/// The large "disable the bound" settings are asserted alongside the rejections on
/// purpose, the same way the `git_service_timeout_secs` sibling does it. The ceiling
/// exists to exclude unrepresentable values, not to impose a view of a reasonable
/// budget, so a node already running on such a value must not start failing at boot on
/// upgrade. A test that only checked the boundary would pass with a far tighter cap.
#[test]
fn ipfs_request_budget_rejects_values_no_derived_duration_can_represent() {
let parse = |secs: u64| {
Config::try_parse_from([
"gitlawb-node",
"--ipfs-request-budget-secs",
&secs.to_string(),
])
};
// Large "disable the bound" values that predate the ceiling still parse.
for disable in [1_000_000_000, 999_999_999] {
assert_eq!(
parse(disable)
.unwrap_or_else(|e| panic!("{disable} was a working setting: {e}"))
.ipfs_request_budget_secs,
disable
);
}
// At the ceiling: accepted, and the derived deadline still fits. This knob feeds
// only the `Instant` addition (no multiply derivation like the lease steal bound),
// so there is no `checked_mul` clause to carry over from the sibling test.
let at_max =
parse(GIT_SERVICE_TIMEOUT_SECS_MAX).expect("the documented maximum must parse");
assert_eq!(
at_max.ipfs_request_budget_secs,
GIT_SERVICE_TIMEOUT_SECS_MAX
);
assert!(std::time::Instant::now()
.checked_add(std::time::Duration::from_secs(
at_max.ipfs_request_budget_secs
))
.is_some());
// Past the ceiling, and the top of the u64 range clap used to accept. Only the
// latter actually panics `Instant::now() + Duration::from_secs(..)`; the ceiling
// sits well below that, which is the conservative margin the constant documents.
for over in [GIT_SERVICE_TIMEOUT_SECS_MAX + 1, u64::MAX] {
assert!(
parse(over).is_err(),
"{over} is past the representable ceiling and must be rejected at parse time"
);
}
assert!(std::time::Instant::now()
.checked_add(std::time::Duration::from_secs(u64::MAX))
.is_none());
}
#[test]
fn max_concurrent_reads_per_caller_defaults_and_rejects_out_of_range() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).max_concurrent_reads_per_caller,
16
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-reads-per-caller", "4"])
.max_concurrent_reads_per_caller,
4
);
// 0 would shed every read from a keyed caller; clap must reject it.
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-reads-per-caller", "0"])
.is_err()
);
assert!(Config::try_parse_from([
"gitlawb-node",
"--max-concurrent-reads-per-caller",
"1048577"
])
.is_err());
assert_eq!(
Config::parse_from([
"gitlawb-node",
"--max-concurrent-reads-per-caller",
"1048576"
])
.max_concurrent_reads_per_caller,
1_048_576
);
}
/// #174 F1: a connection-affine write lock pins a pooled connection per
/// concurrent write, so the pool must clear `max_concurrent_git_pushes` by
/// `DB_POOL_APP_HEADROOM` or a push burst starves every other DB path.
/// `validate()` must reject an under-sized pool at boot.
#[test]
fn db_pool_must_clear_the_git_push_cap() {
// Shipped defaults validate (48 >= 32 + 8).
Config::parse_from(["gitlawb-node"])
.validate()
.expect("default config must validate");
// An under-sized pool relative to the push cap is rejected (20 < 32 + 8).
let under = Config::parse_from([
"gitlawb-node",
"--db-max-connections",
"20",
"--max-concurrent-git-pushes",
"32",
]);
assert!(
under.validate().is_err(),
"db_max_connections 20 below max_concurrent_git_pushes 32 + headroom must be rejected"
);
// Exactly at the floor validates (40 == 32 + 8).
let at_floor = Config::parse_from([
"gitlawb-node",
"--db-max-connections",
"40",
"--max-concurrent-git-pushes",
"32",
]);
assert!(
at_floor.validate().is_ok(),
"db_max_connections at the floor (pushes + headroom) must validate"
);
}
}
+1 -1
View File
@@ -170,7 +170,7 @@ pub async fn encrypt_and_pin(
continue;
}
};
let cid = match crate::ipfs_pin::pin_git_object(ipfs_api, oid, &envelope).await {
let cid = match crate::ipfs_pin::pin_git_object(ipfs_api, oid, &envelope, None).await {
Ok(c) if !c.is_empty() => c,
Ok(_) => {
tracing::warn!(oid = %oid, "pin_git_object returned no cid; skipping");
+30 -1
View File
@@ -47,6 +47,9 @@ pub enum AppError {
#[error("git service timed out: {0}")]
Timeout(String),
#[error("server overloaded: {0}")]
Overloaded(String),
#[error("database error: {0}")]
Db(#[from] sqlx::Error),
@@ -147,6 +150,12 @@ impl IntoResponse for AppError {
DB_UNAVAILABLE_CODE,
DB_UNAVAILABLE_MESSAGE.into(),
),
// 503 with a Retry-After (attached after this match — the shared tail
// can't carry per-variant headers). This is the single place Overloaded
// becomes a response, so it can never ship a 503 without the retry hint.
AppError::Overloaded(msg) => {
(StatusCode::SERVICE_UNAVAILABLE, "overloaded", msg.clone())
}
AppError::Db(e) => (StatusCode::INTERNAL_SERVER_ERROR, "db_error", e.to_string()),
AppError::Internal(e) => (
StatusCode::INTERNAL_SERVER_ERROR,
@@ -160,7 +169,17 @@ impl IntoResponse for AppError {
"message": message,
}));
(status, body).into_response()
let mut resp = (status, body).into_response();
// Overloaded advertises when to retry. It rides the shared tail above for
// its body/status, so the header is attached here rather than in a bespoke
// early return — keeping the variant handled in exactly one place.
if matches!(self, AppError::Overloaded(_)) {
resp.headers_mut().insert(
axum::http::header::RETRY_AFTER,
axum::http::HeaderValue::from_static("1"),
);
}
resp
}
}
@@ -182,4 +201,14 @@ mod tests {
StatusCode::INTERNAL_SERVER_ERROR
);
}
#[test]
fn overloaded_maps_to_503_with_retry_after() {
let resp = AppError::Overloaded("x".into()).into_response();
assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE);
assert_eq!(
resp.headers().get("retry-after").unwrap().to_str().unwrap(),
"1"
);
}
}
+458 -96
View File
@@ -29,9 +29,9 @@
use std::collections::HashSet;
use std::path::{Path, PathBuf};
use std::process::Command;
use std::time::{Duration, Instant};
use anyhow::{bail, Context, Result};
use anyhow::Result;
/// Env var that forces the push path to always full-scan, bypassing the delta
/// optimization (KTD7 kill-switch). Reuses the already-tested fallback branch
@@ -64,11 +64,21 @@ pub enum PinCandidates {
/// Fail-closed: any condition where the introduced set cannot be safely
/// determined returns [`PinCandidates::FullScanRequired`] rather than a partial
/// set, so the caller full-scans instead of silently under-pinning.
pub fn resolve_push_delta(repo_path: &Path, new_tips: &[&str], old_tips: &[&str]) -> PinCandidates {
// KTD7 kill-switch: force the (already-tested) full-scan fallback. The env
// read is split out from the pure logic so the resolver stays unit-testable
// without touching process-global state.
resolve_push_delta_inner(repo_path, new_tips, old_tips, force_full_scan())
pub fn resolve_push_delta(
repo_path: &Path,
new_tips: &[&str],
old_tips: &[&str],
git_bin: &str,
deadline: Instant,
) -> PinCandidates {
resolve_push_delta_inner(
repo_path,
new_tips,
old_tips,
force_full_scan(),
git_bin,
deadline,
)
}
/// Whether the force-full-scan kill-switch env var is set.
@@ -83,6 +93,8 @@ fn resolve_push_delta_inner(
new_tips: &[&str],
old_tips: &[&str],
force_full_scan: bool,
git_bin: &str,
deadline: Instant,
) -> PinCandidates {
if force_full_scan {
tracing::debug!("{FORCE_FULL_SCAN_ENV} set — forcing full scan");
@@ -102,7 +114,7 @@ fn resolve_push_delta_inner(
// commit). Bare `cat-file -t` returns `tag` for an annotated tag, and
// `for-each-ref %(*objecttype)` peels only one level — neither is correct.
for tip in new_tips {
match peeled_object_type(repo_path, tip) {
match peeled_object_type(repo_path, tip, git_bin, deadline) {
Some(t) if t == "commit" => {}
other => {
tracing::debug!(
@@ -115,7 +127,7 @@ fn resolve_push_delta_inner(
}
}
match rev_list_delta(repo_path, new_tips, old_tips) {
match rev_list_delta(repo_path, new_tips, old_tips, git_bin, deadline) {
Ok(oids) => PinCandidates::Delta(oids),
Err(e) => {
tracing::debug!(err = %e, "push-delta rev-list failed — forcing full scan");
@@ -126,41 +138,43 @@ fn resolve_push_delta_inner(
/// Return the fully-peeled object type of `sha` (e.g. `commit`, `tree`,
/// `blob`), or `None` if the object is missing/unpeelable or git errored.
fn peeled_object_type(repo_path: &Path, sha: &str) -> Option<String> {
let output = Command::new("git")
.args(["cat-file", "-t", &format!("{sha}^{{}}")])
.current_dir(repo_path)
.output()
.ok()?;
if !output.status.success() {
return None;
}
Some(String::from_utf8_lossy(&output.stdout).trim().to_string())
fn peeled_object_type(
repo_path: &Path,
sha: &str,
git_bin: &str,
deadline: Instant,
) -> Option<String> {
let peel = format!("{sha}^{{}}");
let out = crate::git::visibility_pack::run_bounded_git(
git_bin,
&["cat-file", "-t", &peel],
repo_path,
b"",
deadline,
)
.ok()?;
Some(String::from_utf8_lossy(&out).trim().to_string())
}
/// Run `git rev-list --objects --no-object-names <new> --not <old>` and return
/// the bare OID set. Decides on `status.success()` *before* parsing stdout, so
/// a walk that prints a valid prefix then errors mid-walk is discarded.
fn rev_list_delta(repo_path: &Path, new_tips: &[&str], old_tips: &[&str]) -> Result<Vec<String>> {
fn rev_list_delta(
repo_path: &Path,
new_tips: &[&str],
old_tips: &[&str],
git_bin: &str,
deadline: Instant,
) -> Result<Vec<String>> {
let mut args: Vec<&str> = vec!["rev-list", "--objects", "--no-object-names"];
args.extend_from_slice(new_tips);
if !old_tips.is_empty() {
args.push("--not");
args.extend_from_slice(old_tips);
}
let output = Command::new("git")
.args(&args)
.current_dir(repo_path)
.output()
.context("failed to run git rev-list for push delta")?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
bail!("git rev-list failed: {stderr}");
}
let stdout = String::from_utf8_lossy(&output.stdout);
let out =
crate::git::visibility_pack::run_bounded_git(git_bin, &args, repo_path, b"", deadline)?;
let stdout = String::from_utf8_lossy(&out);
Ok(stdout
.lines()
.map(|l| l.trim().to_string())
@@ -175,23 +189,19 @@ fn rev_list_delta(repo_path: &Path, new_tips: &[&str], old_tips: &[&str]) -> Res
/// reconciliation sweep relies on. It returns *all* objects (including
/// unreachable/dangling ones), which is what the sweep needs to catch
/// stragglers — do not swap it for a reachability walk.
pub fn list_all_objects(repo_path: &Path) -> Result<Vec<String>> {
let output = Command::new("git")
.args([
pub fn list_all_objects(repo_path: &Path, git_bin: &str, deadline: Instant) -> Result<Vec<String>> {
let out = crate::git::visibility_pack::run_bounded_git(
git_bin,
&[
"cat-file",
"--batch-all-objects",
"--batch-check=%(objectname)",
])
.current_dir(repo_path)
.output()
.context("failed to run git cat-file")?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
bail!("git cat-file failed: {stderr}");
}
let stdout = String::from_utf8_lossy(&output.stdout);
],
repo_path,
b"",
deadline,
)?;
let stdout = String::from_utf8_lossy(&out);
Ok(stdout
.lines()
.map(|l| l.trim().to_string())
@@ -203,23 +213,23 @@ pub fn list_all_objects(repo_path: &Path) -> Result<Vec<String>> {
/// `--batch-check='%(objectname) %(objecttype)'`. The pin path's fail-closed
/// filter needs to tell blobs (content, withholdable) from commits/trees
/// (structural, never withheld) without typing the candidate list itself.
pub fn list_all_objects_with_type(repo_path: &Path) -> Result<Vec<(String, String)>> {
let output = Command::new("git")
.args([
pub fn list_all_objects_with_type(
repo_path: &Path,
git_bin: &str,
deadline: Instant,
) -> Result<Vec<(String, String)>> {
let out = crate::git::visibility_pack::run_bounded_git(
git_bin,
&[
"cat-file",
"--batch-all-objects",
"--batch-check=%(objectname) %(objecttype)",
])
.current_dir(repo_path)
.output()
.context("failed to run git cat-file")?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
bail!("git cat-file failed: {stderr}");
}
let stdout = String::from_utf8_lossy(&output.stdout);
],
repo_path,
b"",
deadline,
)?;
let stdout = String::from_utf8_lossy(&out);
Ok(stdout
.lines()
.filter_map(|l| {
@@ -235,8 +245,12 @@ pub fn list_all_objects_with_type(repo_path: &Path) -> Result<Vec<(String, Strin
/// fail-closed pin filter drops any candidate blob absent from the reachable,
/// visibility-allowed set; a dangling private blob is in this set but not the
/// allowed set, so it never replicates (#99).
pub fn all_blob_oids(repo_path: &Path) -> Result<HashSet<String>> {
Ok(list_all_objects_with_type(repo_path)?
pub fn all_blob_oids(
repo_path: &Path,
git_bin: &str,
deadline: Instant,
) -> Result<HashSet<String>> {
Ok(list_all_objects_with_type(repo_path, git_bin, deadline)?
.into_iter()
.filter(|(_, ty)| ty == "blob")
.map(|(oid, _)| oid)
@@ -268,22 +282,68 @@ pub struct PinCandidateSet {
/// it can never leak because the withheld/fail-closed filter still runs on
/// whatever set is returned. `full_scan` rides on the returned set so the caller
/// knows when the dangling-inclusive filter is required.
///
/// `scan_sem` is the post-receive scan admission pool (`git_encrypt_semaphore`,
/// #174 F4): both git-spawning stages — the per-tip `cat-file` probe + delta
/// rev-list, and the full-scan fallback — run under ONE permit held for the
/// whole blocking scan. The deletion-only fast path (no new tips) spawns no git
/// and never parks.
///
/// `force_full_scan` forces the full-scan fallback regardless of the tips — the
/// coalesced-drain `PendingWork::FullScan` marker (#174 F5). It composes with
/// the env kill-switch (either forces), and it disqualifies the empty-tips fast
/// path: a forced-scan call with no tips must still enumerate the repo, never
/// silently resolve to an empty delta (which would pin nothing and lose the
/// drained pushes' work — the F5 bug shape).
pub async fn resolve_candidates_for_push(
scan_sem: std::sync::Arc<tokio::sync::Semaphore>,
repo_path: PathBuf,
new_tips: Vec<String>,
old_tips: Vec<String>,
git_bin: String,
timeout: Duration,
force_full_scan: bool,
) -> PinCandidateSet {
let force = force_full_scan || self::force_full_scan();
// No-git fast path: a deletion-only push resolves to an empty delta without
// spawning any git child, so it must not park on the scan pool. A forced
// full scan (caller flag or kill-switch) disqualifies it — the scan spawns git.
if new_tips.is_empty() && !force {
tracing::info!(delta = 0usize, repo = %repo_path.display(), "pin candidate set from push delta");
return PinCandidateSet {
candidates: Vec::new(),
full_scan: false,
};
}
// Scan admission (#174 F4): DEFER, never shed — a dropped scan would
// silently under-pin this push. The permit moves into the blocking closure
// so a started scan always completes holding it. Residuals at
// `acquire_scan_permit`.
let permit =
crate::state::acquire_scan_permit(scan_sem, &repo_path, "pin-candidate scan").await;
tokio::task::spawn_blocking(move || {
let _permit = permit;
// ONE shared deadline for the whole scan, per jatmn ("the same deadline").
let deadline = Instant::now() + timeout;
let new_refs: Vec<&str> = new_tips.iter().map(String::as_str).collect();
let old_refs: Vec<&str> = old_tips.iter().map(String::as_str).collect();
match resolve_push_delta(&repo_path, &new_refs, &old_refs) {
// `force` already folds in the env kill-switch, so the forced arm skips the
// delta machinery outright; the unforced arm goes through the normal
// resolver (whose own env read is false here by construction).
let resolved = if force {
tracing::debug!("full scan forced (coalesced-drain marker or kill-switch)");
PinCandidates::FullScanRequired
} else {
resolve_push_delta(&repo_path, &new_refs, &old_refs, &git_bin, deadline)
};
match resolved {
PinCandidates::Delta(objs) => {
tracing::info!(delta = objs.len(), repo = %repo_path.display(), "pin candidate set from push delta");
PinCandidateSet { candidates: objs, full_scan: false }
}
PinCandidates::FullScanRequired => {
tracing::warn!(repo = %repo_path.display(), "pin delta unavailable (non-commit tip, git error, or force-full-scan) — full-scan fallback");
match list_all_objects(&repo_path) {
match list_all_objects(&repo_path, &git_bin, deadline) {
Ok(objs) => PinCandidateSet { candidates: objs, full_scan: true },
Err(e) => {
tracing::warn!(repo = %repo_path.display(), err = %e, "full-scan fallback failed; pinning nothing this push (reconciliation sweep backstops)");
@@ -304,8 +364,13 @@ pub async fn resolve_candidates_for_push(
mod tests {
use super::*;
use std::collections::HashSet;
use std::process::Command;
use tempfile::TempDir;
fn td() -> std::time::Instant {
std::time::Instant::now() + std::time::Duration::from_secs(600)
}
/// Minimal git helper for building test repos.
struct Repo {
_td: TempDir,
@@ -363,9 +428,10 @@ mod tests {
let repo = Repo::new();
let c1 = repo.commit_file("a.txt", "one\n");
let c2 = repo.commit_file("b.txt", "two\n");
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&c2], &[&c1]))
.into_iter()
.collect();
let got: HashSet<String> =
delta(resolve_push_delta(&repo.path, &[&c2], &[&c1], "git", td()))
.into_iter()
.collect();
// The new blob b.txt and commit c2 are in the delta; the old blob a.txt
// and commit c1 are not.
let new_blob = repo.rev("HEAD:b.txt");
@@ -383,7 +449,7 @@ mod tests {
// genuinely new objects, never fewer.
let repo = Repo::new();
let c1 = repo.commit_file("a.txt", "one\n");
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&c1], &[]))
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&c1], &[], "git", td()))
.into_iter()
.collect();
assert!(got.contains(&c1));
@@ -399,9 +465,15 @@ mod tests {
// Rewrite history: reset to base, commit a different file.
repo.git(&["reset", "-q", "--hard", &base]);
let new_tip = repo.commit_file("c.txt", "three\n");
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&new_tip], &[&old_tip]))
.into_iter()
.collect();
let got: HashSet<String> = delta(resolve_push_delta(
&repo.path,
&[&new_tip],
&[&old_tip],
"git",
td(),
))
.into_iter()
.collect();
assert!(got.contains(&new_tip), "new tip in delta");
assert!(got.contains(&repo.rev(&format!("{new_tip}:c.txt"))));
// No error; force-push computes new-minus-old cleanly.
@@ -413,7 +485,7 @@ mod tests {
repo.commit_file("a.txt", "one\n");
// All updates were deletions => new_tips empty after the caller strips zeros.
assert_eq!(
resolve_push_delta(&repo.path, &[], &[ZERO]),
resolve_push_delta(&repo.path, &[], &[ZERO], "git", td()),
PinCandidates::Delta(Vec::new())
);
}
@@ -424,7 +496,7 @@ mod tests {
repo.commit_file("a.txt", "one\n");
let blob = repo.rev("HEAD:a.txt");
assert_eq!(
resolve_push_delta(&repo.path, &[&blob], &[]),
resolve_push_delta(&repo.path, &[&blob], &[], "git", td()),
PinCandidates::FullScanRequired,
"a blob tip must force full scan (rev-list would exit 0 and walk it)"
);
@@ -436,7 +508,7 @@ mod tests {
repo.commit_file("a.txt", "one\n");
let tree = repo.rev("HEAD^{tree}");
assert_eq!(
resolve_push_delta(&repo.path, &[&tree], &[]),
resolve_push_delta(&repo.path, &[&tree], &[], "git", td()),
PinCandidates::FullScanRequired
);
}
@@ -449,7 +521,7 @@ mod tests {
repo.git(&["tag", "-a", "treetag", "-m", "x", &tree]);
let tag = repo.rev("treetag");
assert_eq!(
resolve_push_delta(&repo.path, &[&tag], &[]),
resolve_push_delta(&repo.path, &[&tag], &[], "git", td()),
PinCandidates::FullScanRequired,
"annotated tag peeling to a tree must force full scan"
);
@@ -465,7 +537,7 @@ mod tests {
repo.git(&["tag", "-a", "t2", "-m", "x", &t1]);
let t2 = repo.rev("t2");
assert_eq!(
resolve_push_delta(&repo.path, &[&t2], &[]),
resolve_push_delta(&repo.path, &[&t2], &[], "git", td()),
PinCandidates::FullScanRequired
);
}
@@ -480,7 +552,7 @@ mod tests {
let c1 = repo.commit_file("a.txt", "one\n");
repo.git(&["tag", "-a", "rel", "-m", "release", &c1]);
let tag = repo.rev("rel");
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&tag], &[]))
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&tag], &[], "git", td()))
.into_iter()
.collect();
assert!(
@@ -501,7 +573,7 @@ mod tests {
let t1 = repo.rev("t1");
repo.git(&["tag", "-a", "t2", "-m", "x", &t1]);
let t2 = repo.rev("t2");
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&t2], &[]))
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&t2], &[], "git", td()))
.into_iter()
.collect();
assert!(
@@ -516,9 +588,16 @@ mod tests {
let repo = Repo::new();
let c1 = repo.commit_file("a.txt", "one\n");
let c2 = repo.commit_file("b.txt", "two\n");
let set =
resolve_candidates_for_push(repo.path.clone(), vec![c2.clone()], vec![c1.clone()])
.await;
let set = resolve_candidates_for_push(
std::sync::Arc::new(tokio::sync::Semaphore::new(64)),
repo.path.clone(),
vec![c2.clone()],
vec![c1.clone()],
"git".to_string(),
std::time::Duration::from_secs(600),
false,
)
.await;
assert!(!set.full_scan, "happy-path delta is not a full scan");
let got: HashSet<String> = set.candidates.into_iter().collect();
let new_blob = repo.rev("HEAD:b.txt");
@@ -536,20 +615,206 @@ mod tests {
let repo = Repo::new();
repo.commit_file("a.txt", "one\n");
let blob = repo.rev("HEAD:a.txt");
let all: HashSet<String> = list_all_objects(&repo.path).unwrap().into_iter().collect();
let set = resolve_candidates_for_push(repo.path.clone(), vec![blob], vec![]).await;
let all: HashSet<String> = list_all_objects(&repo.path, "git", td())
.unwrap()
.into_iter()
.collect();
let set = resolve_candidates_for_push(
std::sync::Arc::new(tokio::sync::Semaphore::new(64)),
repo.path.clone(),
vec![blob],
vec![],
"git".to_string(),
std::time::Duration::from_secs(600),
false,
)
.await;
assert!(set.full_scan, "non-commit tip is signalled as a full scan");
let got: HashSet<String> = set.candidates.into_iter().collect();
assert_eq!(got, all, "non-commit tip falls back to full repo scan");
}
/// F4 defer proof 2: `resolve_candidates_for_push`'s git stages (the per-tip
/// cat-file type probe + delta rev-list, and the full-scan fallback) run under a
/// scan-admission permit: with a zero-permit pool the call parks and spawns no
/// git; once a permit is available the SAME call runs (defer, not shed). On
/// ungated code the git runs regardless of the pool (RED).
#[cfg(unix)]
#[tokio::test]
async fn resolve_candidates_defers_when_scan_pool_exhausted() {
use std::os::unix::fs::PermissionsExt;
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::Semaphore;
let dir = tempfile::TempDir::new().unwrap();
let marker = dir.path().join("git.ran");
// Fake git records ANY invocation; cat-file reports a commit tip so the
// delta stage proceeds, rev-list yields an empty delta.
let fake = dir.path().join("fakegit");
std::fs::write(
&fake,
format!(
"#!/bin/sh\necho ran >> \"{}\"\ncase \"$1\" in\n cat-file) echo commit ;;\n *) : ;;\nesac\nexit 0\n",
marker.display()
),
)
.unwrap();
let mut perm = std::fs::metadata(&fake).unwrap().permissions();
perm.set_mode(0o755);
std::fs::set_permissions(&fake, perm).unwrap();
let git_bin = fake.to_str().unwrap().to_string();
let tip = "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef".to_string();
let sem: Arc<Semaphore> = Arc::new(Semaphore::new(0));
let blocked = tokio::time::timeout(
Duration::from_millis(500),
resolve_candidates_for_push(
sem.clone(),
dir.path().to_path_buf(),
vec![tip.clone()],
vec![],
git_bin.clone(),
Duration::from_secs(5),
false,
),
)
.await;
assert!(
blocked.is_err(),
"the pin-candidate scan must defer (park on admission) when the pool is exhausted"
);
assert!(
!marker.exists(),
"the scan's git must not spawn while its admission permit is unavailable (F4)"
);
// Release admission: the SAME scan now runs (defer, not shed).
sem.add_permits(1);
let set = resolve_candidates_for_push(
sem,
dir.path().to_path_buf(),
vec![tip],
vec![],
git_bin,
Duration::from_secs(5),
false,
)
.await;
assert!(
marker.exists(),
"once admission is available the deferred scan runs its git"
);
assert!(!set.full_scan, "commit tip + empty rev-list is a delta");
assert!(set.candidates.is_empty());
}
/// F4 fast-path negative arm: a deletion-only push (no new tips, kill-switch
/// off) computes its empty delta without spawning ANY git child, so it must
/// complete without acquiring from the scan pool — even at zero permits. The
/// per-tip cat-file probe means every push with a non-empty new tip DOES spawn
/// git, so this is the only genuinely git-free stage.
#[cfg(unix)]
#[tokio::test]
async fn resolve_candidates_no_git_fast_path_skips_admission() {
use std::os::unix::fs::PermissionsExt;
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::Semaphore;
let dir = tempfile::TempDir::new().unwrap();
let marker = dir.path().join("git.ran");
let fake = dir.path().join("fakegit");
std::fs::write(
&fake,
format!("#!/bin/sh\necho ran >> \"{}\"\nexit 0\n", marker.display()),
)
.unwrap();
let mut perm = std::fs::metadata(&fake).unwrap().permissions();
perm.set_mode(0o755);
std::fs::set_permissions(&fake, perm).unwrap();
let set = tokio::time::timeout(
Duration::from_millis(500),
resolve_candidates_for_push(
Arc::new(Semaphore::new(0)),
dir.path().to_path_buf(),
vec![],
vec!["deadbeefdeadbeefdeadbeefdeadbeefdeadbeef".to_string()],
fake.to_str().unwrap().to_string(),
Duration::from_secs(5),
false,
),
)
.await
.expect("the no-new-tips fast path must not park on the scan pool");
assert_eq!(
set,
PinCandidateSet {
candidates: Vec::new(),
full_scan: false
}
);
assert!(!marker.exists(), "the fast path must spawn no git at all");
}
/// #174 F5: the coalesced-drain FullScan marker is signalled via the explicit
/// `force_full_scan` flag, and a flagged call with NO tips must still enumerate
/// the whole repo (full_scan=true, non-empty candidates). The RED arm of the
/// encoding question: if the marker were encoded as a plain empty-tips call
/// (flag off — the fast path), the drain would resolve to an empty delta and
/// pin nothing, silently losing the coalesced pushes' work.
#[tokio::test]
async fn forced_full_scan_with_no_tips_enumerates_the_repo() {
let repo = Repo::new();
repo.commit_file("a.txt", "one\n");
let all: HashSet<String> = list_all_objects(&repo.path, "git", td())
.unwrap()
.into_iter()
.collect();
assert!(!all.is_empty(), "fixture repo has objects");
let set = resolve_candidates_for_push(
std::sync::Arc::new(tokio::sync::Semaphore::new(64)),
repo.path.clone(),
vec![],
vec![],
"git".to_string(),
std::time::Duration::from_secs(600),
true,
)
.await;
assert!(set.full_scan, "the forced call is signalled as a full scan");
let got: HashSet<String> = set.candidates.into_iter().collect();
assert_eq!(
got, all,
"a forced full scan with no tips enumerates the repo — never an empty delta"
);
// The discriminator: the SAME empty-tips call without the flag is the
// deletion-only fast path (empty delta, pin nothing). The two must differ,
// or the marker encoding has collapsed into the silent-loss shape.
let unforced = resolve_candidates_for_push(
std::sync::Arc::new(tokio::sync::Semaphore::new(64)),
repo.path.clone(),
vec![],
vec![],
"git".to_string(),
std::time::Duration::from_secs(600),
false,
)
.await;
assert!(!unforced.full_scan);
assert!(unforced.candidates.is_empty());
}
#[test]
fn missing_oid_tip_forces_full_scan() {
let repo = Repo::new();
repo.commit_file("a.txt", "one\n");
let bogus = "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef";
assert_eq!(
resolve_push_delta(&repo.path, &[bogus], &[]),
resolve_push_delta(&repo.path, &[bogus], &[], "git", td()),
PinCandidates::FullScanRequired,
"a missing/corrupt tip OID must force full scan"
);
@@ -564,7 +829,7 @@ mod tests {
let c1 = repo.commit_file("a.txt", "one\n");
let c2 = repo.commit_file("b.txt", "two\n");
let old_tree = repo.rev(&format!("{c1}^{{tree}}"));
let result = resolve_push_delta(&repo.path, &[&c2], &[&old_tree]);
let result = resolve_push_delta(&repo.path, &[&c2], &[&old_tree], "git", td());
match result {
PinCandidates::FullScanRequired => {} // safe: caller full-scans
PinCandidates::Delta(objs) => {
@@ -583,10 +848,15 @@ mod tests {
// branch2 from base advances independently
repo.git(&["checkout", "-q", "-b", "branch2", &base]);
let b2 = repo.commit_file("c.txt", "three\n");
let got: HashSet<String> =
delta(resolve_push_delta(&repo.path, &[&b1, &b2], &[&base, &base]))
.into_iter()
.collect();
let got: HashSet<String> = delta(resolve_push_delta(
&repo.path,
&[&b1, &b2],
&[&base, &base],
"git",
td(),
))
.into_iter()
.collect();
assert!(got.contains(&b1), "branch1 new commit");
assert!(got.contains(&b2), "branch2 new commit");
}
@@ -595,7 +865,7 @@ mod tests {
fn empty_repo_no_tips_yields_empty_delta() {
let repo = Repo::new();
assert_eq!(
resolve_push_delta(&repo.path, &[], &[]),
resolve_push_delta(&repo.path, &[], &[], "git", td()),
PinCandidates::Delta(Vec::new())
);
}
@@ -609,12 +879,12 @@ mod tests {
let repo = Repo::new();
let c1 = repo.commit_file("a.txt", "one\n");
assert_eq!(
resolve_push_delta_inner(&repo.path, &[&c1], &[], true),
resolve_push_delta_inner(&repo.path, &[&c1], &[], true, "git", td()),
PinCandidates::FullScanRequired
);
// And with the flag off, the same push yields a Delta.
assert!(matches!(
resolve_push_delta_inner(&repo.path, &[&c1], &[], false),
resolve_push_delta_inner(&repo.path, &[&c1], &[], false, "git", td()),
PinCandidates::Delta(_)
));
}
@@ -624,7 +894,7 @@ mod tests {
let repo = Repo::new();
repo.commit_file("a.txt", "one\n");
repo.commit_file("b.txt", "two\n");
let all = list_all_objects(&repo.path).unwrap();
let all = list_all_objects(&repo.path, "git", td()).unwrap();
// 2 commits + 2 trees + 2 blobs = 6 objects.
assert_eq!(all.len(), 6, "got: {all:?}");
}
@@ -642,7 +912,7 @@ mod tests {
std::fs::write(repo.path.join("orphan.bin"), b"dangling\n").unwrap();
let dangling = repo.git(&["hash-object", "-w", "orphan.bin"]);
let blobs = all_blob_oids(&repo.path).unwrap();
let blobs = all_blob_oids(&repo.path, "git", td()).unwrap();
assert!(blobs.contains(&reachable_blob), "reachable blob present");
assert!(
blobs.contains(&dangling),
@@ -652,7 +922,7 @@ mod tests {
assert!(!blobs.contains(&tree), "tree is not a blob");
// The typed lister tags each object; spot-check the dangling blob's type.
let typed = list_all_objects_with_type(&repo.path).unwrap();
let typed = list_all_objects_with_type(&repo.path, "git", td()).unwrap();
assert!(
typed
.iter()
@@ -660,4 +930,96 @@ mod tests {
"dangling object is typed as a blob"
);
}
#[cfg(unix)]
#[test]
fn list_all_objects_times_out_on_a_hung_git_instead_of_blocking() {
use std::os::unix::fs::PermissionsExt;
use std::time::{Duration, Instant};
let dir = tempfile::TempDir::new().unwrap();
// Fake git: hang on cat-file (bounded to 30s so a broken test can't wedge).
let fake = dir.path().join("fakegit");
std::fs::write(
&fake,
"#!/bin/sh\ncase \"$1\" in\n cat-file) i=0; while [ $i -lt 30 ]; do sleep 1; i=$((i+1)); done ;;\n *) : ;;\nesac\n",
)
.unwrap();
let mut perm = std::fs::metadata(&fake).unwrap().permissions();
perm.set_mode(0o755);
std::fs::set_permissions(&fake, perm).unwrap();
let git_bin = fake.to_str().unwrap().to_string();
let (tx, rx) = std::sync::mpsc::channel();
let path = dir.path().to_path_buf();
std::thread::spawn(move || {
let _ = tx.send(list_all_objects(
&path,
&git_bin,
Instant::now() + Duration::from_millis(150),
));
});
let res = rx.recv_timeout(Duration::from_secs(10)).expect(
"list_all_objects must return within the watchdog budget, not block on a hung git",
);
assert!(
res.is_err(),
"a hung git must make list_all_objects error out, not hang"
);
}
/// #174 finding 2: the DELTA-path children — `peeled_object_type` (cat-file),
/// `rev_list_delta` (rev-list) — and `list_all_objects_with_type` (cat-file) are
/// the remaining candidate-discovery execs (the common push path). Each must
/// return within the watchdog budget on a hung git rather than block and pin the
/// write permit. Revert any one to a bare `Command::output()` and its arm blocks
/// past the recv budget (RED).
#[cfg(unix)]
#[test]
fn delta_path_exec_fns_time_out_on_a_hung_git() {
use std::os::unix::fs::PermissionsExt;
use std::time::{Duration, Instant};
let dir = tempfile::TempDir::new().unwrap();
// Fake git hangs on BOTH cat-file and rev-list (bounded 30s so a broken
// test can't leak a permanent orphan or wedge the suite).
let fake = dir.path().join("fakegit");
std::fs::write(
&fake,
"#!/bin/sh\ncase \"$1\" in\n cat-file|rev-list) i=0; while [ $i -lt 30 ]; do sleep 1; i=$((i+1)); done ;;\n *) : ;;\nesac\n",
)
.unwrap();
let mut perm = std::fs::metadata(&fake).unwrap().permissions();
perm.set_mode(0o755);
std::fs::set_permissions(&fake, perm).unwrap();
let git_bin = fake.to_str().unwrap().to_string();
let path = dir.path().to_path_buf();
// Run `f` on a thread and require it to RETURN (not block) within 10s.
fn returns_within<T: Send + 'static>(f: impl FnOnce() -> T + Send + 'static) -> T {
let (tx, rx) = std::sync::mpsc::channel();
std::thread::spawn(move || {
let _ = tx.send(f());
});
rx.recv_timeout(Duration::from_secs(10)).expect(
"a delta-path exec fn must return within the watchdog budget, not block on a hung git",
)
}
let dl = || Instant::now() + Duration::from_millis(150);
let sha = "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef";
let (p, g, d) = (path.clone(), git_bin.clone(), dl());
assert!(
returns_within(move || peeled_object_type(&p, sha, &g, d)).is_none(),
"peeled_object_type must time out to None on a hung cat-file"
);
let (p, g, d) = (path.clone(), git_bin.clone(), dl());
assert!(
returns_within(move || rev_list_delta(&p, &[sha], &[], &g, d)).is_err(),
"rev_list_delta must error out on a hung rev-list"
);
let (p, g, d) = (path.clone(), git_bin.clone(), dl());
assert!(
returns_within(move || list_all_objects_with_type(&p, &g, d)).is_err(),
"list_all_objects_with_type must error out on a hung cat-file"
);
}
}
+845 -21
View File
@@ -13,7 +13,8 @@ use std::path::{Path, PathBuf};
use std::sync::Arc;
use anyhow::{Context, Result};
use sqlx::PgPool;
use sqlx::pool::PoolConnection;
use sqlx::{PgPool, Postgres};
use tokio::sync::Mutex;
use tracing::{debug, info, warn};
@@ -30,6 +31,12 @@ pub struct RepoStore {
/// Tracks repos already confirmed to exist in Tigris — avoids redundant
/// HEAD checks and background uploads for repos we've already migrated.
migrated: Arc<Mutex<HashSet<String>>>,
/// Test-only seam: armed here, copied into every `RepoWriteGuard` this store
/// hands out, so a test that only holds the `AppState` (not the guard) can
/// still park `release` at its pre-unlock point. See
/// `RepoWriteGuard::test_pre_unlock_gate`. Never set outside tests.
#[cfg(test)]
pre_unlock_gate: Option<Arc<tokio::sync::Notify>>,
}
impl RepoStore {
@@ -40,15 +47,27 @@ impl RepoStore {
tigris: None,
pool,
migrated: Arc::new(tokio::sync::Mutex::new(std::collections::HashSet::new())),
pre_unlock_gate: None,
}
}
/// Test-only: every guard from this store parks in `release` right before the
/// `pg_advisory_unlock` await, until `gate` is notified. Dropping the future
/// while it is parked reproduces a client disconnect inside `release`.
#[cfg(test)]
pub fn with_pre_unlock_gate(mut self, gate: Arc<tokio::sync::Notify>) -> Self {
self.pre_unlock_gate = Some(gate);
self
}
pub fn new(repos_dir: PathBuf, tigris: Option<TigrisClient>, pool: PgPool) -> Self {
Self {
repos_dir,
tigris,
pool,
migrated: Arc::new(Mutex::new(HashSet::new())),
#[cfg(test)]
pre_unlock_gate: None,
}
}
@@ -157,13 +176,44 @@ impl RepoStore {
let (owner_slug, local_path) = self.local_path(owner_did, repo_name)?;
let lock_key = advisory_lock_key(&owner_slug, repo_name);
// Acquire Postgres advisory lock with retry using pg_try_advisory_lock
// to avoid blocking indefinitely on stale locks from crashed connections.
// Pin a dedicated pooled connection and build the guard holding it BEFORE
// issuing the lock query. Session-level pg advisory locks are
// connection-affine (they can only be released on the session that took
// them), so the guard must own the locking connection; and building the
// guard first means any cancellation after the lock is taken — a
// `tokio::time::timeout` firing during the Tigris download below — drops a
// guard that CAN release, closing the leak the outer timeout otherwise
// opened (#174 F1).
let conn = self
.pool
.acquire()
.await
.context("acquiring db connection for the write advisory lock")?;
let mut guard = RepoWriteGuard {
owner_slug: owner_slug.clone(),
repo_name: repo_name.to_string(),
local_path: local_path.clone(),
lock_key,
conn: Some(conn),
locked: false,
released: false,
tigris: self.tigris.clone(),
#[cfg(test)]
test_pre_unlock_gate: self.pre_unlock_gate.clone(),
};
// Acquire the advisory lock with retry, through the guard's OWN connection,
// so the matching unlock (in release, or the Drop backstop) runs on the same
// session — pg_advisory_unlock on a different pooled connection is a no-op.
let mut acquired = false;
for attempt in 0..60 {
let c = guard
.conn
.as_deref_mut()
.expect("write guard holds its connection during acquisition");
let row: (bool,) = sqlx::query_as("SELECT pg_try_advisory_lock($1)")
.bind(lock_key)
.fetch_one(&self.pool)
.fetch_one(&mut *c)
.await
.context("trying advisory lock")?;
if row.0 {
@@ -177,9 +227,11 @@ impl RepoStore {
if !acquired {
anyhow::bail!("could not acquire advisory lock after 60s — possible stale lock for {owner_slug}/{repo_name}");
}
guard.locked = true;
// Always download the latest from Tigris before writing.
// Local disk may be stale if another machine pushed since our last access.
// Always download the latest from Tigris before writing. Local disk may be
// stale if another machine pushed since our last access. The guard already
// owns the lock + its connection, so a cancellation here drops through Drop.
if let Some(ref tigris) = self.tigris {
if tigris.exists(&owner_slug, repo_name).await.unwrap_or(false) {
debug!(repo = %repo_name, "write acquire: downloading latest from tigris");
@@ -197,14 +249,7 @@ impl RepoStore {
}
}
Ok(RepoWriteGuard {
owner_slug,
repo_name: repo_name.to_string(),
local_path,
lock_key,
pool: self.pool.clone(),
tigris: self.tigris.clone(),
})
Ok(guard)
}
/// Initialize a new bare repo on local disk and upload to Tigris.
@@ -479,8 +524,62 @@ pub struct RepoWriteGuard {
repo_name: String,
pub local_path: PathBuf,
lock_key: i64,
pool: PgPool,
/// The pooled connection that took the advisory lock. Session-level pg
/// advisory locks are connection-affine, so the guard pins that connection
/// for its whole lifetime and unlocks on it (in `release`, or the `Drop`
/// backstop). `None` only after the connection has been taken, either to run
/// the detached unlock in `Drop` or to be closed when `release`'s unlock
/// errored (#174 F3b).
conn: Option<PoolConnection<Postgres>>,
/// Set once the advisory lock has actually been taken. A guard dropped
/// before the lock is held (or after `release`) performs no unlock.
locked: bool,
/// Set once `release` has run its unlock, making the `Drop` backstop inert.
released: bool,
tigris: Option<TigrisClient>,
/// Test-only seam: when set, `release` parks on this gate at the exact point
/// it is about to await `pg_advisory_unlock` (connection still owned, not yet
/// released). Dropping the `release` future while it is parked reproduces a
/// mid-unlock cancellation, so a test can assert the `Drop` backstop still
/// frees the session lock. Never set outside tests.
#[cfg(test)]
test_pre_unlock_gate: Option<Arc<tokio::sync::Notify>>,
}
/// Deadline for tearing down the connection that saw a failing `pg_advisory_unlock`.
/// Long enough that a healthy socket always finishes well inside it, short enough that
/// a blackholed one does not pin admission resources for a TCP timeout.
const UNLOCK_ERROR_CLOSE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
/// Await `close` under a deadline (#174 F3c).
///
/// `release` awaits this INLINE while the global write permit, the per-source permit
/// and the write lease are all still held, and sqlx puts no deadline on `close()`:
/// it writes Terminate and then tears the socket down. The branch that reaches here is
/// by definition a connection whose last statement errored, and a blackholed TCP path
/// to Postgres (a cloud failover that drops packets without an RST) is a plausible
/// cause, so an unbounded await here parks every later push to the repo behind three
/// pinned admission resources until the steal bound.
///
/// On elapsed the future is simply dropped, which drops the `PoolConnection` it owns.
/// Dropping it closes the socket, and closing the socket is what actually ends the
/// session and makes Postgres release the lock, so the deadline costs nothing the
/// graceful path was buying.
async fn close_conn_bounded(
repo_name: &str,
close: impl std::future::Future<Output = Result<(), sqlx::Error>>,
) {
match tokio::time::timeout(UNLOCK_ERROR_CLOSE_TIMEOUT, close).await {
Ok(Ok(())) => {}
Ok(Err(e)) => {
warn!(repo = %repo_name, err = %e,
"closing the write-lock connection failed, the session teardown still frees the lock server-side");
}
Err(_) => {
warn!(repo = %repo_name, timeout_secs = UNLOCK_ERROR_CLOSE_TIMEOUT.as_secs(),
"closing the write-lock connection timed out, dropping it instead; the socket goes down either way, which is what frees the lock server-side");
}
}
}
impl RepoWriteGuard {
@@ -494,7 +593,7 @@ impl RepoWriteGuard {
/// half-applied or otherwise inconsistent repo would propagate corruption to
/// Tigris (and to every node that later downloads it). The lock is always
/// released regardless, to avoid stale locks blocking future writes.
pub async fn release(self, success: bool) {
pub async fn release(mut self, success: bool) {
// Upload to Tigris only on success.
if success {
if let Some(ref tigris) = self.tigris {
@@ -509,11 +608,124 @@ impl RepoWriteGuard {
warn!(repo = %self.repo_name, "write failed — skipping tigris upload to avoid propagating an inconsistent repo");
}
// Release advisory lock
let _ = sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(self.lock_key)
.execute(&self.pool)
.await;
// Release the advisory lock on the SAME connection that took it (session
// advisory locks are connection-affine). Unlock through the connection
// while it is STILL owned by `self` — do not `take()` it first. If this
// future is cancelled during the unlock await, `self` is dropped with
// `conn == Some(..)` and `released == false`, so the `Drop` backstop still
// runs the detached unlock. `released` is set only AFTER the await
// resolves, so a cancellation cannot make the backstop inert (#174 F4).
if self.locked {
#[cfg(test)]
let pre_unlock_gate = self.test_pre_unlock_gate.clone();
let unlock = if let Some(conn) = self.conn.as_deref_mut() {
// Test-only: park right before the unlock await so a test can drop
// this future mid-unlock (connection owned, not yet released).
#[cfg(test)]
if let Some(gate) = pre_unlock_gate {
gate.notified().await;
}
Some(
sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(self.lock_key)
.execute(&mut *conn)
.await,
)
} else {
None
};
// An unlock that ERRORS is a different failure from a cancellation: the
// await resolved, so `Drop` is about to be made inert by `released`
// below, but the session is still alive and still holds the lock
// (statement timeout, admin cancel, aborted transaction). Returning that
// `PoolConnection` to the pool would hand the next caller a connection
// holding a lock nobody tracks (#174 F3b). Connection disposal is the
// single mechanism here, and it is why we do not instead try to keep the
// `Drop` backstop armed: disposal needs `conn.take()`, and `Drop`
// early-returns on `conn == None`. Ending the session is what frees the
// lock, so `released = true` still holds.
if let Some(Err(e)) = unlock {
warn!(repo = %self.repo_name, err = %e,
"advisory unlock failed, closing the connection so the session ends and postgres drops the lock");
if let Some(conn) = self.conn.take() {
// `close()` over `detach()`: both consume the `PoolConnection` by
// value in sqlx 0.8.6, but we are in an async fn, so `close()`
// sends Terminate and waits for the socket to go down before
// `release` returns. `detach()` would only end the session
// whenever the returned `PgConnection` is dropped and its
// background close completes. If this future is cancelled during
// `close()`, the connection is dropped mid-close, which still
// tears the session down. That last point is also why the await is
// safe to bound: see `close_conn_bounded`, which gives it the
// deadline sqlx does not.
close_conn_bounded(&self.repo_name, conn.close()).await;
}
}
}
self.released = true;
}
}
impl Drop for RepoWriteGuard {
/// Cancellation-safe backstop: if the guard is dropped while still holding the
/// advisory lock (a `tokio::time::timeout` cancelled `acquire_write`, or a
/// handler future was dropped before `release`), unlock on the pinned
/// connection. This is NOT the backstop for an unlock that ran and returned an
/// error: that case is closed inside `release` by disposing of the connection,
/// because `Drop` early-returns on `conn == None` and the two mechanisms cannot
/// both apply (#174 F3b). `Drop` cannot await, so spawn a detached unlock — it runs on the
/// same session (connection-affine). An off-runtime drop has nothing to spawn onto,
/// so it disposes of the connection instead. On runtime
/// SHUTDOWN the spawned unlock task may be dropped before it polls, so the unlock
/// may not run — but shutdown tears down the pool, and closing the connection
/// releases the session-level advisory lock server-side, so this too is bounded.
fn drop(&mut self) {
if self.released || !self.locked {
return;
}
let Some(mut conn) = self.conn.take() else {
return;
};
let lock_key = self.lock_key;
let repo_name = self.repo_name.clone();
match tokio::runtime::Handle::try_current() {
Ok(handle) => {
handle.spawn(async move {
let unlock = sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(lock_key)
.execute(&mut *conn)
.await;
// Same failure as `release`'s (#174 F3b), one level down: the await
// RESOLVED with an error, so the session is alive and still holds
// the lock. Letting this async block end here would drop `conn` and
// RETURN it to the pool, handing the next caller a connection
// holding a lock nobody tracks. Close it instead, which both keeps
// it out of the pool and ends the session that holds the lock.
if let Err(e) = unlock {
warn!(repo = %repo_name, err = %e, "detached advisory-unlock on write-guard drop failed, closing the connection so the session ends and postgres drops the lock");
close_conn_bounded(&repo_name, conn.close()).await;
}
});
}
Err(_) => {
// No runtime to spawn the unlock onto, and the connection is already
// out of the guard, so there is no path that unlocks on this session.
// Returning it to the pool would hand the next caller a connection
// still holding the lock. `PoolConnection`'s own drop also spawns its
// return-to-pool task, which panics with no runtime. `detach` gives up
// the pool slot and yields a plain `PgConnection`; dropping that closes
// the socket, which ends the session and is what frees the lock
// server-side. `Drop` cannot await, so this is the whole disposal:
// `close_conn_bounded` is not available here.
drop(conn.detach());
warn!(
repo = %repo_name,
"RepoWriteGuard dropped off a Tokio runtime; no detached unlock is \
possible, so the pinned connection is disposed of instead: ending \
the session is what releases the advisory lock"
);
}
}
}
}
@@ -932,4 +1144,616 @@ mod tests {
);
}
}
// ── advisory-lock cancellation-safety (#174 F1, RED-before/GREEN-after) ──
/// F1 (P1): dropping a `RepoWriteGuard` WITHOUT calling `release()` — the
/// state a `tokio::time::timeout` cancellation leaves `acquire_write` in when
/// it fires during the Tigris await — must still release the session advisory
/// lock. A checker connection is held OUT of the pool first, so `acquire_write`
/// is forced onto a distinct session; the checker (a different session) then
/// probes the lock, so advisory-lock re-entrancy cannot mask a leak.
///
/// Load-bearing: RED today (no `Drop` releases the lock → held by
/// `acquire_write`'s session → checker's `pg_try_advisory_lock` returns
/// false). GREEN after the connection-affine `Drop` backstop.
#[sqlx::test]
async fn write_guard_drop_without_release_frees_the_lock(pool: sqlx::PgPool) {
let dir = tempfile::TempDir::new().unwrap();
let store = RepoStore::for_testing(dir.path().to_path_buf(), pool.clone());
let owner = "did:key:z6MkDropBackstopProofAAAAAAAAAAAAAAAAAAAAAA";
let name = "leaktest";
let slug = owner.replace([':', '/'], "_");
let key = advisory_lock_key(&slug, name);
// Distinct session for the probe: hold it out of the pool BEFORE acquiring,
// so acquire_write cannot use it and a re-entrant probe cannot falsely read free.
let mut checker = pool.acquire().await.expect("checker connection");
let guard = store.acquire_write(owner, name).await.expect("acquire");
// The cancellation shape: drop without release().
drop(guard);
// Let the detached unlock task run.
tokio::time::sleep(std::time::Duration::from_millis(400)).await;
let (free,): (bool,) = sqlx::query_as("SELECT pg_try_advisory_lock($1)")
.bind(key)
.fetch_one(&mut *checker)
.await
.unwrap();
assert!(
free,
"advisory lock must be released when the guard is dropped without release()"
);
let _ = sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(key)
.execute(&mut *checker)
.await;
}
/// F1 (latent non-affine release): `release()` must unlock on the SAME
/// session that locked, so the lock is freed regardless of which pooled
/// connection would service a fresh query. Observed from a distinct session.
#[sqlx::test]
async fn write_guard_release_frees_the_lock_from_a_distinct_session(pool: sqlx::PgPool) {
let dir = tempfile::TempDir::new().unwrap();
let store = RepoStore::for_testing(dir.path().to_path_buf(), pool.clone());
let owner = "did:key:z6MkAffineReleaseProofBBBBBBBBBBBBBBBBBBBB";
let name = "affinetest";
let slug = owner.replace([':', '/'], "_");
let key = advisory_lock_key(&slug, name);
let mut checker = pool.acquire().await.expect("checker connection");
let guard = store.acquire_write(owner, name).await.expect("acquire");
guard.release(false).await;
let (free,): (bool,) = sqlx::query_as("SELECT pg_try_advisory_lock($1)")
.bind(key)
.fetch_one(&mut *checker)
.await
.unwrap();
assert!(
free,
"release() must free the advisory lock via connection-affine unlock"
);
let _ = sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(key)
.execute(&mut *checker)
.await;
}
// ── cancellation-safe unlock (#174 F4, RED-before/GREEN-after) ──────────
/// F4 (P1): a cancellation DURING the unlock await must still free the session
/// advisory lock. `release` unlocks through the connection while `self` still
/// owns it, so if the future is dropped mid-unlock, `Drop` sees `conn == Some`
/// + `locked && !released` and runs its detached-unlock backstop. A test-only
/// gate parks `release` at the exact pre-unlock point; dropping the future
/// there reproduces the cancellation.
///
/// Load-bearing: RED on the original ordering (`self.conn.take()` before the
/// await → at cancellation `self.conn == None` → `Drop` skips → the local
/// connection returns to the pool with the session lock still held → the
/// checker's `pg_try_advisory_lock` returns false). GREEN after the reorder.
#[sqlx::test]
async fn write_guard_release_cancelled_mid_unlock_frees_the_lock(pool: sqlx::PgPool) {
let dir = tempfile::TempDir::new().unwrap();
let store = RepoStore::for_testing(dir.path().to_path_buf(), pool.clone());
let owner = "did:key:z6MkCancelMidUnlockProofCCCCCCCCCCCCCCCCCC";
let name = "canceltest";
let slug = owner.replace([':', '/'], "_");
let key = advisory_lock_key(&slug, name);
// Distinct session for the probe, held out of the pool before acquiring.
let mut checker = pool.acquire().await.expect("checker connection");
let mut guard = store.acquire_write(owner, name).await.expect("acquire");
// Arm the pre-unlock gate; it is never notified, so `release` parks on it
// with the connection still owned and `released` still false.
let gate = Arc::new(tokio::sync::Notify::new());
guard.test_pre_unlock_gate = Some(gate);
// Box the future so we can drop it ourselves (tokio::pin! keeps it alive to
// end of scope, which would defer the guard's Drop past the assertions).
let mut fut = Box::pin(guard.release(false));
let parked =
tokio::time::timeout(std::time::Duration::from_millis(300), fut.as_mut()).await;
assert!(
parked.is_err(),
"release should park on the pre-unlock gate, not complete"
);
// Cancel mid-unlock: dropping the boxed future runs RepoWriteGuard::drop.
drop(fut);
// Let the Drop backstop's detached unlock task run.
tokio::time::sleep(std::time::Duration::from_millis(400)).await;
let (free,): (bool,) = sqlx::query_as("SELECT pg_try_advisory_lock($1)")
.bind(key)
.fetch_one(&mut *checker)
.await
.unwrap();
assert!(
free,
"advisory lock must be freed when release is cancelled mid-unlock — the \
connection must stay owned by the guard so Drop's backstop can run"
);
let _ = sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(key)
.execute(&mut *checker)
.await;
}
/// F4: the ordinary success path still frees the lock, and a second write
/// acquire on the same repo returns promptly (no stale-lock retry loop).
#[sqlx::test]
async fn write_guard_release_true_frees_lock_and_second_acquire_succeeds(pool: sqlx::PgPool) {
let dir = tempfile::TempDir::new().unwrap();
let store = RepoStore::for_testing(dir.path().to_path_buf(), pool.clone());
let owner = "did:key:z6MkReleaseTrueProofDDDDDDDDDDDDDDDDDDDDDD";
let name = "reltruetest";
let guard = store
.acquire_write(owner, name)
.await
.expect("first acquire");
guard.release(true).await;
let again = tokio::time::timeout(
std::time::Duration::from_secs(2),
store.acquire_write(owner, name),
)
.await
.expect("second acquire_write must not hit the ~60s stale-lock retry loop")
.expect("second acquire");
again.release(true).await;
}
// ── unlock error disposes the connection (#174 F3b, RED-before/GREEN-after) ─
/// Put the guard's pinned connection into a failed-transaction state, so the
/// next statement on it errors while the SESSION stays alive and keeps holding
/// the session-level advisory lock (those survive a transaction abort; only
/// `pg_advisory_xact_lock` would not). This is the smallest injection that
/// reproduces F3b's shape: `pg_advisory_unlock` returning `Err` on a live,
/// still-locking session. No production seam is needed because the tests live
/// in this module and can reach `conn` directly.
async fn poison_guard_connection(guard: &mut RepoWriteGuard) {
let conn = guard
.conn
.as_deref_mut()
.expect("guard holds its connection before release");
sqlx::query("BEGIN")
.execute(&mut *conn)
.await
.expect("open a transaction on the guard connection");
let poisoned = sqlx::query("SELECT 1 / 0").execute(&mut *conn).await;
assert!(
poisoned.is_err(),
"the poison statement must fail so the transaction is aborted"
);
}
/// How long the polls below may wait for an asynchronous server-side effect.
/// Postgres frees a session advisory lock when the backend exits, which happens
/// asynchronously to our socket close, so these are polled to a generous deadline
/// rather than slept for a fixed interval: a constant sleep is a flake under load,
/// and one that is long enough to be safe is dead time on every run.
const RELEASE_POLL_DEADLINE: std::time::Duration = std::time::Duration::from_secs(20);
/// Poll `cond` until it holds, failing at the deadline so a regression fails the
/// test rather than hanging the suite.
async fn wait_until(mut cond: impl FnMut() -> bool, what: &str) {
let deadline = std::time::Instant::now() + RELEASE_POLL_DEADLINE;
while !cond() {
assert!(
std::time::Instant::now() < deadline,
"timed out waiting for {what}"
);
tokio::time::sleep(std::time::Duration::from_millis(25)).await;
}
}
/// Poll the advisory lock from `checker` until it is free. `pg_try_advisory_lock`
/// ACQUIRES on success, so a true result both answers the question and leaves the
/// checker session holding the lock; the caller unlocks it.
async fn wait_until_lock_free(checker: &mut sqlx::PgConnection, key: i64, what: &str) {
let deadline = std::time::Instant::now() + RELEASE_POLL_DEADLINE;
loop {
let (free,): (bool,) = sqlx::query_as("SELECT pg_try_advisory_lock($1)")
.bind(key)
.fetch_one(&mut *checker)
.await
.unwrap();
if free {
return;
}
assert!(
std::time::Instant::now() < deadline,
"timed out waiting for {what}"
);
tokio::time::sleep(std::time::Duration::from_millis(25)).await;
}
}
/// A second pool over the same test database with the idle reaper DISABLED.
///
/// `#[sqlx::test]`'s own pool sets `idle_timeout(1s)`, so a connection returned to
/// it is closed by the reaper about a second later, which ends the session and
/// frees the advisory lock all on its own. The old fixed 400ms sleep landed inside
/// that window by luck; polling to a deadline long enough to be flake-proof would
/// land outside it and go green whether or not `release` disposed of the
/// connection, so the tests below would stop testing anything (measured: with the
/// reaper in play, the disposal shows up at ~2s even with the fix reverted). With
/// no reaper, `release` is the only thing that can end that session, so the poll
/// measures exactly the property these two tests exist for.
async fn pool_without_idle_reaper(pool: &sqlx::PgPool) -> sqlx::PgPool {
sqlx::postgres::PgPoolOptions::new()
.max_connections(5)
.idle_timeout(None)
.max_lifetime(None)
.connect_with(pool.connect_options().as_ref().clone())
.await
.expect("a second pool over the test database")
}
/// F3c (P2): the connection teardown on the failing-unlock path must be BOUNDED.
/// `release` awaits it inline while the global write permit, the per-source permit
/// and the write lease are all still held, and sqlx's `close()` carries no deadline
/// of its own, so a blackholed socket would park every later push to that repo
/// behind three pinned admission resources.
///
/// What this covers: the deadline itself. A close that never resolves still lets
/// `close_conn_bounded` return, which is the property `release` depends on. What it
/// does NOT cover, and is reasoned rather than run: that sqlx's own `close()` is
/// what stalls in production. Making a real `PgConnection::close` hang needs a
/// blackholed TCP path to Postgres, and the flip has to land after the unlock
/// statement round-trips but before the Terminate write, which is not a seam this
/// module exposes. A never-resolving future is the faithful stand-in for that
/// close, and the F3b tests above already cover that `release` really routes its
/// close through here.
///
/// Time is paused, so nothing here depends on wall clock: the runtime auto-advances
/// to the next timer, and the assertion is on which timer fired, not on elapsed
/// time. The outer bound is what turns a removed deadline into a failure rather
/// than a hung suite.
///
/// Load-bearing: drop the `tokio::time::timeout` in `close_conn_bounded` and the
/// inner future never resolves, so the outer bound fires and this fails.
#[tokio::test(start_paused = true)]
async fn unlock_error_connection_close_is_bounded() {
let hanging = std::future::pending::<Result<(), sqlx::Error>>();
let outcome = tokio::time::timeout(
UNLOCK_ERROR_CLOSE_TIMEOUT * 4,
close_conn_bounded("boundedclosetest", hanging),
)
.await;
assert!(
outcome.is_ok(),
"a connection close that never completes must not hold the write lease and \
both admission permits open-endedly: close_conn_bounded must give up and \
drop the connection"
);
}
/// F3b (P1): when `pg_advisory_unlock` ERRORS while the session is still alive
/// (statement timeout, admin cancel, aborted transaction), the lock must not
/// survive `release`. The old code discarded the error with `let _ =` and set
/// `released = true` anyway, so `Drop` early-returned and the `PoolConnection`
/// went back to the pool still holding the session lock.
///
/// Observed from a SEPARATE connection held out of the pool before acquiring:
/// session advisory locks are re-entrant and counted, so probing from the
/// holding session (or via a fresh `acquire_write` that may be handed the same
/// connection) would report free whether or not the fix is present.
///
/// Load-bearing: RED before the fix (lock still held → `pg_try_advisory_lock`
/// returns false), GREEN after (the errored connection is closed, so the
/// session ends and Postgres drops the lock).
#[sqlx::test]
async fn write_guard_release_with_failing_unlock_frees_the_lock(pool: sqlx::PgPool) {
let dir = tempfile::TempDir::new().unwrap();
let store_pool = pool_without_idle_reaper(&pool).await;
let store = RepoStore::for_testing(dir.path().to_path_buf(), store_pool.clone());
let owner = "did:key:z6MkUnlockErrorProofFFFFFFFFFFFFFFFFFFFFFF";
let name = "unlockerrtest";
let slug = owner.replace([':', '/'], "_");
let key = advisory_lock_key(&slug, name);
// Distinct session for the probe, held out of the pool before acquiring.
let mut checker = pool.acquire().await.expect("checker connection");
let mut guard = store.acquire_write(owner, name).await.expect("acquire");
poison_guard_connection(&mut guard).await;
// Sanity: the poisoned session is still alive and still holds the lock, so
// the assertion below measures the release path and not a dead session.
let (free_before,): (bool,) = sqlx::query_as("SELECT pg_try_advisory_lock($1)")
.bind(key)
.fetch_one(&mut *checker)
.await
.unwrap();
assert!(
!free_before,
"the poisoned session must still hold the lock before release"
);
guard.release(false).await;
// Postgres drops the lock when the disposed session's backend exits, which is
// asynchronous to our socket close: poll for it rather than sleeping a
// constant. The deadline is what keeps this load-bearing: a release that
// leaves the lock held never satisfies the probe and fails here.
wait_until_lock_free(
&mut checker,
key,
"an errored pg_advisory_unlock must not leave the lock held: release must \
dispose of the connection so the session ends",
)
.await;
let _ = sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(key)
.execute(&mut *checker)
.await;
}
/// F3b: the connection that saw the unlock error must leave the pool entirely,
/// rather than being handed to the next caller while still holding the lock.
/// `PgPool::size()` counts the connections the pool owns, so closing the
/// errored one is observable as a drop in that count.
#[sqlx::test]
async fn write_guard_release_with_failing_unlock_does_not_return_the_connection(
pool: sqlx::PgPool,
) {
let dir = tempfile::TempDir::new().unwrap();
let store_pool = pool_without_idle_reaper(&pool).await;
let store = RepoStore::for_testing(dir.path().to_path_buf(), store_pool.clone());
let owner = "did:key:z6MkUnlockErrorPoolProofGGGGGGGGGGGGGGGGGG";
let name = "unlockerrpooltest";
let mut guard = store.acquire_write(owner, name).await.expect("acquire");
poison_guard_connection(&mut guard).await;
let size_before = store_pool.size();
assert!(size_before > 0, "the pool owns the guard's connection");
guard.release(false).await;
// The pool's size drops when the closed connection's slot is given up, which
// is not synchronous with `release` returning: poll rather than sleep.
wait_until(
|| store_pool.size() == size_before - 1,
"the connection that saw the unlock error to be closed rather than returned \
to the pool still holding the session lock",
)
.await;
}
/// F3b regression guard on the success path: a normal unlock keeps the
/// connection in the pool and marks the guard released, so the disposal branch
/// is confined to the error case.
#[sqlx::test]
async fn write_guard_release_success_keeps_the_connection_and_frees_the_lock(
pool: sqlx::PgPool,
) {
let dir = tempfile::TempDir::new().unwrap();
let store = RepoStore::for_testing(dir.path().to_path_buf(), pool.clone());
let owner = "did:key:z6MkUnlockOkProofHHHHHHHHHHHHHHHHHHHHHHHH";
let name = "unlockoktest";
let slug = owner.replace([':', '/'], "_");
let key = advisory_lock_key(&slug, name);
let mut checker = pool.acquire().await.expect("checker connection");
let guard = store.acquire_write(owner, name).await.expect("acquire");
let size_before = pool.size();
guard.release(false).await;
tokio::time::sleep(std::time::Duration::from_millis(400)).await;
assert_eq!(
pool.size(),
size_before,
"a successful unlock must leave the connection in the pool"
);
let (free,): (bool,) = sqlx::query_as("SELECT pg_try_advisory_lock($1)")
.bind(key)
.fetch_one(&mut *checker)
.await
.unwrap();
assert!(free, "the success path must still free the lock");
let _ = sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(key)
.execute(&mut *checker)
.await;
}
// ── the Drop backstop disposes its connection too (#174 U8) ─────────────
/// U8 (P2): the `Drop` backstop carries the same hazard F3b closed in `release`.
/// When the detached `pg_advisory_unlock` ERRORS on a live session, the async
/// block ends and drops the moved `PoolConnection`, which RETURNS it to the pool
/// while that session may still hold the lock, handing the next caller a
/// connection holding a lock nobody tracks. The errored connection must be closed
/// instead: that keeps it out of the pool and ends the session, which is what
/// frees the lock server-side.
///
/// Run against a pool with the idle reaper disabled, for the reason spelled out on
/// `pool_without_idle_reaper`: with the reaper in play the session dies on its own
/// about a second later and the assertion stops measuring the disposal.
///
/// Load-bearing: RED before the fix (the connection goes back to the pool, so the
/// size never drops and this times out), GREEN after.
#[sqlx::test]
async fn write_guard_drop_with_failing_unlock_does_not_return_the_connection(
pool: sqlx::PgPool,
) {
let dir = tempfile::TempDir::new().unwrap();
let store_pool = pool_without_idle_reaper(&pool).await;
let store = RepoStore::for_testing(dir.path().to_path_buf(), store_pool.clone());
let owner = "did:key:z6MkDropUnlockErrProofIIIIIIIIIIIIIIIIIIII";
let name = "dropunlockerrtest";
let slug = owner.replace([':', '/'], "_");
let key = advisory_lock_key(&slug, name);
// Distinct session for the probe, held out of the store's pool entirely.
let mut checker = pool.acquire().await.expect("checker connection");
let mut guard = store.acquire_write(owner, name).await.expect("acquire");
poison_guard_connection(&mut guard).await;
let size_before = store_pool.size();
assert!(size_before > 0, "the pool owns the guard's connection");
// The backstop shape: dropped without release(), with an unlock that errors.
drop(guard);
wait_until(
|| store_pool.size() == size_before - 1,
"the connection whose detached unlock errored to be closed rather than \
returned to the pool still holding the session lock",
)
.await;
wait_until_lock_free(
&mut checker,
key,
"an errored detached unlock must not leave the lock held: Drop must dispose \
of the connection so the session ends",
)
.await;
let _ = sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(key)
.execute(&mut *checker)
.await;
}
/// U8 regression guard on the success path: a detached unlock that SUCCEEDS must
/// still return the connection to the pool. Without this, "close the connection on
/// Drop" could be widened to "always close" and the test above would not notice.
#[sqlx::test]
async fn write_guard_drop_with_successful_unlock_keeps_the_connection(pool: sqlx::PgPool) {
let dir = tempfile::TempDir::new().unwrap();
let store_pool = pool_without_idle_reaper(&pool).await;
let store = RepoStore::for_testing(dir.path().to_path_buf(), store_pool.clone());
let owner = "did:key:z6MkDropUnlockOkProofJJJJJJJJJJJJJJJJJJJJ";
let name = "dropunlockoktest";
let slug = owner.replace([':', '/'], "_");
let key = advisory_lock_key(&slug, name);
let mut checker = pool.acquire().await.expect("checker connection");
let guard = store.acquire_write(owner, name).await.expect("acquire");
let size_before = store_pool.size();
assert!(size_before > 0, "the pool owns the guard's connection");
drop(guard);
// The connection goes back only once the detached unlock task has finished.
wait_until(
|| store_pool.num_idle() > 0,
"the detached unlock to finish and hand the connection back",
)
.await;
assert_eq!(
store_pool.size(),
size_before,
"a successful detached unlock must leave the connection in the pool"
);
wait_until_lock_free(
&mut checker,
key,
"the Drop backstop's successful unlock to free the lock",
)
.await;
let _ = sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(key)
.execute(&mut *checker)
.await;
}
/// U8, the off-runtime arm: with no Tokio runtime there is nothing to spawn the
/// unlock onto, and the connection has already been taken out of the guard, so the
/// old code dropped it with no unlock attempted at all, back to the pool, session
/// lock still held. Dropping a `PoolConnection` off a runtime is worse than that:
/// sqlx's return-to-pool path spawns, and its no-runtime fallback panics, so the
/// old arm also panicked in a destructor.
///
/// Reached by dropping the guard on a plain `std::thread`, where
/// `Handle::try_current()` fails.
///
/// Load-bearing: RED before the fix (the join sees the "requires a Tokio context"
/// panic from sqlx's return-to-pool spawn), GREEN after (`detach` gives up the
/// pool slot, so nothing is spawned and dropping the detached connection closes
/// the socket, which ends the session and frees the lock).
#[sqlx::test]
async fn write_guard_dropped_off_runtime_disposes_the_connection(pool: sqlx::PgPool) {
let dir = tempfile::TempDir::new().unwrap();
let store_pool = pool_without_idle_reaper(&pool).await;
let store = RepoStore::for_testing(dir.path().to_path_buf(), store_pool.clone());
let owner = "did:key:z6MkDropOffRuntimeProofKKKKKKKKKKKKKKKKKK";
let name = "dropoffruntimetest";
let slug = owner.replace([':', '/'], "_");
let key = advisory_lock_key(&slug, name);
let mut checker = pool.acquire().await.expect("checker connection");
let guard = store.acquire_write(owner, name).await.expect("acquire");
let size_before = store_pool.size();
assert!(size_before > 0, "the pool owns the guard's connection");
let dropped = std::thread::spawn(move || drop(guard)).join();
assert!(
dropped.is_ok(),
"dropping a write guard off a Tokio runtime must not panic"
);
wait_until(
|| store_pool.size() == size_before - 1,
"the connection of a guard dropped off a runtime to be disposed of rather \
than returned to the pool with no unlock attempted",
)
.await;
wait_until_lock_free(
&mut checker,
key,
"a guard dropped off a runtime to end its session so postgres drops the lock",
)
.await;
let _ = sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(key)
.execute(&mut *checker)
.await;
}
/// F4: releasing a guard that never took the lock (`locked == false`, the state
/// acquire_write leaves after a failed acquisition) must not unlock or panic.
#[sqlx::test]
async fn write_guard_release_when_not_locked_does_not_unlock_or_panic(pool: sqlx::PgPool) {
let dir = tempfile::TempDir::new().unwrap();
let owner = "did:key:z6MkNotLockedProofEEEEEEEEEEEEEEEEEEEEEE";
let name = "notlockedtest";
let slug = owner.replace([':', '/'], "_");
let key = advisory_lock_key(&slug, name);
let guard = RepoWriteGuard {
owner_slug: slug,
repo_name: name.to_string(),
local_path: dir.path().to_path_buf(),
lock_key: key,
conn: Some(pool.acquire().await.expect("conn")),
locked: false,
released: false,
tigris: None,
test_pre_unlock_gate: None,
};
// Must complete without panic and issue no unlock.
guard.release(false).await;
let mut checker = pool.acquire().await.expect("checker");
let (free,): (bool,) = sqlx::query_as("SELECT pg_try_advisory_lock($1)")
.bind(key)
.fetch_one(&mut *checker)
.await
.unwrap();
assert!(free, "release on an unlocked guard must not touch the key");
let _ = sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(key)
.execute(&mut *checker)
.await;
}
}
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+19
View File
@@ -31,6 +31,25 @@ impl TigrisClient {
})
}
/// Test-only constructor with an explicit S3 endpoint, region, and static
/// credentials — no env-var reads, so parallel tests cannot race each other's
/// `AWS_*` environment the way the env-based `new` would. Lets a test point
/// the client at a non-routable endpoint to exercise acquire-stall paths.
#[cfg(test)]
pub(crate) async fn for_testing_with_endpoint(bucket: &str, endpoint_url: &str) -> Self {
let creds = aws_sdk_s3::config::Credentials::new("test", "test", None, None, "test");
let config = aws_config::defaults(aws_config::BehaviorVersion::latest())
.endpoint_url(endpoint_url)
.region(aws_config::Region::new("auto"))
.credentials_provider(creds)
.load()
.await;
Self {
s3: S3Client::new(&config),
bucket: bucket.to_string(),
}
}
/// S3 key for a given repo: `repos/v1/{owner_slug}/{repo_name}.tar.zst`
fn repo_key(owner_slug: &str, repo_name: &str) -> String {
format!("repos/v1/{owner_slug}/{repo_name}.tar.zst")
+744 -94
View File
@@ -4,11 +4,321 @@
//! content is held back.
use crate::db::VisibilityRule;
use crate::git::store;
use crate::visibility::{visibility_check, Decision};
use anyhow::{Context, Result};
use std::collections::{BTreeSet, HashMap, HashSet};
use std::path::Path;
use std::process::Stdio;
use std::sync::mpsc;
use std::time::{Duration, Instant};
/// Fixed budget bounding the whole withheld-blob classification walk (#174 U3).
/// The walk is fast for a real repo; this bound exists to reap a hung or
/// pathologically slow git child so it cannot pin a served-git permit (the read
/// permit on the upload-pack serve path, the write permit on the receive-pack
/// post-push replication path) past the deadline. Every caller funnels through
/// `blob_paths`, so bounding here bounds both paths at one seam. Production callers
/// pass the operator-configured `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` instead; this
/// fixed budget only backs the `git_bin`-less test wrappers.
#[cfg(test)]
const WALK_TIMEOUT: Duration = Duration::from_secs(600);
/// How long the process-group watchdog waits after SIGTERM before escalating to
/// SIGKILL, giving a well-behaved git child time to clean up its `*.lock` files. Only
/// paid on a timeout (already the exceptional path).
#[cfg(unix)]
const WATCHDOG_TERM_GRACE: Duration = Duration::from_secs(1);
/// Run one git child under a shared `deadline` with process-group teardown,
/// BLOCKING, and return its stdout. The child runs in its own process group; a
/// watchdog thread SIGTERMs (lets git clean up its `*.lock` files), then SIGKILLs,
/// the whole group if the deadline passes before the child is reaped, so a hung or
/// slow git can pin neither a served-git permit nor a blocking thread past the
/// deadline (jatmn's "retain admission until they are reaped"). This is the
/// blocking-side counterpart of `smart_http::drive_git_child`, needed because the
/// walk's callers run it inside `spawn_blocking`, which an async timeout cannot
/// cancel. Returns [`crate::git::smart_http::GitServiceTimeout`] on the deadline so
/// the serve handler maps it to 504. `git_bin` is injectable so a fake `git` can
/// drive the teardown in tests without mutating the process-global PATH;
/// `stdin_bytes` feeds children that read stdin (empty for the arg-only children).
/// Returns true if `pid` (a process-group leader we spawned) has terminated, WITHOUT
/// reaping it. `waitid(..., WNOWAIT)` reports the exit state but leaves the child
/// waitable, so the caller's later `child.wait()` still collects the status and the
/// pid/pgid stays live until then — which is what keeps the watchdog's `kill(-pgid)`
/// teardown from ever racing a recycled pgid. Used to distinguish "the child actually
/// exited" from "the child merely closed stdout" after the drain returns (#174 P1-a).
#[cfg(unix)]
fn child_terminated_without_reaping(pid: i32) -> bool {
// SAFETY: waitid writes only into the zeroed siginfo and borrows no Rust memory;
// WNOWAIT leaves the child unreaped, WNOHANG makes the probe non-blocking.
let mut info: libc::siginfo_t = unsafe { std::mem::zeroed() };
let rc = unsafe {
libc::waitid(
libc::P_PID,
pid as libc::id_t,
&mut info,
libc::WEXITED | libc::WNOHANG | libc::WNOWAIT,
)
};
// rc == 0 with si_pid == 0 means "no state change yet" (still running); a non-zero
// si_pid means the child has entered a waitable, exited state. EINTR/other errors
// (rc != 0) are treated as "not yet terminated" and the caller re-polls.
rc == 0 && unsafe { info.si_pid() } != 0
}
#[cfg(unix)]
pub(crate) fn run_bounded_git_raw(
git_bin: &str,
args: &[&str],
repo_path: &Path,
stdin_bytes: &[u8],
deadline: Instant,
) -> Result<(std::process::ExitStatus, Vec<u8>, Vec<u8>)> {
use std::io::{Read, Write};
use std::os::unix::process::CommandExt;
use std::sync::mpsc::RecvTimeoutError;
let label = args.first().copied().unwrap_or("git");
let mut child = std::process::Command::new(git_bin)
.args(args)
.current_dir(repo_path)
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.process_group(0)
.spawn()
.with_context(|| format!("failed to spawn git {label}"))?;
// With process_group(0) the child leads its own group, so pgid == its pid.
let pgid = child.id() as i32;
// Watchdog: on the deadline, tear the WHOLE process group down — SIGTERM, a grace
// for a well-behaved child to clean up its `*.lock` files, then an UNCONDITIONAL
// SIGKILL of the group. It never stands down on leader-reap alone: a group member
// that ignores SIGTERM while the leader exits cleanly would otherwise escape the
// SIGKILL and keep running past the deadline (finding 3, #174). The main thread
// defers reaping the leader until this thread returns (see below), so the leader's
// pid is still unreaped while every `kill(-pgid)` fires and the pgid cannot have
// been recycled — which is why this no longer needs the old `reaped` short-circuit.
// Kept off the main thread because the main thread's stdout drain is exactly what
// blocks until a hung child is torn down.
let (done_tx, done_rx) = mpsc::channel::<()>();
let watchdog = std::thread::spawn(move || -> bool {
let wait = deadline.saturating_duration_since(Instant::now());
match done_rx.recv_timeout(wait) {
Ok(()) | Err(RecvTimeoutError::Disconnected) => false,
Err(RecvTimeoutError::Timeout) => {
// SAFETY: kill(2) takes only integers and borrows no Rust memory;
// ESRCH on an already-gone group is ignored.
unsafe { libc::kill(-pgid, libc::SIGTERM) };
// Fixed grace: because the main thread defers the leader's reap, a
// fully-exited group still shows a zombie leader here, so polling for
// ESRCH cannot detect early completion — just wait the grace, then
// SIGKILL. On a group of only zombies the SIGKILL is a harmless no-op;
// on a SIGTERM-ignoring member it is what actually kills it.
std::thread::sleep(WATCHDOG_TERM_GRACE);
unsafe { libc::kill(-pgid, libc::SIGKILL) };
// Brief settle so the SIGKILL is delivered before the main thread
// reaps the leader and frees the pgid. A wedged (D-state) member
// survives even SIGKILL — the documented residual, as in smart_http.
std::thread::sleep(Duration::from_millis(20));
if unsafe { libc::kill(-pgid, 0) } == 0 {
tracing::warn!(
pgid,
"withheld-walk git survived SIGKILL past the watchdog cap (uninterruptible I/O?)"
);
}
true
}
}
});
// Feed stdin on a writer thread and drain stderr on a reader thread so the main
// thread can drain stdout concurrently; writing all of stdin (or draining one
// pipe) before the others can deadlock once a pipe buffer fills.
let mut stdin = child.stdin.take();
let input = stdin_bytes.to_vec();
let writer = std::thread::spawn(move || {
if let Some(mut s) = stdin.take() {
let _ = s.write_all(&input);
}
});
let mut stderr = child.stderr.take().context("git stderr was not piped")?;
let err_reader = std::thread::spawn(move || {
let mut err = Vec::new();
let _ = stderr.read_to_end(&mut err);
err
});
let mut stdout = child.stdout.take().context("git stdout was not piped")?;
let mut out = Vec::new();
// Blocking drain, unblocked by the child closing stdout on exit. The watchdog's
// SIGTERM/SIGKILL is what makes a hung child exit; a git wedged in uninterruptible
// (D-state) I/O survives even SIGKILL, so this drain and the wait below can block
// until the kernel returns, pinning the walk thread and its permit. That residual
// is unreachable in userspace (no signal reaps a D-state process) and matches the
// async `reap_group_on_timeout`, which likewise only warns and gives up there.
let read_result = stdout.read_to_end(&mut out);
// The drain has returned, but that only means all stdout write ends are closed —
// NOT that the child has exited. A group member, or the leader itself, can close
// stdout and keep running; standing the watchdog down on the drain alone (as the
// old code did) would then let `child.wait()` block forever on that live child,
// past the deadline, pinning the walk thread and its permit (finding P1-a, #174).
// So stand the watchdog down only once the child has ACTUALLY terminated, detected
// WITHOUT reaping (waitid + WNOWAIT) so the leader's pid stays unreaped and its
// pgid un-recycled until the watchdog finishes and we join it below. Past the
// deadline the watchdog owns the teardown, so we stop polling and let it run the
// full SIGTERM -> grace -> SIGKILL; joining it before `child.wait()` keeps every
// `kill(-pgid)` firing while the pid is still unreaped and guarantees a
// stdout-closing-then-hanging member has been SIGKILLed rather than left running.
loop {
if child_terminated_without_reaping(pgid) {
let _ = done_tx.send(());
break;
}
if Instant::now() >= deadline {
break;
}
std::thread::sleep(Duration::from_millis(2));
}
let killed = watchdog.join().unwrap_or(false);
let status = child.wait().context("git wait failed")?;
let err = err_reader.join().unwrap_or_default();
let _ = writer.join();
read_result.context("failed to read git stdout")?;
// The watchdog runs off a wall clock that can race a child finishing right at the
// deadline. A child that exited on its own (success) is not a timeout even if the
// watchdog fired late; only a child that did not exit successfully is a genuine
// timeout, which keeps a walk completing at its budget from a spurious 504.
if killed && !status.success() {
return Err(crate::git::smart_http::GitServiceTimeout.into());
}
Ok((status, out, err))
}
/// Bounded git returning only stdout, `bail!`ing on any nonzero exit. The thin
/// wrapper the walk callers use. Probes that must distinguish exit classes —
/// `git cat-file` absence vs an object-store access failure — call
/// [`run_bounded_git_raw`] and classify the status/stderr themselves.
#[cfg(unix)]
pub(crate) fn run_bounded_git(
git_bin: &str,
args: &[&str],
repo_path: &Path,
stdin_bytes: &[u8],
deadline: Instant,
) -> Result<Vec<u8>> {
let label = args.first().copied().unwrap_or("git");
let (status, out, err) = run_bounded_git_raw(git_bin, args, repo_path, stdin_bytes, deadline)?;
if !status.success() {
anyhow::bail!("git {label} failed: {}", String::from_utf8_lossy(&err));
}
Ok(out)
}
/// Non-Unix fallback for [`run_bounded_git`]. Windows and other non-Unix targets
/// have no process-group teardown (`process_group(0)` / `kill(-pgid)` are Unix-only),
/// so this bounds a single child on its own: threads feed stdin and drain stderr
/// while the main thread drains stdout, and a watchdog thread kills the child at the
/// deadline (which closes stdout and unblocks the drain). The child is shared with
/// the watchdog behind a mutex that the main thread does NOT hold while draining, so
/// the watchdog can always acquire it to kill. Best-effort — it reaps only the direct
/// child, not a descendant group — which is why the hardened, group-aware path above
/// is gated to Unix, the only target the served node actually runs on (the Windows
/// release binary is best-effort / `continue-on-error` in CI). Kept in lockstep with
/// the Unix version's signature and result semantics so every caller compiles on all
/// targets (#174).
#[cfg(not(unix))]
pub(crate) fn run_bounded_git_raw(
git_bin: &str,
args: &[&str],
repo_path: &Path,
stdin_bytes: &[u8],
deadline: Instant,
) -> Result<(std::process::ExitStatus, Vec<u8>, Vec<u8>)> {
use std::io::{Read, Write};
use std::sync::mpsc::RecvTimeoutError;
let label = args.first().copied().unwrap_or("git");
let mut child = std::process::Command::new(git_bin)
.args(args)
.current_dir(repo_path)
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.with_context(|| format!("failed to spawn git {label}"))?;
let mut stdin = child.stdin.take();
let input = stdin_bytes.to_vec();
let writer = std::thread::spawn(move || {
if let Some(mut s) = stdin.take() {
let _ = s.write_all(&input);
}
});
let mut stderr = child.stderr.take().context("git stderr was not piped")?;
let err_reader = std::thread::spawn(move || {
let mut err = Vec::new();
let _ = stderr.read_to_end(&mut err);
err
});
let mut stdout = child.stdout.take().context("git stdout was not piped")?;
// Share the child with the watchdog. The main thread drains stdout WITHOUT
// holding this lock, so the watchdog can always acquire it to kill on timeout;
// killing closes stdout and unblocks the drain below.
let child = std::sync::Arc::new(std::sync::Mutex::new(child));
let (done_tx, done_rx) = mpsc::channel::<()>();
let watchdog = {
let child = child.clone();
std::thread::spawn(move || -> bool {
let wait = deadline.saturating_duration_since(Instant::now());
match done_rx.recv_timeout(wait) {
Ok(()) | Err(RecvTimeoutError::Disconnected) => false,
Err(RecvTimeoutError::Timeout) => {
if let Ok(mut c) = child.lock() {
let _ = c.kill();
}
true
}
}
})
};
let mut out = Vec::new();
let read_result = stdout.read_to_end(&mut out);
// The drain has returned (child exited or was killed), so taking the lock here
// cannot deadlock against the watchdog.
let status = child
.lock()
.expect("git child mutex poisoned")
.wait()
.context("git wait failed")?;
let _ = done_tx.send(());
let killed = watchdog.join().unwrap_or(false);
let err = err_reader.join().unwrap_or_default();
let _ = writer.join();
read_result.context("failed to read git stdout")?;
if killed && !status.success() {
return Err(crate::git::smart_http::GitServiceTimeout.into());
}
Ok((status, out, err))
}
/// Non-Unix thin wrapper matching the Unix [`run_bounded_git`] semantics.
#[cfg(not(unix))]
pub(crate) fn run_bounded_git(
git_bin: &str,
args: &[&str],
repo_path: &Path,
stdin_bytes: &[u8],
deadline: Instant,
) -> Result<Vec<u8>> {
let label = args.first().copied().unwrap_or("git");
let (status, out, err) = run_bounded_git_raw(git_bin, args, repo_path, stdin_bytes, deadline)?;
if !status.success() {
anyhow::bail!("git {label} failed: {}", String::from_utf8_lossy(&err));
}
Ok(out)
}
/// Fail closed unless every ref ultimately resolves to a commit (a ref pointing
/// directly at a blob or tree, or an annotated tag — even a nested one — of such
@@ -21,19 +331,15 @@ use std::path::Path;
/// Full peeling is why this is not `for-each-ref %(*objecttype)`, which
/// dereferences only one tag level and so misclassifies a tag-of-a-tag-of-a-
/// commit as a non-commit.
fn assert_all_refs_are_commits(repo_path: &Path) -> Result<()> {
let refs = std::process::Command::new("git")
.args(["for-each-ref", "--format=%(refname)"])
.current_dir(repo_path)
.output()
.context("git for-each-ref failed")?;
if !refs.status.success() {
anyhow::bail!(
"git for-each-ref failed: {}",
String::from_utf8_lossy(&refs.stderr)
);
}
let refs_stdout = String::from_utf8_lossy(&refs.stdout);
fn assert_all_refs_are_commits(repo_path: &Path, git_bin: &str, deadline: Instant) -> Result<()> {
let refs_out = run_bounded_git(
git_bin,
&["for-each-ref", "--format=%(refname)"],
repo_path,
b"",
deadline,
)?;
let refs_stdout = String::from_utf8_lossy(&refs_out);
let refnames: Vec<&str> = refs_stdout
.lines()
.map(str::trim)
@@ -43,59 +349,25 @@ fn assert_all_refs_are_commits(repo_path: &Path) -> Result<()> {
return Ok(());
}
// Peel every ref in one `git cat-file --batch-check` pass: one
// `<refname>^{}` query per line, one output line per input line, in order.
// The stdin write runs on a separate thread so this thread can drain stdout
// concurrently. cat-file echoes the full query on a `<query> missing` line,
// so output scales with refname length (not a fixed size per ref); writing
// all of stdin before reading any stdout would deadlock both pipes once the
// child's stdout buffer fills. Dropping `stdin` at the end of the closure
// sends EOF.
// Peel every ref in one `git cat-file --batch-check` pass: one `<refname>^{}`
// query per line, one output line per input line, in order. cat-file echoes the
// full query on a `<query> missing` line, so output scales with refname length;
// run_bounded_git drains stdout concurrently with the stdin write, so the pipe
// cannot deadlock, and the whole peel is bounded by the shared walk deadline.
let queries = refnames
.iter()
.map(|r| format!("{r}^{{}}"))
.collect::<Vec<_>>()
.join("\n");
use std::io::Write;
let mut child = std::process::Command::new("git")
.args(["cat-file", "--batch-check=%(objecttype)"])
.current_dir(repo_path)
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped())
.spawn()
.context("failed to spawn git cat-file")?;
// Feed stdin on a writer thread so this thread can drain stdout via
// wait_with_output concurrently; a None handle (the pipe vanished) becomes a
// broken-pipe write error. wait_with_output reaps the child unconditionally
// before any error is surfaced, so no path drops it unwaited (#53), and the
// writer is joined only after the drain so the join cannot deadlock.
let writer = child
.stdin
.take()
.map(|mut stdin| std::thread::spawn(move || stdin.write_all(queries.as_bytes())));
let peel_result = child.wait_with_output();
let write_result = match writer {
Some(handle) => handle
.join()
.map_err(|_| anyhow::anyhow!("git cat-file stdin writer thread panicked"))?,
None => Err(std::io::Error::new(
std::io::ErrorKind::BrokenPipe,
"git cat-file stdin unavailable",
)),
};
// Surface a write error only if the process didn't already fail with a
// clearer status.
let peel = peel_result.context("git cat-file failed")?;
if !peel.status.success() {
anyhow::bail!(
"git cat-file --batch-check failed: {}",
String::from_utf8_lossy(&peel.stderr)
);
}
write_result.context("failed to write to git cat-file stdin")?;
let peel_out = run_bounded_git(
git_bin,
&["cat-file", "--batch-check=%(objecttype)"],
repo_path,
queries.as_bytes(),
deadline,
)?;
let peel_stdout = String::from_utf8_lossy(&peel.stdout);
let peel_stdout = String::from_utf8_lossy(&peel_out);
let types: Vec<&str> = peel_stdout.lines().map(str::trim).collect();
// A short read means at least one ref went unclassified — fail closed.
if types.len() != refnames.len() {
@@ -147,47 +419,46 @@ fn assert_all_refs_are_commits(repo_path: &Path) -> Result<()> {
/// Fails closed: if commit enumeration or any tree walk fails, returns an error so
/// the caller aborts the serve/pin rather than producing a partial (under-withheld)
/// set.
fn blob_paths(repo_path: &Path) -> Result<Vec<(String, String)>> {
assert_all_refs_are_commits(repo_path)?;
fn blob_paths(repo_path: &Path, git_bin: &str, timeout: Duration) -> Result<Vec<(String, String)>> {
// One deadline spans the whole walk (the ref check, the HEAD probe, rev-list,
// and every per-commit ls-tree), so a slow or hung walk is bounded as a unit
// rather than granting each git child a fresh timeout.
let deadline = Instant::now() + timeout;
assert_all_refs_are_commits(repo_path, git_bin, deadline)?;
// Enumerate every reachable commit, not just ref tips. `--all` walks all refs;
// append HEAD so a detached HEAD (reachable by rev-list/upload-pack but in no
// ref) is still classified. When HEAD does not resolve (unborn branch on an
// empty repo) `--all` alone yields nothing, which is correct — no objects exist.
let head = store::head_commit(repo_path).context("resolve HEAD failed")?;
// empty repo) `--all` alone yields nothing, which is correct: no objects exist.
// The HEAD probe is a bounded `git rev-parse --verify HEAD` (a clean exit means
// HEAD resolves), replacing the previously unbounded `store::head_commit` child.
let head_resolves = run_bounded_git(
git_bin,
&["rev-parse", "--verify", "HEAD"],
repo_path,
b"",
deadline,
)
.is_ok();
let mut rev_args = vec!["rev-list", "--all"];
if head.is_some() {
if head_resolves {
rev_args.push("HEAD");
}
let commits = std::process::Command::new("git")
.args(&rev_args)
.current_dir(repo_path)
.output()
.context("git rev-list --all failed")?;
if !commits.status.success() {
anyhow::bail!(
"git rev-list --all failed: {}",
String::from_utf8_lossy(&commits.stderr)
);
}
let commits_stdout = String::from_utf8_lossy(&commits.stdout);
let commits_out = run_bounded_git(git_bin, &rev_args, repo_path, b"", deadline)?;
let commits_stdout = String::from_utf8_lossy(&commits_out);
let mut out: HashSet<(String, String)> = HashSet::new();
for commit in commits_stdout.lines() {
let commit = commit.trim();
if commit.is_empty() {
continue;
}
let listing = std::process::Command::new("git")
.args(["ls-tree", "-rz", commit])
.current_dir(repo_path)
.output()
.context("git ls-tree -rz failed")?;
if !listing.status.success() {
anyhow::bail!(
"git ls-tree -rz {commit} failed: {}",
String::from_utf8_lossy(&listing.stderr)
);
}
let listing_out = run_bounded_git(
git_bin,
&["ls-tree", "-rz", commit],
repo_path,
b"",
deadline,
)?;
// `-z` NUL-delimits records and emits paths raw; plain `git ls-tree -r`
// C-quotes any path with non-ASCII or special bytes (e.g. café.txt becomes
// "secret/caf\303\251.txt"), and that quoted literal would not match a
@@ -198,7 +469,7 @@ fn blob_paths(repo_path: &Path) -> Result<Vec<(String, String)>> {
// path (e.g. a non-UTF-8 directory name) with U+FFFD, and the mangled string
// would no longer match its deny rule — the same under-withholding class, one
// layer down. Fail closed instead so the caller aborts rather than leaks.
let Ok(listing_stdout) = std::str::from_utf8(&listing.stdout) else {
let Ok(listing_stdout) = std::str::from_utf8(&listing_out) else {
anyhow::bail!(
"git ls-tree -rz {commit} returned a non-UTF-8 path; \
refusing to produce a partial (under-withheld) set"
@@ -229,6 +500,7 @@ fn blob_paths(repo_path: &Path) -> Result<Vec<(String, String)>> {
///
/// The whole-repo "/" gate is handled by the caller before this function runs:
/// if "/" denies, the caller gets a 404 and never reaches the filtered serve.
#[cfg(test)]
pub fn withheld_blob_oids(
repo_path: &Path,
rules: &[VisibilityRule],
@@ -236,7 +508,33 @@ pub fn withheld_blob_oids(
owner_did: &str,
caller: Option<&str>,
) -> Result<HashSet<String>> {
let pairs = blob_paths(repo_path)?;
withheld_blob_oids_bounded(
repo_path,
"git",
WALK_TIMEOUT,
rules,
is_public,
owner_did,
caller,
)
}
/// [`withheld_blob_oids`] with an injectable `git_bin` and walk `timeout`. Served
/// handlers call this with the operator-configured git binary and
/// `GITLAWB_GIT_SERVICE_TIMEOUT_SECS`, so the whole walk is bounded by the same
/// budget as the other served-git ops and a fake `git` can drive its teardown in
/// tests. The `git_bin`-less wrapper above keeps the fixed [`WALK_TIMEOUT`] for the
/// classification tests that run against real git.
pub fn withheld_blob_oids_bounded(
repo_path: &Path,
git_bin: &str,
timeout: Duration,
rules: &[VisibilityRule],
is_public: bool,
owner_did: &str,
caller: Option<&str>,
) -> Result<HashSet<String>> {
let pairs = blob_paths(repo_path, git_bin, timeout)?;
Ok(withheld_from_pairs(
&pairs, rules, is_public, owner_did, caller,
))
@@ -304,6 +602,7 @@ pub fn replicable_objects(all: Vec<String>, withheld: &HashSet<String>) -> Vec<S
/// at an allowed path is included even when also denied elsewhere (its content
/// is public elsewhere). A dangling blob is absent from the reachable walk, so
/// it is never in this set and the fail-closed filter drops it (#99).
#[cfg(test)]
pub fn replicable_blob_set(
repo_path: &Path,
rules: &[VisibilityRule],
@@ -313,6 +612,21 @@ pub fn replicable_blob_set(
allowed_blob_set_for_caller(repo_path, rules, is_public, owner_did, None)
}
/// [`replicable_blob_set`] with an injectable `git_bin` and walk `timeout`, for the
/// fail-closed full-scan pin path on the receive-pack side.
pub fn replicable_blob_set_bounded(
repo_path: &Path,
git_bin: &str,
timeout: Duration,
rules: &[VisibilityRule],
is_public: bool,
owner_did: &str,
) -> Result<HashSet<String>> {
allowed_blob_set_for_caller_bounded(
repo_path, git_bin, timeout, rules, is_public, owner_did, None,
)
}
/// Reachable blob OIDs that visibility ALLOWS `caller` at some path. The
/// caller-aware generalization of `replicable_blob_set` (which is the anonymous
/// `caller = None` case). Used by `GET /ipfs/{cid}` to gate fail-closed against
@@ -325,6 +639,7 @@ pub fn replicable_blob_set(
/// elsewhere (its content is readable to this caller elsewhere). Trees and
/// commits are NOT included here; the caller decides per object type whether
/// the allow-set applies (it does not for trees/commits — KTD3).
#[cfg(test)]
pub fn allowed_blob_set_for_caller(
repo_path: &Path,
rules: &[VisibilityRule],
@@ -332,7 +647,29 @@ pub fn allowed_blob_set_for_caller(
owner_did: &str,
caller: Option<&str>,
) -> Result<HashSet<String>> {
let pairs = blob_paths(repo_path)?;
allowed_blob_set_for_caller_bounded(
repo_path,
"git",
WALK_TIMEOUT,
rules,
is_public,
owner_did,
caller,
)
}
/// [`allowed_blob_set_for_caller`] with an injectable `git_bin` and walk `timeout`,
/// for the `GET /ipfs/{cid}` gate.
pub fn allowed_blob_set_for_caller_bounded(
repo_path: &Path,
git_bin: &str,
timeout: Duration,
rules: &[VisibilityRule],
is_public: bool,
owner_did: &str,
caller: Option<&str>,
) -> Result<HashSet<String>> {
let pairs = blob_paths(repo_path, git_bin, timeout)?;
let mut allowed = HashSet::new();
for (oid, path) in &pairs {
if visibility_check(rules, is_public, owner_did, caller, path) == Decision::Allow {
@@ -365,14 +702,28 @@ pub fn replicable_objects_fail_closed(
/// owner plus any reader DID that `visibility_check` Allows at some path the
/// blob appears at. Least-privilege: a reader of one private subtree is not a
/// recipient of a blob that only lives in another.
#[cfg(test)]
pub fn withheld_blob_recipients(
repo_path: &Path,
rules: &[VisibilityRule],
is_public: bool,
owner_did: &str,
) -> Result<HashMap<String, BTreeSet<String>>> {
withheld_blob_recipients_bounded(repo_path, "git", WALK_TIMEOUT, rules, is_public, owner_did)
}
/// [`withheld_blob_recipients`] with an injectable `git_bin` and walk `timeout`, for
/// the receive-pack encrypt-then-pin path.
pub fn withheld_blob_recipients_bounded(
repo_path: &Path,
git_bin: &str,
timeout: Duration,
rules: &[VisibilityRule],
is_public: bool,
owner_did: &str,
) -> Result<HashMap<String, BTreeSet<String>>> {
// One history walk feeds both the withheld set and the recipient mapping.
let pairs = blob_paths(repo_path)?;
let pairs = blob_paths(repo_path, git_bin, timeout)?;
let withheld = withheld_from_pairs(&pairs, rules, is_public, owner_did, None);
if withheld.is_empty() {
return Ok(HashMap::new());
@@ -402,6 +753,300 @@ pub fn withheld_blob_recipients(
#[cfg(test)]
mod tests {
use super::*;
/// Write an executable fake `git` shell script into `dir` and return its path,
/// so a test can drive the walk's process-group teardown without a real git and
/// without mutating the process-global PATH (the crate's only injection seam).
#[cfg(unix)]
fn write_fake_git(dir: &Path, body: &str) -> String {
use std::os::unix::fs::PermissionsExt;
let p = dir.join("fakegit");
std::fs::write(&p, body).unwrap();
let mut perm = std::fs::metadata(&p).unwrap().permissions();
perm.set_mode(0o755);
std::fs::set_permissions(&p, perm).unwrap();
p.to_str().unwrap().to_string()
}
/// #174 U3: the withheld-blob walk is bounded at the shared `blob_paths` seam, so
/// a hung git child cannot pin the caller's permit past the deadline. A fake git
/// that hangs on `rev-list` must make `blob_paths` return `GitServiceTimeout`
/// within the watchdog budget (not block for the child's lifetime), and the
/// child's process group must be reaped (its recorded leader PID gone). Every
/// caller (upload-pack serve, receive-pack replication) funnels through
/// `blob_paths`, so this seam-level proof covers both permit pools. Neutralize
/// the watchdog SIGTERM and this hangs past the recv budget (RED).
#[cfg(unix)]
#[test]
fn blob_paths_times_out_and_reaps_a_hung_walk() {
use std::time::Duration;
let tmp = TempDir::new().unwrap();
// Fast on every stage except rev-list, which records its own (group-leader)
// PID and then hangs. `sleep 30` bounds the worst case if the watchdog is
// ever broken, so a regression cannot wedge the suite for 300s.
let body = "#!/bin/sh\ncase \"$1\" in\n rev-list) echo $$ > revlist.pid ; sleep 30 ;;\n rev-parse) echo deadbeef ;;\n *) : ;;\nesac\nexit 0\n";
let git_bin = write_fake_git(tmp.path(), body);
// Run the walk on a thread with a short budget; the recv_timeout succeeding
// is itself proof the walk did not block on the hung child.
let (tx, rx) = mpsc::channel();
let path = tmp.path().to_path_buf();
std::thread::spawn(move || {
let _ = tx.send(blob_paths(&path, &git_bin, Duration::from_millis(200)));
});
let result = rx.recv_timeout(Duration::from_secs(10)).expect(
"blob_paths must return within the watchdog budget, not hang on a stuck git child",
);
let err = result.expect_err("a hung rev-list must abort the walk with an error");
assert!(
err.downcast_ref::<crate::git::smart_http::GitServiceTimeout>()
.is_some(),
// `{err:#}` prints the whole anyhow chain. Plain `{err}` shows only the top
// context ("failed to spawn git for-each-ref") and drops the underlying io
// error, which left a real beta-lane CI failure undiagnosable.
"a hung walk must abort with GitServiceTimeout (mapped to 504), got: {err:#}"
);
// The recorded process-group leader must be gone: the watchdog reaps the
// whole group before blob_paths returns, so no orphaned git lingers.
let pid: i32 = std::fs::read_to_string(tmp.path().join("revlist.pid"))
.expect("the fake git must have recorded its rev-list PID")
.trim()
.parse()
.expect("recorded PID must parse");
let mut gone = false;
for _ in 0..200 {
// SAFETY: kill(2) with signal 0 only probes existence; ESRCH (-1) means
// the process is gone. Borrows no Rust memory.
if unsafe { libc::kill(pid, 0) } != 0 {
gone = true;
break;
}
std::thread::sleep(Duration::from_millis(10));
}
assert!(
gone,
"the hung git child (pid {pid}) must be reaped, not orphaned, after the walk aborts"
);
}
/// #174 (F1 status-gate, vetted by execution): a child that exits SUCCESSFULLY is
/// never reported as a timeout even when the watchdog fires, so a walk finishing
/// right at its deadline is not a spurious 504. The fake only exits when signalled
/// and exits 0 on SIGTERM, so with a deadline already elapsed the watchdog always
/// reaches its kill path (killed == true) yet the child's status is success.
/// Drop the `!status.success()` guard and this returns GitServiceTimeout (RED).
#[cfg(unix)]
#[test]
fn run_bounded_git_success_at_the_deadline_is_not_a_timeout() {
use std::time::{Duration, Instant};
let tmp = TempDir::new().unwrap();
let body = "#!/bin/sh\ntrap 'exit 0' TERM\nsleep 30 &\nwait\n";
let git_bin = write_fake_git(tmp.path(), body);
let out = run_bounded_git(
&git_bin,
&["rev-list"],
tmp.path(),
b"",
Instant::now() + Duration::from_millis(100),
);
assert!(
out.is_ok(),
"a child that exited successfully must not be reported as a timeout even if the watchdog fired: {out:?}"
);
}
/// #174 (F3, vetted by execution): a child that IGNORES SIGTERM is still reaped
/// via the watchdog's SIGKILL escalation, so it cannot pin the walk thread or its
/// permit. The fake traps SIGTERM and keeps sleeping; run_bounded_git must still
/// return (via SIGKILL at the grace step) with a timeout error and the group must
/// be gone. (A truly uninterruptible D-state child, which no signal can reap, is
/// the documented residual this teardown, like the async twin, cannot cover.)
#[cfg(unix)]
#[test]
fn run_bounded_git_reaps_a_sigterm_ignoring_child_via_sigkill() {
use std::time::{Duration, Instant};
let tmp = TempDir::new().unwrap();
let body = "#!/bin/sh\ntrap '' TERM\necho $$ > pid\nwhile true; do sleep 1; done\n";
let git_bin = write_fake_git(tmp.path(), body);
let (tx, rx) = std::sync::mpsc::channel();
let path = tmp.path().to_path_buf();
std::thread::spawn(move || {
let _ = tx.send(run_bounded_git(
&git_bin,
&["rev-list"],
&path,
b"",
Instant::now() + Duration::from_millis(100),
));
});
let out = rx
.recv_timeout(Duration::from_secs(10))
.expect("run_bounded_git must return via SIGKILL even for a SIGTERM-ignoring child");
assert!(
out.is_err(),
"a SIGTERM-ignoring child killed by SIGKILL is a timeout, not a success: {out:?}"
);
let pid: i32 = std::fs::read_to_string(tmp.path().join("pid"))
.unwrap()
.trim()
.parse()
.unwrap();
let mut gone = false;
for _ in 0..300 {
if unsafe { libc::kill(pid, 0) } != 0 {
gone = true;
break;
}
std::thread::sleep(Duration::from_millis(10));
}
assert!(
gone,
"the SIGTERM-ignoring child (pid {pid}) must be reaped via SIGKILL, not left running"
);
}
/// #174 finding 3 (jatmn/CodeRabbit): a group MEMBER that ignores SIGTERM must
/// still be SIGKILLed even when the group LEADER exits cleanly on SIGTERM. The
/// leader traps SIGTERM to exit 0, but first spawns a descendant (`sh -c`, so its
/// `$$` is its OWN pid — a `( )` subshell's `$$` is the parent's) that ignores
/// SIGTERM and closes its inherited stdout/stderr. When the watchdog SIGTERMs the
/// group, the leader exits, its stdout closes, the main drain unblocks, and the
/// leader is reaped — the exact window a `reaped`-gated watchdog stands down in,
/// before escalating to SIGKILL. The descendant must be dead when run_bounded_git
/// returns; a teardown that stands down on leader-reap leaves it running (RED).
#[cfg(unix)]
#[test]
fn run_bounded_git_sigkills_a_sigterm_ignoring_descendant_after_leader_exits() {
use std::time::{Duration, Instant};
let tmp = TempDir::new().unwrap();
// Both loops are bounded (~30s) so a broken teardown cannot leak a permanent
// orphan or wedge the suite; the assertion fires well before then.
let body = "#!/bin/sh\n\
case \"$1\" in\n\
rev-list)\n\
sh -c 'trap \"\" TERM; echo $$ > desc.pid; exec 1>&- 2>&-; i=0; while [ $i -lt 30 ]; do sleep 1; i=$((i+1)); done' &\n\
trap 'exit 0' TERM\n\
i=0; while [ $i -lt 30 ]; do sleep 1; i=$((i+1)); done ;;\n\
*) : ;;\n\
esac\n";
let git_bin = write_fake_git(tmp.path(), body);
let (tx, rx) = std::sync::mpsc::channel();
let path = tmp.path().to_path_buf();
std::thread::spawn(move || {
let _ = tx.send(run_bounded_git(
&git_bin,
&["rev-list"],
&path,
b"",
Instant::now() + Duration::from_millis(100),
));
});
let _ = rx
.recv_timeout(Duration::from_secs(10))
.expect("run_bounded_git must return within the watchdog budget");
// Wait for the descendant to record its OWN pid, then assert it is gone.
let desc_pid_path = tmp.path().join("desc.pid");
let mut desc: Option<i32> = None;
for _ in 0..200 {
if let Some(p) = std::fs::read_to_string(&desc_pid_path)
.ok()
.and_then(|s| s.trim().parse::<i32>().ok())
{
desc = Some(p);
break;
}
std::thread::sleep(Duration::from_millis(10));
}
let desc = desc.expect("the fake leader must have spawned and recorded a descendant");
let mut gone = false;
for _ in 0..300 {
if unsafe { libc::kill(desc, 0) } != 0 {
gone = true;
break;
}
std::thread::sleep(Duration::from_millis(10));
}
// Kill it regardless so a RED run leaks no orphan.
unsafe { libc::kill(desc, libc::SIGKILL) };
assert!(
gone,
"a SIGTERM-ignoring descendant (pid {desc}) must be SIGKILLed even after the leader exits cleanly, not orphaned"
);
}
/// #174 U1 (P1-a, RED-before/GREEN-after): the group LEADER closes its own
/// stdout/stderr BEFORE the deadline and then keeps running. On the pre-fix code
/// the stdout drain returns EOF early, `done_tx.send` stands the watchdog down
/// before it ever fires (`recv` gets `Ok` -> `false`, no kill), and `child.wait()`
/// then blocks on the still-alive leader — pinning the walk thread and its read/
/// write permit past the deadline, bypassing GITLAWB_GIT_SERVICE_TIMEOUT_SECS.
/// This is distinct from the descendant case above: there the leader sleeps until
/// the deadline so the watchdog DOES time out; here the drain-EOF races ahead of
/// the deadline. The fix keeps the watchdog armed until the child is actually
/// reaped, so the deadline SIGTERM still fires and the call returns within budget.
/// A pre-fix build blocks on `child.wait()` past the recv budget (RED).
#[cfg(unix)]
#[test]
fn run_bounded_git_reaps_a_leader_that_closes_stdout_then_hangs() {
use std::time::{Duration, Instant};
let tmp = TempDir::new().unwrap();
// rev-list records its (leader) pid, closes stdout+stderr so the drain EOFs
// immediately, then sleeps without trapping TERM. `sleep 30` bounds the worst
// case so a RED run cannot wedge the suite; the recv budget fires first.
//
// The deadline below is deliberately far larger than the work it bounds: the
// watchdog tears the whole group down at the deadline, so a deadline tight
// enough to race `/bin/sh` reaching its first statement kills the leader
// BEFORE it records its pid, and the `leader.pid` read below then fails with
// NotFound. That is a harness race, not a product failure, and it is exactly
// what a loaded CI runner reproduced (test (beta), 2026-08-04). Sizing it at
// 2s keeps the property under test intact, since the pre-fix build blocks on
// `child.wait()` for the child's full `sleep 30` and still trips the 10s recv
// budget below (RED), while leaving a 20x margin over the ~100ms that a
// healthy start actually needs.
let body = "#!/bin/sh\ncase \"$1\" in\n rev-list) echo $$ > leader.pid; exec 1>&- 2>&-; sleep 30 ;;\n *) : ;;\nesac\nexit 0\n";
let git_bin = write_fake_git(tmp.path(), body);
let (tx, rx) = std::sync::mpsc::channel();
let path = tmp.path().to_path_buf();
std::thread::spawn(move || {
let _ = tx.send(run_bounded_git(
&git_bin,
&["rev-list"],
&path,
b"",
Instant::now() + Duration::from_secs(2),
));
});
let out = rx.recv_timeout(Duration::from_secs(10)).expect(
"run_bounded_git must return within the watchdog budget when the leader closes stdout then hangs, not block on child.wait()",
);
assert!(
out.is_err(),
"a leader killed at the deadline (no TERM trap) is a timeout, not a success: {out:?}"
);
let pid: i32 = std::fs::read_to_string(tmp.path().join("leader.pid"))
.unwrap()
.trim()
.parse()
.unwrap();
let mut gone = false;
for _ in 0..300 {
if unsafe { libc::kill(pid, 0) } != 0 {
gone = true;
break;
}
std::thread::sleep(Duration::from_millis(10));
}
// Kill it regardless so a RED run leaks no orphan.
unsafe { libc::kill(pid, libc::SIGKILL) };
assert!(
gone,
"the hung leader (pid {pid}) must be killed and reaped at the deadline, not left running"
);
}
use crate::db::VisibilityMode;
use chrono::Utc;
use std::process::Command;
@@ -735,7 +1380,12 @@ mod tests {
String::from_utf8_lossy(&out.stdout).trim().to_string()
};
let all_blobs = crate::git::push_delta::all_blob_oids(&work).unwrap();
let all_blobs = crate::git::push_delta::all_blob_oids(
&work,
"git",
std::time::Instant::now() + std::time::Duration::from_secs(600),
)
.unwrap();
assert!(
all_blobs.contains(&dangling_oid),
"precondition: the dangling blob is in the all-objects universe"
+164
View File
@@ -14,6 +14,68 @@ use subscription::SubscriptionRoot;
pub type GitlawbSchema = Schema<QueryRoot, MutationRoot, SubscriptionRoot>;
/// Client-facing message for GraphQL resolver failures that wrap a real
/// `sqlx::Error`. The real error is logged server-side; never put sqlx/Postgres
/// detail in the GraphQL `errors` array (#250).
///
/// Kept as its own constant on this PR's base (main still renders
/// `AppError::Db` with `e.to_string()`). If/when #247's `DB_ERROR_MESSAGE`
/// lands, fold this into that shared constant.
pub const GRAPHQL_DB_ERROR_MESSAGE: &str = "a database error occurred";
fn anyhow_has_sqlx(e: &anyhow::Error) -> bool {
e.chain().any(|c| c.downcast_ref::<sqlx::Error>().is_some())
}
/// Map an `anyhow` failure from the db layer to a GraphQL error.
///
/// - Real DB faults (`sqlx::Error` anywhere in the chain) → opaque client
/// message + `error!` log with the full `{e:#}` cause chain.
/// - Application/business errors (e.g. claim race, not-in-claimed-state) →
/// keep the actionable message; log at `warn!` so they are not mistaken for
/// infrastructure failures (#250 review).
pub(crate) fn graphql_db_err(e: anyhow::Error) -> async_graphql::Error {
if anyhow_has_sqlx(&e) {
tracing::error!(error = %format!("{e:#}"), "graphql database error");
async_graphql::Error::new(GRAPHQL_DB_ERROR_MESSAGE)
} else {
tracing::warn!(error = %format!("{e:#}"), "graphql application error");
async_graphql::Error::new(e.to_string())
}
}
/// Map an `AppError` from a shared collector (e.g. ref-update feed) to a
/// GraphQL error.
///
/// Fail closed: only explicitly curated variants surface their `Display`
/// text. Unnamed variants (including `Git`, which may embed on-disk paths)
/// render opaque so a future addition cannot leak by default (#255 review).
pub(crate) fn graphql_app_err(e: crate::error::AppError) -> async_graphql::Error {
match e {
crate::error::AppError::Db(sql) => graphql_db_err(sql.into()),
crate::error::AppError::Internal(err) => {
tracing::error!(error = %format!("{err:#}"), "graphql internal error");
async_graphql::Error::new(GRAPHQL_DB_ERROR_MESSAGE)
}
// Curated client-safe variants — `Display` is intentional API text.
safe @ (crate::error::AppError::RepoNotFound(_)
| crate::error::AppError::RepoExists(_)
| crate::error::AppError::NotFound(_)
| crate::error::AppError::Unauthorized(_)
| crate::error::AppError::Forbidden(_)
| crate::error::AppError::BadRequest(_)
| crate::error::AppError::TooManyRequests(_)
| crate::error::AppError::Incomplete(_)) => {
tracing::warn!(error = %safe, "graphql application error");
async_graphql::Error::new(safe.to_string())
}
other => {
tracing::error!(error = %other, "graphql unclassified AppError (opaque)");
async_graphql::Error::new(GRAPHQL_DB_ERROR_MESSAGE)
}
}
}
pub fn build_schema(
db: Arc<Db>,
ref_update_tx: tokio::sync::broadcast::Sender<RefUpdateBroadcast>,
@@ -25,3 +87,105 @@ pub fn build_schema(
.data(task_event_tx)
.finish()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn graphql_db_err_opaques_sqlx_chain() {
let leak = "error returned from database: column \"is_public\" does not exist";
// Context layer must not hide sqlx from the chain walk (db helpers
// wrap with `.context(...)` in several places).
let err = graphql_db_err(
anyhow::Error::from(sqlx::Error::Protocol(leak.into())).context("loading repos"),
);
assert_eq!(err.message, GRAPHQL_DB_ERROR_MESSAGE);
assert!(!err.message.contains("is_public"));
assert!(!err.message.contains(leak));
assert!(!err.message.contains("loading repos"));
}
#[test]
fn graphql_db_err_keeps_business_message() {
let msg = "task not claimable: not found or already claimed";
let err = graphql_db_err(anyhow::anyhow!("{msg}"));
assert_eq!(err.message, msg);
}
#[test]
fn graphql_app_err_opaques_db_and_internal() {
let leak = "column \"is_public\" does not exist";
let db_err = graphql_app_err(crate::error::AppError::Db(sqlx::Error::Protocol(
leak.into(),
)));
assert_eq!(db_err.message, GRAPHQL_DB_ERROR_MESSAGE);
assert!(!db_err.message.contains("is_public"));
let internal = graphql_app_err(crate::error::AppError::Internal(anyhow::anyhow!(
"loading repo: {leak}"
)));
assert_eq!(internal.message, GRAPHQL_DB_ERROR_MESSAGE);
assert!(!internal.message.contains("is_public"));
}
#[test]
fn graphql_app_err_keeps_safe_variant_messages() {
let err = graphql_app_err(crate::error::AppError::NotFound("widget".into()));
assert!(
err.message.contains("widget"),
"safe NotFound message must reach the client: {}",
err.message
);
assert_ne!(err.message, GRAPHQL_DB_ERROR_MESSAGE);
let err = graphql_app_err(crate::error::AppError::BadRequest("bad cid".into()));
assert!(
err.message.contains("bad cid"),
"safe BadRequest message must reach the client: {}",
err.message
);
assert_ne!(err.message, GRAPHQL_DB_ERROR_MESSAGE);
}
#[test]
fn graphql_app_err_opaques_unclassified_variants() {
// `Git` may embed on-disk paths from libgit2; fail closed.
let path = "/var/lib/gitlawb/repos/owner/secret.git";
let err = graphql_app_err(crate::error::AppError::Git(format!(
"failed to open '{path}'"
)));
assert_eq!(err.message, GRAPHQL_DB_ERROR_MESSAGE);
assert!(!err.message.contains(path));
assert!(!err.message.contains("failed to open"));
}
/// Every `.map_err(` in the GraphQL query/mutation resolvers must route
/// through the opaque helpers, or discard the error (`|_|`). Same source-
/// scrape pattern as `api::authz_guard` (#255 review).
#[test]
fn every_graphql_map_err_uses_opaque_helpers() {
for (file, src) in [
("query.rs", include_str!("query.rs")),
("mutation.rs", include_str!("mutation.rs")),
] {
for (lineno, line) in src.lines().enumerate() {
let code = line.split("//").next().unwrap_or(line);
let Some(idx) = code.find(".map_err(") else {
continue;
};
let after = code[idx + ".map_err(".len()..].trim_start();
let ok = after.starts_with("crate::graphql::graphql_db_err")
|| after.starts_with("crate::graphql::graphql_app_err")
|| after.starts_with("|_|")
|| after.starts_with("|_ ");
assert!(
ok,
"{file}:{}: `.map_err(` must use graphql_db_err / graphql_app_err \
or discard (`|_|`): {line}",
lineno + 1
);
}
}
}
}
+50 -8
View File
@@ -54,7 +54,7 @@ impl MutationRoot {
};
db.create_task(&task)
.await
.map_err(|e| async_graphql::Error::new(e.to_string()))?;
.map_err(crate::graphql::graphql_db_err)?;
Ok(AgentTaskType::from(task))
}
@@ -76,7 +76,7 @@ impl MutationRoot {
let task = db
.claim_task(&id, &assignee_did)
.await
.map_err(|e| async_graphql::Error::new(e.to_string()))?;
.map_err(crate::graphql::graphql_db_err)?;
let _ = tx.send(TaskEventBroadcast {
task_id: id,
old_status: "pending".to_string(),
@@ -108,7 +108,7 @@ impl MutationRoot {
let existing = db
.get_task(&id)
.await
.map_err(|e| async_graphql::Error::new(e.to_string()))?
.map_err(crate::graphql::graphql_db_err)?
.ok_or_else(|| async_graphql::Error::new("task not found"))?;
if !crate::api::did_matches(caller, existing.assignee_did.as_deref().unwrap_or_default()) {
return Err(async_graphql::Error::new(
@@ -118,7 +118,7 @@ impl MutationRoot {
let task = db
.finish_task(&id, "completed", input.result.as_deref())
.await
.map_err(|e| async_graphql::Error::new(e.to_string()))?;
.map_err(crate::graphql::graphql_db_err)?;
let _ = tx.send(TaskEventBroadcast {
task_id: id,
old_status: "claimed".to_string(),
@@ -149,7 +149,7 @@ impl MutationRoot {
let existing = db
.get_task(&id)
.await
.map_err(|e| async_graphql::Error::new(e.to_string()))?
.map_err(crate::graphql::graphql_db_err)?
.ok_or_else(|| async_graphql::Error::new("task not found"))?;
if !crate::api::did_matches(caller, existing.assignee_did.as_deref().unwrap_or_default()) {
return Err(async_graphql::Error::new(
@@ -160,7 +160,7 @@ impl MutationRoot {
let task = db
.finish_task(&id, "failed", Some(&reason))
.await
.map_err(|e| async_graphql::Error::new(e.to_string()))?;
.map_err(crate::graphql::graphql_db_err)?;
let _ = tx.send(TaskEventBroadcast {
task_id: id,
old_status: "claimed".to_string(),
@@ -218,8 +218,9 @@ mod tests {
errors(&resp)
);
// 3. Signed as the claimed assignee → passes the auth gate (any remaining
// error is the missing task, not an auth error).
// 3. Signed as the claimed assignee → passes the auth gate. The missing
// task is a business error from claim_task, not a sqlx fault, so the
// actionable message must survive (not the opaque DB string) (#250).
let resp = schema
.execute(Request::new(&q).data(AuthenticatedDid(assignee.into())))
.await;
@@ -228,6 +229,47 @@ mod tests {
!errs.contains("authentication required") && !errs.contains("authenticated signer"),
"matching signer must pass the auth gate: {errs}"
);
assert!(
errs.contains("task not claimable"),
"claim race / missing task must keep its business message: {errs}"
);
assert!(
!errs.contains(crate::graphql::GRAPHQL_DB_ERROR_MESSAGE),
"business error must not be rewritten as opaque DB error: {errs}"
);
}
/// #250: mutation DB faults must be opaque; create_task hits agent_tasks.
#[sqlx::test]
async fn create_task_db_error_message_is_opaque(pool: PgPool) {
let state = crate::test_support::test_state(pool.clone()).await;
sqlx::query("ALTER TABLE agent_tasks DROP COLUMN status")
.execute(&pool)
.await
.unwrap();
let delegator = "did:key:zGQLDELEGATORAAAAAAAAAAAAAAAAAAAAAAAAAA";
let q = format!(
r#"mutation {{
createTask(
delegatorDid: "{delegator}",
input: {{ kind: "build", capability: "repo:write" }}
) {{ id }}
}}"#
);
let resp = state
.graphql_schema
.execute(Request::new(&q).data(AuthenticatedDid(delegator.into())))
.await;
let errs = errors(&resp);
assert!(
errs.contains(crate::graphql::GRAPHQL_DB_ERROR_MESSAGE),
"sqlx fault must be opaque: {errs}"
);
assert!(
!errs.contains("column") && !errs.contains("status"),
"schema text leaked: {errs}"
);
}
/// Adversarial-review GATE-1 (GraphQL): completing a task requires being its
+102 -7
View File
@@ -14,7 +14,7 @@ impl QueryRoot {
let repos = db
.list_all_repos_deduped()
.await
.map_err(|e| async_graphql::Error::new(e.to_string()))?;
.map_err(crate::graphql::graphql_db_err)?;
// Apply the same "/" visibility gate the REST/per-repo endpoints use so
// this surface does not enumerate private repos (#97). The caller DID is
@@ -27,7 +27,7 @@ impl QueryRoot {
let rules_by_repo = db
.list_visibility_rules_for_repos(&ids)
.await
.map_err(|e| async_graphql::Error::new(e.to_string()))?;
.map_err(crate::graphql::graphql_db_err)?;
Ok(repos
.into_iter()
@@ -71,7 +71,7 @@ impl QueryRoot {
let updates =
crate::api::events::collect_visible_ref_updates(db, repo.as_deref(), limit, caller)
.await
.map_err(|e| async_graphql::Error::new(e.to_string()))?;
.map_err(crate::graphql::graphql_app_err)?;
// Resolve the trusted display owner_did per row, identical to the REST
// feed: the stored wire value is untrusted, so it is echoed only when it
@@ -86,7 +86,7 @@ impl QueryRoot {
let owner_dids = db
.resolve_ref_update_owner_dids(&pairs)
.await
.map_err(|e| async_graphql::Error::new(e.to_string()))?;
.map_err(crate::graphql::graphql_db_err)?;
let resolved: Vec<RefUpdateType> = updates
.into_iter()
@@ -110,13 +110,21 @@ impl QueryRoot {
ctx: &Context<'_>,
status: Option<String>,
assignee_did: Option<String>,
#[graphql(default = 50)] limit: i64,
#[graphql(
default = 50,
desc = "Max 200; larger requests are clamped to 200 (no error). Negative values clamp to 0."
)]
limit: i64,
) -> Result<Vec<AgentTaskType>> {
let db = ctx.data_unchecked::<Arc<Db>>();
// Clamp before SQL: a negative LIMIT is a client fault that Postgres
// rejects with 2201W, which would otherwise trip the opaque DB path
// and write an error-level log on every probe (#250 review).
let limit = limit.clamp(0, 200);
let tasks = db
.list_tasks(status.as_deref(), assignee_did.as_deref(), limit)
.await
.map_err(|e| async_graphql::Error::new(e.to_string()))?;
.map_err(crate::graphql::graphql_db_err)?;
Ok(tasks.into_iter().map(AgentTaskType::from).collect())
}
@@ -125,7 +133,7 @@ impl QueryRoot {
let t = db
.get_task(&id)
.await
.map_err(|e| async_graphql::Error::new(e.to_string()))?;
.map_err(crate::graphql::graphql_db_err)?;
Ok(t.map(AgentTaskType::from))
}
}
@@ -419,4 +427,91 @@ mod tests {
let q = r#"{ refUpdates { repo } }"#;
assert_eq!(count(&authed(&schema, q, "did:key:z6MkQuar").await), 0);
}
/// #250: anonymous GraphQL query DB failures must not leak sqlx/schema text.
#[sqlx::test]
async fn repos_query_db_error_message_is_opaque(pool: PgPool) {
let db = db(pool.clone()).await;
db.create_repo(&repo("r1", OWNER, "widget", true))
.await
.unwrap();
sqlx::query("ALTER TABLE repos DROP COLUMN is_public")
.execute(&pool)
.await
.unwrap();
let schema = schema(db);
let resp = anon(&schema, "{ repos { name ownerDid } }").await;
assert!(
!resp.errors.is_empty(),
"DB failure must surface as a GraphQL error"
);
for err in &resp.errors {
assert_eq!(
err.message,
crate::graphql::GRAPHQL_DB_ERROR_MESSAGE,
"raw DB detail leaked into GraphQL error: {}",
err.message
);
assert!(
!err.message.contains("is_public") && !err.message.contains("column"),
"schema text leaked: {}",
err.message
);
}
}
/// #250: negative tasks(limit) must not hit Postgres (and must not 500-log).
#[sqlx::test]
async fn tasks_negative_limit_clamped(pool: PgPool) {
let db = db(pool).await;
let schema = schema(db);
let resp = anon(&schema, "{ tasks(limit: -1) { id } }").await;
assert!(
resp.errors.is_empty(),
"negative limit must clamp, not fail: {:?}",
resp.errors
);
assert_eq!(count_tasks(&resp), 0);
}
/// #255: ceiling of the tasks(limit) clamp must be held (schema promises Max 200).
#[sqlx::test]
async fn tasks_limit_ceiling_clamped_to_200(pool: PgPool) {
let db = db(pool).await;
let now = Utc::now().to_rfc3339();
for i in 0..201 {
db.create_task(&crate::db::AgentTask {
id: format!("task-ceil-{i}"),
repo_id: None,
kind: "build".into(),
status: "pending".into(),
delegator_did: OWNER.into(),
assignee_did: None,
capability: "repo:write".into(),
ucan_token: None,
payload: None,
result: None,
created_at: now.clone(),
updated_at: now.clone(),
deadline: None,
})
.await
.unwrap();
}
let schema = schema(db);
let resp = anon(&schema, "{ tasks(limit: 5000) { id } }").await;
assert_eq!(count_tasks(&resp), 200, "limit above 200 must clamp to 200");
}
fn count_tasks(resp: &async_graphql::Response) -> usize {
assert!(resp.errors.is_empty(), "graphql errors: {:?}", resp.errors);
let async_graphql::Value::Object(obj) = &resp.data else {
panic!("data not an object: {:?}", resp.data);
};
let async_graphql::Value::List(rows) = obj.get("tasks").expect("tasks key") else {
panic!("tasks not a list");
};
rows.len()
}
}
File diff suppressed because it is too large. Load diff
+142 -10
View File
@@ -74,6 +74,13 @@ async fn main() -> Result<()> {
// bootstrap peers. Operators can opt out via GITLAWB_BOOTSTRAP_DISABLE_SEEDS.
bootstrap::merge_seeds(&mut config);
// Fail fast on config combinations that are individually in-range but jointly
// unsafe — notably a DB pool too small for the concurrent-write cap, which
// would let a push burst starve every other DB path (#174 F1).
config
.validate()
.map_err(|e| anyhow::anyhow!("invalid configuration: {e}"))?;
if !config.public_read {
warn!(
"GITLAWB_PUBLIC_READ=false is reserved; per-repository private-read enforcement is not wired in alpha"
@@ -378,7 +385,81 @@ async fn main() -> Result<()> {
sync_trigger_rate_limiter,
peer_write_rate_limiter,
shutdown_tx: shutdown_tx.clone(),
git_read_semaphore: Arc::new(tokio::sync::Semaphore::new(config.max_concurrent_git_ops)),
git_write_semaphore: Arc::new(tokio::sync::Semaphore::new(
config.max_concurrent_git_pushes,
)),
// Anon receive-pack advertisements get their OWN pool, same size as the
// write pool but disjoint, so filling it (which takes many source IPs, each
// capped by git_push_advert_per_caller) never occupies a permit the
// authenticated POST needs (#174).
git_push_advert_semaphore: Arc::new(tokio::sync::Semaphore::new(
config.max_concurrent_git_pushes,
)),
// Bounds concurrent detached post-push encryption walks, sized from the push
// pool (no separate knob — Q1): completed pushes cannot outnumber active
// encryption walks past this (#174 P1-e).
git_encrypt_semaphore: Arc::new(tokio::sync::Semaphore::new(
config.max_concurrent_git_pushes,
)),
// Bounds how many post-push pin loops run concurrently across all repos (#174 F6),
// independent of the per-repo encrypt-task coalescing below. Not a bound on the
// MB-scale object-id lists themselves: parked tasks still hold theirs (see the
// field doc on AppState::pin_semaphore).
pin_semaphore: Arc::new(tokio::sync::Semaphore::new(config.max_concurrent_pin_tasks)),
// Coalesces the DETACHED post-push encryption tasks per repo so a rapid pusher
// cannot grow the outstanding parked-waiter set past one task per repo (#174
// P2-2). No knob: it is a natural cap (one entry per distinct repo), not a
// sized pool.
encrypt_inflight: crate::state::EncryptInflight::new(),
// Per-repo in-process write-lease serializer (#174 U2/F3): supplements the pg
// advisory lock so a disconnected push's still-reaping git group can't be raced
// by a second same-node push. The map is naturally capped (one entry per contended
// repo, freed when unreferenced); the sized knob is how many pushes may PARK on
// one repo, since each parked push holds a fully buffered pack.
repo_write_leases: crate::state::RepoWriteLeases::new(config.repo_lease_max_waiters),
git_read_per_caller: rate_limit::PerCallerConcurrency::with_default_max_keys(
config.max_concurrent_reads_per_caller,
),
// Per-source cap on the receive-pack advertisement, sized to an eighth of the
// write pool (min 1): one resolved client key (rate_limit::client_key) can hold
// at most this many slots in the DEDICATED advert pool (git_push_advert_semaphore,
// disjoint from the write pool), so saturating that pool takes ~8 distinct keys
// (#174). That bounds an IPv4 or single-address caller; a caller controlling many
// addresses (an IPv6 /64 is 2^64 keys) still gets one cap per address, since
// client_key uses the full IP with no prefix folding. Narrowing the keying is a
// deferred design call, not something these caps claim to solve. Sized off the
// write pool only because the advert pool is created at the same size; an advert
// flood cannot touch a write permit.
git_push_advert_per_caller: rate_limit::PerCallerConcurrency::with_default_max_keys(
rate_limit::per_source_push_cap(config.max_concurrent_git_pushes),
),
// Per-source cap on the authenticated receive-pack POST, sized like the advert
// cap: one resolved client key can hold at most this many write-pool slots, so
// monopolizing the pool takes ~8 distinct keys (#174 P1-d). Same residual as
// above: keys are full IPs, so a caller with many addresses has many caps.
git_write_per_caller: rate_limit::PerCallerConcurrency::with_default_max_keys(
rate_limit::per_source_push_cap(config.max_concurrent_git_pushes),
),
// Bounds concurrent /ipfs visibility walks — a distinct public cost center, so
// its own pool + per-source sub-cap + per-IP rate limiter, never a git pool
// (#174 P1-3). The per-source map is bounded (reject-before-insert, INV-15).
git_ipfs_walk_semaphore: Arc::new(tokio::sync::Semaphore::new(
config.max_concurrent_ipfs_walks,
)),
git_ipfs_walk_per_caller: rate_limit::PerCallerConcurrency::with_default_max_keys(
config.ipfs_walk_per_source,
),
ipfs_rate_limiter: rate_limit::RateLimiter::new_bounded(
config.ipfs_rate_limit,
std::time::Duration::from_secs(3600),
200_000,
),
git_bin: "git".to_string(),
};
if config.ipfs_rate_limit == 0 {
tracing::warn!("GITLAWB_IPFS_RATE_LIMIT=0 — per-IP /ipfs rate limiting disabled");
}
// Periodic peer-count poll for the metrics gauge. If p2p is disabled
// we still set the gauge to 0 so dashboards don't show "no data".
@@ -408,22 +489,14 @@ async fn main() -> Result<()> {
// Periodic cleanup of expired rate limit entries + consumed-proof ledger
{
let rl = state.rate_limiter.clone();
let create_ip_rl = state.create_ip_rate_limiter.clone();
let push_rl = state.push_rate_limiter.clone();
let sync_trigger_rl = state.sync_trigger_rate_limiter.clone();
let peer_write_rl = state.peer_write_rate_limiter.clone();
let sweep_state = state.clone();
let db = state.db.clone();
let mut shutdown_rx = state.subscribe_shutdown();
tokio::spawn(async move {
loop {
tokio::select! {
_ = tokio::time::sleep(std::time::Duration::from_secs(300)) => {
rl.cleanup().await;
create_ip_rl.cleanup().await;
push_rl.cleanup().await;
sync_trigger_rl.cleanup().await;
peer_write_rl.cleanup().await;
sweep_rate_limiters(&sweep_state).await;
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs() as i64)
@@ -959,6 +1032,65 @@ async fn gossip_task(
}
}
#[cfg(test)]
mod rate_limiter_sweep_tests {
use crate::rate_limit::RateLimiter;
use std::time::Duration;
// Every per-key limiter the router mounts must be swept by the periodic
// task, the `/ipfs` one included: a limiter left out keeps expired keys
// until its map fills and the inline capacity sweep fires. Fails on the
// pre-fix sweeper, which skipped `ipfs_rate_limiter`.
#[tokio::test]
async fn sweep_evicts_expired_keys_from_every_limiter() {
let window = Duration::from_millis(30);
let mut state = crate::test_support::test_state_lazy();
state.rate_limiter = RateLimiter::new(10, window);
state.create_ip_rate_limiter = RateLimiter::new(10, window);
state.push_rate_limiter = RateLimiter::new(10, window);
state.sync_trigger_rate_limiter = RateLimiter::new(10, window);
state.peer_write_rate_limiter = RateLimiter::new(10, window);
state.ipfs_rate_limiter = RateLimiter::new(10, window);
let limiters = |s: &crate::state::AppState| {
[
s.rate_limiter.clone(),
s.create_ip_rate_limiter.clone(),
s.push_rate_limiter.clone(),
s.sync_trigger_rate_limiter.clone(),
s.peer_write_rate_limiter.clone(),
s.ipfs_rate_limiter.clone(),
]
};
for l in limiters(&state) {
assert!(l.check("1.2.3.4").await);
assert_eq!(l.tracked_keys().await, 1);
}
tokio::time::sleep(window * 3).await;
super::sweep_rate_limiters(&state).await;
for (i, l) in limiters(&state).into_iter().enumerate() {
assert_eq!(l.tracked_keys().await, 0, "limiter {i} was not swept");
}
}
}
/// Evict expired entries from every per-key rate limiter on the state.
///
/// Named and driven off `AppState` so the periodic sweeper stays in step with
/// the limiters the router actually mounts: adding a limiter field and
/// forgetting it here leaves its keys pinned until the map hits `max_keys` and
/// the inline capacity sweep runs (the `/ipfs` limiter was missed this way).
async fn sweep_rate_limiters(state: &AppState) {
state.rate_limiter.cleanup().await;
state.create_ip_rate_limiter.cleanup().await;
state.push_rate_limiter.cleanup().await;
state.sync_trigger_rate_limiter.cleanup().await;
state.peer_write_rate_limiter.cleanup().await;
state.ipfs_rate_limiter.cleanup().await;
}
/// Build the shared node HTTP client used for every outbound fan-out (sync
/// trigger, profile/repo fetches, gossip announce + peer pings).
///
+42 -8
View File
@@ -57,10 +57,15 @@ static PEERS_CONNECTED: OnceLock<IntGauge> = OnceLock::new();
/// more than once is a silent no-op. MUST be called from `main()` after
/// the node DID is known.
pub fn init(version: &str, node_did: &str) {
if REGISTRY.get().is_some() {
return;
}
// Guard with Once so two concurrent callers cannot both pass an unsynchronized
// `REGISTRY.get()` check and then race on `OnceLock::set(...).expect(...)`,
// panicking the loser (#192 F4). Once runs the body exactly once, so the
// `.expect("set X once")` calls below can never observe an already-set slot.
static INIT: std::sync::Once = std::sync::Once::new();
INIT.call_once(|| init_inner(version, node_did));
}
fn init_inner(version: &str, node_did: &str) {
let registry = Registry::new();
let info = IntGaugeVec::new(
@@ -218,6 +223,17 @@ pub fn record_fetch(repo: &str) {
}
}
/// Test-only: current `gitlawb_fetches_total` value for `repo` (0 if the registry
/// is not initialized). Lets api-layer tests assert the completed-fetch count with
/// a unique label instead of scraping the encoded text.
#[cfg(test)]
pub fn fetch_count_for_test(repo: &str) -> u64 {
FETCHES
.get()
.map(|c| c.with_label_values(&[repo]).get())
.unwrap_or(0)
}
/// Record one HTTP signature check that passed.
#[allow(dead_code)] // wired in a follow-up; helpers are part of the public metrics surface
pub fn record_auth_success(route: &str) {
@@ -286,11 +302,11 @@ pub fn encode() -> Result<String, prometheus::Error> {
mod tests {
use super::*;
// Note: these tests are not run in parallel by default (cargo runs
// tests in a binary in parallel via threads but the OnceLock guard
// means only the first init() call succeeds; subsequent ones are
// no-ops). The encode test below is structured so it's safe to run
// alongside other tests in the same binary.
// Note: cargo runs tests in a binary in parallel via threads, and several
// tests here call init(). init() is guarded by std::sync::Once, so concurrent
// callers run the body exactly once and never race on the OnceLock setters
// (#192 F4). The encode test below is structured so it's safe to run alongside
// other tests in the same binary regardless of ordering.
#[test]
fn encode_after_init_returns_prometheus_text() {
@@ -321,6 +337,24 @@ mod tests {
);
}
/// #192 F4: `init` is idempotent and safe to call repeatedly. The panic that
/// hit concurrent callers (both passing the old unsynchronized `REGISTRY.get()`
/// check, then racing on `OnceLock::set(...).expect(...)`) is fixed by the
/// `Once` guard. That race is inherently non-deterministic to assert in the
/// shared-binary suite — a 32-thread barrier stress-test reproduces it only in
/// isolation (verified: RED before the `Once` fix, GREEN after). This guard
/// covers the deterministic half of the contract: a repeat call is a no-op,
/// never a panic, and the registry stays usable.
#[test]
fn init_is_idempotent_no_panic_on_repeat() {
init("0.0.0-a", "did:key:a");
init("0.0.0-b", "did:key:b");
assert!(
encode().is_ok(),
"registry must stay usable after a repeated init"
);
}
#[test]
fn record_helpers_are_noops_before_init() {
// These don't panic. They also don't show up in encode() output
+742 -9
View File
@@ -7,6 +7,7 @@
//! no-op, so nodes without Pinata backing work fine.
use anyhow::Result;
use std::time::{Duration, Instant};
/// Pin a single git object's raw bytes on Pinata (v3 API).
///
@@ -72,25 +73,77 @@ pub async fn pin_object(
/// `object_list` is the already-withheld-filtered OID set to pin: the caller
/// applies `visibility_pack::replicable_objects` on the delta path or the
/// `..._fail_closed` filter on the full-scan path before calling. `repo_path` is
/// still needed to read each object's bytes. The twin in `ipfs_pin.rs` mirrors
/// this shape — change both in lockstep. Objects already recorded with a
/// `pinata_cid` are skipped. Returns `(sha_hex, cid)` pairs for each newly
/// pinned object.
/// still needed to read each object's bytes, and `git_bin` names the binary those
/// reads run: the production caller passes the literal `"git"`, and a test passes a
/// fake so the loop's own bound can be driven with a git that never answers.
/// Objects already recorded with a `pinata_cid` are skipped. Returns
/// `(sha_hex, cid)` pairs for each newly pinned object.
///
/// # What `batch_budget` does and does not bound
///
/// The loop runs under a `pin_semaphore` permit and that pool defers rather than
/// sheds, so the hold has to be bounded by something other than the pusher's object
/// count. Two things here are:
///
/// - this loop's own wall-clock: the deadline is taken once at loop start and
/// checked at the top of every iteration, so no object's work begins with less
/// than the read floor left. It is a gate, not a hard ceiling, since a started
/// iteration still runs to completion;
/// - the git read: `store::read_object_bounded` runs under `spawn_blocking` against the
/// ABSOLUTE batch deadline (not the loop-top remainder, which the `has_pinata_cid`
/// round-trip sitting between the two would push past it), with SIGTERM-then-SIGKILL
/// process-group teardown, so a hung `git cat-file` costs this batch its remaining
/// budget plus one watchdog teardown instead of holding the permit for the child's
/// whole lifetime and blocking a runtime worker while it does.
///
/// So the LOOP's hold is bounded by roughly `batch_budget`, plus one watchdog
/// teardown and one upload (the shared client's whole-request timeout bounds the
/// upload; `pin_object` takes no per-request override). The PERMIT's hold is NOT
/// bounded by any of this: `api::repos` acquires the permit and then re-derives the
/// object list with `pinata_object_list_for_refs` BEFORE this function is entered,
/// and that walk carries no aggregate deadline. The DB round-trips
/// (`has_pinata_cid`, `record_pinata_cid`) are untimed inside the budgeted region
/// too.
///
/// The twin in `ipfs_pin.rs` shares the budget gate and the bounded read with this
/// loop, so both are at parity again. Change them in lockstep: the skip-if-pinned
/// check, the fault arms, the returned pairs, and the budget handling.
// Eight arguments, one over clippy's threshold: the two the budget adds (`git_bin`,
// `batch_budget`) are what put the read under test injection and under a deadline, and
// grouping them into a struct would only move the same values behind a name the twin in
// `ipfs_pin.rs` does not use. Same allow as the sibling call sites in `api::repos`.
#[allow(clippy::too_many_arguments)]
pub async fn pin_new_objects(
client: &reqwest::Client,
upload_url: &str,
jwt: &str,
repo_path: &std::path::Path,
git_bin: &str,
object_list: Vec<String>,
db: &crate::db::Db,
batch_budget: Duration,
) -> Vec<(String, String)> {
if jwt.is_empty() {
return vec![];
}
let deadline = Instant::now() + batch_budget;
let total = object_list.len();
let mut pinned = Vec::new();
for sha in object_list {
for (attempted, sha) in object_list.into_iter().enumerate() {
// Top of the iteration, before any of this object's work: an object is never
// started with a remainder too small to cover a bounded read's teardown. The
// gate is shared with the IPFS loop so the two cannot drift apart in how they
// report a truncated batch. Consumed as a guard only: the read below runs against
// the absolute batch deadline, and `pin_object` takes no per-request override, so
// the remainder has no other consumer here.
if crate::ipfs_pin::batch_budget_gate("Pinata", deadline, pinned.len(), total - attempted)
.is_none()
{
break;
}
match db.has_pinata_cid(&sha).await {
Ok(true) => continue,
Ok(false) => {}
@@ -100,13 +153,73 @@ pub async fn pin_new_objects(
}
}
let data = match crate::git::store::read_object(repo_path, &sha) {
Ok(Some((_kind, bytes))) => bytes,
Ok(None) => continue,
Err(e) => {
// Read raw object content, bounded and reaped, under `spawn_blocking`: this is
// synchronous blocking work (child spawn, pipe drain, watchdog join), so calling
// it from the runtime task would block a worker thread for its whole duration.
// Placement mirrors the `/ipfs` serve path and the IPFS pin loop.
//
// The read runs against the ABSOLUTE batch deadline, not against the remainder
// measured at the top of the iteration: the `has_pinata_cid` round-trip above sits
// between the two, so `Instant::now() + budget_left` would land past `deadline` by
// however long the DB took, and under a saturated pool that is the dominant term.
// A slow DB check must not push the read's own bound out.
let read_deadline = deadline;
let read_path = repo_path.to_path_buf();
let read_sha = sha.clone();
let read_git = git_bin.to_string();
let read = tokio::task::spawn_blocking(move || {
crate::git::store::read_object_bounded(&read_git, &read_path, &read_sha, read_deadline)
})
.await;
let data = match read {
Ok(Ok(Some((_kind, bytes)))) => bytes,
// A verified absence, and the only outcome that is not a fault.
Ok(Ok(None)) => continue,
// A Transient fault does NOT by itself mean the store is gone. It also covers
// a spawn or watchdog-timeout failure of the reaped child, an unaffordable
// confirming re-probe, and, because readability is judged FOR one oid, a
// single unreadable `objects/<xx>` fan-out, which is 1/256 of the store. So
// re-check store-wide before deciding what the fault costs.
Ok(Err(e @ crate::git::store::ProbeError::Transient(_))) => {
if !crate::git::store::object_store_readable_store_wide(repo_path) {
// Genuinely store-wide: every remaining object fails identically, and
// continuing would spawn one doomed bounded child per object and spend
// the batch budget reaping them.
tracing::warn!(
sha = %sha,
err = %e,
unattempted = total - attempted,
"object store unreadable while pinning to Pinata; stopping the batch"
);
break;
}
// The store still reads store-wide, so the fault is object-scoped or
// transient to this read. Breaking would forfeit a healthy store's
// remaining objects permanently: the documented recovery re-derives the
// same list and breaks at the same index.
tracing::warn!(
sha = %sha,
err = %e,
"transient fault reading git object for Pinata; the object store is \
still readable store-wide, so this costs only this object"
);
continue;
}
// The store is readable and git still failed: a corrupt object, or a
// repo-wide fault git reports immediately. Either way it is per-object work
// that stays inside the budget, and breaking would forfeit a healthy store's
// remaining objects over one bad one, permanently (a later full-scan push
// re-offers the same object and breaks in the same place).
Ok(Err(e)) => {
tracing::warn!(sha = %sha, err = %e, "failed to read git object for Pinata");
continue;
}
// A panic in the read closure leaves no evidence that the failure is
// object-scoped, so fail toward the conservative arm.
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "bounded git read task failed; stopping the batch");
break;
}
};
match pin_object(client, upload_url, jwt, &sha, &data).await {
@@ -132,6 +245,626 @@ pub async fn pin_new_objects(
mod tests {
use super::*;
/// Write `n` loose blobs into a fresh bare repo and return their oids. The read
/// path shells to `git cat-file`, so the objects must genuinely exist on disk: a
/// fabricated oid would `continue` past the upload and the loop scenarios below
/// would prove nothing. Copied from `ipfs_pin.rs`'s test mod rather than shared,
/// since test mods are private.
fn seed_loose_blobs(repo_path: &std::path::Path, n: usize) -> Vec<String> {
crate::git::store::init_bare(repo_path).expect("init bare repo");
(0..n)
.map(|i| {
let mut cmd = std::process::Command::new("git");
cmd.args(["hash-object", "-w", "--stdin"])
.current_dir(repo_path)
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::piped());
let mut child = cmd.spawn().expect("spawn git hash-object");
{
use std::io::Write;
child
.stdin
.as_mut()
.expect("stdin")
.write_all(format!("pinata loop object {i}\n").as_bytes())
.expect("write stdin");
}
let out = child.wait_with_output().expect("hash-object output");
assert!(
out.status.success(),
"git hash-object: {}",
String::from_utf8_lossy(&out.stderr)
);
String::from_utf8_lossy(&out.stdout).trim().to_string()
})
.collect()
}
/// A live Pinata-shaped endpoint that answers each upload with a v3 `data.cid`
/// body after `delays[i]` for the i-th request it accepts (the last entry
/// repeats), counting the requests it received.
///
/// Hand rolled rather than driven with `mockito` like the `pin_object` tests
/// above: mockito has no per-response delay primitive, and the batch-budget test
/// needs uploads that are slow enough to exhaust the budget partway. Drains the
/// full request, headers plus the declared `Content-Length` body, before
/// sleeping: answering early and closing would surface as a write failure on the
/// client and turn a slow-but-healthy upload into a different failure shape.
/// Same fixture shape as `ipfs_pin.rs`'s `delaying_endpoint`.
async fn delaying_pinata_endpoint(
delays: Vec<Duration>,
requests: std::sync::Arc<std::sync::atomic::AtomicUsize>,
) -> String {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let endpoint = format!("http://{}", listener.local_addr().unwrap());
tokio::spawn(async move {
let mut seen = 0usize;
while let Ok((mut sock, _)) = listener.accept().await {
let delay = *delays
.get(seen)
.or_else(|| delays.last())
.unwrap_or(&Duration::ZERO);
seen += 1;
requests.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
tokio::spawn(async move {
let mut acc = Vec::new();
let mut buf = [0u8; 4096];
loop {
let n = match sock.read(&mut buf).await {
Ok(0) | Err(_) => break,
Ok(n) => n,
};
acc.extend_from_slice(&buf[..n]);
if let Some(hdr_end) =
acc.windows(4).position(|w| w == b"\r\n\r\n").map(|p| p + 4)
{
let headers = String::from_utf8_lossy(&acc[..hdr_end]).to_lowercase();
let len: usize = headers
.lines()
.find_map(|l| l.strip_prefix("content-length:"))
.and_then(|v| v.trim().parse().ok())
.unwrap_or(0);
if acc.len() >= hdr_end + len {
break;
}
}
}
tokio::time::sleep(delay).await;
let body = br#"{"data":{"cid":"QmPinataBatchTestCid","name":"git.bin"}}"#;
let head = format!(
"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\n\r\n",
body.len()
);
let _ = sock.write_all(head.as_bytes()).await;
let _ = sock.write_all(body).await;
let _ = sock.flush().await;
});
}
});
endpoint
}
/// A `tracing` sink a test can read back, so the truncation warn and its sink
/// label can be asserted on rather than assumed. Installed with `set_default`,
/// which is thread-local and scoped to the guard, so it cannot bleed into any
/// other test in the binary.
#[derive(Clone, Default)]
struct CapturedLogs(std::sync::Arc<std::sync::Mutex<Vec<u8>>>);
impl CapturedLogs {
fn text(&self) -> String {
String::from_utf8_lossy(&self.0.lock().unwrap()).to_string()
}
}
impl std::io::Write for CapturedLogs {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
self.0.lock().unwrap().extend_from_slice(buf);
Ok(buf.len())
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
impl<'a> tracing_subscriber::fmt::MakeWriter<'a> for CapturedLogs {
type Writer = CapturedLogs;
fn make_writer(&'a self) -> Self::Writer {
self.clone()
}
}
fn capture_logs() -> (CapturedLogs, tracing::subscriber::DefaultGuard) {
let logs = CapturedLogs::default();
let subscriber = tracing_subscriber::fmt()
.with_writer(logs.clone())
.with_max_level(tracing::Level::WARN)
.with_ansi(false)
.finish();
let guard = tracing::subscriber::set_default(subscriber);
(logs, guard)
}
/// Write an executable `/bin/sh` script. Copied per module rather than shared:
/// `store.rs`, `visibility_pack.rs` and `ipfs_pin.rs` each keep their own, since
/// their test mods are private and not reachable from here.
#[cfg(unix)]
fn write_script(path: &std::path::Path, body: &str) {
use std::os::unix::fs::PermissionsExt;
std::fs::write(path, body).expect("write fake git");
let mut perm = std::fs::metadata(path).unwrap().permissions();
perm.set_mode(0o755);
std::fs::set_permissions(path, perm).unwrap();
}
/// The permit-hold bound on this branch. `pin_new_objects` runs under the same
/// deferring `pin_semaphore` as the IPFS loop, so without a batch deadline the
/// hold is O(N) with N chosen by the pusher. Five objects against an endpoint
/// that takes 2s each, under a 5.5s budget, must stop partway: the batch is
/// truncated and the remainder is left unattempted with exactly one warn naming
/// how many, labelled for this sink.
///
/// The windows are deliberately loose. Four pins would need every upload to
/// answer in under 1.4s, which the endpoint's own 2s sleep forbids, and one pin
/// needs only the first upload to land inside 5.5s, so both bounds hold with
/// more than a second of slack on a loaded box.
#[sqlx::test]
async fn pin_new_objects_stops_the_batch_at_its_deadline(pool: sqlx::PgPool) {
let db = crate::db::Db::for_testing(pool);
db.run_migrations().await.expect("migrations");
let tmp = tempfile::TempDir::new().unwrap();
let repo_path = tmp.path().join("slow.git");
let oids = seed_loose_blobs(&repo_path, 5);
let requests = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
let endpoint = delaying_pinata_endpoint(
vec![Duration::from_secs(2)],
std::sync::Arc::clone(&requests),
)
.await;
let (logs, _guard) = capture_logs();
let client = reqwest::Client::new();
let pinned = tokio::time::timeout(
Duration::from_secs(30),
pin_new_objects(
&client,
&endpoint,
"test-jwt",
&repo_path,
"git",
oids,
&db,
Duration::from_millis(5500),
),
)
.await
.expect("wedge guard: a 5.5s budget cannot take 30s");
assert!(
(1..=3).contains(&pinned.len()),
"the batch must stop partway, not pin all five and not stall on the first: pinned {}",
pinned.len()
);
assert_eq!(
requests.load(std::sync::atomic::Ordering::SeqCst),
pinned.len(),
"no upload may be issued for an object the budget stopped short of"
);
let text = logs.text();
let warns: Vec<&str> = text
.lines()
.filter(|l| l.contains("pin batch deadline reached"))
.collect();
assert_eq!(
warns.len(),
1,
"the deadline must be reported exactly once for the batch, not per object: {text}"
);
assert!(
warns[0].contains("Pinata"),
"the truncation warn must name this sink, not the twin's: {}",
warns[0]
);
let unattempted: usize = warns[0]
.split("unattempted=")
.nth(1)
.and_then(|s| {
s.split(|c: char| !c.is_ascii_digit())
.next()
.and_then(|d| d.parse().ok())
})
.unwrap_or_else(|| panic!("the deadline warn must name the unattempted count: {text}"));
assert!(
unattempted >= 1 && unattempted + pinned.len() <= 5,
"unattempted={unattempted} with {} pinned is not a partial batch of five",
pinned.len()
);
}
/// #174 F3 on this branch: the git read runs while the `pin_semaphore` permit is
/// held, so a wedged `git cat-file` used to hold that permit for as long as the
/// child lived, with no deadline and no reaping, on a path a pusher drives. With
/// the read bounded, a git that never answers costs the batch its budget plus one
/// watchdog teardown and no more.
///
/// The fake traps SIGTERM and sleeps a BOUNDED 30s, following the fixture in
/// `visibility_pack.rs`: with the deadline neutralized the read would otherwise
/// leave the blocking closure and its child alive long after the test-level
/// timeout fires, wedging the run instead of reporting a failure. The endpoint is
/// never reached, since no object's bytes are ever produced.
///
/// The batch ends on the BUDGET, not on the fault arm: the repo is a healthy bare
/// store, so the timeout's `Transient` verdict is object-scoped and the loop moves on,
/// only to find the budget spent. Capturing that warn is not decoration. `tracing`
/// caches a callsite's interest globally the first time it is hit, and a hit from a
/// thread with no subscriber caches it as never-interested for the whole binary, which
/// silently blinds the deadline tests running beside this one.
#[cfg(unix)]
#[sqlx::test]
async fn pin_new_objects_returns_by_budget_with_a_hung_git(pool: sqlx::PgPool) {
let db = crate::db::Db::for_testing(pool);
db.run_migrations().await.expect("migrations");
let tmp = tempfile::TempDir::new().unwrap();
let repo_path = tmp.path().join("wedged.git");
let oids = seed_loose_blobs(&repo_path, 3);
let fake = tmp.path().join("hanging-git");
write_script(&fake, "#!/bin/sh\ntrap '' TERM\necho $$ > pid\nsleep 30\n");
let (logs, _guard) = capture_logs();
let client = reqwest::Client::new();
let started = std::time::Instant::now();
let pinned = tokio::time::timeout(
Duration::from_secs(25),
pin_new_objects(
&client,
"http://127.0.0.1:9",
"test-jwt",
&repo_path,
fake.to_str().unwrap(),
oids,
&db,
Duration::from_secs(2),
),
)
.await
.expect(
"a wedged git must not hold the pin permit past the batch budget: the read is \
bounded and reaped, so this cannot reach the outer timeout",
);
let elapsed = started.elapsed();
assert!(
pinned.is_empty(),
"a git that never answers cannot produce a pinned object: {pinned:?}"
);
assert!(
elapsed < Duration::from_secs(20),
"elapsed {elapsed:?} must stay inside the budget plus one watchdog teardown"
);
let text = logs.text();
assert_eq!(
text.lines()
.filter(|l| l.contains("pin batch deadline reached"))
.count(),
1,
"one wedged read must spend the whole budget and stop the batch there, exactly \
once: {text}"
);
// The child's process group must be gone once the call returns; a bounded read
// that leaves the child running has only moved the hold somewhere else.
let pid: i32 = std::fs::read_to_string(repo_path.join("pid"))
.expect("the fake git must have recorded its pid, or it was never on the read path")
.trim()
.parse()
.unwrap();
let mut gone = false;
for _ in 0..200 {
// SAFETY: kill(2) with signal 0 only probes existence; ESRCH means gone.
if unsafe { libc::kill(pid, 0) } != 0 {
gone = true;
break;
}
std::thread::sleep(Duration::from_millis(10));
}
assert!(
gone,
"the reaped fake git ({pid}) must not outlive the call"
);
}
/// A `git_bin` wrapper that records every invocation's arguments and then execs the
/// real git, so a test can tell which objects the loop actually attempted. The returned
/// pin list cannot: it is empty both when the loop broke after one object and when it
/// continued past all of them. Copied from `ipfs_pin.rs`'s test mod, like the fixtures
/// above, since test mods are private.
#[cfg(unix)]
fn counting_git(dir: &std::path::Path, log: &std::path::Path) -> String {
let fake = dir.join("counting-git");
write_script(
&fake,
&format!(
"#!/bin/sh\necho \"$*\" >> {}\nexec git \"$@\"\n",
log.display()
),
);
fake.to_str().unwrap().to_string()
}
/// How many objects the loop actually attempted, read off the invocation log.
///
/// Counts `--batch-check` invocations, not log lines and not oid occurrences: the type
/// probe carries its oid on stdin rather than in argv, so an oid appears in the log only
/// once an object has already got past its probe, and a healthy object costs two
/// invocations to a faulting one's one.
fn objects_attempted(log: &std::path::Path) -> usize {
std::fs::read_to_string(log)
.unwrap_or_default()
.lines()
.filter(|l| l.contains("--batch-check"))
.count()
}
/// The store-wide fault arm, the twin of the IPFS loop's. When the object store cannot
/// be read at all every remaining object fails identically, so continuing would spawn
/// one doomed bounded child per object and burn the batch budget on reaping them.
///
/// The fixture looks wrong and is not: with the objects LOOSE and only `objects/pack`
/// unreadable, git still resolves each object, but it prints an `error:` diagnostic
/// that the probe routes to a fault before the present/missing parse, so the read
/// reaches the fault classification and (the store being unreadable) returns
/// `Transient`, which the store-wide re-check then confirms really is store-wide.
#[cfg(unix)]
#[sqlx::test]
async fn pin_new_objects_breaks_the_batch_on_an_unreadable_store(pool: sqlx::PgPool) {
use std::os::unix::fs::PermissionsExt;
let db = crate::db::Db::for_testing(pool);
db.run_migrations().await.expect("migrations");
let tmp = tempfile::TempDir::new().unwrap();
let repo_path = tmp.path().join("unreadable.git");
let oids = seed_loose_blobs(&repo_path, 5);
let log = tmp.path().join("calls.log");
let git_bin = counting_git(tmp.path(), &log);
let requests = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
let endpoint =
delaying_pinata_endpoint(vec![Duration::ZERO], std::sync::Arc::clone(&requests)).await;
let client = reqwest::Client::new();
let pack_dir = repo_path.join("objects").join("pack");
let chmod = |mode: u32| {
let mut perms = std::fs::metadata(&pack_dir).unwrap().permissions();
perms.set_mode(mode);
std::fs::set_permissions(&pack_dir, perms).unwrap();
};
chmod(0o000);
// Root bypasses permission bits, so witness the exact operation the probe performs
// and skip rather than falsely fail.
let genuinely_unreadable = std::fs::read_dir(&pack_dir).is_err();
let pinned = tokio::time::timeout(
Duration::from_secs(30),
pin_new_objects(
&client,
&endpoint,
"test-jwt",
&repo_path,
&git_bin,
oids,
&db,
Duration::from_secs(60),
),
)
.await
.expect("an immediately-faulting store cannot take 30s");
let attempted = objects_attempted(&log);
chmod(0o755); // restore BEFORE any assertion that can panic, so TempDir cleans up
if genuinely_unreadable {
assert!(
pinned.is_empty(),
"nothing can be pinned through a store that cannot be read: {pinned:?}"
);
assert_eq!(
attempted, 1,
"a store-wide fault must break the batch after the first object, not spawn \
one doomed bounded child per object: {attempted} of 5 objects were read"
);
assert_eq!(
requests.load(std::sync::atomic::Ordering::SeqCst),
0,
"no upload may be issued for an object whose bytes were never read"
);
}
}
/// The must-not direction of the arm above, the twin of the IPFS loop's. One corrupt
/// loose object among healthy ones is a `Deterministic` fault (the store is readable,
/// git still fails), and the documented recovery path cannot repair it: a later
/// full-scan push re-offers the same object and would break at the same place, so
/// breaking here stops the repo replicating permanently.
///
/// Deliberately not the bad-config corruption, which is repo-wide: all five objects
/// would fault and the test would pin the store-wide case rather than the object-scoped
/// one this arm rests on.
#[cfg(unix)]
#[sqlx::test]
async fn pin_new_objects_continues_past_a_deterministic_fault(pool: sqlx::PgPool) {
use std::os::unix::fs::PermissionsExt;
let db = crate::db::Db::for_testing(pool);
db.run_migrations().await.expect("migrations");
let tmp = tempfile::TempDir::new().unwrap();
let repo_path = tmp.path().join("corrupt.git");
let oids = seed_loose_blobs(&repo_path, 5);
let log = tmp.path().join("calls.log");
let git_bin = counting_git(tmp.path(), &log);
let requests = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
let endpoint =
delaying_pinata_endpoint(vec![Duration::ZERO], std::sync::Arc::clone(&requests)).await;
let client = reqwest::Client::new();
// Overwrite exactly one loose object with non-zlib garbage (0o444 by default).
let victim = repo_path
.join("objects")
.join(&oids[0][0..2])
.join(&oids[0][2..]);
assert!(victim.is_file(), "fixture must leave the blob loose");
let mut perms = std::fs::metadata(&victim).unwrap().permissions();
perms.set_mode(0o644);
std::fs::set_permissions(&victim, perms).unwrap();
std::fs::write(&victim, b"garbage not a zlib stream").unwrap();
let pinned = tokio::time::timeout(
Duration::from_secs(30),
pin_new_objects(
&client,
&endpoint,
"test-jwt",
&repo_path,
&git_bin,
oids,
&db,
Duration::from_secs(60),
),
)
.await
.expect("an immediate endpoint and four healthy objects cannot take 30s");
assert_eq!(
objects_attempted(&log),
5,
"an object-scoped fault must not stop the batch: every object must be read"
);
assert_eq!(
pinned.len(),
4,
"one corrupt object must cost only itself: the other four must still pin"
);
assert_eq!(
requests.load(std::sync::atomic::Ordering::SeqCst),
4,
"exactly the readable objects may reach the endpoint"
);
}
/// The normal direction, and the dedup branch driven both ways: on a healthy
/// store with a healthy endpoint and a generous budget every object pins and the
/// CID is recorded, and a second call over the same list uploads nothing because
/// `has_pinata_cid` now answers true for all of them.
#[sqlx::test]
async fn pin_new_objects_pins_every_object_then_skips_the_recorded_ones(pool: sqlx::PgPool) {
let db = crate::db::Db::for_testing(pool);
db.run_migrations().await.expect("migrations");
let tmp = tempfile::TempDir::new().unwrap();
let repo_path = tmp.path().join("healthy.git");
let oids = seed_loose_blobs(&repo_path, 3);
let requests = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
let endpoint =
delaying_pinata_endpoint(vec![Duration::ZERO], std::sync::Arc::clone(&requests)).await;
let client = reqwest::Client::new();
let pinned = tokio::time::timeout(
Duration::from_secs(30),
pin_new_objects(
&client,
&endpoint,
"test-jwt",
&repo_path,
"git",
oids.clone(),
&db,
Duration::from_secs(60),
),
)
.await
.expect("an immediate endpoint and three healthy objects cannot take 30s");
assert_eq!(pinned.len(), 3, "every healthy object must pin: {pinned:?}");
for (i, (sha, cid)) in pinned.iter().enumerate() {
assert_eq!(sha, &oids[i], "the pairs must carry the objects' own oids");
assert_eq!(cid, "QmPinataBatchTestCid");
assert!(
db.has_pinata_cid(sha).await.unwrap(),
"a pinned object must be recorded so the next batch skips it"
);
}
assert_eq!(requests.load(std::sync::atomic::Ordering::SeqCst), 3);
let again = tokio::time::timeout(
Duration::from_secs(30),
pin_new_objects(
&client,
&endpoint,
"test-jwt",
&repo_path,
"git",
oids,
&db,
Duration::from_secs(60),
),
)
.await
.expect("a fully deduped batch cannot take 30s");
assert!(again.is_empty(), "already-recorded objects must be skipped");
assert_eq!(
requests.load(std::sync::atomic::Ordering::SeqCst),
3,
"the skip must happen before the upload, not after it"
);
}
/// The no-op configuration still short-circuits with the budgeted signature: an
/// empty JWT must return before any git child is spawned and before any request
/// is issued. The `git_bin` here records every invocation, so "git was never
/// touched" is observed rather than assumed.
#[cfg(unix)]
#[sqlx::test]
async fn pin_new_objects_with_an_empty_jwt_touches_neither_git_nor_the_endpoint(
pool: sqlx::PgPool,
) {
let db = crate::db::Db::for_testing(pool);
db.run_migrations().await.expect("migrations");
let tmp = tempfile::TempDir::new().unwrap();
let repo_path = tmp.path().join("unconfigured.git");
let oids = seed_loose_blobs(&repo_path, 2);
let log = tmp.path().join("calls.log");
let fake = tmp.path().join("counting-git");
write_script(
&fake,
&format!(
"#!/bin/sh\necho \"$*\" >> {}\nexec git \"$@\"\n",
log.display()
),
);
let requests = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
let endpoint =
delaying_pinata_endpoint(vec![Duration::ZERO], std::sync::Arc::clone(&requests)).await;
let client = reqwest::Client::new();
let pinned = tokio::time::timeout(
Duration::from_secs(30),
pin_new_objects(
&client,
&endpoint,
"",
&repo_path,
fake.to_str().unwrap(),
oids,
&db,
Duration::from_secs(60),
),
)
.await
.expect("an unconfigured sink returns immediately");
assert!(pinned.is_empty(), "an empty JWT pins nothing");
assert!(
!log.exists(),
"no git child may be spawned when the sink is not configured"
);
assert_eq!(requests.load(std::sync::atomic::Ordering::SeqCst), 0);
}
#[tokio::test]
async fn test_pin_skipped_when_jwt_empty() {
let client = reqwest::Client::new();
+168
View File
@@ -121,6 +121,13 @@ impl RateLimiter {
true
}
/// Number of keys currently tracked. Tests use it to observe what a sweep
/// reclaimed; there is no production reader.
#[cfg(test)]
pub async fn tracked_keys(&self) -> usize {
self.state.lock().await.len()
}
pub async fn cleanup(&self) {
let now = Instant::now();
let mut state = self.state.lock().await;
@@ -132,6 +139,110 @@ impl RateLimiter {
}
}
/// Per-source concurrency cap derived from the write-pool size: one resolved client
/// key (see [`client_key`]) may hold at most an eighth of the pool, so saturating it
/// takes ~8 distinct keys. Real for an IPv4 or single-address caller; a caller with a
/// routed IPv6 /64 has 2^64 keys, because `client_key` returns the full address with
/// no prefix folding. Narrowing that keying is a deferred decision, so this cap is a
/// bound per key, not a bound per operator.
///
/// Floored at 1 because the value feeds [`PerCallerConcurrency`], where a cap of 0
/// would shed EVERY receive-pack advertisement and break all pushes. The floor is
/// load-bearing at the minimum write-pool size (1), which integer-divides to 0.
pub(crate) fn per_source_push_cap(max_concurrent_git_pushes: usize) -> usize {
(max_concurrent_git_pushes / 8).max(1)
}
/// A bounded per-caller CONCURRENCY limiter — distinct from [`RateLimiter`], which
/// caps request RATE. Each caller key may hold at most `per_caller` in-flight
/// permits at once; beyond that [`try_acquire`](Self::try_acquire) returns `None`
/// and the caller sheds. Used to stop one caller (a single anonymous source-IP or
/// DID) monopolizing the served-git read pool (#174).
///
/// The key map is self-bounding: a key is removed the instant its in-flight count
/// reaches zero, so it never holds more keys than there are concurrently-active
/// callers (itself bounded by the read semaphore). A `max_keys` reject-before-insert
/// backstop guarantees a key farm can never grow the map even if that invariant
/// weakened — a NEW key at the cap is rejected WITHOUT allocating an entry (INV-15).
///
/// Uses a `std::sync::Mutex` (not the file's `tokio::sync::Mutex`) because the
/// permit's `Drop` must release synchronously; the critical section holds no await.
#[derive(Clone)]
pub struct PerCallerConcurrency {
state: Arc<std::sync::Mutex<HashMap<String, usize>>>,
per_caller: usize,
max_keys: usize,
}
/// RAII permit from [`PerCallerConcurrency::try_acquire`]. On drop it decrements
/// the caller's in-flight count and removes the key when it reaches zero.
pub struct PerCallerPermit {
state: Arc<std::sync::Mutex<HashMap<String, usize>>>,
key: String,
}
impl PerCallerConcurrency {
pub fn new(per_caller: usize, max_keys: usize) -> Self {
Self {
state: Arc::new(std::sync::Mutex::new(HashMap::new())),
per_caller: per_caller.max(1),
max_keys: max_keys.max(1),
}
}
/// Convenience constructor with the default key bound.
pub fn with_default_max_keys(per_caller: usize) -> Self {
Self::new(per_caller, DEFAULT_MAX_KEYS)
}
/// `Some(permit)` when the caller is under its cap and the map has room;
/// `None` (shed) otherwise. Reject-before-insert: a new key at `max_keys` is
/// rejected without allocating.
pub fn try_acquire(&self, key: &str) -> Option<PerCallerPermit> {
// Recover from a poisoned lock rather than panicking: the critical section
// is pure counter arithmetic and cannot itself panic, so a poisoned mutex
// would only ever come from an unrelated abort, and a slightly-off count
// self-heals as permits drop. A panic here would instead brick the limiter
// for every caller (each subsequent lock re-panics).
let mut map = self.state.lock().unwrap_or_else(|e| e.into_inner());
match map.get_mut(key) {
Some(count) => {
if *count >= self.per_caller {
return None;
}
*count += 1;
}
None => {
if map.len() >= self.max_keys {
return None;
}
map.insert(key.to_string(), 1);
}
}
Some(PerCallerPermit {
state: self.state.clone(),
key: key.to_string(),
})
}
#[cfg(test)]
pub fn tracked_keys(&self) -> usize {
self.state.lock().unwrap().len()
}
}
impl Drop for PerCallerPermit {
fn drop(&mut self) {
let mut map = self.state.lock().unwrap_or_else(|e| e.into_inner());
if let Some(count) = map.get_mut(&self.key) {
*count -= 1;
if *count == 0 {
map.remove(&self.key);
}
}
}
}
pub async fn rate_limit_by_did(request: Request, next: Next) -> Response {
let limiter = request.extensions().get::<RateLimiter>().cloned();
@@ -288,6 +399,63 @@ pub async fn rate_limit_by_ip(request: Request, next: Next) -> Response {
mod tests {
use super::*;
#[test]
fn per_caller_concurrency_caps_one_caller_and_frees_on_drop() {
let lim = PerCallerConcurrency::new(2, 100);
let p1 = lim.try_acquire("did:key:zA").expect("first under cap");
let p2 = lim.try_acquire("did:key:zA").expect("second under cap");
assert!(
lim.try_acquire("did:key:zA").is_none(),
"a third in-flight op for the same caller sheds (over the per-caller cap)"
);
// A DIFFERENT caller is unaffected — the cap is per-caller, not global.
let _other = lim
.try_acquire("did:key:zB")
.expect("a different caller has its own budget");
drop(p1);
assert!(
lim.try_acquire("did:key:zA").is_some(),
"freeing one in-flight slot lets the same caller back in"
);
drop(p2);
}
#[test]
fn per_caller_concurrency_map_is_self_bounding_and_reject_before_insert() {
// Self-bounding: acquire+drop many distinct keys — the map never grows
// because a key is removed the instant its in-flight count hits zero.
let lim = PerCallerConcurrency::new(4, 3);
for i in 0..50 {
let _p = lim.try_acquire(&format!("k{i}"));
}
assert_eq!(
lim.tracked_keys(),
0,
"keys with zero in-flight ops are removed, so an acquire+drop flood leaves the map empty"
);
// Reject-before-insert: HOLD max_keys distinct keys, then a new key sheds
// WITHOUT growing the map past the cap (INV-15 — a rejected request never
// allocates an entry).
let held: Vec<_> = (0..3)
.map(|i| lim.try_acquire(&format!("h{i}")).unwrap())
.collect();
assert_eq!(
lim.tracked_keys(),
3,
"three distinct callers held concurrently"
);
assert!(
lim.try_acquire("h3").is_none(),
"a new key at max_keys is rejected"
);
assert_eq!(
lim.tracked_keys(),
3,
"the rejected new key did not allocate an entry (reject-before-insert)"
);
drop(held);
}
#[tokio::test]
async fn allows_within_limit() {
let limiter = RateLimiter::new(3, Duration::from_secs(60));
+11
View File
@@ -214,9 +214,20 @@ pub fn build_router(state: AppState) -> Router {
// identity and can apply per-repo visibility (#110); anonymous callers stay
// anonymous and still read genuinely public content. `/api/v1/ipfs/pins`
// stays unsigned — gating the pin index is tracked separately (#121).
// `/ipfs/{cid}` also carries a per-IP flood brake: it is anon-reachable and each
// request can drive a full-history git walk, so the per-IP rate limiter is the
// outermost layer (rejects a flood before the walk-admission work), mirroring the
// push/create routers. The extension MUST be attached or rate_limit_by_ip is a
// silent no-op. `/api/v1/ipfs/pins` (no walk) is merged in unbraked, as before.
let ipfs_limiter = rate_limit::IpRateLimiter {
limiter: state.ipfs_rate_limiter.clone(),
trust: state.push_limiter_trust,
};
let ipfs_routes = Router::new()
.route("/ipfs/{cid}", get(ipfs::get_by_cid))
.layer(middleware::from_fn(auth::optional_signature))
.layer(middleware::from_fn(rate_limit::rate_limit_by_ip))
.layer(axum::Extension(ipfs_limiter))
.merge(Router::new().route("/api/v1/ipfs/pins", get(ipfs::list_pins)));
// ── Arweave permanent anchors ──────────────────────────────────────────
File diff suppressed because it is too large. Load diff
+1 -1
View File
@@ -605,7 +605,7 @@ async fn replicate_encrypted_blobs(
continue;
}
};
match crate::ipfs_pin::pin_git_object(ipfs_api, &blob.oid, &envelope).await {
match crate::ipfs_pin::pin_git_object(ipfs_api, &blob.oid, &envelope, None).await {
Ok(cid) if !cid.is_empty() => {
if cid != blob.cid {
warn!(oid = %blob.oid, expected = %blob.cid, got = %cid, "replicated envelope CID mismatch; skipping record");
+255 -10
View File
@@ -82,6 +82,26 @@ fn build_state(db: Arc<crate::db::Db>, pool: PgPool) -> AppState {
sync_trigger_rate_limiter: RateLimiter::new(60, Duration::from_secs(3600)),
peer_write_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
shutdown_tx: tokio::sync::watch::channel(false).0,
// Generous — no test drives the handler-level shed (git_permit is unit-tested).
git_read_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_write_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_push_advert_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_encrypt_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
pin_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
encrypt_inflight: crate::state::EncryptInflight::new(),
repo_write_leases: crate::state::RepoWriteLeases::new(8),
git_read_per_caller: crate::rate_limit::PerCallerConcurrency::with_default_max_keys(16),
git_push_advert_per_caller: crate::rate_limit::PerCallerConcurrency::with_default_max_keys(
8,
),
git_write_per_caller: crate::rate_limit::PerCallerConcurrency::with_default_max_keys(8),
// Generous — a test that drives the /ipfs walk shed overrides these directly.
git_ipfs_walk_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_ipfs_walk_per_caller: crate::rate_limit::PerCallerConcurrency::with_default_max_keys(
16,
),
ipfs_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
git_bin: "git".to_string(),
}
}
@@ -109,6 +129,25 @@ pub(crate) fn signed_request_as(did: &str, method: Method, uri: &str, body: Body
.expect("request builder")
}
/// A local endpoint whose TCP accept succeeds instantly but that never writes an
/// HTTP response, so any request against it stalls deterministically until the
/// caller's own timeout. (A non-routable address hangs only if the network
/// blackholes the SYN — a fast RST would end the stall early and make a timeout
/// test pass for the wrong reason.) The accepted sockets are parked in the
/// spawned task, which dies with the test's runtime, so the peer never sees a
/// close mid-test.
pub(crate) async fn silent_http_endpoint() -> String {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let endpoint = format!("http://{}", listener.local_addr().unwrap());
tokio::spawn(async move {
let mut held = Vec::new();
while let Ok((sock, _)) = listener.accept().await {
held.push(sock);
}
});
endpoint
}
#[cfg(test)]
mod tests {
use super::*;
@@ -187,6 +226,207 @@ mod tests {
);
}
/// PR3 (#62): the served-git concurrency cap sheds at the HTTP layer before the
/// DB. The held `git_permit` acquire now sits after the per-source cap, so the
/// cheap early shed is carried by an explicit `available_permits() == 0` check at
/// the top of the handler (the held permit remains the authoritative bound further
/// down). That check is a permit-less snapshot: it spares a request's DB work once
/// the pool is ALREADY saturated, which is the case this test drives, and it does
/// not bound the DB window in general. DB-free here because an exhausted semaphore
/// sheds before any DB/disk access, so a lazy state works. Remove the early-shed block
/// from git_info_refs and this goes red (the request falls through to the DB and
/// returns something other than 503).
#[tokio::test]
async fn git_info_refs_sheds_with_503_when_semaphore_exhausted() {
let mut state = test_state_lazy();
state.git_read_semaphore = Arc::new(tokio::sync::Semaphore::new(0));
let router = Router::new()
.route(
"/{owner}/{repo}/info/refs",
axum::routing::get(crate::api::repos::git_info_refs),
)
.with_state(state);
let resp = router
.oneshot(anon_get(
"/alice/repo.git/info/refs?service=git-upload-pack",
))
.await
.unwrap();
assert_eq!(
resp.status(),
StatusCode::SERVICE_UNAVAILABLE,
"an exhausted git semaphore must shed info/refs with 503 before touching the DB"
);
assert_eq!(
resp.headers()
.get("retry-after")
.and_then(|h| h.to_str().ok()),
Some("1"),
"the 503 shed must carry Retry-After"
);
}
/// PR3 (#62) sibling of the info/refs shed test: git-upload-pack carries the same
/// explicit `available_permits() == 0` early check at the top, so an ALREADY
/// exhausted semaphore must shed the request with a 503 before its DB/disk work.
/// That is the case the permit-less snapshot does deliver; it is not an admission
/// bound on the DB window. Anonymous-reachable, so no auth injection is needed.
/// Remove the early-shed block from git_upload_pack and this goes red.
#[tokio::test]
async fn git_upload_pack_sheds_with_503_when_semaphore_exhausted() {
let mut state = test_state_lazy();
state.git_read_semaphore = Arc::new(tokio::sync::Semaphore::new(0));
let router = Router::new()
.route(
"/{owner}/{repo}/git-upload-pack",
axum::routing::post(crate::api::repos::git_upload_pack),
)
.with_state(state);
let req = Request::builder()
.method(Method::POST)
.uri("/alice/repo.git/git-upload-pack")
.body(Body::from(&b"0000"[..]))
.unwrap();
let resp = router.oneshot(req).await.unwrap();
assert_eq!(
resp.status(),
StatusCode::SERVICE_UNAVAILABLE,
"an exhausted git semaphore must shed git-upload-pack with 503 before touching the DB"
);
assert_eq!(
resp.headers()
.get("retry-after")
.and_then(|h| h.to_str().ok()),
Some("1"),
"the 503 shed must carry Retry-After"
);
}
/// PR3 (#62) receive-pack sibling of the info/refs shed test: the early shed
/// selects the dedicated ADVERT pool for a git-receive-pack advertisement (#174),
/// so an ALREADY exhausted advert pool sheds the advert with 503 before its DB/disk
/// work (the case the permit-less snapshot delivers, not an admission bound on the
/// DB window), while the write pool (reserved for authenticated POSTs) is left
/// free here.
/// Flip the pool selection back to the write pool, or remove the early-shed
/// block, and this goes red.
#[tokio::test]
async fn git_info_refs_receive_pack_sheds_with_503_when_advert_pool_exhausted() {
let mut state = test_state_lazy();
state.git_push_advert_semaphore = Arc::new(tokio::sync::Semaphore::new(0));
let router = Router::new()
.route(
"/{owner}/{repo}/info/refs",
axum::routing::get(crate::api::repos::git_info_refs),
)
.with_state(state);
let resp = router
.oneshot(anon_get(
"/alice/repo.git/info/refs?service=git-receive-pack",
))
.await
.unwrap();
assert_eq!(
resp.status(),
StatusCode::SERVICE_UNAVAILABLE,
"an exhausted ADVERT pool must shed the receive-pack advertisement with 503 before touching the DB"
);
assert_eq!(
resp.headers()
.get("retry-after")
.and_then(|h| h.to_str().ok()),
Some("1"),
"the 503 shed must carry Retry-After"
);
}
/// PR3 (#62) sibling for the push path: git-receive-pack requires an
/// AuthenticatedDid extension (production: require_signature injects it), so the
/// request carries one via signed_request_as — without it the Extension
/// extractor 500s before the handler body reaches the shed. What sits at the top of
/// the handler is a permit-less `available_permits() == 0` peek, NOT the permit
/// itself: the authoritative held acquire is taken after the per-repo lease, so a
/// lease-blocked waiter pins no write slot. An ALREADY exhausted pool is the case
/// the peek delivers, so the request sheds 503 before its DB work here. Remove the
/// early-shed block from git_receive_pack and this goes red.
#[tokio::test]
async fn git_receive_pack_sheds_with_503_when_semaphore_exhausted() {
let mut state = test_state_lazy();
state.git_write_semaphore = Arc::new(tokio::sync::Semaphore::new(0));
let router = Router::new()
.route(
"/{owner}/{repo}/git-receive-pack",
axum::routing::post(crate::api::repos::git_receive_pack),
)
.with_state(state);
let owner = "did:key:zRECVSHEDOWNERAAAAAAAAAAAAAAAAAAAAAAAAAA";
let resp = router
.oneshot(signed_request_as(
owner,
Method::POST,
"/alice/repo.git/git-receive-pack",
Body::from(&b"0000"[..]),
))
.await
.unwrap();
assert_eq!(
resp.status(),
StatusCode::SERVICE_UNAVAILABLE,
"an exhausted write pool must shed git-receive-pack with 503 before touching the DB"
);
assert_eq!(
resp.headers()
.get("retry-after")
.and_then(|h| h.to_str().ok()),
Some("1"),
"the 503 shed must carry Retry-After"
);
}
/// #174 (SC1, load-bearing): a saturated READ pool must NOT shed an
/// authenticated push — the write pool is a separate budget. Read pool at zero,
/// write pool with capacity: the push proceeds PAST admission (it then errors on
/// the placeholder DB, but crucially it is not a 503). Route git-receive-pack
/// back to the read pool and this goes red — that is the isolation proof.
#[tokio::test]
async fn git_receive_pack_not_shed_by_exhausted_read_pool() {
let mut state = test_state_lazy();
// Read pool exhausted as if a flood of anonymous clones held every slot.
state.git_read_semaphore = Arc::new(tokio::sync::Semaphore::new(0));
// Write pool keeps its default capacity from test_state_lazy.
let router = Router::new()
.route(
"/{owner}/{repo}/git-receive-pack",
axum::routing::post(crate::api::repos::git_receive_pack),
)
.with_state(state);
let owner = "did:key:zRECVCROSSBOUNDARYAAAAAAAAAAAAAAAAAAAAA";
let resp = router
.oneshot(signed_request_as(
owner,
Method::POST,
"/alice/repo.git/git-receive-pack",
Body::from(&b"0000"[..]),
))
.await
.unwrap();
assert_ne!(
resp.status(),
StatusCode::SERVICE_UNAVAILABLE,
"an exhausted READ pool must not shed a push — the write pool is a separate budget (#174)"
);
}
/// N7: merge_pr is owner-only. A non-owner is rejected by require_repo_owner
/// before any git work (so no on-disk repo is needed for the rejection).
#[sqlx::test]
@@ -3285,7 +3525,10 @@ mod tests {
/// the handler skips the whole repo rather than serving. Asserts no leak of the
/// withheld blob AND that even the *public* blob in that repo is withheld — the
/// latter distinguishes fail-closed-skip from normal per-blob withholding and
/// would serve 200 if the error arm wrongly proceeded.
/// would serve 200 if the error arm wrongly proceeded. The skip carries no
/// VERDICT (F2), so the response is the retryable truncation 503, not a 404
/// claiming the object is absent — never-serve-unproven and never-404-unproven
/// hold together.
#[sqlx::test]
async fn ipfs_cid_walk_error_fails_closed(pool: PgPool) {
use crate::db::VisibilityMode;
@@ -3330,7 +3573,8 @@ mod tests {
.await
.expect("deny rule");
// Withheld secret CID under a walk error → 404, no leak.
// Withheld secret CID under a walk error → the repo is skipped without a
// verdict, so the scan is truncated (503), and nothing leaks.
let (st, body) = cid_parts(
cid_router(&state)
.oneshot(cid_anon(&secret_cid))
@@ -3340,17 +3584,17 @@ mod tests {
.await;
assert_eq!(
st,
StatusCode::NOT_FOUND,
"walk error must not serve the withheld blob"
StatusCode::SERVICE_UNAVAILABLE,
"walk error must not serve the withheld blob — the unproven skip sheds 503"
);
assert!(
!body.contains("TOP SECRET"),
"walk-error 404 must not leak the secret"
"walk-error 503 must not leak the secret"
);
// The PUBLIC blob in the same repo is also 404: the walk error fails closed
// by skipping the whole repo, not by serving. Without the fail-closed arm
// this would serve 200, so this assertion is the load-bearing discriminator.
// The PUBLIC blob in the same repo is also not served: the walk error fails
// closed by skipping the whole repo. Without the fail-closed arm this would
// serve 200, so this assertion is the load-bearing discriminator.
let (st, _) = cid_parts(
cid_router(&state)
.oneshot(cid_anon(&public_cid))
@@ -3360,8 +3604,9 @@ mod tests {
.await;
assert_eq!(
st,
StatusCode::NOT_FOUND,
"walk error fails closed: repo skipped, even the public blob is not served"
StatusCode::SERVICE_UNAVAILABLE,
"walk error fails closed: repo skipped without a verdict, even the public \
blob is not served and the scan sheds 503"
);
}
+489
View File
@@ -0,0 +1,489 @@
//! #174 U6 — INV-22 completeness guard (rung-raising).
//!
//! INV-22: a permit held per op recovers only if every path that holds it is also
//! duration-bounded and reaps the process group before releasing admission, and every
//! detached git/blocking task carries its own admission. PR #174 fixed five paths
//! (U1-U5) that violated this. Each fix has a per-unit RED/GREEN regression; together
//! those form the five-revert matrix. This guard adds the missing piece: a source-scan
//! tripwire that fails when a NEW site reintroduces the class, or when one of the five
//! gates is removed.
//!
//! It lives in `tests/` (a separate crate) on purpose: a guard that scanned the same
//! file it lives in would match its own identifier literals and pass vacuously. Here
//! the scanned `src/` files never contain this file's literals, so each check is
//! load-bearing — reverting the named gate turns the assertion red.
//!
//! These are deliberately coarse structural checks, not a parser. They cannot prove a
//! gate is *correct* (the per-unit tests do that); they prove a gate is *present and
//! not bypassed*, which is what stops the class from silently regressing.
use std::path::Path;
fn src(rel: &str) -> String {
let p = Path::new(env!("CARGO_MANIFEST_DIR")).join("src").join(rel);
std::fs::read_to_string(&p).unwrap_or_else(|e| panic!("read {}: {e}", p.display()))
}
#[test]
fn inv22_concurrency_gates_present_and_not_bypassed() {
let repos = src("api/repos.rs");
let smart_http = src("git/smart_http.rs");
let vis = src("git/visibility_pack.rs");
// U1 / P1-a: run_bounded_git stands the watchdog down only after confirming the
// child actually terminated (WNOWAIT), not on the raw stdout-drain EOF — otherwise
// a child that closes stdout then hangs pins the permit past the deadline. The
// probe is defined and called, so >= 2 occurrences; reverting the fix removes both.
assert!(
vis.matches("child_terminated_without_reaping").count() >= 2,
"U1/P1-a gate missing: run_bounded_git must confirm child exit via \
child_terminated_without_reaping before signalling the watchdog"
);
// U2 / P1-c: on client disconnect KillGroupOnDrop must launch a detached reaper
// that runs the full TERM/grace/KILL/reap, not a lone SIGTERM. The reaper is spawned
// via a runtime handle in Drop; `Handle::try_current` is unique to that launch (the
// timeout path already has an async context and never calls it).
assert!(
smart_http.contains("Handle::try_current"),
"U2/P1-c gate missing: KillGroupOnDrop::drop must launch the full \
TERM/grace/KILL reaper on disconnect (Handle::try_current), not a lone SIGTERM"
);
// U4 / P1-d: git_receive_pack must acquire the per-source write sub-cap before the
// global write permit. The acquire reads `state.git_write_per_caller`; comments name
// the field without the `state.` prefix, so this targets the real acquire site.
assert!(
repos.contains("state.git_write_per_caller"),
"U4/P1-d gate missing: git_receive_pack must acquire the per-source write cap \
(state.git_write_per_caller) before the global write permit"
);
// U5 / P1-e: the detached post-push encryption walk must run through the
// admission-gated helper, which is wired to the shared encrypt pool.
assert!(
repos.contains("fn withheld_recipients_gated")
&& repos.contains("state.git_encrypt_semaphore"),
"U5/P1-e gate missing: the encryption walk must run through \
withheld_recipients_gated, which acquires git_encrypt_semaphore"
);
// P1-e non-bypass tripwire: the bounded recipients walk is spawn_blocking'd nowhere
// but inside withheld_recipients_gated. A second call site (count > 1) is a new
// detached git walk that skips the admission gate — exactly the class U5 closed.
assert_eq!(
repos.matches("withheld_blob_recipients_bounded").count(),
1,
"P1-e bypass: the bounded recipients walk must be invoked only inside \
withheld_recipients_gated; a new call site bypasses the encrypt-walk admission cap"
);
// U4 / P2-2: the detached post-push encryption task must be gated by the per-repo
// coalescing set (`encrypt_inflight.try_begin`) so the OUTSTANDING parked-task set is
// bounded to <=1 per repo. Removing the gate lets N rapid pushes spawn N parked
// waiters (the unbounded set U4 closed); the semaphore only caps active walks.
assert!(
repos.contains("state.encrypt_inflight.try_begin"),
"U4/P2-2 gate missing: the detached post-push encryption spawn must consult \
encrypt_inflight.try_begin to coalesce per repo (bound the outstanding-task set)"
);
// F5: coalescing must REQUEUE, not shed. The in-flight task pins only its own
// pre-spawn snapshot, so a coalesced push's tip pairs are recorded on its key and
// the task must loop-drain them (`finish_or_take_pending`) before releasing it.
// Removing the drain reverts to the silent loss: a coalesced push's pins and
// recovery copies are absent until an unrelated later push. Scan only the
// production half of the file — the u5 tests in its `mod tests` also name the
// drain call, and matching them would make this check vacuous.
let repos_production = repos
.split("#[cfg(test)]")
.next()
.expect("split always yields a first chunk");
assert!(
repos_production.contains("finish_or_take_pending"),
"F5 gate missing: the post-push encryption task must loop-drain coalesced \
pushes via finish_or_take_pending before releasing its repo key"
);
// F4: every post-receive scan helper admits itself to the shared scan pool via
// `crate::state::acquire_scan_permit` BEFORE its spawn_blocking git walk, so a
// push burst cannot accumulate unbounded concurrent scans once the write permit
// is released. Two halves, both load-bearing: the helper body must actually
// acquire the pool (state.rs sits the helper at the end of the file, so the
// definition tail contains no other `acquire_owned` to match vacuously), and
// within each scan helper the first qualified call precedes the first
// `spawn_blocking`. Severing a call site pushes the next occurrence past the
// helper's own `spawn_blocking` (or off the end of the file), turning the
// assertion red; comments name the helper without the `crate::state::` prefix,
// so this targets the real call sites.
let state_rs = src("state.rs");
let helper_def = state_rs
.find("fn acquire_scan_permit")
.expect("F4 gate missing: state.rs no longer defines acquire_scan_permit");
assert!(
state_rs[helper_def..].contains("acquire_owned"),
"F4 gate gutted: acquire_scan_permit must acquire the scan pool via acquire_owned"
);
let push_delta = src("git/push_delta.rs");
for (file_src, file, helper) in [
(&repos, "api/repos.rs", "fn replication_withheld_set"),
(&repos, "api/repos.rs", "fn fail_closed_full_scan_objects"),
(
&push_delta,
"git/push_delta.rs",
"fn resolve_candidates_for_push",
),
] {
let start = file_src
.find(helper)
.unwrap_or_else(|| panic!("{file}: `{helper}` not found"));
let tail = &file_src[start..];
let acquire = tail
.find("crate::state::acquire_scan_permit(")
.unwrap_or_else(|| {
panic!("F4 gate missing: {file} `{helper}` no longer acquires a scan permit")
});
let spawn = tail.find("spawn_blocking").unwrap_or_else(|| {
panic!("{file}: `{helper}` lost its spawn_blocking walk — update this guard")
});
assert!(
acquire < spawn,
"F4 gate bypassed: {file} `{helper}` must acquire its git_encrypt_semaphore \
permit BEFORE dispatching the blocking git scan"
);
}
}
/// F4 (repo_store advisory-unlock cancellation safety): `RepoWriteGuard::release`
/// must await `pg_advisory_unlock` while `self` still owns the pooled connection,
/// and must not mark itself `released` until that await resolves. Either shape,
/// reintroduced, re-opens the mid-unlock cancellation leak: taking the connection
/// early leaves `Drop` with `conn == None`, and setting `released = true` early
/// leaves the `Drop` backstop inert — both strand the session lock on cancellation.
///
/// Scoped to the `release` fn body: the `Drop` impl legitimately takes the
/// connection and unlocks, so a whole-file scan would match it and read as a false
/// pass. Reverting either ordering turns this red (proven load-bearing).
#[test]
fn f4_release_keeps_conn_owned_until_unlock_resolves() {
let repo_store = src("git/repo_store.rs");
let rel_start = repo_store
.find("pub async fn release(mut self")
.expect("F4 gate: repo_store.rs no longer defines RepoWriteGuard::release");
let rel_end = repo_store[rel_start..]
.find("impl Drop for RepoWriteGuard")
.map(|off| rel_start + off)
.expect("F4 gate: release fn / Drop impl markers moved — update this guard");
let release_body = &repo_store[rel_start..rel_end];
let unlock = release_body
.find("pg_advisory_unlock")
.expect("F4 gate: release must still issue pg_advisory_unlock");
let before_unlock = &release_body[..unlock];
// (a) the connection must still be owned by `self` at the unlock await.
assert!(
!before_unlock.contains("self.conn.take()"),
"F4 regression: RepoWriteGuard::release takes self.conn BEFORE awaiting \
pg_advisory_unlock. A cancellation during the unlock await then strands the \
session advisory lock (Drop sees conn == None and skips its backstop). \
Unlock through the still-owned connection instead."
);
// (b) `released` must not be set before the unlock await — the other
// reintroduction shape a single-reorder check on (a) alone is blind to.
assert!(
!before_unlock.contains("released = true"),
"F4 regression: RepoWriteGuard::release sets `released = true` BEFORE awaiting \
pg_advisory_unlock. A cancellation during the await then leaves the Drop \
backstop inert (it early-returns on released). Set released only AFTER the \
unlock await resolves."
);
}
/// F6/KTD-5 (initial IPFS metadata queries deadline-wrapped): `get_by_cid` acquires
/// the scarce walk permits (RAII, held for the whole request) BEFORE its two initial
/// metadata queries, and the per-repo loop's first budget gate runs only later. So
/// both `list_all_repos` and `list_visibility_rules_for_repos` must be clamped to the
/// remaining request budget — otherwise a query blocked in Postgres pins the walk slot
/// for the whole stall, past the budget. This scans the PRODUCTION half of `api/ipfs.rs`
/// (the `mod tests` half names the same calls in its own harness and would make the
/// check vacuous) and asserts each query call is immediately preceded by a
/// `tokio::time::timeout(` wrapper. Removing either wrapper turns this red (proven
/// load-bearing).
#[test]
fn f6_ipfs_metadata_queries_are_deadline_wrapped() {
let ipfs = src("api/ipfs.rs");
let production = ipfs
.split("#[cfg(test)]")
.next()
.expect("split always yields a first chunk");
for call in [".list_all_repos()", ".list_visibility_rules_for_repos("] {
assert_eq!(
production.matches(call).count(),
1,
"F6 guard stale: `{call}` must appear exactly once in the production half \
of api/ipfs.rs (the deadline-wrapped handler call) — update this guard"
);
let idx = production
.find(call)
.unwrap_or_else(|| panic!("F6 gate: api/ipfs.rs no longer calls `{call}`"));
// The wrapper opens a few lines above the call (match tokio::time::timeout( ...
// remaining budget ..., <query> )); a 240-char lookback covers it without
// reaching the previous statement.
let window = &production[idx.saturating_sub(240)..idx];
assert!(
window.contains("tokio::time::timeout("),
"F6 gate missing: `{call}` must be wrapped in tokio::time::timeout(...) \
clamped to the remaining request budget. An unwrapped await pins the held \
walk permit for the whole DB stall, past GITLAWB_IPFS_REQUEST_BUDGET_SECS."
);
}
}
/// #174 F2 / KTD-3: the detached post-receive Pinata replication task must enqueue
/// only the push's `(ref, old, new)` tuples and RE-DERIVE its object set inside the
/// worker once a pin slot frees — it must NOT move the full per-push object list into
/// the closure and hold it across the `pin_semaphore` acquire. Retaining the list makes
/// every parked task (under a slow Pinata backend) hold an MB-scale OID list, so
/// outstanding memory grows O(pushes x object-list). Coalescing/shedding the task is
/// forbidden (its per-ref effects are non-idempotent), so the fix bounds the retained
/// data, not the task count.
///
/// Two load-bearing checks, both against the PRODUCTION half of `api/repos.rs` (the
/// `mod tests` half names the same identifiers in its own harness and would make the
/// scan vacuous): (a) the closure-local `object_list_pinata` binding — the retain form —
/// must be gone; reintroducing `let object_list_pinata = object_list;` turns this red.
/// (b) the re-derivation (`pinata_object_list_for_refs`) must appear AFTER the Pinata
/// pin permit is acquired, so the object list is materialized only inside the pin-bounded
/// section and a parked task holds O(ref tuples).
#[test]
fn f2_pinata_enqueues_refs_not_retained_object_lists() {
let repos = src("api/repos.rs");
let production = repos
.split("#[cfg(test)]")
.next()
.expect("split always yields a first chunk");
// (a) the retained-list form must be gone from production.
assert!(
!production.contains("object_list_pinata"),
"F2/KTD-3 regression: the Pinata task retains a full per-push object list \
(`object_list_pinata`) across the pin-permit acquire. Enqueue only the ref \
tuples and re-derive the object set inside the worker (pinata_object_list_for_refs)."
);
// (b) the re-derivation runs AFTER the pin permit is acquired. Anchor on the
// Pinata pin-admission clone so the window is the Pinata task, not the sibling
// IPFS/encrypt spawn (which shares `pin_semaphore` but never re-derives).
let anchor = production
.find("let pin_sem_pinata")
.expect("F2 gate stale: the Pinata pin-admission clone (pin_sem_pinata) moved");
let tail = &production[anchor..];
let acquire = tail
.find(".acquire_owned()")
.expect("F2 gate: the Pinata task no longer acquires a pin permit (acquire_owned)");
let rederive = tail.find("pinata_object_list_for_refs(").expect(
"F2 gate missing: the Pinata task must re-derive its object set via \
pinata_object_list_for_refs; it can no longer move a pre-resolved list into the closure",
);
assert!(
acquire < rederive,
"F2 gate bypassed: the Pinata object-set re-derivation must run AFTER the pin \
permit is acquired, so a parked task never holds the MB-scale object list"
);
// The re-derivation is driven by the push's ref tuples (the enqueued unit), not a
// retained object list — tie "enqueue ref_updates" to the call explicitly.
assert!(
tail[rederive..].starts_with("pinata_object_list_for_refs(")
&& tail[rederive..]
.get(..400)
.map(|w| w.contains("ref_updates_clone"))
.unwrap_or(false),
"F2 gate: pinata_object_list_for_refs must re-derive from the push's ref tuples \
(ref_updates_clone), the small unit the parked task retains"
);
}
/// #174 U2 / F3 (second same-repo writer serialized until a disconnected first push's
/// git group is reaped): `git_receive_pack` must take the per-repo in-process write
/// lease and CARRY it on the write-path `AdmissionGuard` (via `.with_lease`), so the
/// lease rides `KillGroupOnDrop`'s detached reaper and frees only after the group is
/// reaped — supplementing the pg advisory lock, which `RepoWriteGuard::Drop` releases at
/// the disconnect instant. Tying the lease to `RepoWriteGuard` instead (or dropping the
/// `.with_lease` carry) reopens the race; the behavioral RED/GREEN test
/// (`f3_second_push_serialized_until_disconnected_group_reaped`) is the real bar, this
/// is the completeness tripwire. Scanned against the PRODUCTION half of `api/repos.rs`
/// (the `mod tests` half names these identifiers in its own harness).
#[test]
fn f3_second_writer_leased_until_reap() {
let repos = src("api/repos.rs");
let smart_http = src("git/smart_http.rs");
let repos_production = repos
.split("#[cfg(test)]")
.next()
.expect("split always yields a first chunk");
// The lease is acquired, then carried by the AdmissionGuard (.with_lease), before
// receive_pack runs the write. Severing any of the three turns this red.
let lease_acquire = repos_production.find("repo_write_leases").expect(
"F3 gate missing: git_receive_pack no longer takes the per-repo write lease \
(state.repo_write_leases)",
);
let with_lease = repos_production.find(".with_lease(").expect(
"F3 gate missing: the write-path AdmissionGuard no longer carries the lease \
(.with_lease). The lease must ride the disconnect reaper via the AdmissionGuard, \
NOT RepoWriteGuard (which drops at the disconnect instant, reopening F3).",
);
let receive = repos_production
.find("smart_http::receive_pack(")
.expect("F3 gate stale: git_receive_pack no longer calls smart_http::receive_pack");
assert!(
lease_acquire < with_lease && with_lease < receive,
"F3 gate bypassed: the write lease must be acquired, then carried by the \
AdmissionGuard (.with_lease), BEFORE receive_pack runs the write"
);
// The AdmissionGuard must actually hold the lease (Option<RepoWriteLease>) via a
// with_lease setter, so it travels into the detached reaper. Removing the field or
// method turns this red.
assert!(
smart_http.contains("_lease: Option<crate::state::RepoWriteLease>")
&& smart_http.contains("pub fn with_lease("),
"F3 gate missing: AdmissionGuard must hold an Option<RepoWriteLease> set via \
with_lease, so the lease rides the guard into KillGroupOnDrop's reaper"
);
}
/// #174 U1 — every blocking walk in the `/ipfs` scan carries the request's walk
/// admission.
///
/// The admission is an `Arc<WalkAdmission>` cloned into each `spawn_blocking`
/// closure, so the global + per-source permits release only when the last holder
/// drops: a client disconnect leaves the abandoned closure holding the slot, and a
/// panicking closure leaves the handler holding it.
///
/// Only the walk site runs under `state.git_bin`, so only it can be pinned by the
/// fake-git harness and mutation-verified dynamically (see
/// `get_by_cid_walk_permit_held_through_blocking_walk`). The probe and the content
/// read deliberately shell to the real `git`, which is why this structural check
/// exists: it binds all three sites, and any blocking site added to this loop
/// later, without reversing that deliberate independence.
///
/// MUTATION (RED): delete any one `Arc::clone(&admission)` binding, or drop the
/// clone from inside its closure, and the count falls below three.
#[test]
fn inv22_ipfs_walk_admission_reaches_every_blocking_site() {
let ipfs = src("api/ipfs.rs");
// The shared owner must exist and be built once per request.
assert!(
ipfs.contains("struct WalkAdmission") && ipfs.contains("Arc::new(WalkAdmission {"),
"U1 gate missing: the /ipfs walk admission must be a shared WalkAdmission, \
not a handler-local permit pair"
);
// Every blocking site in the scan takes its own clone...
let clones = ipfs.matches("Arc::clone(&admission)").count();
assert!(
clones >= 3,
"U1 gate bypassed: expected an admission clone for each of the three \
/ipfs spawn_blocking sites (probe, walk, read); found {clones}. A blocking \
walk that does not hold the admission lets a disconnect or a panic free the \
slot while its git child is still running."
);
// ...and each clone is actually moved INTO the blocking closure, not merely
// created in the async frame (which would hold nothing across the join).
let held = ipfs.matches("let _admission = ").count();
assert!(
held >= 3,
"U1 gate bypassed: each admission clone must be bound inside its \
spawn_blocking closure so the blocking work owns it; found {held} of 3."
);
// The count of blocking sites is itself the thing being covered: if a fourth
// appears, it needs an admission clone too and this gate must be revisited.
let sites = ipfs.matches("spawn_blocking(move ||").count();
assert_eq!(
sites, 3,
"the /ipfs scan grew or lost a spawn_blocking site ({sites} found, expected 3); \
give any new blocking walk its own Arc::clone(&admission) and update this gate"
);
}
/// #174 U5: the post-receive replication tail is spawned at the DURABILITY BOUNDARY,
/// which is the moment receive-pack returns success, not the end of the handler and
/// not after `guard.release()`.
///
/// The tail owes this push its pins, recovery copy, and announcements. Everything
/// below the spawn stays in the cancellable request future, so anything the tail is
/// spawned after is a window where a client disconnect drops that work while the pack
/// is already durable on disk. `guard.release()` is such a window: on success it
/// awaits the Tigris upload and then the advisory unlock.
///
/// The lower bound matters just as much as the upper one: `release` runs on failure
/// too, so an ungated spawn would fire for a push git rejected, pinning and announcing
/// a half-applied repo. Above `release` the `?` on `receive_result` can no longer be
/// what gates it, so the success check is explicit and this gate binds it: the spawn
/// must sit inside `if push_succeeded`, and `release` must consume the same flag so
/// the two cannot drift apart.
///
/// This is an ordering check rather than a cancellation-race test on purpose: it is
/// the companion to `receive_pack_tail_survives_a_disconnect_during_release`, which
/// drives the actual disconnect through a parked `release`. Same instrument the F3
/// gate above uses.
///
/// MUTATION (RED): move the `tokio::spawn(post_receive_replication_tail` call below
/// `guard.release(` and the ordering assertion fails; take it out of the
/// `if push_succeeded` block and the failed-push assertion fails.
#[test]
fn inv22_replication_tail_spawns_at_the_durability_boundary() {
let repos = src("api/repos.rs");
// Production half only — the tests below name these identifiers too.
let production = repos
.split("#[cfg(test)]")
.next()
.expect("split always yields a first chunk");
let success_flag = production
.find("let push_succeeded = receive_result.is_ok();")
.expect(
"U5 gate missing: the tail's success gate must be bound from receive_result.is_ok()",
);
let gate_open = production
.find("if push_succeeded {")
.expect("U5 gate missing: the tail spawn must be gated on the push having succeeded");
let spawn = production
.find("tokio::spawn(post_receive_replication_tail(")
.expect("U5 gate missing: the replication tail must be spawned by git_receive_pack");
let release = production
.find("guard.release(push_succeeded)")
.expect("U5 gate stale: release must consume the same success flag as the tail gate");
let touch = production
.find("state.db.touch_repo(")
.expect("U5 gate stale: git_receive_pack no longer calls touch_repo");
let webhook = production
.find("webhooks::fire_event(")
.expect("U5 gate stale: git_receive_pack no longer fires push webhooks");
assert!(
success_flag < gate_open && gate_open < spawn,
"U5 gate bypassed: the tail must be spawned inside `if push_succeeded`, or a \
rejected push spawns a tail that pins and announces a half-applied repo"
);
// Still inside that block: no `}` may close it between the gate and the spawn.
assert!(
!production[gate_open + "if push_succeeded {".len()..spawn].contains('}'),
"U5 gate bypassed: the tail spawn left the `if push_succeeded` block, so a \
rejected push now spawns a tail"
);
assert!(
spawn < release && spawn < touch && spawn < webhook,
"U5 gate bypassed: the tail must be spawned BEFORE guard.release, touch_repo \
and the webhook fan-out, so a disconnect in any of those windows cannot drop \
this push's pins, recovery copy, and announcements"
);
}
+41
View File
@@ -720,4 +720,45 @@ mod tests {
assert!(failures.is_empty(), "\n{}", failures.join("\n"));
}
#[test]
fn new_normalizes_empty_api_key_to_none_on_trusted_origin() {
let guard = ICAPTCHA_ENV_LOCK.lock().unwrap();
let prev_key = std::env::var_os("GITLAWB_ICAPTCHA_API_KEY");
let prev_op = std::env::var_os("GITLAWB_ICAPTCHA_URL");
// Operator origin configured and resolved as trusted (no advert), so
// api_key reaches the env read and the emptiness filter is the only
// thing standing between a blank var and an empty bearer token.
std::env::set_var("GITLAWB_ICAPTCHA_URL", "https://icap.mynode.example");
// "Operator set the var but left it blank" must stay keyless: a
// Some("") would attach `Authorization: Bearer ` (empty token) to the
// challenge and answer requests instead of omitting the header.
std::env::set_var("GITLAWB_ICAPTCHA_API_KEY", "");
let empty = IcaptchaCfg::new("did:key:zTEST", None, None);
// Contrast arm: a real key on the same trusted path must survive the
// filter, so the filter can't regress to dropping every key.
std::env::set_var("GITLAWB_ICAPTCHA_API_KEY", "secret-bearer");
let non_empty = IcaptchaCfg::new("did:key:zTEST", None, None);
// Restore env and release the lock BEFORE asserting, so a failing
// assertion can't poison the shared lock or cascade into a sibling.
match prev_key {
Some(v) => std::env::set_var("GITLAWB_ICAPTCHA_API_KEY", v),
None => std::env::remove_var("GITLAWB_ICAPTCHA_API_KEY"),
}
match prev_op {
Some(v) => std::env::set_var("GITLAWB_ICAPTCHA_URL", v),
None => std::env::remove_var("GITLAWB_ICAPTCHA_URL"),
}
drop(guard);
assert_eq!(
empty.api_key, None,
"empty GITLAWB_ICAPTCHA_API_KEY must normalize to None"
);
assert_eq!(non_empty.api_key.as_deref(), Some("secret-bearer"));
}
}
@@ -0,0 +1,146 @@
//! Tripwire: a committed negative case that locks the blackhole mechanism.
//!
//! The guard against non-loopback egress works because `obtain_proof` builds a
//! stock `reqwest::blocking::Client` that inherits the process's proxy env vars.
//! If the builder later gains `.no_proxy()` or a custom connector, the guard
//! disarms silently and every existing test stays green (no live call is made
//! during the normal mock-consumed run because the mock is on loopback and
//! `NO_PROXY` covers it — the blackhole isn't even exercised).
//!
//! This test asserts that a destination the blackhole's `NO_PROXY` does NOT
//! cover fails *while the blackhole is armed*. The destination is a local HTTP
//! server on `[::1]` (IPv6 loopback), which `NO_PROXY` (`127.0.0.1, localhost`)
//! does not list. With the blackhole active the proxy intercepts the
//! connection and blocks it; without the blackhole the request would reach the
//! server directly and succeed, turning this test RED.
//!
//! A positive control (disarmed blackhole) runs first, proving the fixture is
//! valid and `[::1]` is reachable.
//!
//! Design constraints (see issue #211):
//! - An unresolvable host would keep the request failing even when the guard
//! is disarmed (DNS failure masquerading as the blackhole), so we must use
//! a reachable address.
//! - A real external host would make a live network call the moment the guard
//! disarms, and on any runner that cannot reach that host the connect error
//! again masquerades as the blackhole. A local loopback address avoids both
//! problems — it is always reachable when unblocked and never reaches the
//! real network.
//! - The address must NOT be covered by `NO_PROXY`, yet must exist on every
//! platform. An arbitrary `127/8` alias is not portable: macOS only binds
//! `127.0.0.1` by default. `[::1]` (IPv6 loopback) is present by default
//! on Linux, macOS, and Windows, and `NO_PROXY` only lists `127.0.0.1` and
//! `localhost`, so the blackhole still intercepts it.
//! - This depends on hyper-util's proxy matcher having no implicit loopback
//! bypass — a host is only proxied-around when literally listed in
//! `NO_PROXY` (verified in the pinned hyper-util; recheck on upgrade).
use std::io::{Read, Write};
use std::net::TcpListener;
use std::thread;
use icaptcha_client::{obtain_proof, Challenge, IcaptchaCfg};
mod support;
/// A minimal HTTP server that responds to iCaptcha challenge and answer
/// requests on the IPv6 loopback address NOT covered by NO_PROXY.
///
/// When the blackhole is working, the proxy should intercept these requests
/// and `obtain_proof` should fail. When the blackhole is disarmed, the
/// requests reach this server and the flow succeeds.
fn serve_icaptcha(listener: TcpListener) {
thread::spawn(move || {
for stream in listener.incoming() {
let Ok(mut stream) = stream else { continue };
// Read until the first \n (end of request line) so TCP
// segmentation cannot split the path out of a single read.
let mut buf = [0u8; 4096];
let mut n = 0usize;
while n < buf.len() {
match stream.read(&mut buf[n..]) {
Ok(0) | Err(_) => break,
Ok(read) => {
n += read;
if buf[..n].contains(&b'\n') {
break;
}
}
}
}
if n == 0 {
continue;
}
let request = String::from_utf8_lossy(&buf[..n]);
let body = if request.contains("/v1/answer") {
r#"{"status":"passed","proof":"PROOF-TRIP"}"#
} else {
r#"{"challengeId":"c1","type":"anagram","difficulty":1,"prompt":"listen","token":"tok-1"}"#
};
let response = format!(
"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
body.len(),
);
let _ = stream.write_all(response.as_bytes());
}
});
}
fn make_cfg(url: &str) -> IcaptchaCfg {
IcaptchaCfg {
url: url.to_string(),
did: "did:key:zTEST".to_string(),
level: 1,
api_key: None,
}
}
#[test]
fn obtain_proof_blackhole_tripwire() {
// Bind to [::1], IPv6 loopback, which NO_PROXY (127.0.0.1, localhost)
// does not cover. Unlike an arbitrary 127/8 alias, IPv6 loopback is
// present by default on Linux, macOS, and Windows, so the fixture is
// portable across platforms. Fail loudly if it cannot be bound: a
// silent skip would report this tripwire as green while exercising
// none of the negative control, recreating the very gap it exists to
// close.
let listener = TcpListener::bind(("::1", 0)).expect(
"cannot bind [::1] for the blackhole tripwire. IPv6 loopback (::1) \
is present by default on Linux, macOS, and Windows. This test fails \
rather than silently skipping so a platform without it cannot count \
the negative control as passed.",
);
let port = listener.local_addr().unwrap().port();
let url = format!("http://[::1]:{port}");
serve_icaptcha(listener);
// ── Positive control: disarmed blackhole → obtain_proof succeeds ──────
let _disarmed_guard = support::disarm_proxy_env();
let solve: &dyn Fn(&Challenge) -> Option<String> = &|_c| Some("silent".to_string());
let cfg = make_cfg(&url);
let proof = obtain_proof(&cfg, Some(solve))
.expect("positive control: obtain_proof should succeed when the proxy is disarmed");
assert_eq!(
proof, "PROOF-TRIP",
"positive control: unexpected proof value"
);
drop(_disarmed_guard);
// ── Negative control: armed blackhole → obtain_proof fails with connect error ──
let _armed_guard = support::arm_blackhole("http://127.0.0.1:1");
let result = obtain_proof(&cfg, Some(solve));
let err = result.expect_err(
"blackhole tripwire: obtain_proof succeeded against [::1], \
meaning the proxy blackhole did not intercept the request; \
the no-live-call guard is disarmed",
);
assert!(
err.chain().any(|c| c
.downcast_ref::<reqwest::Error>()
.is_some_and(|e| e.is_connect())),
"expected a connect/proxy error, got: {err:#}",
);
}
@@ -1,4 +1,4 @@
//! INV-19 guard: the icaptcha flow must actually call the service (consume the
//! INV-19 guard: the icaptcha flow must actually call the service (consuming the
//! mock), not short-circuit or make a live call. If a future change let
//! `obtain_proof` skip the network (e.g. a stray `cfg(test)` relaxation, which
//! INV-19/INV-20 warn is inert across crates), the `.assert()` calls below go
@@ -8,50 +8,144 @@
//! see a request that ALSO leaks the DID / answer / API key to `DEFAULT_URL` or
//! any other origin (a different host the mock never observes). To make the
//! "no live call" half of the guard load-bearing, the test blackholes every
//! non-loopback destination: `NO_PROXY` lets the loopback mock through while a
//! closed proxy port routes any other host nowhere, so the client contacting
//! anything but the injected mock fails the request instead of silently
//! succeeding. Point `cfg.url` at `DEFAULT_URL` and this test goes RED.
//! non-loopback destination via a proxy observer: `NO_PROXY` lets the loopback
//! mock through while every proxy variable routes any other host through a TCP
//! listener that counts connections. Zero observed connections proves no leak
//! reached the network on the exercised path.
//!
//! The observer design counts every accepted connection regardless of whether
//! the caller propagates the transport error, so it catches leaks that a
//! closed-port blackhole (which only surfaces propagating errors) would miss.
//! To close the timing race inherent in a detached accept loop, the count is
//! read only after a bounded drain: the test opens a flush connection and
//! blocks until the accept thread has accepted *that specific connection*
//! (tagged by its source port), not merely until any next counter tick.
//! Because `accept()` hands out connections FIFO, every leak connection that
//! completed its TCP handshake before the flush is counted before the flush —
//! the assertion therefore covers all egress in flight when `obtain_proof`
//! returned. A leak that starts its handshake after the drain is outside the
//! window; that is the irreducible limit of observing a detached accept loop,
//! and it is the remaining, well-defined gap this guard accepts.
//!
//! SCOPE: The observer only witnesses leaks issued on the path this test drives
//! (a single passing round, 200 on both endpoints, no PoW, no Continue/Failed,
//! no non-2xx response). A leak living on an error or retry branch makes no
//! connection during the run and the zero-count assertion stays green. The
//! guarantee covers only the exercised happy path and proxy-honoring transports;
//! a future `.no_proxy()` or raw `TcpStream` bypasses the observer entirely.
//!
//! `ALL_PROXY` alone is NOT enough: reqwest honors the scheme-specific
//! `HTTPS_PROXY` / `https_proxy` (and the http equivalents) OVER `ALL_PROXY`, so
//! a runner with a working `HTTPS_PROXY` in its environment would route an https
//! leak to `DEFAULT_URL` (which is https) through that proxy while the loopback
//! mock is still consumed and this test stayed green. So we blackhole the
//! scheme-specific variables too. No restore is needed: this file has a single
//! `#[test]`, so it runs in its own process and the override never races a
//! sibling test.
//! mock is still consumed and this test stayed green. So we blackhole the
//! scheme-specific variables too.
use std::net::{TcpListener, TcpStream};
use std::sync::{Arc, Condvar, Mutex};
use icaptcha_client::{obtain_proof, Challenge, IcaptchaCfg};
mod support;
/// A TCP listener that counts every accepted connection.
///
/// Acts as the proxy target: any leaked connection to a non-loopback
/// destination will be routed through this listener and counted. Every
/// accepted connection is recorded by its source port so the test's own
/// flush connection ([`ProxyObserver::drain`]) can be recognised and
/// excluded from the leak count.
struct ProxyObserver {
_listener: Arc<TcpListener>,
port: u16,
/// Source ports of every accepted connection, in accept order.
accepted: Arc<Mutex<Vec<u16>>>,
/// Source ports of the test's own flush connections (never counted).
flush_ports: Arc<Mutex<Vec<u16>>>,
/// Signalled after each accept so `drain` can wait on the flush.
ack: Arc<Condvar>,
}
impl ProxyObserver {
fn bind() -> Self {
let listener = Arc::new(TcpListener::bind("127.0.0.1:0").expect("bind proxy observer"));
let port = listener.local_addr().unwrap().port();
let accepted = Arc::new(Mutex::new(Vec::new()));
let flush_ports = Arc::new(Mutex::new(Vec::new()));
let ack = Arc::new(Condvar::new());
let a = Arc::clone(&accepted);
let c = Arc::clone(&ack);
let l = Arc::clone(&listener);
std::thread::spawn(move || {
for stream in l.incoming() {
let Ok(stream) = stream else { continue };
a.lock()
.unwrap()
.push(stream.peer_addr().map(|a| a.port()).unwrap_or(0));
c.notify_all();
}
});
ProxyObserver {
_listener: listener,
port,
accepted,
flush_ports,
ack,
}
}
fn port(&self) -> u16 {
self.port
}
/// Number of accepted connections that are not the test's own flush
/// connections — i.e. the leak count.
fn count(&self) -> usize {
let flush = self.flush_ports.lock().unwrap();
let accepted = self.accepted.lock().unwrap();
accepted.iter().filter(|p| !flush.contains(p)).count()
}
/// Bounded acknowledgement barrier for the accept loop.
///
/// Opens a flush connection and blocks until the accept thread has
/// accepted *that connection* (identified by its source port), bounded by
/// a timeout. Because `accept()` hands out connections FIFO, once the
/// flush is accepted every connection that completed its handshake before
/// it has been accepted and recorded too — so counting on afterwards
/// cannot miss a leak that was already in flight. The flush connection
/// itself is never counted.
///
/// A counter-tick barrier would be unsound here: sampling "current + 1"
/// and waiting for any increment lets a leak accepted after the sample
/// satisfy the wait, and the flush is then miscounted as that leak.
fn drain(&self) {
let flush =
TcpStream::connect(("127.0.0.1", self.port)).expect("observer flush connection");
let flush_port = flush.local_addr().unwrap().port();
self.flush_ports.lock().unwrap().push(flush_port);
let mut accepted = self.accepted.lock().unwrap();
while !accepted.contains(&flush_port) {
let (guard, timed_out) = self
.ack
.wait_timeout(accepted, std::time::Duration::from_secs(5))
.unwrap();
accepted = guard;
assert!(
!timed_out.timed_out(),
"proxy observer drain timed out; accept loop stalled"
);
}
}
}
#[test]
fn obtain_proof_consumes_the_mocked_service_and_makes_no_live_call() {
// Blackhole every non-loopback origin: the loopback mock bypasses the proxy
// (NO_PROXY); any other host is forced through a closed port and fails.
// reqwest reads these at client-build time, and obtain_proof builds its
// client fresh, so this bounds the transport for the whole flow.
//
// Set the scheme-specific vars too (both cases), not just ALL_PROXY: reqwest
// gives HTTPS_PROXY/https_proxy precedence over ALL_PROXY for https URLs, so
// an ALL_PROXY-only blackhole leaves an https leak to DEFAULT_URL open on a
// runner whose environment already has a working HTTPS_PROXY.
let blackhole = "http://127.0.0.1:1";
std::env::set_var("NO_PROXY", "127.0.0.1,localhost");
std::env::set_var("no_proxy", "127.0.0.1,localhost");
for var in [
"ALL_PROXY",
"all_proxy",
"HTTPS_PROXY",
"https_proxy",
"HTTP_PROXY",
"http_proxy",
] {
std::env::set_var(var, blackhole);
}
let observer = ProxyObserver::bind();
let _guard = support::arm_blackhole(&format!("http://127.0.0.1:{}", observer.port()));
let mut server = mockito::Server::new();
// The two endpoints the flow hits, each expected exactly once.
let challenge = server
.mock("POST", "/v1/challenge")
.with_status(200)
@@ -77,18 +171,29 @@ fn obtain_proof_consumes_the_mocked_service_and_makes_no_live_call() {
api_key: None,
};
// "anagram" is not a built-in solvable type, so the flow consults this
// solver callback. The answer body itself does not matter: the mocked
// /v1/answer returns "passed" regardless of what is submitted.
let solve = |_c: &Challenge| Some("silent".to_string());
let solver: &dyn Fn(&Challenge) -> Option<String> = &solve;
let count_before = observer.count();
let proof = obtain_proof(&cfg, Some(solver))
.expect("obtain_proof should complete against the mocked service");
assert_eq!(proof, "PROOF-XYZ");
// INV-19: prove the client actually called the mocked endpoints. Had it made
// a live call or skipped the network, these would fail (mock never hit).
challenge.assert();
answer.assert();
// Drain the accept loop before asserting: the flush connection is not
// counted, and because accept() is FIFO every leak connection that
// completed its handshake before the flush is recorded before the flush
// is — so the count after drain reflects all egress in flight during the
// call.
observer.drain();
let leak_count = observer.count() - count_before;
assert_eq!(
leak_count, 0,
"proxy observer caught leaked connections; \
the blackhole failed to contain egress"
);
}
@@ -0,0 +1,93 @@
//! Shared helpers for the icaptcha-client integration tests.
//!
//! CONCURRENCY CONSTRAINT: [`arm_blackhole`] and [`disarm_proxy_env`] mutate
//! process-wide proxy env vars and restore them on drop. Each test binary
//! that consumes this module currently has exactly one `#[test]`, so there is
//! no concurrent use. `cargo test` runs tests within a binary in parallel by
//! default, so the NEXT test added to either file MUST be serialized against
//! these helpers (e.g. a shared `Mutex` in this module, or the `serial_test`
//! crate) or it will race on the proxy vars.
use std::ffi::OsString;
const PROXY_VARS: &[&str] = &[
"NO_PROXY",
"no_proxy",
"ALL_PROXY",
"all_proxy",
"HTTPS_PROXY",
"https_proxy",
"HTTP_PROXY",
"http_proxy",
"REQUEST_METHOD",
];
/// Captures the current values of a set of env vars and restores them on drop.
pub struct EnvSnapshot(Vec<(String, Option<OsString>)>);
impl EnvSnapshot {
pub fn capture(vars: &[&str]) -> Self {
EnvSnapshot(
vars.iter()
.map(|v| (v.to_string(), std::env::var_os(v)))
.collect(),
)
}
}
impl Drop for EnvSnapshot {
fn drop(&mut self) {
for (var, val) in &self.0 {
match val {
Some(v) => std::env::set_var(var, v),
None => std::env::remove_var(var),
}
}
}
}
/// Guard that restores env vars captured at construction time.
pub struct EnvGuard {
_prev: EnvSnapshot,
}
/// Override every proxy-related env var so that non-loopback destinations are
/// forced through `proxy_url`, with NO_PROXY covering loopback. Also clears
/// REQUEST_METHOD, which causes hyper-util to ignore all proxy variables when
/// set. Restores prior values on drop.
///
/// Not safe to call from concurrent tests (see module docs).
pub fn arm_blackhole(proxy_url: &str) -> EnvGuard {
let prev = EnvSnapshot::capture(PROXY_VARS);
std::env::set_var("NO_PROXY", "127.0.0.1,localhost");
std::env::set_var("no_proxy", "127.0.0.1,localhost");
for var in [
"ALL_PROXY",
"all_proxy",
"HTTPS_PROXY",
"https_proxy",
"HTTP_PROXY",
"http_proxy",
] {
std::env::set_var(var, proxy_url);
}
std::env::remove_var("REQUEST_METHOD");
EnvGuard { _prev: prev }
}
/// Clear every proxy-related env var so connections go direct. Restores
/// prior values on drop.
///
/// Not safe to call from concurrent tests (see module docs).
#[allow(dead_code)]
pub fn disarm_proxy_env() -> EnvGuard {
let prev = EnvSnapshot::capture(PROXY_VARS);
for var in PROXY_VARS {
std::env::remove_var(var);
}
EnvGuard { _prev: prev }
}
+23
View File
@@ -0,0 +1,23 @@
#!/usr/bin/env bash
set -euo pipefail
tag="${1-}"
: "${GITHUB_OUTPUT:?GITHUB_OUTPUT must be set}"
case "$tag" in
""|*[!A-Za-z0-9._-]*)
printf '%s\n' "::error::release tag is empty or contains invalid characters"
exit 1
;;
esac
case "$tag" in
v[0-9]*) ;;
*)
printf '%s\n' "::error::'$tag' does not look like a release tag (vX.Y.Z)"
exit 1
;;
esac
printf '%s\n' "tag=$tag" >> "$GITHUB_OUTPUT"
printf '%s\n' "version=${tag#v}" >> "$GITHUB_OUTPUT"
+163
View File
@@ -0,0 +1,163 @@
#!/usr/bin/env bash
set -euo pipefail
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
resolver="$repo_root/scripts/resolve-release-tag.sh"
test_tmp="$(mktemp -d)"
trap 'rm -r -- "$test_tmp"' EXIT
valid_output="$test_tmp/valid-output"
GITHUB_OUTPUT="$valid_output" "$resolver" "v1.2.3"
expected_output="$test_tmp/expected-output"
printf '%s\n' "tag=v1.2.3" "version=1.2.3" > "$expected_output"
cmp "$expected_output" "$valid_output"
newline_output="$test_tmp/newline-output"
newline_stdout="$test_tmp/newline-stdout"
newline_stderr="$test_tmp/newline-stderr"
if GITHUB_OUTPUT="$newline_output" "$resolver" $'v1.2.3\nname=owned' \
> "$newline_stdout" 2> "$newline_stderr"
then
printf '%s\n' "newline-containing release tag unexpectedly passed" >&2
exit 1
fi
test ! -s "$newline_output"
grep -qxF "::error::release tag is empty or contains invalid characters" "$newline_stdout"
test ! -s "$newline_stderr"
empty_output="$test_tmp/empty-output"
if GITHUB_OUTPUT="$empty_output" "$resolver" ""; then
printf '%s\n' "empty release tag unexpectedly passed" >&2
exit 1
fi
test ! -s "$empty_output"
invalid_output="$test_tmp/invalid-output"
for invalid_tag in \
"v1.2.3 tag" \
"v1.2.3;name=owned" \
"v1.2.3+build.5" \
"release-1.2.3" \
"v-1" \
"vlatest" \
"v" \
"v1.2.3/../../x" \
"V1.2.3"
do
: > "$invalid_output"
if GITHUB_OUTPUT="$invalid_output" "$resolver" "$invalid_tag"; then
printf '%s\n' "invalid release tag unexpectedly passed: $invalid_tag" >&2
exit 1
fi
test ! -s "$invalid_output"
done
release_workflow="$repo_root/.github/workflows/release.yml"
actual_resolver_steps="$test_tmp/actual-resolver-steps"
expected_resolver_steps="$test_tmp/expected-resolver-steps"
# Pin each resolver step and the checkout step that supplies its script. Any
# change to one of these reviewed blocks must be reflected here deliberately.
awk '
function emit_step() {
if (in_step && (is_rel || is_workflow_scripts_checkout)) {
printf "job=%s\n%s", job, step
}
in_step = 0
is_rel = 0
is_workflow_scripts_checkout = 0
step = ""
}
/^ [A-Za-z0-9_-]+:[[:space:]]*$/ {
emit_step()
job = $0
sub(/^ /, "", job)
sub(/:[[:space:]]*$/, "", job)
next
}
/^ - / {
emit_step()
in_step = 1
is_workflow_scripts_checkout = ($0 ~ /^ - name:[[:space:]]*Check out workflow scripts[[:space:]]*$/)
step = $0 ORS
next
}
in_step {
step = step $0 ORS
if ($0 ~ /^ id:[[:space:]]*rel[[:space:]]*$/) {
is_rel = 1
}
}
END {
emit_step()
}
' "$release_workflow" > "$actual_resolver_steps"
cat > "$expected_resolver_steps" <<'EOF'
job=docker
- name: Check out workflow scripts
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
job=docker
- name: Resolve release tag
id: rel
env:
DISPATCH_TAG: ${{ inputs.docker_backfill_tag }}
RELEASE_TAG: ${{ needs.release-please.outputs.tag_name }}
run: |
set -euo pipefail
scripts/resolve-release-tag.sh "${DISPATCH_TAG:-$RELEASE_TAG}"
# ghcr requires a lowercase repository path, and unlike metadata-action,
# buildx's `--output name=` does no lowercasing — a mixed-case owner
# makes the digest push fail with "invalid reference format".
echo "image=ghcr.io/${GITHUB_REPOSITORY,,}" >> "$GITHUB_OUTPUT"
job=docker-manifest
- name: Check out workflow scripts
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
job=docker-manifest
- name: Resolve release tag
id: rel
env:
DISPATCH_TAG: ${{ inputs.docker_backfill_tag }}
RELEASE_TAG: ${{ needs.release-please.outputs.tag_name }}
run: |
set -euo pipefail
scripts/resolve-release-tag.sh "${DISPATCH_TAG:-$RELEASE_TAG}"
job=npm-publish
- name: Check out workflow scripts
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
job=npm-publish
- name: Resolve release tag
id: rel
env:
DISPATCH_TAG: ${{ inputs.npm_backfill_tag }}
RELEASE_TAG: ${{ needs.release-please.outputs.tag_name }}
run: |
set -euo pipefail
scripts/resolve-release-tag.sh "${DISPATCH_TAG:-$RELEASE_TAG}"
EOF
if ! cmp "$expected_resolver_steps" "$actual_resolver_steps"; then
printf '%s\n' \
"release workflow resolver steps differ from the three reviewed blocks" >&2
diff -u "$expected_resolver_steps" "$actual_resolver_steps" >&2 || true
exit 1
fi
printf '%s\n' "release tag validation tests passed"