Compare commits

..
Author SHA1 Message Date
beardthelion 370a364b0f test(node): cover the sweep's boot wiring and prove an interrupted push never reaches the upload
Two claims from earlier commits were argued rather than executed. Both are now
observed.

The legacy-CID sweep's spawn sat inline in main() where no test could reach it,
so the batch size, the inter-batch delay, the detachment and the shutdown select
were all reasoned. The block moves verbatim into `spawn_legacy_cid_sweep`, which
a test calls directly. It seeds rows whose keys are already canonical, so the
cost gate skips each and the persisted cursor reads out walk progress cleanly,
then asserts the cursor stops at the configured batch boundary, the call returns
without blocking, the task is still alive inside its delay, and shutdown ends it.
Never spawning, reading the batch or delay from the wrong knob, or removing the
shutdown branch each turns it red. What remains reasoned is the single call line
inside main(), down from a twenty-six line block.

The claim that a disconnect-interrupted push never uploads to Tigris was
structural: the tests build the store with no Tigris client, so a counter inside
the client arm could never move and would have proved nothing. The counter now
sits at the upload decision, before the client is consulted, and the disconnect
test asserts zero while the success test asserts exactly one. That positive
control is what makes the zero meaningful, and it is itself load-bearing:
releasing the success path with failure semantics drops it to zero, and adding a
Drop impl that uploads raises the disconnect count to one.

The counter observes reaching the upload site, not an S3 request. TigrisClient
wraps a concrete SDK client with no injectable endpoint, and plumbing one through
would be out of proportion to the risk here.
2026-07-27 21:44:31 -05:00
beardthelion 03aee267e6 fix(node): shed 503 from the issue and PR write paths when the lock pool is exhausted
The lock-pool work gave `acquire_write` a distinct exhaustion error and mapped it
to 503 with Retry-After, but only on the push handler. The three other callers,
both issue writes and the PR merge, still turned it into a generic git error and
returned 500. Those are exactly the callers that hold no concurrency permit, so
they are the likeliest to meet an exhausted pool in the first place.

All three now go through the push handler's helper rather than repeating the
mapping, so the log level and the message cannot drift between the four sites.
Each has a test that occupies the single connection of a one-slot pool with a
guard on a different repo, so the failure is pool capacity rather than
advisory-lock contention, and asserts status 503 plus retry-after through
IntoResponse. Reverting the mapping at one site turns only that site's test red.

Worth knowing for review, and left alone deliberately: close_issue takes the
write guard before its author check, so this shed is reachable by any
authenticated caller whether or not they could close the issue. That ordering
predates this change and reordering the handler is not this commit's business,
but it does mean the 503 discloses that the repo's write pool is busy.
2026-07-27 21:14:55 -05:00
beardthelion c519d54fc1 fix(node): keep the incompleteness marker until a source is really recorded, and let the sweep retry
Four follow-ups from an adversarial pass over the previous commits.

The marker clear ran unconditionally inside `record_pin_source`'s transaction,
but its INSERT is `ON CONFLICT DO NOTHING` guarded by the cap, so it records
nothing when the repo is already a source. That is the common case, not a rare
one: the pin path calls `record_pin_source` for every already-pinned object, and
on a requeue pass that is the whole-repo enumeration. So a re-push from the repo
that was already recorded cleared a marker set by a different repo's failure and
the public copy went back to 404ing. The clear is now gated on the insert
actually adding a row. A genuine record from a third repo still clears the
marker, which needs a per-(oid, repo) table to close and is documented rather
than papered over.

The sweep advanced its cursor before every skip, which stops a wedge but made a
transient skip permanent: once the walk reached the end of the table the cursor
parked there and no later boot read another row. On a Tigris-backed node the
common skip is "repo is not on local disk", so the sweep would repair whatever
happened to be warm at boot and then never run again, and the new advertise
filter turned those rows from "advertised but 404" into "never advertised, never
repaired". Repairs now report whether a skip was retryable or terminal, and a run
that skipped anything retryable rewinds the cursor once, after the walk, so the
next run re-walks. A permanently unrepairable row is terminal and does not
trigger a rewind, so this cannot become a hot loop.

The sweep also reached its repo paths through the raw join helper, skipping the
three-layer traversal barrier `RepoStore::local_path` applies. A test with a repo
row named `../../escapee` showed it reading and repairing bytes from outside
`repos_dir`. That barrier is now extracted and shared, so the two callers cannot
drift. And the repair's synchronous `git cat-file` moved under `spawn_blocking`,
matching the existing treatment of the withheld walk, so a wedged read no longer
parks a worker thread for the full service timeout.

Finally the resolver's two source-set queries ran ahead of the work-budget peek
that exists to shed a spent caller before doing work. The peek now runs first. A
throttled caller whose set turns out complete gets 429 where it used to get 404,
which is the honest answer since its search never ran, and it removes an oracle
that let a throttled caller tell a complete source set from an incomplete one.
2026-07-27 18:29:01 -05:00
beardthelion 1c82060c2b fix(node): stop the lock pool from being occupied by spinners and racing the reaper
Three regressions from the previous four commits, each demonstrated before being
fixed.

The advisory-lock retry loop checked its connection out before spinning, so a
caller that lost the lock held a pool connection through all sixty one-second
attempts. The old pool-per-query shape returned it between iterations, so a
spinner occupied nothing. That matters more than the pool sizing suggested:
`acquire_write` has three non-push callers (two in issues, one in pulls) that
hold no concurrency permit, and the issue routes carry no rate limit, so any
self-minted DID could fire enough concurrent closes at one repo to strand the
pool and fail authenticated pushes on every repo. The checkout now happens per
attempt and a losing attempt returns the connection before sleeping, so only the
winner retains one. Lock-pool exhaustion also carries its own error now and sheds
503 with Retry-After instead of surfacing as a 500.

Holding the lock on one connection then meant a client disconnect released it
early: the handler future dies mid-receive-pack, the guard drops without
`release`, and `after_release` frees the lock while the detached reaper is still
giving the process group its SIGTERM grace. A second writer was admitted 90ms
after the drop, well inside that window, which is the invariant smart_http
already documents. The write guard now rides the AdmissionGuard into the reaper,
the same seam the admission permits already use, so the lock outlives the group.
The success path still reclaims the guard and releases it synchronously, and a
dropped guard performs no Tigris upload, so an interrupted push cannot publish a
half-applied repo.

The requeue loop's exit on an exhausted re-read discarded a push that coalesced
during the retry window: `requeue_or_release` had cleared the dirty bit without
marking the guard released, so `Drop` removed the key and no pass was ever
attempted. It now falls through with nothing to replicate and lets the tail's
atomic check-and-clear decide. A lap only happens when a push actually coalesced
and each pays a full bounded re-read, so a sustained outage still terminates,
which is tested with ten thousand injected faults under a watchdog rather than
argued.

Also clamps the lock pool's derived size, which previously followed
max_concurrent_git_pushes up to its million-connection ceiling, and documents the
node's total connection budget on that knob.
2026-07-27 17:47:56 -05:00
beardthelion 73429c7a56 fix(node): sweep legacy provider-CID pins instead of waiting for a re-push
`repair_legacy_provider_cid` had exactly one trigger: the already-pinned skip
branch inside `pin_new_objects`, which a push only reaches when it re-carries the
object. Normal git negotiation omits objects the node already has, so on an
upgraded node that push generally never arrives. Meanwhile `list_pinned_cids`
kept advertising the stored provider CID that this branch's `/ipfs/{cid}`
deliberately withholds, so the node handed out CIDs it would then refuse to
serve, and previously pinned data stayed unresolvable.

Two halves, both needed. A detached background sweep walks `pinned_cids` in
bounded batches with an inter-batch delay, resolves each row's repo from its
recorded provenance, and reuses the existing repair (and its cost gate, so a row
already keyed on a raw CIDv1 reads no bytes). Migration v16 adds a single-row
cursor table so a restart mid-walk resumes rather than rewinding; the cursor
advances on every row read, before any skip, so an unrepairable row cannot wedge
the walk. A row whose bytes are gone, whose provenance is NULL, or whose repo is
not on local disk is left exactly as it is. And `list_pinned_cids` now omits rows
whose key is not a raw CIDv1, so the window before the sweep catches up
advertises nothing the resolver withholds.

The sweep resolves paths directly rather than through `repo_store.acquire`, which
would pull cold repos back from Tigris and turn a repair pass into a bulk
restore. Defaults are deliberately quiet: 64 rows per batch, 60s between batches.

Ten tests cover the repair, the bytes-gone and unrepairable arms, the cost gate,
the batch bound, cursor resumption, both states of one row through the advertise
filter, and the degenerate empty and no-legacy-rows tables. Removing the filter,
the batch bound, the cursor advance, the bytes-gone early return, or the
inter-batch sleep each turns a test red.

One pre-existing test used placeholder resolver keys that the new filter drops;
its values are now real raw CIDv1s with every assertion unchanged.
2026-07-27 15:53:26 -05:00
beardthelion fccdfc1820 fix(node): record a failed pin-source write so the resolver keeps its fallback
The resolver treated a non-empty, below-cap pin-source set as proof that every
source had been recorded, and skipped the bounded scan on that basis. But
`record_pin_source` is best effort at every call site: the ipfs_pin sites were
warn-only once their retries exhausted, and both pinata sites had no retry at
all. An object first pinned from a private repo and later pushed from a public
one whose source write failed keeps a set naming only the private source, so
`GET /ipfs/{cid}` returned 404 for an object the public repo would have served.

Migration v15 adds `pinned_cids.pin_sources_incomplete`, NOT NULL DEFAULT FALSE
so every existing row reads as complete. A record that fails outright sets it; a
later successful record clears it, inside the same transaction as the insert so
the two paths cannot drift. `needs_scan` now ORs the marker in, short-circuiting
after the existing empty and at-cap checks so it costs nothing on the serve path.
Pinata's two bare calls now share `retry_db_record` with ipfs_pin. The ipfs_pin
first-pin path additionally records the CID and its source in one transaction;
Pinata's remains two retried calls.

INV-10 is preserved: a complete set still tail-404s with no preload, asserted on
the existing preload counter. The fallback is not an authorization bypass, and
the test for that asserts the scan actually ran before checking the denial, so
the denial cannot pass vacuously. Dropping the marker term from the gate, forcing
it true, or removing the clear each turns a test red.

The marker is per-object rather than per-(object, repo): a successful record from
one repo clears a marker set by another's failure. That is the conservative
direction, since the marker only ever adds the fallback back.
2026-07-27 14:40:03 -05:00
beardthelion 8e6e2ad0fe fix(node): retry the requeue re-read so a coalesced push is not silently dropped
`requeue_or_release` clears the dirty flag atomically with the decision to loop,
which is correct and stays as it is. The problem was what the pass did next: a
failed `get_repo_by_id` fell into an Err arm that yielded no rules, and
`list_visibility_rules(...).ok()` collapsed its own error the same way. With no
rules, `replication_withheld_set` returns `(false, None)`, the pass skipped its
work entirely, and the loop exited having consumed the dirty bit that stood for
the coalesced push. There is no reconciliation sweep to re-derive it, so that
push's objects were never pinned and its withheld blobs never sealed.

`requeue_refresh_state` now separates the three cases the old code collapsed.
`Ok(Some)` proceeds. `Ok(None)` means the repo really is gone and releases
without touching the retry budget. `Err` from either read is transient and is
retried with doubling backoff up to three attempts. A rules-read error is no
longer indistinguishable from a repo that has no rules.

Exhausting the retries still drops the pass, which is the pre-existing residual,
but it now logs at error level with the repo id and attempt count instead of
disappearing behind a warning. Six tests cover the retried-then-lands path, the
bounded-exhaustion path, repo-gone, the rules-read arm, freshness of the applied
rules, and the coalescing property that had to survive. Reverting the Err arm,
restoring the `.ok()`, raising the bound, or downgrading the error log each turns
a test red.
2026-07-27 13:52:44 -05:00
beardthelion 9a70d75d13 fix(node): own the repo write lock on one connection so cancellation cannot leak it
`acquire_write` took `pg_try_advisory_lock` through the shared pool, then awaited
the Tigris exists/download path before constructing `RepoWriteGuard`. Two bugs
fell out of that. The `tokio::time::timeout` this branch added around the call
can drop the future inside that window, after the lock is held and before the
only unlock (`RepoWriteGuard::release`) exists, wedging every later push to the
repo. And because a session advisory lock belongs to the connection that took
it, running the lock and the unlock through the pool lets them land on different
connections: the unlock returns false and leaks, while a competing acquire that
draws the holding connection re-enters the lock and two writers run against the
same repo concurrently.

`RepoStore` now owns a dedicated lock pool built with an `after_release` hook
that runs `pg_advisory_unlock_all()`. `acquire_write` checks out one connection,
takes the lock on it, and moves it into the guard, which unlocks on that same
connection. sqlx's `PoolConnection::drop` spawns `return_to_pool`, which runs the
hook, so a connection dropped by cancellation still clears its locks; the unlock
is asynchronous with respect to the drop, which the tests account for. The pool
is separate from the main query pool because a push holds its connection for the
whole receive-pack and `db_max_connections` (20) sits below
`max_concurrent_git_pushes` (32).

Seven tests cover the cancellation window, the same-session release, and the
serialization the pool-mismatch bug was breaking. Both the cancellation
regression and the concurrent-writer case were observed red before the fix, and
removing the `after_release` hook or moving the unlock back to the pool turns
them red again.
2026-07-27 13:22:45 -05:00
t 62a99270a9 fix(node): bound the post-push pin and repair reads so a wedge cannot pin the coalescing key
A cross-model adversarial pass found that run_post_push_replication holds the
per-repo coalescing key until requeue_or_release, but pin_new_objects reached it
only after two unbounded git reads: the U7 legacy-repair read and the
pre-existing pin read, both plain Command::output with no teardown. A wedged
cat-file on a stuck backend hung the task forever, so the key was held until
process death and later pushes only marked it dirty without spawning a
replacement (the same class this PR closed on the /ipfs serve path).

Add read_object_bounded (the bounded twins under one shared deadline) and use it
at both sites; a timeout skips that object and lets the task proceed to
requeue_or_release. Adds a wedge-reap regression proven load-bearing two ways.
2026-07-20 21:06:10 -05:00
t fba3c6fda3 fix(review): share one deadline across the /ipfs size+content reads
Code review found each candidate's size and content reads were each granted a
full git_service_timeout, so a single served candidate could hold the /ipfs
walk permit for 2x the timeout. Share one deadline across both stages (mirrors
build_filtered_pack) so the read holds admission for one timeout total, and
correct the admission-bound comment: the worst case is O(candidates x timeout)
bounded by the walk concurrency cap, not a single deadline.
2026-07-20 20:27:35 -05:00
t 45e269b2ae fix(node): opportunistically repair legacy provider-CID pins
Before this PR the pin path stored the provider CID (Kubo dag-pb / Pinata) in
pinned_cids.cid, which the new resolver recomputes-and-404s, while
/api/v1/ipfs/pins still advertises the stale value. Repair a legacy row in the
already-pinned skip branch: gate on a cheap stored-CID codec check (only a
non-raw-codec CID reads bytes, so raw rows keep the DB-only skip cost),
recompute the raw CID, and rewrite it while stashing the old value in a new v14
legacy_provider_cid column (distinct from pinata_cid, which gates the Pinata
pin-skip). Rows whose bytes are gone stay withheld. Documents that the deferred
one-shot sweep, not this opportunistic path, fully retires the window.
2026-07-20 20:11:42 -05:00
t b774c435a6 fix(node): retry pin-source and pinned-CID recording on transient errors
The three warn-only record sites in the detached post-push task dropped a
transient DB failure silently, leaving a permanently incomplete pin-source set
that makes the resolver 404 a valid public copy. Wrap them in a bounded retry
(inserts are idempotent via ON CONFLICT DO NOTHING); on exhaustion the warn
still fires so behavior degrades to today's. Documents the residual crash/outage
window that only a reconciliation sweep retires.
2026-07-20 19:50:23 -05:00
t 0a1618f7fb fix(node): split /ipfs work accounting off the route rate limiter
The route middleware charged ipfs_rate_limiter once per request while the
legacy scan and provenance walk charged the same bucket per probe/walk, so a
one-probe request cost two tokens and GITLAWB_IPFS_RATE_LIMIT=1 admitted the
request at the route then 429'd it at the pre-scan peek.

Add a bounded ipfs_work_rate_limiter, move the in-scan, per-walk, and pre-scan
charges onto it, and leave the route limiter as the pure once-per-request
brake. The work-budget capacity derives from the route limit with a floor of
one full legacy-probe budget per window (no new operator knob), so a
default-config deep search never self-throttles. Adds it to the periodic sweep,
updates all three AppState construction sites, and rewrites the contradictory
limiter comments.
2026-07-20 19:41:57 -05:00
t c2fce67bae fix(node): wire GITLAWB_IPFS_MAX_REPOS_WALKED to the legacy-probe budget
The knob was parsed and documented as the per-request /ipfs fan-out cap but
never read in production: AppState seeded the legacy-probe budget from a fixed
256 constant, so setting the knob did nothing and an operator could still incur
256 acquire/probe operations per request.

Seed ipfs_max_legacy_probes from the knob (default raised to 256 to preserve
shipped behavior); leave the history-walk ceiling on its MAX_PIN_SOURCES+1
constant so a provenanced request is never truncated into a false 503. Updates
the README and .env.example to the shipped behavior and adds cap-honored and
plumbing tests.
2026-07-20 19:17:12 -05:00
t 3351e09c08 fix(node): requeue coalesced pushes instead of dropping them
Per-repo coalescing dropped every push that arrived while a repo's encryption
task was in flight, and no reconciliation ever processed it, so a withheld
blob added by the coalesced push had its recovery copy permanently absent. The
drop also silently skipped the coalesced push's local-IPFS pin work.

EncryptInflight becomes a dirty-flag map; the detached task loops, and the
atomic check-and-clear sits at the task tail, unconditional on the
has_path_scoped_rule gate and walk success (a public or rules-free repo would
otherwise exit before it ran). Each requeue re-reads repo state fresh and
re-enumerates the pin half through the fail-closed full scan, never bare
list_all_objects, so a coalesced rule change is honored and no withheld or
dangling object leaks. At-most-one-task-per-repo is preserved.
2026-07-20 19:04:55 -05:00
t 757edf560c fix(node): own /ipfs admission for the life of the walk
get_by_cid held the global walk permit and the per-source permit as handler
locals, so dropping the request future released both while the spawn_blocking
probe, visibility walk, and object read kept running; a cancel-spam or timeout
client could exceed MAX_CONCURRENT_IPFS_WALKS and the per-source cap, and the
bare cat-file probe/read children had no teardown at all.

Move the gated serve pipeline into a detached task that owns an AdmissionGuard,
awaited by the handler, so admission releases only after the bounded work is
gone; back the /ipfs probe and read with run_bounded_git twins for
process-group teardown. Adds cancel-mid-walk, timeout-reap, and cancel-spam
regressions plus two load-bearing INV-22 rows. Also reflows one U1 test line to
satisfy cargo fmt.
2026-07-20 18:34:09 -05:00
t 86828046ea fix(node): hold read admission through the filtered-pack reaper
The path-scoped upload-pack branch kept the read and per-caller permits as
handler locals and called upload_pack_excluding with no AdmissionGuard, so a
cancelled filtered clone released admission while KillGroupOnDrop was still
reaping the git group, bypassing the read and per-caller concurrency caps.

drive_git_child now returns the disarmed guard on success so one guard rides
both build_filtered_pack stages (rev-list then pack-objects); the handler builds
the guard from the permits as the plain path does. Adds a disconnect regression
and an INV-22 gate row, both proven load-bearing.
2026-07-20 17:54:49 -05:00
t d26cf6dd1d fix(node): close the /ipfs pin-source griefing hole via a bounded scan fallback
An attacker who pins a public object from MAX_PIN_SOURCES repos before the
legitimate public source registers fills pin_repo_sources (record_pin_source is
first-N-wins and drops later sources silently); the buried public source is then
never reached because a non-empty provenance set suppressed the legacy scan, so
anon GET /ipfs/{cid} 404s forever for a public object (re-breaks F1).

U1: db::pin_sources_at_cap reports whether pin_repo_sources is at MAX_PIN_SOURCES
for an oid (the only observable signal that a servable source may have been
dropped, since the write cap never overshoots).

U2: get_by_cid falls back to the bounded legacy scan on a provenance miss when the
set is empty OR at_cap. The scan gates every repo through the real per-caller gate,
so it finds the buried public copy. A complete (non-full) set still fast-404s, so
ordinary denials never fan out to O(repos) (INV-10/F3); the fallback honors the
is_throttled peek.

Regression ipfs_cid_buried_public_source_still_serves_via_scan_fallback: 404 before,
200 after; pin_sources_at_cap_flips_at_max covers the boundary.
2026-07-17 10:15:56 -05:00
t 94ca744269 fix(node): align ipfs_pin return with pinata; add Retry-After to SearchIncomplete
- ipfs_pin::pin_new_objects returns the provider Hash (not the raw resolver key),
  matching the pinata twin's contract; the DB cid stays the raw content CID. The
  return is logging-only here, so this is drift-avoidance, not a functional fix.
- AppError::SearchIncomplete now carries Retry-After: 1 like Overloaded, so both
  retryable 503s from the /ipfs handler advertise the retry hint consistently.
2026-07-17 10:15:56 -05:00
t 2eea354188 fix(node): tie the /ipfs walk ceiling to MAX_PIN_SOURCES + 1
A provenanced object has a bounded source set (first-pinner + up to
MAX_PIN_SOURCES additional). With the walk ceiling at 16 == MAX_PIN_SOURCES, an
authorizing public source sorting after 16 path-scoped denials was never reached:
the ceiling truncated the search and returned a false 503 for a readable object.
Raise the ceiling to MAX_PIN_SOURCES + 1 so the whole bounded provenance set is
always tried; the legacy scan stays bounded by MAX_LEGACY_PROBES_PER_REQUEST.
2026-07-17 10:15:56 -05:00
t c767658b7b fix(node): bound the /ipfs cat-file probes under the walk permit; backfill pinata provenance
P2-E: the object-type probe and the F6 size+read ran bare git cat-file inside
spawn_blocking while the /ipfs walk permit was held, so a wedged cat-file could
pin the global walk slot for the request's life. Bound both under
git_service_timeout_secs; on timeout free the slot (truncated -> retryable 503)
and skip the repo.

P2-D: the pinata already-pinned branch now backfills NULL first-pinner
provenance in lockstep with the ipfs_pin skip branch (consistency; the object
was already resolvable via the pin_repo_sources union).
2026-07-17 10:15:56 -05:00
t 8afc9eaf97 fix(node): close grok P1 findings on the #173/#174 IPFS integration
P1-A: gate_and_serve held the /ipfs walk permit across a bare repo_store.acquire;
wrap it in git_acquire_timeout_secs (mirroring #174's own handler) so a cold/hung
Tigris acquire cannot pin the global walk slot. On expiry skip the repo and mark
the search truncated (retryable 503, not a false 404).

P1-B: the local-IPFS pin path recorded Kubo's provider Hash as pinned_cids.cid;
for objects above the block size that is a dag-pb root that does not hash the raw
content, so GET /ipfs/{cid} listed then 404'd them under the F2 integrity check.
Record the locally-computed raw-content CID, mirroring the pinata twin.
2026-07-17 10:15:56 -05:00
t 433c8915bf feat(node): integrate the #173 IPFS CID tree-gate onto the #174 walk-concurrency base
Rebased onto #174 (fix/served-git-concurrency-cap). #173's incremental history
cannot replay commit-by-commit because #174 rewrote the same handler files in
parallel; this single commit carries the fully-integrated tree (identical to the
verified merge result), with the follow-up fixes as separate commits on top.

Brings in #173: GET /ipfs/{cid} CID->oid resolution via pinned_cids, per-caller
path-scoped blob/tree/commit-tag gating (#135/#173 F1-F6), and the pin-source
provenance table. Adapts #173's tree/commit-tag walks to #174's run_bounded_git
so the held /ipfs walk-concurrency permit is duration-safe (F5).
2026-07-17 10:15:56 -05:00
t 413d6cf0bc fix(node): reject unsupported git service before the read slot; document the new admission knobs (#174)
P2-1: git_info_refs treated every service other than git-receive-pack as a read op
and did the DB/visibility/Tigris work under a read permit, validating the service
string only downstream in smart_http. So an unauthenticated `?service=anything` to
a public repo consumed a read slot + storage work before rejection. Validate the
service is exactly git-upload-pack or git-receive-pack immediately after parsing,
returning 400 before the pre-DB shed and the permit. Handler-layer regression:
with the read pool exhausted, `?service=git-explode` returns 400 (validated first),
not 503 — removing the validation makes it 503 (RED).

U5/INV-24 docs: document the new knobs from U2/U3 in .env.example and the README
env-var table — GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS, GITLAWB_MAX_CONCURRENT_IPFS_WALKS,
GITLAWB_IPFS_WALK_PER_SOURCE, GITLAWB_IPFS_MAX_REPOS_WALKED, GITLAWB_IPFS_RATE_LIMIT.
Each is already enforced on the path it names (U2/U3 tests), so the docs match the
implemented routing.
2026-07-15 13:16:40 -05:00
t 394768711e fix(node): bound the post-push encryption task set by per-repo coalescing without dropping recovery copies (#174)
After a successful path-scoped push the handler spawns a DETACHED task that runs
pin_new_objects then withheld_recipients_gated, which acquire_owned().await's on
git_encrypt_semaphore — it DEFERS (blocks) when the pool is full. The semaphore
caps active walks but nothing capped how many detached tasks were spawned and
PARKED on that await: N rapid path-scoped pushes spawned N tasks, each holding
cloned object lists/rules/paths/keys — an unbounded parked-waiter set (P2-2).

2a54c15 deliberately chose defer-not-shed because dropping the walk loses the
withheld-blob recovery copy and there is no reconciliation sweep to rebuild it.
So bound the OUTSTANDING-TASK set by per-repo coalescing rather than shedding:
EncryptInflight (a bounded Arc<Mutex<HashSet<repo_id>>>) + try_begin — before the
spawn, if a task for the repo is already in-flight skip the duplicate (the pending
walk covers the newer objects); otherwise spawn and move an RAII
EncryptInflightGuard into the task that removes the key on drop (completion, error,
or panic-unwind). This bounds the set to <=1 pending task per repo and never drops
work; withheld_recipients_gated's defer is unchanged.

The second detached post-push spawn (Pinata replication, also runs pin_new_objects)
is scoped out with rationale: it parks on no semaphore (no unbounded waiter set)
and does per-push per-ref work (branch->CID, gossip, subscription, Arweave,
peer-notify) keyed to this push's ref_updates — coalescing it would DROP a later
push's announcements, a correctness regression.

Regressions (mutation-verified RED->GREEN): the outstanding set is bounded to 1
per repo under saturation (never-coalesce -> 32 tasks RED); a coalesced repo's key
is released when its task ends so it is reprocessed, never permanently skipped
(guard-drop no-op -> repo locked out, recovery copy lost forever, RED — the exact
durability regression the fix prevents); distinct repos each admit; cold-set first
push admits; plus an inv22_gates structural tripwire that fails if the handler gate
is removed.
2026-07-15 13:10:04 -05:00
t f424ccbbcd fix(node): gate the /ipfs walk with bounded concurrency admission + per-source cap + route rate limit (#174)
GET /ipfs/{cid} (get_by_cid, publicly reachable via optional_signature) ran
allowed_blob_set_for_caller_bounded in spawn_blocking inside a per-repo loop with
NO global/per-source admission and NO route rate limit — the surface had zero
concurrency or rate control. A permissionless caller could fan out concurrent
full-history git walks (each up to the walk timeout) exhausting blocking-pool
threads and PIDs outside every served-git pool (P1-3).

Acquire a global git_ipfs_walk permit (try_acquire_owned, shed 503) plus a
per-source sub-cap (with_default_max_keys, reject-before-insert, keyed via
client_key/TrustedProxy so XFF can't spoof it) ONCE after CID validation, held as
handler locals across the whole request — including every spawn_blocking walk, so
the slot reflects real blocking-thread occupancy (a tokio timeout cannot cancel
spawn_blocking). None key -> global pool only. Cap repos walked per request so one
request can't serialize N full-history walks. Attach rate_limit_by_ip +
IpRateLimiter to the /ipfs route (mirrors the git/create/peer routers; the
extension is required or the middleware is a silent no-op). Visibility/authz
semantics unchanged.

New knobs: GITLAWB_MAX_CONCURRENT_IPFS_WALKS (32), GITLAWB_IPFS_WALK_PER_SOURCE
(4), GITLAWB_IPFS_MAX_REPOS_WALKED (64), GITLAWB_IPFS_RATE_LIMIT (600/hr, 0
disables) — all range-validated and enforced on-path.

Handler-layer regressions (all mutation-verified RED->GREEN): shed-at-capacity
503 (delete acquire -> falls through), per-source cap sheds same source / admits
another, None-key arm sheds on the global pool, map self-bounds reject-before-insert,
the walk permit is held while the spawn_blocking walk runs (available_permits()==0;
dropping it before the loop -> RED), repos-walked cap (remove break -> 2 walks),
and the route IP rate limit actually fires 429 (drop the Extension -> RED no-op).
2026-07-15 12:42:55 -05:00
t 8d17038e5b fix(node): bound the served-git storage-acquisition phase with a release-on-expiry deadline (#174)
The served-git admission permit is acquired, then RepoStore::acquire*/advisory-lock
runs before the bounded git runner starts — with no local deadline. acquire/acquire_fresh
await Tigris HEAD/GET, and acquire_write's advisory-lock loop awaits a per-iteration
pg_try_advisory_lock that can block indefinitely on a hung Postgres pool. Since
GITLAWB_GIT_SERVICE_TIMEOUT_SECS only starts once git spawns, a stalled backend pins
the permit and drains the pool until every later request 503s (P1-2).

Wrap each git-path acquire (git_info_refs read/advert, git_upload_pack, git_receive_pack)
and the /ipfs per-repo acquire loop in tokio::time::timeout(git_acquire_timeout_secs, ..).
On expiry the git paths return AppError::Overloaded (503 + Retry-After); the permit is a
handler-local at that point (moved into the AdmissionGuard only after acquire) so the early
return frees the slot. The /ipfs per-repo acquire keeps its fail-closed `continue` on
timeout (never serves an un-acquired repo). No repo_store.rs signature change: the outer
timeout cancels a mid-sleep/mid-fetch_one future, so no in-loop deadline is needed.

New knob GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS (default 30, range 1..), modeled on
git_service_timeout_secs, kept separate because acquisition and git execution are distinct
cost centers.

Handler-layer regression receive_pack_acquire_deadline_sheds_and_releases_permit: holds the
same pg advisory lock acquire_write derives on a second connection so the real loop must
retry, drives git-receive-pack with the deadline at 2s and write pool size 1, and asserts
503 at ~2s + the permit recovers + a follow-up admits once the lock frees. Removing the
timeout wrapper turns it RED (blocks ~31s to the test ceiling on the ~59s advisory loop).
2026-07-15 12:00:02 -05:00
t 72e899dae0 fix(node): hold served-git admission until the process group is reaped on the plain spawn paths (#174)
On the plain (non-path-scoped) info_refs / upload-pack / receive-pack paths the
global + per-source admission permits were handler-locals that dropped the
instant the handler future dropped on a client disconnect, while KillGroupOnDrop
reaps the git process group in a DETACHED task (SIGTERM -> grace -> SIGKILL ->
reap). So a disconnect-spammer admitted replacements while prior groups were
still alive, defeating the process cap and the per-source cap during teardown.
be0cdd6 already fixed this for the path-scoped upload-pack walk; this closes the
residual plain paths (P1-a).

Introduce AdmissionGuard, a move-only holder of the permits, threaded through
info_refs/upload_pack/receive_pack/run_git_service into drive_git_child's
KillGroupOnDrop. On disconnect the guard moves into the detached reaper and drops
only after the group is ESRCH-confirmed reaped; on success/timeout disarm()
returns it for the earliest provably-free drop. The rev-list/pack-objects
visibility-walk callers hold no admission and pass None.

Thread git_bin through upload_pack/receive_pack (they hardcoded "git") so the
plain POST path is drivable by an injected fake git, matching the existing
injectable seam on info_refs/run_git_service/upload_pack_excluding.

Handler-layer regression upload_pack_plain_permit_held_through_group_reap_after_disconnect:
isolates the global read pool (size 1, per-source + rate limiter permissive), a
SIGTERM-ignoring descendant keeps the group alive across the reap window, and
asserts available_permits()==0 held on disconnect + a cross-source replacement
sheds 503, then frees after ESRCH. Reverting the guard-into-reaper move (release
immediately) turns it RED at the held-through-reap assertion. Plus a None-key arm.
2026-07-15 12:00:02 -05:00
t 5749df6e67 test(node): INV-22 completeness guard for the served-git concurrency surface (#174)
A source-scan tripwire that fails when a new site reintroduces the concurrency-cap
class PR #174 fixed, or when one of the five gates (U1-U5) is removed. INV-22: a
permit held per op recovers only if every path is duration-bounded and reaps the
group before releasing admission, and every detached git task carries admission.

Lives in tests/ (a separate crate) on purpose — a guard scanning the file it lives
in would match its own identifier literals and pass vacuously; scanning src/ from
here keeps every check load-bearing. Checks: run_bounded_git confirms child exit via
child_terminated_without_reaping (U1); KillGroupOnDrop launches the reaper via
Handle::try_current (U2); git_receive_pack acquires state.git_write_per_caller (U4);
the encryption walk runs through withheld_recipients_gated over git_encrypt_semaphore
(U5); and the bounded recipients walk is invoked exactly once — only inside the gated
helper — so a new spawn_blocking bypass (count > 1) fails (P1-e non-bypass).

Proven load-bearing: a synthetic second call site turns the count tripwire RED, and
each named gate maps to a fix whose own regression already goes RED on revert.
2026-07-14 20:33:01 -05:00
t 2a54c159e6 fix(node): bound concurrent post-push encryption walks with an admission pool (#174)
After a successful path-scoped push the handler released its write permit and
then ran a DETACHED tokio task whose spawn_blocking(withheld_blob_recipients_bounded)
performed another full-history git walk under no admission. N fast completed
pushes spawned N concurrent full-history walks past GITLAWB_MAX_CONCURRENT_GIT_PUSHES
(which bounds only the in-handler phase), and the per-child bounding from 71389e6
caps each walk's duration but not their count (P1-e).

Add git_encrypt_semaphore, a pool of its own (sized from max_concurrent_git_pushes,
no new knob — Q1) so a long background walk never holds a foreground write slot and
a handler holding a write permit can't self-deadlock. Route the walk through a new
withheld_recipients_gated helper that acquires the pool before the walk and DEFERS
(blocks) when full rather than shedding — dropping the walk would lose the
withheld-blob recovery copy, so durability stays fail-closed.

Regression: encrypt_walk_defers_when_pool_exhausted drives the gated helper with the
pool exhausted and asserts the walk blocks and does not run its rev-list, then runs
once a permit frees. RED before (the walk runs regardless of the pool), GREEN after.
The detached push task calls this exact helper.
2026-07-14 20:28:52 -05:00
t 12643573f6 fix(node): cap concurrent receive-pack pushes per source IP (#174)
The authenticated git-receive-pack POST acquired only the global write semaphore
— no per-source sub-cap, unlike the read and anon-advert pools. Owner enforcement
defaults off, so one host minting disposable did:key identities could open
max_concurrent_git_pushes slow POSTs from a single source IP and 503 every other
source's push; the 600/hour push limiter bounds arrival rate, not in-flight
concurrency (P1-d).

Add git_write_per_caller (a PerCallerConcurrency sized like the advert cap,
max_concurrent_git_pushes/8), and acquire it before the global write permit,
keyed on the resolved source IP via read_caller_key — never the signed DID (a DID
farm defeats a DID key). This required adding the PeerAddr + HeaderMap extractors
to git_receive_pack, which it lacked (without them the key is None and the cap is
inert). Neutralized the shared acquire helper's log wording now that it serves
both read and write paths.

Regression: receive_pack_per_source_write_cap_sheds_capped_source_not_others — a
source at its write sub-cap sheds Overloaded/503 (proving the extractors resolve a
key), while a different source is not shed. RED before (the capped source proceeds
past admission to a git error instead of shedding), GREEN after; the existing
receive-pack advert cap test still passes.
2026-07-14 20:20:49 -05:00
t be0cdd6a9e fix(node): hold upload-pack admission through the walk on disconnect (#174)
On the path-scoped upload-pack path the per-source and global read permits were
handler locals, so a client disconnect dropped the handler future and released
both permits while the awaited spawn_blocking withheld-blob walk kept running
(spawn_blocking can't be cancelled). A permitless caller could disconnect-spam to
exceed the read cap while real git work continued (P1-b).

Move both permits into the blocking task and return them out: on success they
flow back so the serve phase keeps them; on a dropped future the returned tuple
(with the permits) is discarded only when the blocking task completes, so
admission tracks the walk's real duration.

Regression: upload_pack_permit_held_through_walk_after_disconnect drives the
handler to mid-walk (fake git hangs on rev-list), disconnects, and asserts the
global read slot stays held (available_permits == 0). RED before (the slot frees
to 1 the instant the future drops), GREEN after. The test roots the repo store at
its own TempDir so it is isolated per run.
2026-07-14 20:09:40 -05:00
t ac59bd7e01 fix(node): reap the served-git process group on disconnect, not just SIGTERM (#174)
On a client disconnect KillGroupOnDrop sent a lone SIGTERM with no escalation or
reap, so a group member that ignores SIGTERM (a wedged pack-objects, a malicious
helper) survived while the handler released its concurrency permit — letting
disconnect-spam accumulate live git processes and defeat the load shed (P1-c).
The timeout path already ran the full teardown.

Make the disconnect teardown as strong as the timeout path: the guard now owns
the child and, on drop, launches a detached reaper that runs the same
TERM -> grace -> SIGKILL -> reap-the-group sequence (reap_group_on_timeout).
Owning the tokio Child in the reaper keeps it the sole reaper of the leader, so
it never races tokio's own orphan reaper (a raw waitpid(-pgid) would).

Regression: run_git_service_sigkills_a_sigterm_ignoring_child_on_disconnect, RED
before (the descendant survives the lone SIGTERM), GREEN after; all existing
teardown/timeout/disarm tests still pass (the three hand-built-guard tests were
updated to the owned-child shape).

Residual: capacity (the handler's semaphore permit) still releases at disconnect
rather than strictly after the reaper confirms the group dead, so accumulation is
now bounded to ~one teardown window (<=4s) rather than eliminated; gating the
permit on the reap needs threading it through the serve handlers and is left for
a follow-up (it also touches the receive-pack permit lifetime U4 changes).
2026-07-14 19:50:41 -05:00
t 3cc01890fb fix(node): keep the visibility-walk watchdog armed until the child is reaped (#174)
A group member (or the leader itself) that closes stdout before the deadline made
the stdout drain return EOF early, which stood the watchdog down via done_tx
before it could ever fire; child.wait() then blocked on the still-live child,
pinning the walk thread and its read/write permit past
GITLAWB_GIT_SERVICE_TIMEOUT_SECS. This is distinct from the descendant case
already covered: there the leader sleeps until the deadline so the watchdog does
time out; here the drain-EOF races ahead of it.

Stand the watchdog down only once the child has actually terminated, detected
without reaping (waitid + WNOWAIT) so the leader's pid stays unreaped and its
pgid un-recycled until the watchdog finishes its teardown and is joined. Past the
deadline the watchdog still owns the full SIGTERM -> grace -> SIGKILL sequence.

Regression: run_bounded_git_reaps_a_leader_that_closes_stdout_then_hangs, RED
before (blocks on child.wait past the recv budget), GREEN after; the three
existing teardown tests still pass (no recycle-hazard regression).
2026-07-14 19:27:12 -05:00
t a030eefb70 test(node): retry the ETXTBSY exec race in build_filtered_pack timeout tests (#174)
build_filtered_pack_times_out_a_hung_rev_list and _a_hung_pack_objects
spawn a freshly-written fake git directly, bypassing the fake_git_run_with_pids
retry the rest of the module uses. Under cargo test fork-storm load a concurrent
worker forks while the fake's write fd is still open, so execve returns ETXTBSY
and the raw io::Error surfaces where the test asserts GitServiceTimeout, failing
test (stable) intermittently.

Route both tests through build_filtered_pack_or_exec_race_retry, which retries
ONLY errno 26 (ETXTBSY) and keeps the outer 10s watchdog per attempt, so a
genuinely-missing async timeout bound still trips the watchdog and a wrong error
type still fails at the assertion. Production paths unchanged.
2026-07-13 16:01:49 -05:00
t cf97405bc1 test(node): execution-cover the delta-path push-scan timeout wiring (#174)
The finding-2 wiring test covered only list_all_objects (the full-scan fallback).
Add delta_path_exec_fns_time_out_on_a_hung_git covering the common push path:
peeled_object_type (cat-file), rev_list_delta (rev-list), and list_all_objects_with_type
(cat-file, which all_blob_oids delegates to). Each must return within the watchdog
budget on a hung git; reverting any to a bare Command::output() blocks past the recv
budget (RED-verified).
2026-07-13 10:55:57 -05:00
t 4410b834df docs(node): correct concurrency + timeout docs to match the final git routing (#174)
jatmn P2 (INV-24): the config help, .env.example, and README described routing the
handler no longer does.

- The read pool serves upload-pack and the UPLOAD-PACK info/refs advertisement only;
  the anon receive-pack info/refs advertisement draws from its own dedicated pool
  (sized like the write pool but disjoint), not the read pool and not the write pool.
- The write pool is the authenticated push POST only.
- The per-caller read cap is keyed on the resolved source IP, never the DID: a
  signature does not move a caller off it (the docs claimed a per-DID budget, which
  would tell a NATed authenticated client it escapes the shared cap when it does not).
- git_service_timeout_secs now also bounds the push-side pin-candidate discovery
  (rev-list / cat-file), reaped via process-group teardown, alongside the info/refs
  advertisements and the withheld-blob walk.
2026-07-13 10:46:24 -05:00
t 9412aa9c5b fix(node): bound and reap push-candidate git children under the write permit (#174)
Route push_delta's candidate-discovery git children (cat-file / rev-list) through the
bounded run_bounded_git (deadline + process-group SIGKILL teardown) instead of bare
Command::output(). A hung scan no longer pins the git-receive-pack write permit past
the deadline or a client disconnect. Threads git_bin + a shared per-scan deadline
(git_service_timeout_secs) through resolve_candidates_for_push and all_blob_oids.
2026-07-13 10:41:31 -05:00
t 7a6bd359eb fix(node): SIGKILL a SIGTERM-ignoring member of the visibility-walk process group (#174)
jatmn/CodeRabbit P1: the walk watchdog stood down as soon as the group LEADER was
reaped. A leader that exits cleanly on SIGTERM while a background member ignores
SIGTERM and closes its inherited pipes let the main drain unblock, the leader be
reaped, and the watchdog skip its SIGKILL escalation — so the handler returned while
the descendant kept running. jatmn's suggested "continue until ESRCH" could not work:
the main thread reaping the leader is what frees the pgid, so continuing to kill(-pgid)
after that races a recycled group.

Restructure the coordination instead: the main thread now defers reaping the leader
until the watchdog thread returns. The watchdog always escalates through an
unconditional group SIGKILL (no leader-reap short-circuit), and because the leader is
still unreaped while every kill(-pgid) fires, the pgid cannot have been recycled. The
grace before SIGKILL is a fixed budget (only paid on the exceptional timeout path).

Regression run_bounded_git_sigkills_a_sigterm_ignoring_descendant_after_leader_exits:
a leader that exits 0 on SIGTERM spawns an sh -c descendant (its own $$, not a ( )
subshell's parent pid) that ignores SIGTERM and closes its pipes; the descendant must
be dead when the runner returns. RED on the old teardown (survives), GREEN now. The
existing leader-ignores-SIGTERM and hung-walk teardown tests still pass.
2026-07-13 10:11:41 -05:00
t 473924dcfe fix(node): compile the bounded visibility git runner on non-Unix targets (#174)
jatmn P1: run_bounded_git was gated #[cfg(unix)], but its callers
(assert_all_refs_are_commits, blob_paths) and the module are compiled on all
targets, so a non-Unix build (the continue-on-error Windows release target) could
not resolve the function.

Add a #[cfg(not(unix))] twin with the same signature and result semantics. It bounds
a single child (no process-group teardown, which is Unix-only): threads feed stdin
and drain stderr, the main thread drains stdout, and a watchdog thread kills the
child at the deadline. The child is shared with the watchdog behind a mutex the main
thread does not hold while draining, so the watchdog can always acquire it to kill,
and killing closes stdout to unblock the drain. Best-effort (reaps only the direct
child), which is why the hardened group-aware path stays Unix-only.

Verified the twin type-checks by compiling its (portable-std) body on the native
target; the full non-Unix link is CI's Windows job.
2026-07-13 09:49:33 -05:00
t f19f2813a2 fix(node): give the receive-pack advert its own pool so it can't shed authed pushes (#174)
jatmn P1: the anon-reachable GET info/refs?service=git-receive-pack drew from the
same git_write_semaphore as the authenticated git-receive-pack POST. The per-source
advert cap reserves no global slots, so ~8 sources (each within the per-IP push rate
limit) could hold all 32 write permits across acquire_fresh + the advertisement, and
the next authenticated push then 503s at admission. That is an anonymous caller
shedding an authenticated push across the auth boundary (INV-10).

Add a dedicated git_push_advert_semaphore, disjoint from the write pool. Both the
pre-DB early-shed peek and the held acquire for the receive-pack advertisement now
draw from it; the write pool is left exclusively for the POST. An advert flood can at
worst exhaust the advert pool (each source still capped by git_push_advert_per_caller
and the per-IP push rate limiter); the authenticated push pool is untouched.

Test receive_pack_advertisement_draws_from_dedicated_advert_pool proves at the handler
layer that the advert sheds when its own pool is saturated AND survives when the write
pool is saturated (the reservation). Both flip under revert, verified independently for
the pre-DB peek and the held acquire.
2026-07-13 09:44:53 -05:00
t 79b9ee5c89 fix(node): shed the served-git pool before the DB and cap per-source before it (#174)
CodeRabbit review: the held git_permit was acquired at the top of git_info_refs
and git_upload_pack, before the per-source cap, so one source could occupy the
global pool during the DB/visibility window before its sub-cap rejected the
excess. Move the held git_permit to after acquire_read_caller_permit (still
before acquire_fresh/git, so INV-10's bound on the fresh Tigris acquire holds),
and add a cheap pre-DB early shed (available_permits() == 0 -> 503, holds no
permit) so the #62 shed-before-DB property is preserved without holding a permit
across the DB work.

The receive-pack POST is unchanged (owner-only; its top-of-handler git_permit is
deliberate for acquire_write bounding).

Tests: the two #62 shed-before-DB tests now exercise the explicit early check
(load-bearing: disable it and they fall through to the DB); three new
*_per_source_cap_sheds_with_global_capacity tests prove the per-source sub-cap
sheds a capped source even with global capacity free (load-bearing on the caller
cap). Full workspace green.
2026-07-12 14:40:59 -05:00
t 454e59ba53 docs(node): the withheld-walk is bounded by the service timeout, not a fixed const (#174)
Follow-up to threading the configured timeout into the walk: the walk now honors
GITLAWB_GIT_SERVICE_TIMEOUT_SECS on both the serve and replication paths, so the
README no longer says a fixed internal deadline.
2026-07-12 02:12:37 -05:00
t c35452ed41 test(node): vet the withheld-walk bound end to end; use the configured timeout (#174)
Close the reasoned-not-run gaps from the code review by making the walk's git
binary and timeout injectable, then driving the missing branches with a real
handler and a fake git instead of reasoning about them.

- Add state.git_bin and *_bounded variants of the walk entry points taking
  (git_bin, timeout); the served handlers (upload-pack serve, receive-pack
  replication and full-scan and encrypt-pin, and the ipfs gate) now pass the
  operator-configured GITLAWB_GIT_SERVICE_TIMEOUT_SECS, so the whole walk is bounded
  by the same budget as the other served-git ops rather than a fixed constant. The
  git_bin-less wrappers stay for the real-git classification tests.

Newly vetted by execution (not reasoning):
- receive-pack replication path is bounded: replication_withheld_set with an injected
  hung git returns within the budget and fails closed, so it cannot pin the write
  permit git_receive_pack holds across it.
- a hung withheld-blob walk on the upload-pack POST returns 504 (real handler, real
  repo on disk, injected hung git), proving the GitServiceTimeout -> git_service_app_error
  wiring end to end.
- the watchdog status-gate: a child that exits successfully is not reported as a
  timeout even when the watchdog fired (mutation-checked: drop the guard -> RED).
- SIGKILL escalation: a SIGTERM-ignoring child is still reaped via SIGKILL and the
  group is gone; a truly uninterruptible D-state child (unreapable by any signal) is
  the documented residual, matching the async teardown.
- the advertisement per-source cap sizing never derives 0.

Full gitlawb-node suite 515 green.
2026-07-12 02:12:08 -05:00
t e750c3042d fix(review): cap the receive-pack advertisement per source so anon cannot starve the write pool (#174)
U2 moved the receive-pack info/refs advertisement onto git_write_semaphore to keep
an anonymous read flood from starving the push handshake. But the advertisement is
anon-reachable on public repos and holds its write permit across the slow
acquire_fresh Tigris download, and the only per-source brake on it was the push
RATE limiter, not a concurrency cap. So a multi-source flood of receive-pack
advertisements could hold the write pool's slots across those downloads and shed
authenticated pushes (both the advertisement and the owner-gated git-receive-pack
POST draw from the same pool). U2 thus introduced the first anonymous consumer of
the write pool the state doc promised anon could never reach; the plan's residual
note (no worse than the POST) was wrong, because the POST is owner-gated and the
advertisement is not.

Add git_push_advert_per_caller, a per-source concurrency sub-cap on the receive-pack
advertisement keyed on the resolved source IP (the same PerCallerConcurrency
mechanism U1 uses for reads), sized to an eighth of the write pool so a single
source holds at most that share and saturating the pool takes many distinct source
IPs, each also braked by the per-IP push rate limiter. The upload-pack advertisement
keeps its read-pool per-caller cap; the owner-gated POST is unchanged. Correct the
state doc for git_write_semaphore accordingly.

Handler-layer proof: a source at its receive-pack advertisement cap sheds 503 (RED
before the acquisition, 500-not-503), while a different source and the upload-pack
advertisement are unaffected. Full suite 510 green.
2026-07-12 00:56:24 -05:00
t 87a7e40513 fix(review): close the withheld-walk watchdog reused-pgid race (#174)
Code review found run_bounded_git's watchdog could return a spurious 504 and
signal a recycled process group. The watchdog runs off a wall clock on its own
thread; done_tx.send() only fires after child.wait() reaps the leader, so a walk
that finished within microseconds of the deadline took the watchdog's Timeout
branch, discarded a fully-captured successful result, and returned GitServiceTimeout
(a 504 for a walk that actually completed). Worse, the Timeout branch SIGTERMed
-pgid unconditionally after the leader was reaped, so a recycled pgid could be
signalled, the exact hazard smart_http guards via disarm-after-wait.

Set a reaped AtomicBool the instant the main thread reaps the child; the watchdog
checks it before every kill and stands down if the leader is already reaped. Gate
the timeout verdict on !status.success(), so a child that exited on its own is never
reported as a timeout even if the watchdog fired late. Add the survived-SIGKILL warn
smart_http's reap already emits, for operator visibility on a wedged (D-state) git.

The hung-walk test stays green (a killed child exits by signal, not success, so it
still surfaces GitServiceTimeout and reaps the group) and the 32 real-git
classification tests stay green (a fast walk is never spuriously killed).
2026-07-12 00:36:42 -05:00
t ab0ea240fc docs(node): correct the git-service-timeout coverage note (#174)
GITLAWB_GIT_SERVICE_TIMEOUT_SECS bounds the info/refs advertisement too:
smart_http::info_refs drives it through drive_git_child under this timeout, with a
passing test proving the 504. The old note claimed it does not. It also claimed the
withheld-blob path is unbounded; after the blob_paths seam bound (this PR) the walk
is bounded and reaped, by a fixed internal deadline rather than this env var, so the
line now states that precisely instead of overclaiming this setting covers it.
2026-07-12 00:20:07 -05:00
t 71389e621a fix(node): bound the withheld-blob walk at the shared blob_paths seam (#174)
The withheld-blob classification walk (blob_paths) fanned out blocking git children
with no deadline and no process-group teardown: git for-each-ref, git cat-file, git
rev-list, a git ls-tree per commit, and an uncounted git rev-parse (via
store::head_commit). A hung or pathologically slow child pinned the caller's
served-git permit for the whole hang, and on client disconnect the spawn_blocking
task and its git children ran on, orphaned. blob_paths is the shared core of five
callers: the upload-pack serve path (holds a read permit) AND, inside
git_receive_pack, the post-push replication and encrypt-then-pin walks (hold the
write permit U2 reserves for pushes). So the same unbounded walk could pin either
pool, and leaving the write-side twin unbounded would have made U2's reservation a
claim that does not match behavior.

Bound every git child at the blob_paths spawn seam on the blocking side: each child
runs in its own process group with a watchdog thread that SIGTERMs (then SIGKILLs)
the group on one shared deadline spanning the whole walk, and retains admission
until the group is reaped. This is the blocking-side counterpart of
smart_http::drive_git_child (spawn_blocking cannot be cancelled by an async
timeout). blob_paths stays sync, so all five callers keep their signatures and the
32 classification tests are unchanged; because every caller funnels through
blob_paths, one seam bounds both the serve and replication paths. The previously
unbounded store::head_commit child becomes a bounded git rev-parse inside the walk.
A walk that hits its deadline carries GitServiceTimeout, which the serve handler now
maps to 504 rather than a generic 500.

Proof: a fake git that hangs on rev-list makes blob_paths return GitServiceTimeout
within the watchdog budget (not block on the child) and the recorded process-group
leader is reaped, not orphaned; neutralizing the watchdog kill makes it hang past
the budget (RED). The 32 real-git classification tests stay green through the
refactor, including detached-HEAD, non-standard-ref, and deleted-in-history cases.
2026-07-12 00:20:07 -05:00
t e444bc0ae5 fix(node): draw the receive-pack advertisement from the write pool (#174)
git_info_refs acquired git_read_semaphore for BOTH services, so the push handshake
(GET /info/refs?service=git-receive-pack) competed in the global read pool. An
anonymous clone flood could exhaust that pool and shed a legitimate push with 503
during its required advertisement phase, before it ever reached git_write_semaphore
on the POST. The write pool exists precisely so anonymous reads cannot shed an
authenticated push, but only the POST drew from it.

Select the pool by service: the receive-pack advertisement (phase one of a push)
now draws from git_write_semaphore, like the git-receive-pack POST, so a saturated
read pool cannot starve it. The per-IP push_rate_limiter that already brakes the
advertisement stays as the anti-flood control, and the advertisement stays
reader-visible with no new auth requirement. Because the receive-pack branch is now
a write-path op, it no longer consumes a read per-caller slot.

Handler-layer proofs: with the read pool at zero the receive-pack advertisement
survives while the upload-pack advertisement sheds; with the write pool at zero the
receive-pack advertisement sheds while upload-pack is unaffected; and a receive-pack
advertisement from an IP whose read per-caller budget is full still gets through
(mutation-checked, RED when the skip is neutralized).
2026-07-11 23:54:21 -05:00
t 4983b854d1 fix(node): key the per-caller read cap on source IP, not the signed DID (#174)
read_caller_key returned the authenticated DID when a caller signed, dropping the
source-IP key. Public read routes accept any valid did:key via optional_signature
with no admission step, so one host could mint N disposable DIDs and hold
max_concurrent_reads_per_caller slots under each, multiplying its budget N-fold and
filling the global read pool, while the same host unsigned was capped on its IP.

Key the read sub-cap on the resolved source IP for every caller, signed or not,
mirroring the push path's IpRateLimiter which already throttles on source IP for
this exact DID-farm reason. Drops the now-unused caller_did parameter at both call
sites (git_info_refs, git_upload_pack).

Inverts info_refs_per_caller_cap_keys_on_did_not_ip into
info_refs_per_caller_cap_keys_on_ip_not_did: fill one source IP's slot, then two
requests signed under different DIDs from that same IP both shed 503 (farm
defeated), while a signed request from a different IP keeps its own budget. RED on
the DID-keyed tree, GREEN after.
2026-07-11 23:44:13 -05:00
t 362ee3462e fix(node,git): bound the rev-list stage under drive_git_child so a hung enumeration can't pin a git permit (#174) 2026-07-11 13:02:24 -05:00
t 3b1fa54b73 docs(config): add GITLAWB_MAX_CONCURRENT_GIT_OPS to .env.example (#174)
The read-pool knob was referenced by the push and per-caller entries'
comments but had no example line of its own, so operators couldn't
discover it from the template. Add it with the config default (128).
2026-07-10 16:05:29 -05:00
t 5069cd13ca fix(node,concurrency): resolve #174 code-review findings
Review of the served-git concurrency cap found no P0/P1; these are the
verified P2 follow-ups.

config: the max_concurrent_git_ops doc overclaimed that "every capped path
is duration-bounded." The rev-list object enumeration in the withheld-blob
path still runs in an uncancellable spawn_blocking, so a stuck rev-list can
hold its slot until git exits. Scope the guarantee to the streaming stages
and name the residual. Also tighten the fairness claim: the receive-pack
advertisement shares the read pool (a shed advertisement is a cheap retryable
GET); only the push POST is on the isolated write pool.

api/repos: add info_refs_per_caller_cap_keys_on_did_not_ip, the missing
handler proof that a signed caller is keyed by its DID, not its source IP.
Filling the DID slot sheds a request from a free IP; collapsing read_caller_key
to its IP arm turns the assertion green-not-503 (mutation-verified RED).

api/repos: extract acquire_read_caller_permit so both read handlers share one
shed path instead of a duplicated match block.

rate_limit: recover from a poisoned PerCallerConcurrency mutex instead of
panicking. The critical section is pure counter arithmetic and cannot poison
the lock, but a panic there would brick the limiter for every caller.

505 tests pass; clippy -D warnings and fmt clean.

Part of #174.
2026-07-10 13:45:38 -05:00
t 6336657efc docs(node,config): reconcile the concurrency-cap comments with the fix (#62)
The max_concurrent_git_ops and git_service_timeout_secs doc-comments (and .env.example) described the info/refs and withheld-blob paths as unbounded follow-up gaps. Both are now closed (#174): the comments reflect the read/write pool split, the per-caller sub-cap, and that every capped path is duration-bounded with process-group teardown. Verified the pattern-doc pre-ship checklist: every git_permit / write-permit / per-caller site holds only a timeout+teardown git path, and all three size knobs are range(1..=1_048_576).

Closes the #174 work. No behavior change.
2026-07-10 13:20:18 -05:00
t 57c6e01c26 fix(node,git): bound and reap the withheld-blob pack build (#62)
The filtered-pack path ran the whole rev-list + pack-objects build inside a spawn_blocking, so an outer tokio timeout could not cancel the blocking thread and a client disconnect orphaned the git child while the permit freed (jatmn P1b, the second gap path). Split it: rev-list enumeration stays blocking off the runtime (rev_list_keep), but the streaming pack-objects stage now runs under the shared drive_git_child on the async side, so it is duration-bounded (GitServiceTimeout -> 504) and its process group is reaped on disconnect. build_filtered_pack becomes async and takes a git_bin seam + timeout; upload_pack_excluding threads the git_service_timeout through.

A watchdog-bounded test proves a hung pack-objects times out (rev-list fast, pack-objects hangs). The refactor's happy path is covered by the existing filtered-pack correctness and real-git partial-clone/fetch tests, all still green; disconnect/group-teardown is the shared drive_git_child code proven by the run_git_service tests. 504 tests pass.

Part of #174.
2026-07-10 13:16:41 -05:00
t 2c264e08e3 fix(node,git): bound and reap the info/refs advertisement (#62)
info_refs ran a bare Command::output() with no timeout and no process-group teardown, so a hung git pinned its concurrency slot indefinitely and a client disconnect orphaned the child (jatmn P1b). Extract the timeout + process_group(0) + KillGroupOnDrop core from run_git_service into a shared drive_git_child, and route info_refs through it with an injectable git_bin. A hung advertisement now aborts with GitServiceTimeout (mapped to 504); disconnect reaps the group.

run_git_service's teardown tests all pass through the shared core (proving the group teardown info_refs inherits), the real-git filter tests cover the advertisement happy path, and a new watchdog-bounded test proves a hung advertisement times out. 503 tests pass.

Part of #174.
2026-07-10 13:09:34 -05:00
t fc4c206c3a fix(node,git): per-caller concurrency sub-cap on the read pool (#62)
Adds PerCallerConcurrency, a bounded-keyed in-flight limiter (distinct from the request-rate RateLimiter) so no single caller monopolizes the served-git read pool. Each caller (per-DID when signed via optional_signature, else per-source-IP via client_key) may hold at most max_concurrent_reads_per_caller concurrent reads; over that it sheds 503. The key map is self-bounding (a key is dropped when its in-flight count hits zero) with a reject-before-insert max_keys backstop so a key farm can't grow it (INV-15). Applied in git_upload_pack and both info/refs advertisements, acquired after the visibility gate so a denied request never consumes a slot (KTD7).

Primitive unit-tested (cap + self-bounding + reject-before-insert) and mutation-checked. Handler-layer SC2: same-caller sheds while a different caller passes, proven on BOTH git_info_refs and git_upload_pack with independent mutation probes; plus a None-key bypass test. Per-source-IP keying is trust-config dependent, documented on the config knob. 502 tests pass.

Part of #174.
2026-07-10 12:50:46 -05:00
t 2434dde42f fix(node,git): isolate authed pushes in a dedicated write pool (#62)
git-receive-pack now draws from a separate git_write_semaphore (max_concurrent_git_pushes) instead of the shared pool, so a flood of anonymous reads can no longer shed an authenticated push at admission (jatmn P1a). The shared field is renamed git_read_semaphore and continues to gate upload-pack and both info/refs advertisements. The write permit stays above acquire_write so it precedes the Tigris fresh-acquire (INV-10).

Handler-layer tests: write-pool shed (503), and a cross-boundary proof that an exhausted read pool does NOT shed a push; both mutation-checked (routing receive-pack back to the read pool flips each RED). 497 tests pass.

Part of #174.
2026-07-10 12:29:30 -05:00
t db4de844ad feat(node,config): add write-pool and per-caller read-cap knobs (#62)
Add max_concurrent_git_pushes (default 32) and max_concurrent_reads_per_caller (default 16), both clap range(1..=1_048_576) so an oversized value is a clean CLI error, not a Semaphore::new boot panic. The per-caller knob documents that per-source-IP keying is only as granular as GITLAWB_TRUSTED_PROXY. Wiring lands in the following commits; these are the config surface for the #174 concurrency-fairness fix.

Resolves jatmn P1a/P1b groundwork on #174.
2026-07-10 12:18:07 -05:00
t 65ba1374af feat(node,git): cap concurrent served git ops with a 503 load-shed (#62)
PR3 of the #62 served-git hardening stack (timeout #165 and teardown
wiring #150 are merged). A bounded semaphore caps how many upload-pack /
receive-pack / info-refs operations run at once; past the cap a request
is shed with a clean 503 + Retry-After before spawning another git
subprocess, instead of exhausting the PID/thread table. A permit is
acquired at the top of each of the three handlers and held for the whole
op, releasing on return.

The cap is a portable backstop: the compose pids_limit is absent on Fly,
whose 500-connection cap is a different axis. Size --max-concurrent-git-ops
(GITLAWB_MAX_CONCURRENT_GIT_OPS, default 128) below the process budget.
Range 1..=1_048_576 so 0 (shed everything) and an oversized value that
would panic tokio's Semaphore at boot are clean CLI errors.

Known gap, tracked separately: info/refs and the withheld-blob
(upload_pack_excluding) path are not duration-bounded and do not reap
their git child on client disconnect, so a hung git on those two paths
holds its slot until it exits and live git can briefly exceed the cap.
The main pack path (run_git_service) tears its group down on drop.

Tests: Overloaded maps to 503 + Retry-After; the config knob defaults
and rejects out-of-range; git_permit sheds at capacity and releases; and
each of the three endpoints sheds with 503 when the semaphore is
exhausted (load-bearing: drop the permit line and the endpoint test goes
red).
2026-07-10 12:10:30 -05:00
GravireiandGravirei ad7c2b2b83 test(gl): add tests for iCaptcha transparent retry flow (#167) (#168)
* fix(node): gate GET /ipfs/{cid} on reachable allowed-set, not deny-set (#126)

The IPFS visibility gate used withheld_blob_oids (a deny-set enumerating
only reachable blobs), so a dangling/unreachable blob was absent from the
set and served in cleartext to anonymous callers. Flip to an allowed-set
(allowed_blob_set_for_caller) that enumerates reachable blobs the caller
may read: a dangling blob has no path, is never in the set, and 404s.

* perf(ipfs): check object existence before allowed-blob walk

Move store::read_object before the allowed_blob_set_for_caller
spawn_blocking call so random-CID spray against repos with
path-scoped rules cannot trigger full-history git walks on
repos that don't carry the object.

* refactor(ipfs): improve formatting and readability in get_by_cid function

✓ P3 blocker fixed: cargo fmt applied — the format gate will pass.

  ✓ P3 cleanup resolved: withheld_blob_oids is still used by replication code in repos.rs, so it stays.

  • P2 follow-up: Tree/commit disclosure tracked in #135 — out of scope here.

* refactor(ipfs): streamline object retrieval by separating type and content reading

* docs(ipfs): update get_by_cid comment to reflect split object retrieval

* Run cargo fmt on store.rs

* test(gl): add tests for iCaptcha transparent retry flow (#167)

* fix(clippy): remove redundant borrow in format! arg

* fix(review): use runtime GITLAWB_ICAPTCHA_INSECURE flag instead of cfg(test), add mock assertions

* fix(review): set mock expectations at creation time, tighten insecure scheme check

* fix(test): serialize icaptcha integration tests with global Mutex to prevent env-var race

* fix(icaptcha): compare origins (scheme+host+port) in resolve_solver_url; tighten proof assertion in test

* fix(test): save/restore inherited iCaptcha env vars instead of unconditional remove; share lock in icaptcha-client tests

* fix(test): constrain negative send_once mocks with Matcher::Missing so they actually protect the contract

---------

Co-authored-by: Gravirei <[email protected]>
2026-07-10 22:22:30 +08:00
Kevin Codex c98b503f90 feat(icaptcha-client): solve the iCaptcha proof-of-work on answer (#181)
iCaptcha now requires a proof-of-work with each answer (enforce mode) so that
minting a proof costs CPU — the lever that stops a distributed, multi-IP
proof-farming flood that per-IP rate limits cannot. Teach the sanctioned
client to solve it: parse the `pow` field from the challenge, grind a nonce
whose sha256(challenge:nonce) has >= difficulty leading zero bits, and send it
as `powNonce` on /v1/answer. Bounded search so a hostile difficulty can't
hang the client. Mirrors icaptcha/src/pow.ts. Without this, agents on the
updated client cannot obtain a proof once enforce is on.

New pow module + unit tests; verified end-to-end against the live enforce-mode
service (obtained a level-3 proof incl. PoW).
2026-07-10 22:02:58 +08:00
github-actions[bot] 3ec8cbf1e1 chore(main): release 0.5.1 (#158)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-07-10 19:46:16 +08:00
GravireiandGravirei 6b5e5bc7ae fix(node): bound list_ref_certificates with LIMIT and add upsert to prevent unbounded growth (#147) (#149)
* fix(node): bound list_ref_certificates with LIMIT and add upsert to prevent unbounded growth (#147)

* test: add INV-7 v10 upgrade-path dedup test

* fix INV-7 test: correct id-DESC tiebreaker ids and fix fmt

* fix: insert_ref_certificate returns persisted id via RETURNING; issue_ref_certificate uses it

* fix: guard cert upsert on issued_at so older certs cannot regress newer ones

* fix: .max(0) in events limit parsing, re-add lost cert tests after rebase

* address reviewer findings: return full persisted row, add monotonic clock comment, note test drift risk

* fix: add ?prefix= server-side cert lookup for CLI short-ID resolution

---------

Co-authored-by: Gravirei <[email protected]>
2026-07-10 19:44:43 +08:00
dfcaa22b23 fix(node): rate-limit repo/agent creation per client IP to stop DID-farm spam floods (#180)
* fix(node): add per-IP flood brake to repo/agent creation endpoints

The recurring spam-repo floods are not bypassing the iCaptcha gate — they
pass it. The enforced level (3) draws only machine-solvable deterministic
challenges (arithmetic/algebra/sequence), and the /v1/challenge API lets the
caller pin the easy type, so a bot mints a fresh throwaway did:key, solves a
real proof bound to that DID, and creates a repo. The per-DID creation
limiter never trips because each DID makes a single create call.

Add a per-client-IP rate limiter to the creation route group (repos,
register, fork, issues, pulls), reusing the same IpRateLimiter the push path
already uses against the identical DID-farm bypass. It wraps the auth layer
(outermost) so flood traffic is rejected before signature verification.
Default 120/hour/IP, GITLAWB_CREATE_RATE_LIMIT overrides, 0 disables.

* docs(env): document GITLAWB_CREATE_RATE_LIMIT in .env.example

* fix: apply CodeRabbit auto-fixes

Fixed 1 file(s) based on 1 unresolved review comment.

Co-authored-by: CodeRabbit <[email protected]>

* test(node): unit-test the per-IP creation brake runs before auth

The route-level coverage (repo_creation_is_rate_limited_by_ip) is
#[sqlx::test] and needs Postgres, so it is skipped in the plain unit-test
pass. Add a DB-free middleware test that layers the per-IP creation limiter
over an always-401 auth stand-in and asserts an exhausted IP is rejected
with 429 before auth runs — the ordering the DID-farm flood defense relies
on.

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: CodeRabbit <[email protected]>
2026-07-10 19:44:04 +08:00
2109d08e2c Fix/db startup resilience (#170)
* Keep node alive during database outages

* Fix startup-resilience gaps: reachable 503s, bounded retries, seamless handoff

- Route sqlx errors buried in anyhow chains back to AppError::Db and widen
  the unavailable class (Io/Tls/pool), so DB outages return 503 instead of
  500 — the previous 503 arm was unreachable dead code
- Bound each connect+migrate attempt (GITLAWB_DB_CONNECT_TIMEOUT_SECS) so a
  held migration advisory lock cannot wedge the startup retry loop forever
- Share one bound socket between the degraded and full servers and stop
  awaiting the degraded drain: no closed-port window during the handoff and
  no slow client can block the full server from starting
- Probe the database from /ready (2s bound) and gate Fly routing/deploys on
  it, so mid-life outages drain traffic and a bad DATABASE_URL fails the
  deploy visibly instead of going green; classify auth/catalog errors as
  likely-permanent (error-level log, straight to max backoff)
- Start the metrics listener before the DB connects so the degraded window
  is observable; degraded /health now returns 503 so peer pings and uptime
  monitors stop counting a DB-less node as alive
- Move the five GITLAWB_DB_* knobs onto clap Config (validated, in --help),
  drop the redundant outer connect timeout, default the pool to 20 conns,
  fix the backoff exponent cap that defeated GITLAWB_DB_RETRY_MAX_SECS, and
  strip at most one .git suffix in smart-HTTP repo names

* fix: apply CodeRabbit auto-fixes

Fixed 1 file(s) based on 1 unresolved review comment.

Co-authored-by: CodeRabbit <[email protected]>

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: CodeRabbit <[email protected]>
2026-07-10 09:44:02 +08:00
GravireiandGravirei c5e5099def refactor(visibility): route is_owner through the shared did_matches matcher (#153) (#156)
* fix(node): gate GET /ipfs/{cid} on reachable allowed-set, not deny-set (#126)

The IPFS visibility gate used withheld_blob_oids (a deny-set enumerating
only reachable blobs), so a dangling/unreachable blob was absent from the
set and served in cleartext to anonymous callers. Flip to an allowed-set
(allowed_blob_set_for_caller) that enumerates reachable blobs the caller
may read: a dangling blob has no path, is never in the set, and 404s.

* perf(ipfs): check object existence before allowed-blob walk

Move store::read_object before the allowed_blob_set_for_caller
spawn_blocking call so random-CID spray against repos with
path-scoped rules cannot trigger full-history git walks on
repos that don't carry the object.

* refactor(ipfs): improve formatting and readability in get_by_cid function

✓ P3 blocker fixed: cargo fmt applied — the format gate will pass.

  ✓ P3 cleanup resolved: withheld_blob_oids is still used by replication code in repos.rs, so it stays.

  • P2 follow-up: Tree/commit disclosure tracked in #135 — out of scope here.

* refactor(ipfs): streamline object retrieval by separating type and content reading

* docs(ipfs): update get_by_cid comment to reflect split object retrieval

* Run cargo fmt on store.rs

* refactor(visibility): route is_owner through the shared did_matches matcher (#153)

* style(profiles): remove redundant borrow in format arg

---------

Co-authored-by: Gravirei <[email protected]>
2026-07-10 08:55:49 +08:00
beardthelionandt cd67718f49 fix(node,git): bound a hung served git with a total-duration timeout (#62) (#165)
* feat(node,git): bound a hung served git with a total-duration timeout (#62)

Part 1 of #62. A served git upload-pack/receive-pack that neither finishes
nor disconnects parks forever, pinning a git PID (and, for receive-pack,
holding the write lock); on a public repo an anonymous caller can trigger
it. No tower TimeoutLayer exists and axum imposes no default.

Wrap the child interaction in tokio::time::timeout, bounded by a new
GITLAWB_GIT_SERVICE_TIMEOUT_SECS knob (default 600, must be positive).
Drain stdout/stderr concurrently with the stdin write so a large body
can't deadlock. On expiry run_git_service returns a typed GitServiceTimeout
and reaps the WHOLE process group before returning (SIGTERM so git clears
its .git/*.lock, wait for the leader and its grandchildren like index-pack
to exit, escalate to SIGKILL past a grace, hard-capped), so a caller
releasing the receive-pack write lock can't race a still-live git on the
same repo.

The handlers map GitServiceTimeout to a distinct 504 (not the generic 500
git error, and clear of the read-gate 404 / auth 401 the client keys
retries on) via a pure, unit-testable classifier rather than matching the
anyhow string. Surface git's own non-zero exit (its stderr) before a
stdin-write EPIPE so a malformed body is classified 400, not 500.

Out of scope, deferred in #62: the info/refs advertisement and the
withheld-blob path (spawn_blocking, which tokio timeout cannot cancel);
both remain unbounded (noted in the config knob's docs).

Tests: the timeout fires and tears the group down; it reaps the whole
group before returning (RED if the reap is leader-only); the wrap is
load-bearing (RED via an outer bound, not a hang); the error must be the
typed timeout; the 504 mapping and classifier are unit-tested; config
rejects 0. Full suite green.

* fix(review): harden #62 timeout tests and drop a dead reap flag

- test(node): cover the `protocol error` arm of git_service_app_error
  independently (the input has no "bad line length" substring, so it
  isolates the second classifier arm and goes RED if that arm is removed)
- test(node): make the reap-before-return test's grandchild IGNORE SIGTERM
  (`trap "" TERM`) and outlive the ~4s reap cap, so the SIGKILL escalation
  is load-bearing. Previously the grandchild self-exited under the cap, so
  neutering the SIGKILL still passed; the escalation was exercised but not
  required. Verified: RED with SIGKILL disabled, GREEN with it.
- test(node): add a receive-pack timeout test. Every prior timeout test used
  an upload-pack fake; this proves the push path (which also holds the repo
  write lock) is bounded too. RED-verified with the internal timeout removed.
- test(node): poll for the pidfile in the reap-before-return timeout test
  so a loaded runner can't false-panic before the fake git writes it
- remove the redundant `sigkilled` guard in reap_group_on_timeout; step
  == 200 fires exactly once in the 0..400 loop, so no re-entry guard is
  needed and the SIGKILL still fires exactly once

* fix(node): log malformed receive-pack as warn, not error (#165)

git_receive_pack's error classifier routed every non-Timeout error,
including client-caused BadRequest (400), through tracing::error!. Mirror
the git_upload_pack arm so a malformed push logs at warn level instead of
as a server error.

* test(node,git): make smart_http timeout tests pass under the parallel runner (#165)

Addresses jatmn's review of 41eca0b.

[P2] The #62 timeout tests failed under `cargo test --workspace`:
- run_git_service_tears_down_group_when_future_dropped re-polled a completed
  future ("async fn resumed after completion"): the advance-until-pids loop
  ignored the timeout's Ok(_) (future-finished) case and polled the resolved
  future again. It now breaks on early completion.
- Under fork-storm load a freshly-written fake `git` transiently fails to exec
  (ETXTBSY) or is timed out before recording its pids. Added a
  fake_git_run_with_pids retry helper (used by the four pid-reading tests) plus
  a matching inline retry on the drop test, with growing backoff for the
  correlated bursts and a 12-attempt cap that still fails loudly on a genuine
  never-spawns regression. Reverted two tests from a speculative 2000ms bound
  back to 300ms/1000ms now that the retry covers the timeout-vs-pidfile race.

[P3] Documented GITLAWB_GIT_SERVICE_TIMEOUT_SECS in .env.example and the README
config table (default 600; bounds upload-pack/receive-pack, not info/refs or the
withheld-blob path).

The tests stay load-bearing: breaking process_group(0), the post-reap disarm,
the timeout-arm reap, or the internal timeout each still turns the corresponding
test red (mutation-verified).

---------

Co-authored-by: t <t@t>
2026-07-10 08:54:59 +08:00
GravireiandGravirei 6bafaa6dc5 fix(bounties): add tests for claim_bounty repo-read gate (#160) (#169)
* fix(node): gate GET /ipfs/{cid} on reachable allowed-set, not deny-set (#126)

The IPFS visibility gate used withheld_blob_oids (a deny-set enumerating
only reachable blobs), so a dangling/unreachable blob was absent from the
set and served in cleartext to anonymous callers. Flip to an allowed-set
(allowed_blob_set_for_caller) that enumerates reachable blobs the caller
may read: a dangling blob has no path, is never in the set, and 404s.

* perf(ipfs): check object existence before allowed-blob walk

Move store::read_object before the allowed_blob_set_for_caller
spawn_blocking call so random-CID spray against repos with
path-scoped rules cannot trigger full-history git walks on
repos that don't carry the object.

* refactor(ipfs): improve formatting and readability in get_by_cid function

✓ P3 blocker fixed: cargo fmt applied — the format gate will pass.

  ✓ P3 cleanup resolved: withheld_blob_oids is still used by replication code in repos.rs, so it stays.

  • P2 follow-up: Tree/commit disclosure tracked in #135 — out of scope here.

* refactor(ipfs): streamline object retrieval by separating type and content reading

* docs(ipfs): update get_by_cid comment to reflect split object retrieval

* Run cargo fmt on store.rs

* fix(bounties): add tests for claim_bounty repo-read gate (#160)

* fix(profiles): remove redundant borrow in format arg

---------

Co-authored-by: Gravirei <[email protected]>
2026-07-10 08:39:10 +08:00
GravireiandGravirei 26bc3f6987 fix(visibility): gate repo-scoped read surfaces on visibility (#120) (#157)
* fix(node): gate GET /ipfs/{cid} on reachable allowed-set, not deny-set (#126)

The IPFS visibility gate used withheld_blob_oids (a deny-set enumerating
only reachable blobs), so a dangling/unreachable blob was absent from the
set and served in cleartext to anonymous callers. Flip to an allowed-set
(allowed_blob_set_for_caller) that enumerates reachable blobs the caller
may read: a dangling blob has no path, is never in the set, and 404s.

* perf(ipfs): check object existence before allowed-blob walk

Move store::read_object before the allowed_blob_set_for_caller
spawn_blocking call so random-CID spray against repos with
path-scoped rules cannot trigger full-history git walks on
repos that don't carry the object.

* refactor(ipfs): improve formatting and readability in get_by_cid function

✓ P3 blocker fixed: cargo fmt applied — the format gate will pass.

  ✓ P3 cleanup resolved: withheld_blob_oids is still used by replication code in repos.rs, so it stays.

  • P2 follow-up: Tree/commit disclosure tracked in #135 — out of scope here.

* refactor(ipfs): streamline object retrieval by separating type and content reading

* docs(ipfs): update get_by_cid comment to reflect split object retrieval

* Run cargo fmt on store.rs

* fix(visibility): gate repo-scoped read surfaces on visibility (#120)

* fix(visibility): enhance repo read authorization checks and improve test coverage

* Fix review feedback: clamp limit, memoize authz, identical 404 body, resolved issue id in comments

* cargo update -p crossbeam-epoch: fix RUSTSEC-2026-0204 (invalid pointer dereference)

* Fix review findings: correct authz memoization, page past hidden bounties, gate star/unstar

* Fix findings: sign CLI/MCP reads, keyset pagination, gate claim_bounty

---------

Co-authored-by: Gravirei <[email protected]>
2026-07-08 08:46:19 +08:00
beardthelionandt d00d89ae5b fix(node): gate POST /api/v1/sync/trigger and rate-limit the peer-sync routes (#82) (#161)
* feat(node): add per-client-IP limiters + knobs for the peer-sync routes (#82)

Plumb two dedicated per-IP RateLimiters into config/state so the peer-sync
enqueue routes can be braked independently: sync_trigger_rate_limiter (tight,
GITLAWB_SYNC_TRIGGER_RATE_LIMIT, default 60/hr) and peer_write_rate_limiter
(generous, GITLAWB_PEER_WRITE_RATE_LIMIT, default 600/hr). Both are bounded and
swept by the periodic cleanup loop, keyed via the node-wide push_limiter_trust.
No route is wired to them yet; that lands next.

* fix(node): require a signature on /sync/trigger and rate-limit the peer-sync routes (#82)

/api/v1/sync/trigger was reachable unauthenticated in the default config and
drove an O(peers) outbound fan-out + per-repo sync_queue enqueue — an anonymous
amplification/DoS. The same enqueue_sink is reachable via /sync/notify, which
also accepts unsigned known-peer requests and had no rate limit.

- Move /sync/trigger into its own always-signed route group (add_auth_layers),
  independent of require_signed_peer_writes, and give it a tight per-IP brake.
- Add a separate, generous per-IP brake to the peer-write group
  (/peers/announce, /sync/notify); notify's signature behavior is unchanged
  (rolling-upgrade compat), it only gains the brake. Separate buckets so an
  unsigned notify flood can't drain the signed trigger caller's quota.
- Per-IP (not per-DID): a did:key farm self-registers, so a signature/per-DID
  gate does not cap cost (INV-10); the brake keys on the client IP before auth.

Adds an adversarial test matrix (unsigned->401 both modes, IP flood->429,
DID-agnostic brake, forged X-Forwarded-For can't bypass, notify flood->429,
distinct-IP not throttled, cross-route non-exhaustion, 0-disables-but-sig-holds,
announce still accepts unsigned).

* fix(gl): surface node denial on sync trigger and sanitize the error body (#82)

Now that the node gates /api/v1/sync/trigger behind a signature and rate limit,
'gl sync trigger' must not render a 401/429 as a fabricated '✓ sync triggered /
0 peers' success. Check resp.status() before parsing; on a non-2xx, extract the
node's message and bail. Strip C0/C1 control bytes from the node-supplied string
before printing (INV-6) — a hostile node could embed ANSI/OSC escapes in the
body. Adds mockito tests: 401/429 surface as errors (RED without the status
check), control bytes are stripped, and the 200 path is unchanged.

* fix(review): bound gl error-body read, strip bidi controls, genericize 429 message (#82)

Code-review follow-ups on the #82 change:
- gl sync trigger: read the node error body with a capped reader
  (read_body_capped) instead of resp.text(), so a hostile/broken node can't
  force an unbounded allocation to surface a denial (INV-6, read half).
- sanitize_node_msg: also strip Unicode bidi/format controls (U+202A-202E,
  U+2066-2069, LRM/RLM/ALM) — is_control() only covers the Cc category, but a
  right-to-left override can reorder the displayed error line.
- too_many_requests(): the shared per-IP 429 middleware now serves the peer-sync
  routes too, so drop the push-specific wording ('rate limit exceeded'); the
  push-specific 429 in the receive-pack handler is unchanged.
Adds tests for the bidi/length sanitize and the oversized-body path.

* test(#82): close reasoned-not-run gaps with load-bearing tests

Executes the branches that were previously only reasoned:
- read_body_capped byte-cap test (RED against resp.text() — 2MB body -> ≤cap):
  vets the bounded-read fix, not just the display cap.
- sync-route 429 body asserts 'rate limit exceeded' and NOT 'push' (RED before
  the message was genericized); also finishes genericizing the warn log, which
  the response-body change had missed.
- real RFC 9421 signature through the full router reaches the handler (200) in
  both config modes — the positive path through the actual gate, not a
  direct-mount with an injected DID.
- /peers/announce flood -> 429 (the peer_write brake covers announce, not just
  notify).
- trigger_counts extraction unit test (happy + malformed -> 0,0), closing the
  count-formatting gap in the 200 path.
No production behavior change beyond the warn-log wording.

* fix(#82): require an identity for sync trigger and fix the rollout docs

Address review feedback on the always-signed /sync/trigger route.

- gl sync trigger now requires a local identity via load_keypair_from_dir and
  fails locally with "identity not found" before issuing a request, instead of
  .ok() silently sending an unsigned request that can only 401 remotely. Add a
  test proving it fails before dialing the node.
- Split /sync/trigger out of the staged unsigned peer-write rollout in the
  README and the OSS-readiness audit; only /peers/announce and /sync/notify keep
  the rolling-upgrade unsigned behavior.
- Reorder the two peer-sync rate-limit vars in .env.example for dotenv-linter.
- Bump crossbeam-epoch to 0.9.20 (RUSTSEC-2026-0204), matching main, so cargo
  audit passes.

---------

Co-authored-by: t <t@t>
2026-07-08 08:44:55 +08:00
beardthelionandt 67ad2b876c fix(deps): bump crossbeam-epoch to 0.9.20 for RUSTSEC-2026-0204 (#162) (#163)
RUSTSEC-2026-0204 (invalid pointer dereference in crossbeam-epoch's
fmt::Pointer impl for Atomic/Shared) hard-fails cargo audit on every branch,
since the advisory's addition to the RustSec db. crossbeam-epoch is a fully
transitive dependency (moka -> hickory-resolver -> libp2p-dns), and the
advisory's prescribed fix is >= 0.9.20; moka 0.12.15 accepts >=0.9.18,<0.10.0
so the bump resolves with no other graph changes.

Lockfile-only, single crate. Verified: cargo audit exits 0 and the workspace
builds clean with 0.9.20.

Closes #162.

Co-authored-by: t <t@t>
2026-07-07 22:46:44 +08:00
Kevin CodexandClaude Fable 5 6cff5286b4 fix(release): build aarch64-musl natively on arm64 runners, replace retired macos-13 (#155)
The aarch64-unknown-linux-musl leg cross-compiled a musl target with the
glibc toolchain (CC=aarch64-linux-gnu-gcc), which aws-lc-sys cannot build
with — the Build step failed on every release and no aarch64-linux asset
has ever shipped. Build it natively on ubuntu-24.04-arm instead (free for
public repos), using the same stock musl-tools recipe as the x86_64 leg,
and drop the smoke-test skip since the binaries now run on their build
runner.

The x86_64-apple-darwin leg targeted macos-13, which GitHub has retired;
the job queues forever and holds the release concurrency group open. Move
it to macos-15-intel.

Both failures also skipped the downstream npm-publish, homebrew-bump, and
web-sync jobs, so v0.4.0 and v0.5.0 were never published there.

Co-authored-by: Claude Fable 5 <[email protected]>
2026-07-06 10:39:12 +08:00
github-actions[bot] 5f63f28215 chore(main): release 0.5.0 (#142)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-07-05 20:07:23 +08:00
Kevin CodexandClaude Opus 4.8 06388ec26a feat(gl): sanctioned iCaptcha client flow + secure git lifecycle (#138)
* feat(gl): sanctioned iCaptcha client flow + secure git lifecycle

Builds the client side of the iCaptcha gate so a legitimate agent passes the
challenge transparently, with no manual headers and no GITLAWB_ICAPTCHA_PROOF
env var. Server enforcement is unchanged.

New crate `icaptcha-client` (blocking, shared by gl + the git helper):
- deterministic solvers for the computational challenge types (arithmetic,
  algebra, sequence) matching the iCaptcha generators; anagram/logic/LLM fall
  back to a solver hook / interactive prompt.
- obtain_proof(): POST /v1/challenge (requesterId = caller DID), solve, POST
  /v1/answer, handle escalation, return the signed proof. Optional bearer auth
  via GITLAWB_ICAPTCHA_API_KEY.

gl (crates/gl):
- http.rs: unify post/put/delete through one signed sender that, on a 403
  iCaptcha challenge (detected via x-icaptcha-url / x-icaptcha-level headers),
  solves it and retries the same signed request with the x-icaptcha-proof
  header (bounded retry absorbs the ~5 min proof TTL). Removes the env hack.
  Emits an actionable hint on 401 not_an_agent (old-CLI / unregistered).
- repo.rs: `repo info` checks status and returns a real 404 instead of a stub
  card with `?` fields; `repo info`/`clone` resolve the owner from the local
  identity or an explicit owner/name — never the node's own DID.
- doctor.rs: warn on gl-vs-node version drift; add an iCaptcha reachability
  check.

git-remote-gitlawb: same shared client — push stays signed (RFC 9421); on a 403
iCaptcha (safety net; push is signed-only, not gated) it solves and retries.
Clone/fetch unsigned; missing repo keeps its clear 404.

gitlawb-node: the 403 icaptcha_proof_required response now advertises the
service url + required level as JSON fields and x-icaptcha-url / x-icaptcha-level
headers (mirroring the human_detected pattern) so clients discover them instead
of scraping the message. The sub == authenticated-DID and expiry/level checks
are unchanged — enforcement is not weakened.

README: documents the full lifecycle (identity -> register -> repo create with
auto-iCaptcha -> push -> clone), the requesterId == DID rule, and proof TTL.

Tested: solver unit tests; full workspace green against Postgres (node still
rejects missing/expired/wrong-subject proofs).

* fix(icaptcha-client,gl): address PR #138 review feedback

Rebased onto main (resolving the git-remote-gitlawb/main.rs conflict from #119)
and addressed the review findings:

- [beardthelion P2 / jatmn P2 / CodeRabbit] Removed the dead iCaptcha retry loop
  from the push path. git_receive_pack never calls verify_request (only
  create/fork/register are iCaptcha-gated), so a push 403 never carries
  x-icaptcha-* headers and the loop could never solve. Taking main's push path
  (via #119) deletes it, which also resolves the pack-clone (retries no longer
  copy the whole pack) and the interactive-prompt-over-git-stream threads, and
  restores the sanitized info/refs error path. git-remote-gitlawb no longer
  depends on icaptcha-client.

- [beardthelion P1] Sanitize and length-bound iCaptcha error bodies before they
  reach the terminal. The origin is only as trusted as the node that advertised
  it, so its non-2xx bodies (and Failed{reason}) are attacker-influenceable; they
  went raw into bail! (CWE-150, the #137 class). Now bounded-read + C0/C1-stripped
  + capped, same shape as #137.

- [beardthelion P2] Do not trust or hand credentials to a node-chosen iCaptcha
  URL. x-icaptcha-url was used as the base with no scheme/host check and the API
  key was sent to it on the first hop (SSRF + key exfiltration). resolve_solver_url
  now honors an advertised URL only when https + host-allowlisted (public default
  or the operator's GITLAWB_ICAPTCHA_URL), else falls back to the trusted origin;
  the API key is attached only to the operator's own configured origin.

- [jatmn P2 / CodeRabbit] gl repo clone with a bare name now derives the short
  owner key via resolve_owner_did, matching the other commands' gitlawb://z.../name
  URL shape instead of the colon-bearing full DID.

Skipped (with reason): jatmn P3 (doctor GITLAWB_ICAPTCHA_URL) is resolved by the
P2 change — that env var is now read by the write path, so the doctor's probe is
accurate. CodeRabbit User-Agent-version nit lived in the push code reverted to
main. README push wording kept ("signed-only, no per-push challenge") — now
accurate since the retry path is gone; the API-key bullet was updated for the new
trust model.

Tests: added sanitize + resolve_solver_url unit tests; full workspace green.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

---------

Co-authored-by: Claude Opus 4.8 <[email protected]>
2026-07-05 19:47:06 +08:00
2df6ff9d30 fix(node): close two spam-vector root causes (trust upsert + ungated push) (#152)
* fix(node): close two spam-vector root causes (trust upsert + ungated push)

The June 2026 spam wave kept re-materializing after cleanup because two
paths let an unauthenticated-by-captcha actor keep operating:

1. `update_trust_score` was an upsert. Any authenticated push/issue/PR
   from a DID with no agent row silently INSERTed one with a fresh
   `registered_at`, bypassing the iCaptcha gate on /api/register — so
   deregistered spam DIDs came back the moment they pushed. Make it an
   UPDATE-only no-op for unregistered DIDs; registration is the sole way
   into the agents table. Regression test added.

2. `git-receive-pack` had no rate limit. The per-DID limiter on the
   creation routes can't brake a push flood from a DID farm (one
   throwaway identity per repo), so the node absorbed several pushes/sec
   per IP, each a Tigris round-trip + peer-notify fan-out. Add a per-
   client-IP limiter (Fly-Client-IP / X-Forwarded-For) on the push path,
   default 600/h, `GITLAWB_PUSH_RATE_LIMIT` override, 0 disables. Fails
   open without a proxy header so self-hosted nodes aren't broken.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* fix(node): address PR review — advertisement throttle, trusted-proxy IP key, bounded limiter

Resolves the review findings on the push rate limiter:

P1 — advertisement path bypassed the limiter. A push hits
GET info/refs?service=git-receive-pack first, which calls acquire_fresh()
(a Tigris download); it sat in git_read_routes under only optional_signature,
so the flood was reachable via the cheap GET without ever sending the throttled
POST. Now git_info_refs applies the same per-IP brake before the fresh acquire.

P1 — client-supplied forwarded headers were trusted. The key came from
Fly-Client-IP / the first X-Forwarded-For hop, both client-settable, so a
flooder could rotate the header and never fill a bucket. Replaced with a
TrustedProxy policy (GITLAWB_TRUSTED_PROXY = fly | x-forwarded-for | unset):
trust only the operator's edge header, fall back to the real socket peer
(ConnectInfo) otherwise. Fly config trusts Fly-Client-IP; the AWS/Caddy
compose trusts the rightmost X-Forwarded-For hop.

P2 — unbounded limiter key set / insert-before-reject. Added a max_keys cap
(reject-before-insert, inline eviction when full) so a varied key cannot grow
the map, and a rejected request never allocates. Applied to both the per-IP
and per-DID limiters.

P2 — malformed forwarded header disabled the brake. Empty leading XFF hop
returned None and skipped the limiter; empty Fly-Client-IP returned Some("").
client_key now takes the rightmost XFF hop and always falls back to the peer,
so a malformed header can never turn the limiter off.

P2 — missing tests. Added: middleware 429 path, per-peer isolation, key-cap
eviction/rejection, trusted-proxy header resolution, and an integration test
asserting the receive-pack advertisement is throttled before the acquire.

Also serve with into_make_service_with_connect_info so the peer address is
available. New env: GITLAWB_PUSH_RATE_LIMIT, GITLAWB_TRUSTED_PROXY (documented
in .env.example).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* Update infra/aws/compose.yaml.tftpl

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* fix(node): amortize the rate limiter's capacity eviction scan

`RateLimiter::check` ran a full O(max_keys) `retain` sweep on every new-key
miss once the map was at capacity — which is exactly the distinct-key-flood
state the cap defends against, so the limiter serialized all traffic behind a
200k-entry scan per request under the mutex.

Gate the inline sweep to run at most once per `sweep_interval` (min of the
window and 1s), tracked by `last_sweep`. Between sweeps a new key is rejected
without scanning; the background `cleanup()` loop still reclaims independently.
The fast path for existing keys is unchanged, the key cap invariant holds
(map never exceeds max_keys, rejected keys never allocate), and post-flood
self-healing stays bounded to one interval. Added a test asserting a burst of
capacity misses does not trigger repeated eviction scans.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

---------

Co-authored-by: Claude Opus 4.8 <[email protected]>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2026-07-05 17:31:21 +08:00
Mizuki Hayashiandmizuki0x 2620e973e3 fix(repos): log the cause when repo create fails (#103)
create_repo was the one git handler that mapped its error without logging
it (info_refs, receive_pack, acquire all log with repo context). So a
failed init_bare returned an opaque 500 "initializing bare repo" with no
server-side trail to the real cause.

Log the full anyhow chain ({:#}, so the leaf git/filesystem error is kept)
with owner+repo context, matching the other handlers. Response unchanged.

Co-authored-by: mizuki0x <[email protected]>
2026-07-05 17:16:11 +08:00
beardthelionandt e3a20317d5 ci: pin cargo-audit on the blocking gate, float the weekly canary (#151)
The PR-blocking audit job ran `cargo install --locked cargo-audit` with no
version, so the scanner floated to the latest release on every run; a
regressed release could red the PR gate repo-wide with no code change. Pin it
to 0.22.2, the version the unpinned install currently resolves to and the one
the audit-gate learnings were validated against. `cargo install --version X`
is an exact pin (not a caret), so this freezes the binary while the advisory
DB is still fetched fresh at scan time, which is what the gate actually keys on.

Keep a forcing function so the pin cannot silently rot: leave the weekly
Scheduled Audit deliberately unpinned as a drift canary. That job is cron-only
(never pull_request/push-triggered), so it never gates a merge; floating there
surfaces a regressed latest as a red weekly run or a changed report, which is
the drift signal, without ever blocking a PR.

Also assert the installed version after the pinned install so an accidental
future unpin fails loud instead of silently running a different scanner,
include the version in the rust-cache key so cache hits stay deterministic,
and document the yank/bump path in the step comment.

Co-authored-by: t <t@t>
2026-07-05 17:10:45 +08:00
beardthelionandt c296e23678 test(node,git): wire run_git_service teardown end-to-end (#62) (#150)
The #61 tests build KillGroupOnDrop by hand and never call
run_git_service, so dropping process_group(0), the guard-arming, or the
post-reap disarm left them green. Add two tests that drive the real
run_git_service through an injected fake git (a minimal git_bin seam;
production passes "git") and assert the production spawn path actually
wires the teardown:

- future-drop mid-flight SIGTERMs the whole group, reaping git and its
  grandchild (RED without process_group(0) or the guard-arming)
- a completed request disarms the guard, leaving a live grandchild
  untouched (RED without the post-reap disarm)

Each assertion was shown RED against its mutation and green when
restored. Parallel-safe: the fake is reached via the seam, not a PATH
mutation.

Co-authored-by: t <t@t>
2026-07-05 17:10:25 +08:00
beardthelionandt 20d6848846 fix(gl): sign the CLI's /ipfs/pins reads under the #134 auth gate (#146)
* fix(gl,ipfs): sign gl ipfs list so it works under the #134 pins auth gate

/api/v1/ipfs/pins now 401s anonymous callers (#134). Load the caller's
identity (--dir, default keystore) and use NodeClient::get_signed; when no
identity exists, propagate load_keypair_from_dir's existing 'gl identity new'
error instead of a raw 401. Tests: signed-headers present, empty, and
no-identity-issues-no-request (mockito).

* fix(gl,node): sign the gl node status pins panel, degrade gracefully

The pins panel signs its /api/v1/ipfs/pins read (#134 gates it behind auth)
via an injectable fetch_pins helper with four states: pins, empty,
unavailable (signed read rejected/errored), and needs-identity (no keypair,
no request issued). cmd_status loads the identity gracefully so a missing
keystore never aborts status; peers/repos/p2p/events panels stay anonymous.
Tests drive fetch_pins directly against a mock node.

* fix(gl): address review — surface pins HTTP errors, correct empty-panel label

- gl ipfs list: check the /ipfs/pins response status before parsing, so a
  rejected signed read (e.g. 401) surfaces as an error instead of silently
  printing 'No IPFS pins recorded'. Matches the sibling gl ipfs get.
- gl node status: PinsPanel::Empty now renders 'Pinned CIDs: 0' (a reachable
  node with zero pins), reserving 'unavailable' for the failure state; the
  prior label reused 'IPFS not configured' and misreported the zero-pins case.
- Carry the resolved count in PinsPanel::Pins, removing the duplicated
  count-fallback closure between fetch_pins and the render arm.

* test(gl,node): cover fetch_pins error branches and pins-panel rendering

Extract the pins-panel render into a testable pins_status_line helper and
assert all four states' output (closes the untested cmd_status render path,
including the empty->'Pinned CIDs: 0' label). Add fetch_pins tests for the
two error branches that were code-traced but unexecuted: a malformed 200
body and a transport error both degrade to Unavailable without panicking.

* style(gl): rustfmt the ipfs list pins-response parse

* fix(gl,node): honor --dir identity selector in `gl node status` (#146)

`gl node status` always loaded the default keystore via
`load_keypair_from_dir(None)`, so a user who created or selected an identity
with `--dir` saw the status pins panel render "sign in to view" even though
`gl ipfs list --dir` authenticated fine. Add a `--dir` option to
`NodeCmd::Status` and thread it into the signed pins fetch, so both CLI pins
callers select the same identity directory.

An explicit `--dir` that can't be loaded (missing, corrupt, or unreadable) no
longer masquerades as "no identity, run `gl identity new`": the pins identity
is resolved up front into Keyed / Anonymous / DirUnusable, and the unusable
explicit-dir case renders a distinct "no usable identity in <dir>" panel line
that names the path, while a missing default keystore (no `--dir`) still
degrades quietly. A pins failure never aborts the dashboard.

Tests cover both flag branches (Some/None) and the resolver both ways: a good
`--dir` loads Keyed, an unusable one reports DirUnusable/IdentityError without
issuing a request. The new symbols were absent before, so they compile-RED
first.

* test(gl,node): pin the pins-identity resolver to the --dir key

Follow-up test hardening on the #146 --dir wiring after review:

- resolve_pins_auth_loads_keyed_identity_from_explicit_dir now destructures
  Keyed and asserts loaded.did() equals the --dir key's DID. A bare Keyed(_)
  match passed even when the load fell through to the default keystore; the
  DID assertion catches that regression (confirmed red under a simulated
  fallthrough, green on the real code).
- add resolve_pins_auth_none_is_never_dir_unusable: an absent --dir must
  degrade to Anonymous or Keyed, never DirUnusable. It asserts only the
  None to DirUnusable mapping, so it is stable whether or not the test
  runner has a default ~/.gitlawb identity.

Test module only; no production behavior change.

---------

Co-authored-by: t <t@t>
2026-07-05 17:10:02 +08:00
beardthelionandt 4891db3889 fix(node): gate the ref-updates feeds on read visibility (#112, #114) (#143)
* feat(node,visibility): add pure fail-closed feed-visibility filter (#112,#114)

Add ref_update_row_visible: given the deduped local repo set + their
visibility rules, decide whether a received_ref_updates row (keyed by its
peer-supplied slug) is visible to the caller. Matches slug->repo by name
plus prefix-tolerant owner key, and drops any row matching a local repo
the caller cannot read at root; rows matching no local repo (remote/
gossip-only) and malformed slugs pass through. Fail-closed by
construction (no default-keep). Pure/I-O-free so visibility.rs stays
exhaustively unit-tested.

Shared gate for the GraphQL (#112) and REST (#114) ref-updates feeds;
call sites land in the following commits. Includes a DB-layer test
pinning that a private canonical row beats a public mirror row in
list_all_repos_deduped (else the feed gate would over-serve).

* fix(node,graphql): gate refUpdates on read visibility (#112)

The GraphQL ref_updates resolver mapped every received_ref_updates row
straight out with no visibility gate, so an anonymous caller could read
a private repo's ref metadata (refName, SHAs, pusherDid, nodeDid) by
name. Thread the caller DID and retain rows through the shared
ref_update_row_visible filter (deduped local set + rules, same '/'
decision the repos resolver uses); applies to both the repo:Some and
repo:None branches. Remote/gossip-only rows still pass.

Removes the now-live filter's dead_code allow. Tested via the GraphQL
schema: anon leak reproduced before the gate (RED) and closed after
(GREEN); owner still sees their private row; full-DID and truncated-key
slug aliases fail closed; remote rows kept.

* fix(node,events): gate global ref-updates feed on read visibility (#114)

GET /api/v1/events/ref-updates returned every received_ref_updates row
across all repos, including private ones, to any anonymous caller (the
REST analog of #112). Thread the caller DID via optional_signature and
retain rows through the shared ref_update_row_visible filter, deriving
count from the filtered set. Keeps the crate::error::Result return type;
the deduped/rules loads fail closed (500) on a DB error rather than
serving unfiltered. Remote/gossip-only rows still pass.

Tested via the crate's HTTP-API harness: anon leak reproduced before the
gate (RED) and closed after (GREEN); owner still sees their private row;
mixed feed yields only public rows with a filtered count; full-DID and
truncated-key slug aliases fail closed; remote rows kept.

* fix(node,graphql): gate the ref-update subscription broadcast on announce (#112,#114)

Code review surfaced a third reader of the same ref metadata the query
and REST feeds gate: the GraphQL subscription. /graphql/ws is mounted
after the optional_signature layer (no caller to gate against), and the
push handler broadcast every ref update to it unconditionally — outside
the if-announce block that already gates the sibling gossip and Arweave
sends. So an anonymous websocket subscriber received live ref metadata
(repo, refs, SHAs, pusher/node DID) for private repos as they were
pushed — the subscription analog of the #112/#114 leak.

Move the ref_update_tx broadcast inside the existing if-announce guard so
only publicly-readable pushes reach the unauthenticated subscription,
matching gossip and Arweave. Pin the gate decision with a
replication_withheld_set test (announce false for private, true for
public).

Also harden the feed filter's owner-key normalization: normalize the
slug owner to the last ':'-segment (same rule as record_key) instead of
stripping only did🔑, so a multi-segment DID (did:web) and its
full-DID slug match and fail closed rather than over-serving. Regression
test added (RED under the old normalization).

* docs(node,graphql): pin the subscription write-side gate invariant; add did:web keep test

Code review flagged that the unauthenticated /graphql/ws ref_updates
subscription now depends on a single-point invariant: safety rests
entirely on every sender to ref_update_tx being announce-gated, since the
resolver has no caller to gate against. Document that on the resolver so
a future sender can't silently reopen the leak.

Add the symmetric keep-side test for the did:web owner-key normalization
(public multi-segment-DID repo row is kept for anon), so a regression
that over-drops legitimate did:web rows is caught alongside the existing
drop-side test.

* docs(node,visibility): correct ref-update slug normalization rationale (#112,#114)

The comment claimed parity with did_matches/DEDUP_CTE, but the last-segment
normalization deliberately diverges: the emitter (publish_ref_update) broadcasts
the short, method-stripped slug, so a repo's own rows arrive keyed by the
trailing segment and the gate must match on it to stay fail-closed. Reusing the
did:key-aware keep-whole rule would fail open on a did:web repo's short slug.

Add feed_private_didweb_short_slug_dropped_for_anon to pin that load-bearing
canonical case; relabel the full-DID test as the non-canonical hand-forged form.

* fix(node,events): gate ref-update feeds before the limit via paging (#114)

The REST global feed and the GraphQL ref_updates resolver loaded `limit` rows
and dropped private ones in memory, so an anonymous request could get fewer than
`limit` events (or zero) when the newest rows named private local repos, even
though older public ref updates existed.

Add a shared collect_visible_ref_updates helper that applies the fail-closed
gate before the limit: it over-fetches in bounded pages (128 rows, capped at
max(limit, 2048) scanned) until `limit` visible rows are collected or the table
is exhausted. Both feeds route through it so the one gate cannot drift. Back it
with list_ref_updates_page (stable timestamp,id ordering for offset paging) and
drop the now-unused list_ref_updates / list_ref_updates_filtered. The page size
is a parameter on an inner fn so a regression can exercise multi-page paging.

Regressions: newest-private, older-public, small limit still returns the latest
visible rows on both surfaces (RED before, GREEN after); and a page=2 test pins
offset paging across page boundaries (no skip or duplicate).

* docs(node,visibility): correct slug-builder attribution in ref-update gate comment

The comment credited publish_ref_update with building the wire slug, but that
p2p method only broadcasts a pre-built RefUpdateEvent; git_receive_pack assembles
the {last-segment}/{name} slug and hands it over. Fix the attribution.

* fix(node,events): bound feed limit and withhold quarantined mirrors (#114)

Resolve two CodeRabbit findings on the ref-update feeds:

- Clamp the effective limit inside the shared collector (max 200). REST already
  capped, but the GraphQL resolver passed its caller-provided limit uncapped,
  letting a large request return unbounded rows and scan unbounded DB rows. Both
  surfaces are now bounded identically.

- Fold quarantined mirrors into the gate's match universe as unreadable.
  list_all_repos_deduped excludes quarantined rows, so a ref-update naming a
  quarantined local mirror matched nothing, was misclassified as remote, and was
  served to anon, bypassing the withheld-from-every-listing-surface contract. Add
  list_quarantined_repos and drop matching rows fail-closed.

Regressions: oversized-limit clamp (201 rows -> 200) and quarantined-mirror
withheld-from-anon, both RED before, GREEN after.

* fix(node,events): gate the repo-scoped events feed on read visibility (#112, #114)

GET /api/v1/repos/{owner}/{repo}/events (list_repo_events) served a repo's ref
certificates and received gossip ref-updates with no caller identity and no
visibility check, so an anonymous caller who guessed a private repo's owner/name
read its ref metadata. It was the last repo-scoped reader in the #112/#114
ref-metadata leak class.

Gate it in two layers, using the caller DID the optional_signature layer already
supplies:
- The cert half is gated by a repo-root authorize_repo_read (keyed by the unique
  repo record id): a denied, quarantined, or not-hosted repo returns an opaque 404
  (RepoNotFound), never a 403.
- The gossip half is filtered per row through the shared collect_visible_ref_updates.
  received_ref_updates rows are keyed by the lossy, non-unique wire slug
  {last-segment}/{name}, so two owners (did:web:a:alice, did:web:b:alice) collide on
  `alice/name` and a plain slug query would serve a colliding private repo's rows to
  anyone allowed to read this one; the per-row gate drops them fail-closed.

Both halves now propagate DB errors as 500 instead of swallowing them into an empty
200. Removes the now-unused list_repo_ref_updates.

Deliberate behavior change: a repo this node does not host returns 404 (it holds no
visibility record for it and fails closed; remote gossip is read via the global
/api/v1/events/ref-updates feed). Drops list_repo_events from the egress-guard
KNOWN_UNGATED allowlist; the source-scrape staleness assert enforces the removal.

Tests cover anon-private, owner-private (both datasets), anon/authed-public,
quarantined, released-from-quarantine, non-owner, not-hosted, did:web (owner and
anon), the slug collision, oversized-limit clamping, and DB-error-fails-closed on
both the gate and the cert fetch.

* fix(node,events): clamp negative ref-update feed limits to zero (#114)

A caller-supplied `?limit=-1` cleared the `.min(MAX_VISIBLE_REF_UPDATES)`
upper cap unchanged and reached `all_events.truncate(limit as usize)` in
list_repo_events, where `-1 as usize` is usize::MAX, so the truncate was a
no-op and the handler returned the full unbounded local ref-cert set.

Floor both feed handlers' limit parse with `.clamp(0, MAX_VISIBLE_REF_UPDATES)`.
Only the repo handler's clamp is load-bearing: it guards the local `truncate`,
which does not pass through the shared collector. The global feed's clamp is a
consistency measure (its limit is re-clamped in the collector, whose want==0
short-circuits before any scan), kept in lockstep so the two per-handler caps
the module comment calls out don't drift.

Tests, both load-bearing by execution:
- repo_events_negative_limit_clamped: seeds local certs (the vulnerable half),
  RED (3) before the clamp and green (0) after.
- feed_negative_limit_returns_empty: seeds 5 visible rows; green (0) with the
  collector clamp, 5 without it, proving the collector is the global feed's guard.

* fix(node,events): drop quarantined repos before the feed visibility gate (#114)

The ref-update feeds folded quarantined mirrors into the deduped repo set with is_public=false and relied on the row visibility gate to withhold them. That held for anonymous callers but not for one matching the mirror's owner_did: visibility_check short-circuits to Allow for the owner before is_public is read, so an owner-matched quarantined row was served on both the REST global feed and the GraphQL ref_updates resolver.

Drop any row that names a quarantined repo in the shared collector, before the visibility gate runs, so the drop bypasses the owner short-circuit. Quarantine is a status decided at admission and checked separately from the mirror's visibility fields, so it must deny every caller including the owner.

Extract the slug match into one ref_update_row_names_repo predicate reused by both the gate and the quarantine drop, so the two cannot disagree about which rows a repo owns.

Tests cover the owner case at the collector, REST handler, and GraphQL resolver in both bare-key and full-did:key owner forms, plus a must-not case proving the drop withholds only quarantined rows while still serving unrelated visible ones.

* test(node,events): align did:web feed tests with the #141 canonical slug

The rebase onto main adopts #141's normalize_owner_key on the repo-scoped
ref-update read path, which keeps a non-did:key DID intact instead of taking
the last URL segment. Align the two did:web fixtures that still assumed the
old {last-segment}/{name} wire slug:

- repo_events_did_web_owner_reads_own_gossip stored "alice/widget", but the
  emit side (api/repos) publishes "did:web:example.com:alice/widget", so the
  read query no longer matched and the owner saw only the cert half. Store the
  emit-side slug so the gossip KEEP branch is exercised again.

- repo_events_gossip_slug_collision_withheld_from_anon is removed. Its premise
  was that two did:web owners collide on the last-segment slug; #141 gives them
  distinct full-DID slugs, so the repo-scoped query fetches zero rows and the
  per-row drop is never reached (verified by probe: fetched=0). Global-feed
  collision coverage stays in feed_full_did_slug_dropped_for_anon and
  feed_truncated_key_slug_dropped_for_anon.

* docs(node): correct ref-update slug comments for the #141 canonical form

The rebase onto #141 changed the emit-side wire slug from the last-segment
form to normalize_owner_key (bare short key for a single-segment did:key,
full DID otherwise). Four comments still described the old behavior:

- visibility.rs ref_update_row_names_repo: the emitter builds the slug via
  normalize_owner_key, not {last-segment}/{name}. The next_back reduction on
  the matcher is what tolerates the untrusted/crafted short form; it does not
  mirror an already-stripped emit form.
- events.rs list_repo_events gossip half: the residual collision the per-row
  filter still guards is a did:key canonical owner vs its bare short-key
  mirror (both normalize to the same slug), not two did:web owners (#141
  gives those distinct slugs).
- the two did:web feed-visibility tests had their canonical/crafted framing
  backwards: post-#141 the full-DID slug is the emit form and the short slug
  is the crafted one.

Comment-only; no behavior change.

* test(node,events): add load-bearing two-repo owner-key collision deny

A public bare-key mirror (z6MkX) and a private did:key canonical repo
(did:key:z6MkX) normalize to the same owner key post-#141, so their gossip
rows share the z6MkX/... slug space on the untrusted wire. Assert an anon
global-feed read sees the public mirror's row but NOT the private canonical
sibling's: the shared gate keys on the full slug (owner + name), so a
readable public repo under an owner key does not unlock that owner's other
private repos' rows. The removed repo-scoped did:web collision test never
covered this pair.

Verified load-bearing: disabling the per-row visibility drop serves
z6MkX/secret to anon (RED); the gate drops it (GREEN).

* fix(node,events): keyset-paginate the ref-update feed collector

collect_visible_ref_updates paged received_ref_updates with LIMIT/OFFSET,
so a concurrent gossip/notify insert (which sorts to the front under
timestamp DESC) shifted every later offset and let one feed response
duplicate or skip a visible row. Switch the collector to a keyset cursor
on (timestamp, id): each page reads rows strictly older than the last row
fetched, so a newer insert lands above the window and cannot shift it.

One collector feeds the REST global, REST per-repo, and GraphQL paths, so
this covers all three; there is no client-facing cursor, so the internal
cursor tracks the last fetched row. Adds tests that OFFSET duplicates a row
under a concurrent insert and that keyset stays stable.

---------

Co-authored-by: t <t@t>
2026-07-05 17:09:15 +08:00
beardthelionandt 6f36fc07b8 fix(node,git-remote): gate receive-pack advertisement, sign client fetch/push (#119)
* fix(node,git-remote): gate receive-pack advertisement, sign client fetch/push

The info/refs ref advertisement was gated only for service=git-upload-pack
(the visibility check sat inside `if service == "git-upload-pack"`), so an
anonymous GET /{owner}/{repo}.git/info/refs?service=git-receive-pack returned a
private repo's branch names and commit tips, bypassing the read gate. Run the
visibility "/" check for both services.

Now that the advertisement is gated for both services, git-remote-gitlawb has to
authenticate its own requests, or a private repo's owner can neither fetch nor
push it. Sign the Phase-1 advertisement GET and the Phase-2 pack POST for both
services over path_and_query when an identity keypair is present; public repos
stay anonymous. This also repairs a pre-existing break where an owner could not
fetch their own private repo, because the upload-pack POST was sent unsigned.

Tests: node-side gate (anon denied, owner-signed cleared, both services); a
real-signature seam proving the client's own signature verifies under the node's
verification primitives; real served-content withholding against an on-disk repo
(the owner gets the ref bytes, a denied caller's 404 omits them); and mockito
coverage that the client attaches and omits signatures correctly.

* docs(node,git-remote): correct auth-contract comments on receive-pack gating (#119)

receive-pack POST is signature-required and separately owner-gated, not
visibility_check-gated; push implies read, not read-subset-of-push.

* test(node): assert unsigned receive-pack POST is rejected (#119)

No test covered the signature-required half of the receive-pack auth
contract. Wire the POST route behind require_signature as production does and
assert an unsigned POST returns 401 before reaching the handler; 401 (not the
handler's 404/500) proves require_signature rejected it pre-handler.

* test(node): RAII-clean the served-refs test's temp dirs on panic (#119)

advertisement_serves_real_refs_only_to_authorized_callers created /tmp/<slug>/<name>.git and /tmp/gl-seam-src-<short> and only removed them after the final assertion, leaking those dirs on the runner when an earlier assertion failed. Wrap each known path in a DirGuard whose Drop calls remove_dir_all, bound right after creation, so cleanup runs on success or panic. tempfile::TempDir does not fit here because repo_store::for_testing fixes the path layout.

* test(node): drop git_info_refs from KNOWN_UNGATED now that #119 gates it

The repo-scoped egress guard's allowlist (added in #122, merged to main
after this branch was opened) still listed git_info_refs as ungated. This
branch makes the info/refs visibility gate unconditional for both
git-upload-pack and git-receive-pack, so the handler is now gated and the
staleness assert requires removing it from the allowlist.

* style(node,git-remote): tidy #119 review nits (em dashes, dead binding, test header)

Post-rebase ce-code-review pass on #119. No behavior change:
- replace em dashes in the added comments with commas/colons/parentheses
- drop the dead `let _ = COVERED_COMPONENTS;` binding (and its import); the
  `missing_components().is_empty()` assert already covers the intent
- add a section header above the new git-info-refs test block to match the
  module's sectioning convention

* test(node): recurse nested api modules in the egress-gate guard

The completeness scan in every_repo_scoped_handler_is_gated used a flat
read_dir over src/api, so a future api/<module>/mod.rs could add a
repo-scoped handler the scrape never inspects. Walk the api tree recursively
via a new collect_rs_files helper (covered by its own unit test), skipping
only the top-level mod.rs so nested module files are checked too.

* fix(git-remote): keep public fetches anonymous, sign only on a 404 retry (#119)

git-remote-gitlawb signed every git-upload-pack request whenever an identity
existed, disclosing the caller's DID to any node it fetched a public repo from,
even though public repos need no auth. Fetch now sends the advertisement GET
anonymously and escalates to a single signed retry only when the node denies it
with 404 (how it withholds a private repo from an unauthenticated reader); if the
retry was needed, the upload-pack POST is signed too. Push (git-receive-pack) is
unchanged and signs from the first request, since a push can never be anonymous.

Verified against the node by execution: an unsigned private fetch returns 404,
not 401, and a signed retry clears the read gate for the owner and for a granted
non-owner reader, while a signed non-reader stays 404.

Tests drive handle_connect end to end (recording mock): public fetch stays
anonymous for GET and POST, private fetch retries signed once and signs the POST,
a keyless fetch does not retry, a still-denied signed retry surfaces 404 once, and
push signs from the first request.

* fix(node): scan nested api sources by relative path in the egress guard (#119)

The egress-gate completeness scan skipped already-covered files by basename
(listed.contains(fname)), so a nested api/<module>/repos.rs would be skipped
because the top-level repos.rs is listed, letting a future nested module add an
ungated repo-scoped handler the scrape never inspects. Extract the skip logic into
unlisted_source_files and match on the path relative to api_root, so only the exact
top-level files already covered are skipped and a nested file with a colliding
basename is still scanned. Completes CodeRabbit's outstanding request.

---------

Co-authored-by: t <t@t>
2026-07-05 17:08:44 +08:00
beardthelion 563c456803 fix(node,gossip): route gossip HTTP through the no-redirect client (#93) (#140)
gossip_task built its own reqwest::Client::new(), whose default redirect
policy follows up to 10 redirects, and used it for the bootstrap announce
and the periodic peer /health ping. A peer host (public at announce time,
per is_public_http_url) could answer /health with 302 Location:
http://169.254.169.254/... and the node would follow it into cloud
metadata or internal addresses. Blind SSRF: only request status feeds
mark_peer_ping.

Reuse the shared state.http_client (Policy::none, already on AppState),
matching sync.rs and the announce fan-out hardened in #78. Extract
ping_peer_health so the redirect-not-followed behavior is covered by a
mockito test (302 to an internal target registered expect(0), asserted
not hit).
2026-07-03 08:13:20 +08:00
GravireiandGravirei 6c95592d18 fix(node): prefer canonical repo row over mirror row in get_repo (#124) (#141)
* fix(node): prefer canonical repo row over mirror row in get_repo (#124)

get_repo matched both mirror rows (bare short-DID, is_public=true, no
visibility rules) and canonical rows (full did🔑 DID, correct rules),
returning one nondeterministically. When the mirror row won, the
visibility gate saw is_public=true + empty rules, allowing unauthenticated
read of private/withheld content.

Add ORDER BY to prefer canonical rows (UUID id, no slash) over mirror
rows (slash-form id), matching the ranking logic already used by
DEDUP_CTE and dedupe_canonical_repos.

Test seeds mirror FIRST so an unordered fetch_optional returns the
mirror row — proving the test locks in the fix.

* fix(test): date canonical row after mirror to guard CASE term in get_repo fix (#124)

* fix(node): tighten get_repo owner matching to did:key only, not any DID method (#124 P2)

* fix(node): refactor owner DID normalization to a dedicated function in create_repo

* fix(node): align clone URL and slug construction with tightened did:key matching

- Make normalize_owner_key pub(crate) and reuse it in api/repos, api/events, api/peers, and visibility modules.

- Add read-gate authorize_repo_read and to_response clone_url slug tests.

* fix(node): hoist owner-key CASE into shared const + address #124 review feedback

- Hoist 7 verbatim SQL CASE copies into OWNER_KEY_CASE_SQL const
  (dedup_cte(), get_repo, count_repos_deduped), keeping one literal
  copy in the v7 migration with a cross-reference comment.
- Add 9 unit tests for normalize_owner_key over the full boundary set.
- Add LIMIT 1 to get_repo query for single-row intent.
- Add non_key_owner_bare_short_does_not_match is_owner regression test.
- Update is_owner doc to reflect did:key-only rule.
- Document non-key clone_url disk-path constraint in to_response.

* test(node): add SQL round-trip test for normalize_owner_key vs OWNER_KEY_CASE_SQL

Verifies every boundary value produces identical results from the
Rust function and the SQL CASE expression, preventing silent drift.

---------

Co-authored-by: Gravirei <[email protected]>
2026-07-03 08:12:27 +08:00
GravireiandGravirei 466a550915 fix(node): gate GET /ipfs/{cid} on reachable allowed-set, not deny-set (#126) (#133)
* fix(node): gate GET /ipfs/{cid} on reachable allowed-set, not deny-set (#126)

The IPFS visibility gate used withheld_blob_oids (a deny-set enumerating
only reachable blobs), so a dangling/unreachable blob was absent from the
set and served in cleartext to anonymous callers. Flip to an allowed-set
(allowed_blob_set_for_caller) that enumerates reachable blobs the caller
may read: a dangling blob has no path, is never in the set, and 404s.

* perf(ipfs): check object existence before allowed-blob walk

Move store::read_object before the allowed_blob_set_for_caller
spawn_blocking call so random-CID spray against repos with
path-scoped rules cannot trigger full-history git walks on
repos that don't carry the object.

* refactor(ipfs): improve formatting and readability in get_by_cid function

✓ P3 blocker fixed: cargo fmt applied — the format gate will pass.

  ✓ P3 cleanup resolved: withheld_blob_oids is still used by replication code in repos.rs, so it stays.

  • P2 follow-up: Tree/commit disclosure tracked in #135 — out of scope here.

* refactor(ipfs): streamline object retrieval by separating type and content reading

* docs(ipfs): update get_by_cid comment to reflect split object retrieval

* Run cargo fmt on store.rs

---------

Co-authored-by: Gravirei <[email protected]>
2026-07-02 08:11:38 +08:00
Yang Yuandparanoa233 09a0cb23b9 fix(remote): include HTTP error response body (#137)
* fix(remote): include HTTP error response body

* fix(remote): sanitize HTTP error response bodies

---------

Co-authored-by: paranoa233 <[email protected]>
2026-07-01 21:59:06 +08:00
Kevin Codex cd4680ebeb chore(aws): codify iCaptcha enforce in the EC2 node Terraform (#139)
The AWS EC2 node (manila) was upgraded to v0.4.0 and enabled iCaptcha
enforce by hand-editing its running /opt/gitlawb/compose.yaml — which would
be lost on instance replacement or a user-data re-run. Codify it so rebuilds
stay enforced:

- variables.tf: add icaptcha_mode (default "enforce"), icaptcha_pubkey
  (default the pinned live icaptcha.gitlawb.com Ed25519 key), and
  icaptcha_required_level (default 3).
- main.tf: pass the three vars into the compose templatefile render.
- compose.yaml.tftpl: set ICAPTCHA_MODE / ICAPTCHA_PUBKEY /
  ICAPTCHA_REQUIRED_LEVEL on the node service environment.

Operators can override per-deployment via terraform.tfvars.
2026-07-01 09:55:16 +08:00
Kevin Codex 8d96cab4f7 chore(node): seed Gitlawb nodes in bootstrap-peers; ignore node1 fly config (#132)
- bootstrap-peers.json: add node2, node3, and manila (manila.gitlawb.com)
  to the canonical seed list alongside node.gitlawb.com, and normalize the
  Gitlawb operator label to "Gitlawb". DIDs left null — bootstrap merges
  peers on http_url and the parser treats did as optional.
- .gitignore: ignore infra/fly/node1.fly.toml. node1's production Fly
  config is pulled from Fly (fly config save), not tracked in the repo.
2026-06-30 21:59:56 +08:00
72 changed files with 30767 additions and 1364 deletions

No files matched your search

+115
View File
@@ -21,6 +21,23 @@ GITLAWB_REPOS_DIR=/data/repos
# When using the bundled docker-compose, this is wired automatically.
DATABASE_URL=postgresql://gitlawb:changeme@localhost:5432/gitlawb
# ── Database pool & startup resilience ────────────────────────────────────
# Maximum connections in the PostgreSQL pool. A cap, not a floor —
# connections open lazily. Size against the DB server's max_connections,
# remembering admin tooling opens its own pool.
GITLAWB_DB_MAX_CONNECTIONS=20
# Seconds a request waits for a pool connection before failing with 503.
GITLAWB_DB_ACQUIRE_TIMEOUT_SECS=5
# Upper bound on each startup connect+migrate attempt, in seconds. Keep it
# generous enough for a peer instance to finish migrating (the migration
# advisory lock is held across a deploy); on expiry the attempt is retried.
GITLAWB_DB_CONNECT_TIMEOUT_SECS=60
# Startup retry backoff while the database is unavailable: first delay and
# ceiling, in seconds. The node stays up serving 503s (with a truthful
# /ready) instead of crash-looping.
GITLAWB_DB_RETRY_INITIAL_SECS=5
GITLAWB_DB_RETRY_MAX_SECS=60
# ── IPFS pinning (Pinata) ─────────────────────────────────────────────────
# Get a JWT at https://app.pinata.cloud/developers/api-keys
GITLAWB_PINATA_JWT=
@@ -90,6 +107,104 @@ GITLAWB_PUBLIC_READ=true
# Maximum git smart-HTTP pack request size, in bytes.
GITLAWB_MAX_PACK_BYTES=2147483648
# Max seconds a served git upload-pack / receive-pack (clone / push) may run
# before it is aborted with a 504. Bounds a hung git that would otherwise pin a
# worker and, on push, the repo write lock. Also bounds both info/refs
# advertisements, the withheld-blob pack build, and the push-side candidate
# discovery (rev-list / cat-file), all reaped via process-group teardown (#174).
# Default 600.
GITLAWB_GIT_SERVICE_TIMEOUT_SECS=600
# Max seconds the storage-ACQUISITION phase of a served git op may run before the
# request is shed with a 503, separate from the git-run timeout above. A
# concurrency permit is taken before this phase and GITLAWB_GIT_SERVICE_TIMEOUT_SECS
# only starts once git spawns, so without this a stalled backend (a hung Tigris
# HEAD/GET, or a hung pg advisory-lock iteration on push) pins the permit and drains
# the pool until every later request 503s. On expiry the permit is released
# (fail-closed). Kept separate because acquisition and git execution are distinct
# cost centers. Must be positive; set very large to effectively disable. Default 30.
GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS=30
# Max concurrent git READ ops (upload-pack + the upload-pack info/refs
# advertisement) served at once, a global pool separate from the push pool below.
# The anon receive-pack info/refs advertisement has its OWN pool (see below), not
# this one. Over-cap sheds a clean 503 + Retry-After. Anonymous reads draw from
# here, so pair it with GITLAWB_MAX_CONCURRENT_READS_PER_CALLER (below) so one
# caller cannot monopolize the pool. Default 128.
GITLAWB_MAX_CONCURRENT_GIT_OPS=128
# Max concurrent git-receive-pack (push) POST operations, in a pool separate from
# the read pool (GITLAWB_MAX_CONCURRENT_GIT_OPS) so anonymous reads cannot shed an
# authenticated push at admission. The anon receive-pack info/refs advertisement
# runs in a SEPARATE pool of the same size (disjoint from this one), so an
# advertisement flood cannot shed a push either. Over-cap sheds a 503 +
# Retry-After. Default 32.
GITLAWB_MAX_CONCURRENT_GIT_PUSHES=32
# Max concurrent read ops (upload-pack + the upload-pack info/refs advertisement)
# a single caller may hold, so one caller cannot monopolize the read pool. Keyed
# on the resolved SOURCE IP, never the DID: a signature does not move a caller off
# this cap. The source-IP key is only as granular as GITLAWB_TRUSTED_PROXY below:
# left unset, a node behind an edge/NAT keys all callers on the edge IP and this
# collapses to one global cap. Set GITLAWB_TRUSTED_PROXY for per-client keying; a
# high-fanout caller (CI behind one NAT) then needs the operator to raise this.
# Default 16.
GITLAWB_MAX_CONCURRENT_READS_PER_CALLER=16
# ── Push rate limiting (git-receive-pack flood brake) ─────────────────────
# Max receive-pack requests (info/refs advertisement + push POST) per client
# IP per hour. 0 disables. Default 600.
GITLAWB_PUSH_RATE_LIMIT=600
# ── /ipfs/{cid} visibility-walk admission (#174) ──────────────────────────
# GET /ipfs/{cid} runs a per-repo full-history git walk in a blocking thread to
# decide whether the caller may read a path-scoped blob. It is publicly reachable,
# so it is bounded to keep a permissionless caller from fanning out unbounded
# concurrent walks and exhausting blocking-pool threads + PIDs.
# Max concurrent /ipfs walks across all callers (a pool of its own, disjoint from
# the served-git pools). Over-cap sheds a 503. Default 32.
GITLAWB_MAX_CONCURRENT_IPFS_WALKS=32
# Max concurrent /ipfs walks a single SOURCE IP may hold (keyed like the git
# per-caller caps via GITLAWB_TRUSTED_PROXY; reject-before-insert bounded map).
# Default 4.
GITLAWB_IPFS_WALK_PER_SOURCE=4
# Max legacy (NULL-provenance) repos probed per single /ipfs request, bounding the
# scan-fallback fan-out (git cat-file per candidate repo) for an anonymous caller. A
# truncated scan sheds a retryable 503, never a false 404. Default 256.
GITLAWB_IPFS_MAX_REPOS_WALKED=256
# Max /ipfs/{cid} requests per client IP per hour (route flood brake, distinct
# from the concurrency caps above). 0 disables. Default 600.
GITLAWB_IPFS_RATE_LIMIT=600
# ── Creation rate limiting (repo/agent/issue/PR flood brake) ──────────────
# Max creation requests (POST /api/v1/repos, /api/register, fork, issues,
# pulls) per client IP per hour, in addition to the per-DID limit. The per-DID
# limit alone is bypassed by a did:key farm (one throwaway identity per repo),
# which is how spam-repo floods slip past it and the iCaptcha gate; the per-IP
# brake caps a single-source flood. Uses GITLAWB_TRUSTED_PROXY below to resolve
# the client IP. 0 disables. Default 120.
GITLAWB_CREATE_RATE_LIMIT=120
# ── Peer-sync rate limiting (per client IP, uses GITLAWB_TRUSTED_PROXY below) ─
# /api/v1/peers/announce and /api/v1/sync/notify accept unsigned requests from
# known peers and run at higher frequency, so a generous bucket. Separate from
# the trigger bucket so an unsigned notify flood can't drain trigger's quota.
# 0 disables. Default 600.
GITLAWB_PEER_WRITE_RATE_LIMIT=600
# /api/v1/sync/trigger requires a signature and fans out to every peer per call,
# so it gets a tight bucket. 0 disables. Default 60.
GITLAWB_SYNC_TRIGGER_RATE_LIMIT=60
# Which forwarded header the edge is trusted to set, used to resolve the real
# client IP for the push limiter. One of:
# (unset) — no trusted proxy: key on the socket peer address, ignore headers.
# fly — behind Fly's edge (trust Fly-Client-IP).
# x-forwarded-for — behind a single reverse proxy like Caddy/NGINX (trust the
# rightmost X-Forwarded-For hop).
# Only set this when a proxy you control actually fronts the node; trusting a
# forwarded header on a directly-exposed node lets clients spoof the key.
GITLAWB_TRUSTED_PROXY=
# ── Sync ─────────────────────────────────────────────────────────────────
# Enable automatic background sync from known peers
GITLAWB_AUTO_SYNC=false
+12 -1
View File
@@ -37,7 +37,18 @@ jobs:
with:
key: audit-schedule
- name: Install cargo-audit
- name: Install cargo-audit (floating; drift canary)
# Deliberately UNPINNED, unlike the blocking gate in pr-checks.yml. This
# job is cron-only (weekly); it is never pull_request- or push-triggered,
# so it NEVER gates a merge no matter what it does. Floating the scanner
# here is the drift canary: if a newer cargo-audit regresses, at worst
# this weekly run reds (the install step below is unguarded) or its
# visibility report changes, and that is exactly the signal that latest
# has moved and the pinned gate should not be bumped yet. The scan result
# itself never fails the run (the "Full audit report" step is `|| true`
# and the only other hard-fail is a Cargo.lock grep). The point is that
# drift surfaces here without ever blocking a PR. Do NOT pin to match
# pr-checks.yml.
run: cargo install --locked cargo-audit
# Full advisory report with NO suppressions. Run from a scratch dir that
+23 -3
View File
@@ -133,10 +133,30 @@ jobs:
- name: Cache cargo
uses: Swatinem/rust-cache@98c8021b550208e191a6a3145459bfc9fb29c4c0 # v2.8.0
with:
key: audit
key: audit-0.22.2
- name: Install cargo-audit
run: cargo install --locked cargo-audit
- name: Install cargo-audit (pinned)
# Pin the scanner on this BLOCKING gate: an unpinned `cargo install`
# floats to the latest release, so a regressed cargo-audit could red CI
# repo-wide with no code change. Advisory-DB freshness is fetched at scan
# time, independent of the binary, so this does not weaken what the gate
# catches. The weekly Scheduled Audit (audit-schedule.yml) deliberately
# floats to latest as the drift canary; when it shows a newer cargo-audit
# behaving differently, bump this version (and the cache key above) as a
# deliberate maintainer action. If 0.22.2 is ever yanked from crates.io
# this step fails loud until the pin is bumped.
run: |
set -euo pipefail
cargo install --locked --version 0.22.2 cargo-audit
# Assert the pin took effect so an accidental future unpin (or a drift
# off 0.22.2) fails loudly here instead of silently running a different
# scanner. `cargo audit --version` prints "cargo-audit-audit 0.22.2".
installed="$(cargo audit --version)"
echo "cargo-audit installed: $installed"
echo "$installed" | grep -qE '(^| )0\.22\.2($| )' || {
echo "::error::cargo-audit is not the pinned 0.22.2 (got: $installed)"
exit 1
}
# Hard-fail gate. Suppressions live in .cargo/audit.toml (read automatically
# from the repo root), each with no available upstream fix. A green check
+10 -10
View File
@@ -116,10 +116,14 @@ jobs:
include:
- target: x86_64-unknown-linux-musl
os: ubuntu-latest
# Native arm64 runner: cross-compiling this target with the glibc
# toolchain broke aws-lc-sys, and native lets the smoke test run.
- target: aarch64-unknown-linux-musl
os: ubuntu-latest
os: ubuntu-24.04-arm
# macos-13 (the last plain Intel image) is retired and queues forever;
# macos-15-intel is GitHub's supported Intel label.
- target: x86_64-apple-darwin
os: macos-13
os: macos-15-intel
- target: aarch64-apple-darwin
os: macos-14
- target: x86_64-pc-windows-msvc
@@ -137,16 +141,13 @@ jobs:
toolchain: stable
targets: ${{ matrix.target }}
# Both musl targets build natively on a matching-arch runner, so the stock
# musl-gcc wrapper is all that's needed (no cross toolchain).
- name: Install musl tools (linux)
if: contains(matrix.target, 'linux-musl')
run: |
sudo apt-get update
sudo apt-get install -y musl-tools
if [ "${{ matrix.target }}" = "aarch64-unknown-linux-musl" ]; then
sudo apt-get install -y gcc-aarch64-linux-gnu
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
echo "CC_aarch64_unknown_linux_musl=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
fi
- name: Cache cargo
uses: Swatinem/rust-cache@98c8021b550208e191a6a3145459bfc9fb29c4c0 # v2.8.0
@@ -171,10 +172,9 @@ jobs:
cargo build --release --target ${{ matrix.target }} $args
# Run each packaged binary's --version so a broken release artifact fails the
# build instead of shipping. Skipped for cross-compiled targets that can't run
# on the host runner (aarch64 linux on an x86_64 runner).
# build instead of shipping. Every target builds on a matching-arch runner,
# so all of them can execute here.
- name: Smoke test binaries
if: ${{ matrix.target != 'aarch64-unknown-linux-musl' }}
shell: bash
run: |
set -euo pipefail
+3
View File
@@ -23,3 +23,6 @@ keys/
# Logs
*.log
.openclaude-profile.json
# Per-operator Fly config (node1 prod config is pulled from Fly, not tracked)
infra/fly/node1.fly.toml
+1 -1
View File
@@ -1,3 +1,3 @@
{
".": "0.4.0"
".": "0.5.1"
}
+34
View File
@@ -1,5 +1,39 @@
# Changelog
## [0.5.1](https://github.com/Gitlawb/node/compare/v0.5.0...v0.5.1) (2026-07-10)
### Bug Fixes
* **bounties:** add tests for claim_bounty repo-read gate ([#160](https://github.com/Gitlawb/node/issues/160)) ([#169](https://github.com/Gitlawb/node/issues/169)) ([6bafaa6](https://github.com/Gitlawb/node/commit/6bafaa6dc5f05c0dcd61c708397afddbcf8c2e3f))
* **deps:** bump crossbeam-epoch to 0.9.20 for RUSTSEC-2026-0204 ([#162](https://github.com/Gitlawb/node/issues/162)) ([#163](https://github.com/Gitlawb/node/issues/163)) ([67ad2b8](https://github.com/Gitlawb/node/commit/67ad2b876c8d9a336219d1016968de7a88fc4e75))
* **node,git:** bound a hung served git with a total-duration timeout ([#62](https://github.com/Gitlawb/node/issues/62)) ([#165](https://github.com/Gitlawb/node/issues/165)) ([cd67718](https://github.com/Gitlawb/node/commit/cd67718f49ec38726a40f6bcf36f539ccdb42969))
* **node:** bound list_ref_certificates with LIMIT and add upsert to prevent unbounded growth ([#147](https://github.com/Gitlawb/node/issues/147)) ([#149](https://github.com/Gitlawb/node/issues/149)) ([6b5e5bc](https://github.com/Gitlawb/node/commit/6b5e5bc7aee00a2d03295d3620df1ee4d8c024a2))
* **node:** gate POST /api/v1/sync/trigger and rate-limit the peer-sync routes ([#82](https://github.com/Gitlawb/node/issues/82)) ([#161](https://github.com/Gitlawb/node/issues/161)) ([d00d89a](https://github.com/Gitlawb/node/commit/d00d89ae5be992d1f63e95b714ae1bd3735e8457))
* **node:** rate-limit repo/agent creation per client IP to stop DID-farm spam floods ([#180](https://github.com/Gitlawb/node/issues/180)) ([dfcaa22](https://github.com/Gitlawb/node/commit/dfcaa22b23ec91be4c75926956aa994ca89de8d5))
* **release:** build aarch64-musl natively on arm64 runners, replace retired macos-13 ([#155](https://github.com/Gitlawb/node/issues/155)) ([6cff528](https://github.com/Gitlawb/node/commit/6cff5286b436fdee44ad881999e2a5f4bdba18f9))
* **visibility:** gate repo-scoped read surfaces on visibility ([#120](https://github.com/Gitlawb/node/issues/120)) ([#157](https://github.com/Gitlawb/node/issues/157)) ([26bc3f6](https://github.com/Gitlawb/node/commit/26bc3f69870aa77e43c0d92115a5aa59555b7d88))
## [0.5.0](https://github.com/Gitlawb/node/compare/v0.4.0...v0.5.0) (2026-07-05)
### Features
* **gl:** sanctioned iCaptcha client flow + secure git lifecycle ([#138](https://github.com/Gitlawb/node/issues/138)) ([06388ec](https://github.com/Gitlawb/node/commit/06388ec26aa29d356ae311276fdb91be054e9ecc))
### Bug Fixes
* **gl:** sign the CLI's /ipfs/pins reads under the [#134](https://github.com/Gitlawb/node/issues/134) auth gate ([#146](https://github.com/Gitlawb/node/issues/146)) ([20d6848](https://github.com/Gitlawb/node/commit/20d6848846b3a988d604208833167a528b7d8820))
* **node,git-remote:** gate receive-pack advertisement, sign client fetch/push ([#119](https://github.com/Gitlawb/node/issues/119)) ([6f36fc0](https://github.com/Gitlawb/node/commit/6f36fc07b8e10a650c5948b269feac1cb25cae2a))
* **node,gossip:** route gossip HTTP through the no-redirect client ([#93](https://github.com/Gitlawb/node/issues/93)) ([#140](https://github.com/Gitlawb/node/issues/140)) ([563c456](https://github.com/Gitlawb/node/commit/563c456803bf3e958d63869db424b3940472bc3d))
* **node:** close two spam-vector root causes (trust upsert + ungated push) ([#152](https://github.com/Gitlawb/node/issues/152)) ([2df6ff9](https://github.com/Gitlawb/node/commit/2df6ff9d30de62f754fa41473e85db316021718e))
* **node:** gate GET /ipfs/{cid} on reachable allowed-set, not deny-set ([#126](https://github.com/Gitlawb/node/issues/126)) ([#133](https://github.com/Gitlawb/node/issues/133)) ([466a550](https://github.com/Gitlawb/node/commit/466a550915edd711856ef32035f9f474e2577c4f))
* **node:** gate the ref-updates feeds on read visibility ([#112](https://github.com/Gitlawb/node/issues/112), [#114](https://github.com/Gitlawb/node/issues/114)) ([#143](https://github.com/Gitlawb/node/issues/143)) ([4891db3](https://github.com/Gitlawb/node/commit/4891db38892663326ee0c1417a2db931988be4b5))
* **node:** prefer canonical repo row over mirror row in get_repo ([#124](https://github.com/Gitlawb/node/issues/124)) ([#141](https://github.com/Gitlawb/node/issues/141)) ([6c95592](https://github.com/Gitlawb/node/commit/6c95592d188222ac3446dc23ef8d9befbf82ad6f))
* **remote:** include HTTP error response body ([#137](https://github.com/Gitlawb/node/issues/137)) ([09a0cb2](https://github.com/Gitlawb/node/commit/09a0cb23b9f284ccbd69aca6958b70671f3bfb46))
* **repos:** log the cause when repo create fails ([#103](https://github.com/Gitlawb/node/issues/103)) ([2620e97](https://github.com/Gitlawb/node/commit/2620e973e3cd4835ed42ddd4adcd8183b5b3080e))
## [0.4.0](https://github.com/Gitlawb/node/compare/v0.3.9...v0.4.0) (2026-06-30)
Generated
+23 -7
View File
@@ -2293,9 +2293,9 @@ dependencies = [
[[package]]
name = "crossbeam-epoch"
version = "0.9.18"
version = "0.9.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e"
checksum = "2d6914041f254d6e9176c01941b21115dcfb7089e55135a35411081bd106ef3f"
dependencies = [
"crossbeam-utils",
]
@@ -3300,10 +3300,11 @@ dependencies = [
[[package]]
name = "git-remote-gitlawb"
version = "0.3.9"
version = "0.5.1"
dependencies = [
"anyhow",
"gitlawb-core",
"mockito",
"reqwest",
"tracing",
"tracing-subscriber",
@@ -3311,7 +3312,7 @@ dependencies = [
[[package]]
name = "gitlawb-attest"
version = "0.3.9"
version = "0.5.1"
dependencies = [
"base64",
"ed25519-dalek",
@@ -3328,7 +3329,7 @@ dependencies = [
[[package]]
name = "gitlawb-core"
version = "0.3.9"
version = "0.5.1"
dependencies = [
"anyhow",
"base64",
@@ -3355,7 +3356,7 @@ dependencies = [
[[package]]
name = "gitlawb-node"
version = "0.3.9"
version = "0.5.1"
dependencies = [
"alloy",
"anyhow",
@@ -3411,7 +3412,7 @@ dependencies = [
[[package]]
name = "gl"
version = "0.3.9"
version = "0.5.1"
dependencies = [
"alloy",
"anyhow",
@@ -3419,6 +3420,7 @@ dependencies = [
"clap",
"dirs",
"gitlawb-core",
"icaptcha-client",
"mockito",
"reqwest",
"serde",
@@ -3813,6 +3815,20 @@ dependencies = [
"cc",
]
[[package]]
name = "icaptcha-client"
version = "0.4.0"
dependencies = [
"anyhow",
"gitlawb-core",
"reqwest",
"serde",
"serde_json",
"sha2",
"thiserror 2.0.18",
"tracing",
]
[[package]]
name = "icu_collections"
version = "2.1.1"
+1
View File
@@ -6,6 +6,7 @@ members = [
"crates/gl",
"crates/git-remote-gitlawb",
"crates/gitlawb-attest",
"crates/icaptcha-client",
]
[workspace.package]
+51 -3
View File
@@ -185,6 +185,45 @@ git clone gitlawb://did:key:z6Mk.../my-repo
For public-network use, make sure `GITLAWB_NODE` points to the node you want. The helper defaults to localhost for local development.
### Full lifecycle against an iCaptcha-enforcing node
Public nodes (e.g. `node.gitlawb.com`) require two things on writes:
1. **RFC 9421 HTTP Signatures** — every write is signed by your identity key. `gl`
and the `git-remote-gitlawb` helper do this automatically. An old/unsigned CLI
fails with `401 not_an_agent`; `gl` will tell you to upgrade and register.
2. **An iCaptcha proof** on the spam-gated writes (**repo create, fork, register**).
`gl` solves this for you: on the node's `403 icaptcha_proof_required` it reads the
`x-icaptcha-url` / `x-icaptcha-level` hints, requests a challenge, solves it
locally (arithmetic / algebra / sequence), and **retries the same signed request**
with the `x-icaptcha-proof` header — no manual steps, no env vars.
```bash
gl identity new # create did:key identity
gl register --node https://node.gitlawb.com # signed + auto-solves iCaptcha
gl repo create memlawb --node https://node.gitlawb.com # signed + auto-solves iCaptcha
git push origin2 main # origin2 = gitlawb://<your-did>/memlawb (signed)
git clone gitlawb://<your-did>/memlawb # public read, no proof needed
gl doctor # preflight: identity, node, version, iCaptcha
```
Notes:
- **`requesterId` is always your DID.** The proof's `sub` claim must equal the
authenticated signer; `gl`/helper set this automatically and the node enforces
`sub == authenticated DID` (so a proof minted for another identity is rejected).
- **Proofs are short-lived (~5 min TTL) and single-use.** If one expires between
solving and use, the client transparently solves a fresh one and retries.
- **What needs what:** create / fork / register are signed **and** iCaptcha-gated;
`git push` is **signed-only** (owner signature is the gate — no per-push challenge);
reads (clone / fetch / `repo info`) need no proof. A non-existent repo returns a
clear `404`, never a placeholder.
- **API-key iCaptcha deployments:** set `GITLAWB_ICAPTCHA_URL` to your iCaptcha
origin and `GITLAWB_ICAPTCHA_API_KEY` to its key. The client only talks to an
`https` origin whose host is allowlisted (that URL or the public default), and
sends the bearer token **only** to your configured origin — never to a URL a
node advertises — so a hostile node can't capture the key or redirect the solve.
---
## Architecture
@@ -263,20 +302,21 @@ POST /api/v1/bounties/{id}/...
POST /{owner}/{repo}/git-receive-pack
```
Peer write routes support staged rollout:
These peer write routes support staged rollout:
```txt
POST /api/v1/peers/announce
POST /api/v1/sync/notify
POST /api/v1/sync/trigger
```
When `GITLAWB_REQUIRE_SIGNED_PEER_WRITES=false`, unsigned legacy peers are accepted, but signed requests are verified when signature headers are present. Once all live peers upgrade, operators can set:
When `GITLAWB_REQUIRE_SIGNED_PEER_WRITES=false`, unsigned legacy peers are accepted on those two routes, but signed requests are verified when signature headers are present. Once all live peers upgrade, operators can set:
```bash
GITLAWB_REQUIRE_SIGNED_PEER_WRITES=true
```
`POST /api/v1/sync/trigger` is not part of the staged rollout: it always requires a signature in both config modes and returns 401 without one, because each call drives an O(peers) outbound fan-out.
---
## Configuration
@@ -303,12 +343,20 @@ Important node settings:
| `GITLAWB_REQUIRE_SIGNED_PEER_WRITES` | Require signed peer announce/sync writes. |
| `GITLAWB_AUTO_SYNC` | Enable automatic sync from known peers. |
| `GITLAWB_MAX_PACK_BYTES` | Max git pack body size for smart-HTTP routes. |
| `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack, receive-pack, or `info/refs` advertisement may run before it is aborted (504). Default 600. Also bounds the withheld-blob classification walk (on both the upload-pack serve and receive-pack replication paths) and the push-side pin-candidate discovery (`rev-list` / `cat-file`), each reaped via process-group teardown at the deadline. |
| `GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS` | Max seconds the storage-acquisition phase (Tigris HEAD/GET, push advisory-lock) of a served git op may run before the request is shed with a 503, separate from the git-run timeout. The concurrency permit is released on expiry so a stalled backend cannot pin the pool. Default 30. |
| `GITLAWB_MAX_CONCURRENT_IPFS_WALKS` | Max concurrent `GET /ipfs/{cid}` visibility walks across all callers (own pool, disjoint from the served-git pools); over-cap sheds 503. Default 32. |
| `GITLAWB_IPFS_WALK_PER_SOURCE` | Max concurrent `/ipfs` walks a single source IP may hold. Default 4. |
| `GITLAWB_IPFS_MAX_REPOS_WALKED` | Max legacy (NULL-provenance) repos probed per `/ipfs/{cid}` request, bounding the scan-fallback fan-out. A truncated scan returns a retryable 503, not a false 404. Default 256. |
| `GITLAWB_IPFS_RATE_LIMIT` | Max `/ipfs/{cid}` requests per client IP per hour (route flood brake). 0 disables. Default 600. |
| `GITLAWB_TIGRIS_BUCKET` | Optional S3/Tigris shared repo storage bucket. |
| `GITLAWB_PINATA_JWT` | Optional Pinata/IPFS warm-storage pinning. |
| `GITLAWB_IRYS_URL` | Optional Irys/Arweave permanent anchoring. |
Production note: change the default Postgres password before exposing a node publicly.
Legacy-pin window: releases before the CID-resolver work stored the provider CID (Kubo dag-pb / Pinata) as a pinned object's resolver key. The `/ipfs/{cid}` resolver now recomputes the raw-content CID from the object bytes and refuses to serve a key that does not match, so `GET /api/v1/ipfs/pins` can still advertise an unrepaired legacy CID that 404s. Such a row is repaired opportunistically the next time a push carries the object again (its key is rewritten to the raw CID, the old value kept in `legacy_provider_cid`), but git negotiation omits objects the node already has, so most legacy rows never re-enter a push delta. A deferred one-shot startup sweep, not this opportunistic path, is what fully retires the advertise-then-404 window. Rows whose object bytes are gone stay withheld.
---
## Optional node staking
+26 -2
View File
@@ -1,16 +1,40 @@
{
"$comment": "Canonical seed list for the Gitlawb network. Merged with GITLAWB_BOOTSTRAP_PEERS at startup. PRs to add public nodes welcome.",
"version": 1,
"updated": "2026-05-16",
"updated": "2026-06-30",
"peers": [
{
"name": "gitlawb",
"operator": "Gitlawb (Kevin)",
"operator": "Gitlawb",
"did": "did:key:z6MkqDnb7Siv3Cwj7pGJq4T5EsUisECqR8KpnDLwcaZq5TPr",
"http_url": "https://node.gitlawb.com",
"p2p_multiaddr": null,
"added": "2026-04-29"
},
{
"name": "gitlawb-node2",
"operator": "Gitlawb",
"did": null,
"http_url": "https://node2.gitlawb.com",
"p2p_multiaddr": null,
"added": "2026-06-30"
},
{
"name": "gitlawb-node3",
"operator": "Gitlawb",
"did": null,
"http_url": "https://node3.gitlawb.com",
"p2p_multiaddr": null,
"added": "2026-06-30"
},
{
"name": "gitlawb-manila",
"operator": "Gitlawb",
"did": null,
"http_url": "https://manila.gitlawb.com",
"p2p_multiaddr": null,
"added": "2026-06-30"
},
{
"name": "rapybus-gitlawb",
"operator": "RapyBus S.A.C.",
+4 -1
View File
@@ -1,7 +1,7 @@
[package]
name = "git-remote-gitlawb"
description = "Git remote helper — enables 'git clone gitlawb://did:gitlawb:...'"
version = "0.4.0" # x-release-please-version
version = "0.5.1" # x-release-please-version
edition.workspace = true
rust-version.workspace = true
license.workspace = true
@@ -16,3 +16,6 @@ anyhow = { workspace = true }
reqwest = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
[dev-dependencies]
mockito = "1"
+747 -35
View File
@@ -14,7 +14,13 @@
//! # Protocol flow (connect capability)
//! capabilities → "connect\n\n"
//! connect git-upload-pack → GET /info/refs | POST /git-upload-pack
//! connect git-receive-pack → GET /info/refs | POST /git-receive-pack (+ auth header)
//! connect git-receive-pack → GET /info/refs | POST /git-receive-pack
//! Push (git-receive-pack) is RFC-9421 signed from the first request when an
//! identity keypair is present, since a push can never be anonymous. Fetch
//! (git-upload-pack) stays anonymous and is signed only on a single retry, after
//! the node denies the anonymous advertisement with 404 — so a public clone never
//! discloses the caller's DID, while a private repo's owner (or an authorized
//! reader) still authenticates when the node demands it.
use anyhow::{bail, Context, Result};
use gitlawb_core::http_sig::sign_request;
@@ -63,7 +69,10 @@ fn main() -> Result<()> {
let repo_base = format!("{}/{}/{}", node_base, short_owner, repo_name);
tracing::debug!("repo_base: {repo_base}");
// Load keypair for signing push requests (optional — push still works unsigned in v0.1)
// Load keypair for signing requests (optional). Push always signs when a key is
// present; fetch signs only if the node 404s the anonymous advertisement, so
// public fetches stay anonymous. Absent a key, public fetch still works and push
// falls back to unsigned (v0.1 local alpha only).
let keypair = load_keypair();
run_helper(&repo_base, keypair.as_ref())
@@ -115,7 +124,7 @@ fn help_text() -> String {
\n\
ENVIRONMENT:\n\
\x20 GITLAWB_NODE Node base URL (default: http://127.0.0.1:7545)\n\
\x20 GITLAWB_KEY Identity PEM path for signed pushes (default: ~/.gitlawb/identity.pem)\n\
\x20 GITLAWB_KEY Identity PEM path for signed fetch/push (default: ~/.gitlawb/identity.pem)\n\
\x20 GITLAWB_LOG Log filter (default: warn)\n\
\n\
FLAGS:\n\
@@ -179,11 +188,11 @@ fn run_helper(repo_base: &str, keypair: Option<&Keypair>) -> Result<()> {
// ── HTTP proxy for git smart protocol ────────────────────────────────────────
fn handle_connect(
fn handle_connect<R: Read>(
repo_base: &str,
service: &str,
keypair: Option<&Keypair>,
stdin: &mut io::BufReader<io::Stdin>,
stdin: &mut R,
) -> Result<()> {
match service {
"git-upload-pack" | "git-receive-pack" => {}
@@ -198,19 +207,52 @@ fn handle_connect(
//
// The server advertises its refs. git reads this to decide what to fetch/push.
// Signing policy for this exchange, carried through to the Phase-2 POST:
// - git-receive-pack (push): sign from the first request; a push is
// signature-gated and can never be anonymous.
// - git-upload-pack (fetch): start anonymous so a public clone never discloses
// the caller's DID, then escalate to a single signed retry only if the node
// denies the anonymous advertisement with 404 (how it withholds a private
// repo from an unauthenticated reader).
let mut signing_key = if service == "git-receive-pack" {
keypair
} else {
None
};
let refs_url = format!("{}/info/refs?service={}", repo_base, service);
tracing::debug!("GET {refs_url}");
let refs_resp = client
.get(&refs_url)
.header("User-Agent", "git/2.0 git-remote-gitlawb/0.1.0")
let mut refs_resp = build_advertisement_request(&client, &refs_url, signing_key)
.send()
.with_context(|| format!("GET {refs_url}"))?;
// Fetch escalation: retry the advertisement signed once when the anonymous try
// is denied and we hold an identity. A public repo answers 200 on the first
// (anonymous) request and never reaches here, so it stays DID-private.
if service == "git-upload-pack"
&& refs_resp.status() == reqwest::StatusCode::NOT_FOUND
&& keypair.is_some()
{
tracing::debug!("anonymous info/refs returned 404; retrying signed");
signing_key = keypair;
refs_resp = build_advertisement_request(&client, &refs_url, signing_key)
.send()
.with_context(|| format!("GET {refs_url} (signed retry)"))?;
}
if !refs_resp.status().is_success() {
let status = refs_resp.status();
let body = read_error_body(refs_resp);
bail!(
"GET /info/refs returned {} — is the repo registered on this node?",
refs_resp.status()
"{}",
http_error_message(
"GET",
"/info/refs",
status,
&body,
Some("— is the repo registered on this node?")
)
);
}
@@ -266,29 +308,7 @@ fn handle_connect(
let post_url = format!("{}/{}", repo_base, service);
tracing::debug!("POST {post_url} ({} bytes)", request_body.len());
// Extract the URL path for signing (e.g., "/z6Mk.../my-repo/git-receive-pack")
let path_for_sig = url_path(&post_url);
let mut req = client
.post(&post_url)
.header("Content-Type", format!("application/x-{}-request", service))
.header("User-Agent", "git/2.0 git-remote-gitlawb/0.1.0");
// Add RFC 9421 HTTP Signature auth on push operations
if service == "git-receive-pack" {
if let Some(kp) = keypair {
let signed = sign_request(kp, "POST", &path_for_sig, &request_body);
req = req
.header("Content-Digest", signed.content_digest)
.header("Signature-Input", signed.signature_input)
.header("Signature", signed.signature);
tracing::debug!("attached RFC 9421 HTTP Signature (DID: {})", kp.did());
} else {
tracing::warn!(
"no identity keypair found — push will be unsigned (v0.1 local alpha only)"
);
}
}
let req = build_pack_post_request(&client, &post_url, service, &request_body, signing_key);
// Attach the body after signing so the pack bytes are moved, not cloned —
// packs can be large and the clone doubled peak memory on push.
@@ -298,7 +318,10 @@ fn handle_connect(
.with_context(|| format!("POST {post_url}"))?;
if !pack_resp.status().is_success() {
bail!("POST /{} returned {}", service, pack_resp.status());
let status = pack_resp.status();
let body = read_error_body(pack_resp);
let path = format!("/{service}");
bail!("{}", http_error_message("POST", &path, status, &body, None));
}
let pack_bytes = pack_resp.bytes().context("reading pack response")?;
@@ -310,6 +333,68 @@ fn handle_connect(
Ok(())
}
// ── Smart-protocol request builders ───────────────────────────────────────────
const USER_AGENT: &str = "git/2.0 git-remote-gitlawb/0.1.0";
/// Build the Phase-1 ref-advertisement GET, signing it when an identity is
/// present. The node gates the ref advertisement on read visibility for BOTH
/// services, so a private repo's advertisement is denied (404) to an
/// unauthenticated caller; without this signature the repo's own owner can
/// neither fetch (upload-pack) nor push (receive-pack) it. Public repos still
/// work anonymously (no keypair present, or the gate admits anonymous). Sign over
/// the path *and* query (?service=...) because the node verifies the signature
/// over its path_and_query.
fn build_advertisement_request(
client: &reqwest::blocking::Client,
refs_url: &str,
keypair: Option<&Keypair>,
) -> reqwest::blocking::RequestBuilder {
let mut req = client.get(refs_url).header("User-Agent", USER_AGENT);
if let Some(kp) = keypair {
let signed = sign_request(kp, "GET", &url_path(refs_url), b"");
req = req
.header("Content-Digest", signed.content_digest)
.header("Signature-Input", signed.signature_input)
.header("Signature", signed.signature);
tracing::debug!("signed info/refs advertisement (DID: {})", kp.did());
}
req
}
/// Build the Phase-2 pack POST, signing it when an identity is present, for BOTH
/// services. The node read-gates the git-upload-pack POST with visibility_check
/// at "/", and separately owner-gates the git-receive-pack POST (signature
/// required, enforced by middleware). The Phase-1 advertisement signature does
/// not carry to this separate request, so a private repo's owner must
/// authenticate here too or their fetch/push is denied.
/// Public-repo fetch still works anonymously when no keypair is present. The body
/// is signed (content-digest) but NOT attached here, so the caller can move the
/// (possibly large) pack bytes into `.body()` rather than clone them.
fn build_pack_post_request(
client: &reqwest::blocking::Client,
post_url: &str,
service: &str,
body: &[u8],
keypair: Option<&Keypair>,
) -> reqwest::blocking::RequestBuilder {
let mut req = client
.post(post_url)
.header("Content-Type", format!("application/x-{}-request", service))
.header("User-Agent", USER_AGENT);
if let Some(kp) = keypair {
let signed = sign_request(kp, "POST", &url_path(post_url), body);
req = req
.header("Content-Digest", signed.content_digest)
.header("Signature-Input", signed.signature_input)
.header("Signature", signed.signature);
tracing::debug!("signed {service} POST (DID: {})", kp.did());
} else if service == "git-receive-pack" {
tracing::warn!("no identity keypair found, push will be unsigned (v0.1 local alpha only)");
}
req
}
// ── URL helpers ───────────────────────────────────────────────────────────────
/// Parse a `gitlawb://did:key:z6Mk.../repo-name` URL.
@@ -347,7 +432,7 @@ fn parse_gitlawb_url(url: &str) -> Result<(String, String, String)> {
///
/// We also handle the flush-only case (`"0000"`) that git sends when it already
/// has everything it needs (up-to-date clone).
fn read_upload_pack_request(stdin: &mut io::BufReader<io::Stdin>) -> Result<Vec<u8>> {
fn read_upload_pack_request<R: Read>(stdin: &mut R) -> Result<Vec<u8>> {
let mut buf = Vec::new();
loop {
@@ -433,6 +518,78 @@ fn url_path(url: &str) -> String {
.unwrap_or_else(|| "/".to_string())
}
const MAX_ERROR_BODY_BYTES: u64 = 4096;
const MAX_ERROR_BODY_CHARS: usize = 2000;
fn read_error_body(mut response: reqwest::blocking::Response) -> String {
let mut body = Vec::new();
let mut limited = (&mut response).take(MAX_ERROR_BODY_BYTES);
if limited.read_to_end(&mut body).is_err() {
return String::new();
}
String::from_utf8_lossy(&body).into_owned()
}
fn http_error_message(
method: &str,
path: &str,
status: reqwest::StatusCode,
body: &str,
empty_body_hint: Option<&str>,
) -> String {
let mut message = format!("{method} {path} returned {status}");
let body = safe_error_body_excerpt(body);
if !body.is_empty() {
message.push_str(": ");
message.push_str(&body);
} else if let Some(hint) = empty_body_hint {
message.push(' ');
message.push_str(hint);
}
message
}
fn safe_error_body_excerpt(body: &str) -> String {
let mut excerpt = String::new();
let mut pending_space = false;
let mut kept_chars = 0;
for ch in body.trim().chars() {
if kept_chars >= MAX_ERROR_BODY_CHARS {
break;
}
if ch.is_control() {
if matches!(ch, '\n' | '\r' | '\t') {
pending_space = true;
}
continue;
}
if ch.is_whitespace() {
pending_space = true;
continue;
}
if pending_space && !excerpt.is_empty() {
excerpt.push(' ');
kept_chars += 1;
if kept_chars >= MAX_ERROR_BODY_CHARS {
break;
}
}
excerpt.push(ch);
kept_chars += 1;
pending_space = false;
}
excerpt
}
// ── Keypair loading ───────────────────────────────────────────────────────────
fn load_keypair() -> Option<Keypair> {
@@ -476,6 +633,467 @@ fn resolve_key_path() -> std::path::PathBuf {
#[cfg(test)]
mod tests {
use super::*;
use std::net::{TcpListener, TcpStream};
use std::thread::JoinHandle;
struct TestResponse {
request_line: &'static str,
status: &'static str,
body: &'static str,
}
fn serve_http(responses: Vec<TestResponse>) -> (String, JoinHandle<Vec<String>>) {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let base_url = format!("http://{}", listener.local_addr().unwrap());
let handle = std::thread::spawn(move || {
let mut requests = Vec::new();
for response in responses {
let (mut stream, _) = listener.accept().unwrap();
let request = read_http_request(&mut stream);
assert!(
request.starts_with(response.request_line),
"unexpected request:\n{request}"
);
let response_text = format!(
"HTTP/1.1 {}\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{}",
response.status,
response.body.len(),
response.body
);
stream.write_all(response_text.as_bytes()).unwrap();
requests.push(request);
}
requests
});
(base_url, handle)
}
fn read_http_request(stream: &mut TcpStream) -> String {
let mut buf = Vec::new();
let mut chunk = [0u8; 512];
let mut header_len = None;
loop {
let n = stream.read(&mut chunk).unwrap();
if n == 0 {
break;
}
buf.extend_from_slice(&chunk[..n]);
if header_len.is_none() {
header_len = buf.windows(4).position(|w| w == b"\r\n\r\n").map(|p| p + 4);
}
if let Some(header_len) = header_len {
let header = String::from_utf8_lossy(&buf[..header_len]);
let content_len = header
.lines()
.find_map(|line| {
let (name, value) = line.split_once(':')?;
name.eq_ignore_ascii_case("content-length")
.then(|| value.trim().parse::<usize>().ok())
.flatten()
})
.unwrap_or(0);
if buf.len() >= header_len + content_len {
break;
}
}
}
String::from_utf8_lossy(&buf).into_owned()
}
/// The regression that round-1 missed: the Phase-2 `git-upload-pack` POST was
/// left unsigned, so an owner's fetch of a private repo cleared the (now signed)
/// advertisement and then 404'd on the pack POST. Drive BOTH request builders
/// against a mock node and assert the RFC-9421 headers actually go out on the
/// wire for BOTH services when a keypair is present (the mock only matches when
/// the headers exist, so `.assert()` fails if any are dropped).
#[test]
fn advertisement_and_pack_post_are_signed_for_both_services_with_keypair() {
let kp = Keypair::generate();
let client = reqwest::blocking::Client::new();
let body = b"0009done\n".to_vec();
for service in ["git-upload-pack", "git-receive-pack"] {
let mut server = mockito::Server::new();
let repo_base = format!("{}/zOwner/myrepo", server.url());
let get_mock = server
.mock("GET", mockito::Matcher::Regex(r"/info/refs".to_string()))
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.match_header("content-digest", mockito::Matcher::Any)
.with_status(200)
.create();
let refs_url = format!("{repo_base}/info/refs?service={service}");
let resp = build_advertisement_request(&client, &refs_url, Some(&kp))
.send()
.unwrap();
assert!(resp.status().is_success());
get_mock.assert();
let post_mock = server
.mock("POST", mockito::Matcher::Regex(format!("/{service}$")))
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.match_header("content-digest", mockito::Matcher::Any)
.with_status(200)
.create();
let post_url = format!("{repo_base}/{service}");
let resp = build_pack_post_request(&client, &post_url, service, &body, Some(&kp))
.body(body.clone())
.send()
.unwrap();
assert!(resp.status().is_success());
post_mock.assert();
}
}
/// Without an identity, the pack POST must go out UNSIGNED so public fetch (and
/// alpha unsigned push) still work. `Matcher::Missing` matches only when the
/// header is absent, so the mock is hit only if NO signature was attached, for
/// BOTH services (the receive-pack iteration also exercises the no-keypair warn
/// branch and confirms it does not block the request).
#[test]
fn pack_post_is_unsigned_without_keypair_for_both_services() {
let client = reqwest::blocking::Client::new();
let body = b"0009done\n".to_vec();
for service in ["git-upload-pack", "git-receive-pack"] {
let mut server = mockito::Server::new();
let post_url = format!("{}/zOwner/myrepo/{service}", server.url());
let post_mock = server
.mock("POST", mockito::Matcher::Regex(format!("/{service}$")))
.match_header("signature", mockito::Matcher::Missing)
.match_header("signature-input", mockito::Matcher::Missing)
.match_header("content-digest", mockito::Matcher::Missing)
.with_status(200)
.create();
let resp = build_pack_post_request(&client, &post_url, service, &body, None)
.body(body.clone())
.send()
.unwrap();
assert!(resp.status().is_success());
post_mock.assert();
}
}
/// Symmetric negative case for Phase 1: with no identity the advertisement GET
/// carries no signature, so public clones stay anonymous.
#[test]
fn advertisement_get_is_unsigned_without_keypair() {
let client = reqwest::blocking::Client::new();
let mut server = mockito::Server::new();
let refs_url = format!(
"{}/zOwner/myrepo/info/refs?service=git-upload-pack",
server.url()
);
let get_mock = server
.mock("GET", mockito::Matcher::Regex(r"/info/refs".to_string()))
.match_header("signature", mockito::Matcher::Missing)
.match_header("signature-input", mockito::Matcher::Missing)
.match_header("content-digest", mockito::Matcher::Missing)
.with_status(200)
.create();
let resp = build_advertisement_request(&client, &refs_url, None)
.send()
.unwrap();
assert!(resp.status().is_success());
get_mock.assert();
}
// ── Phase-1 retry-on-404 (P2, #119): fetch stays anonymous until the node
// denies it, then escalates to ONE signed retry. Drives handle_connect end to
// end against the recording mock, asserting what actually goes on the wire.
// A request is "signed" iff it carries the `signature-input` header.
/// Public fetch: the node serves the anonymous advertisement (200), so the
/// client never signs — not the advertisement GET, not the upload-pack POST —
/// even though an identity is present. This is the privacy fix: a public clone
/// must not disclose the caller's DID.
#[test]
fn fetch_public_stays_anonymous_even_with_keypair() {
let kp = Keypair::generate();
let (base_url, server) = serve_http(vec![
TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-upload-pack",
status: "200 OK",
body: "0000",
},
TestResponse {
request_line: "POST /zOwner/myrepo/git-upload-pack",
status: "200 OK",
body: "",
},
]);
let repo_base = format!("{base_url}/zOwner/myrepo");
let mut stdin = std::io::Cursor::new(b"0009done\n".to_vec());
handle_connect(&repo_base, "git-upload-pack", Some(&kp), &mut stdin)
.expect("public fetch should succeed");
let requests = server.join().unwrap();
assert_eq!(requests.len(), 2, "public fetch must not retry");
assert!(
!requests[0].to_lowercase().contains("signature-input"),
"the advertisement GET must be anonymous for a public fetch"
);
assert!(
!requests[1].to_lowercase().contains("signature-input"),
"the upload-pack POST must be anonymous for a public fetch"
);
}
/// Private fetch: the anonymous advertisement is denied (404); with an identity
/// present the client retries the GET signed and — because Phase 1 needed
/// signing — also signs the Phase-2 upload-pack POST. The FIRST request is still
/// anonymous: the DID is not disclosed until the node demands authentication.
#[test]
fn fetch_private_retries_signed_on_404_and_signs_the_pack_post() {
let kp = Keypair::generate();
let (base_url, server) = serve_http(vec![
TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-upload-pack",
status: "404 Not Found",
body: r#"{"message":"repository is not registered on this node"}"#,
},
TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-upload-pack",
status: "200 OK",
body: "0000",
},
TestResponse {
request_line: "POST /zOwner/myrepo/git-upload-pack",
status: "200 OK",
body: "",
},
]);
let repo_base = format!("{base_url}/zOwner/myrepo");
let mut stdin = std::io::Cursor::new(b"0009done\n".to_vec());
handle_connect(&repo_base, "git-upload-pack", Some(&kp), &mut stdin)
.expect("signed retry should succeed");
let requests = server.join().unwrap();
assert_eq!(
requests.len(),
3,
"expected anon GET, signed GET retry, signed POST"
);
assert!(
!requests[0].to_lowercase().contains("signature-input"),
"the first advertisement GET must be anonymous"
);
assert!(
requests[1].to_lowercase().contains("signature-input"),
"the retry advertisement GET must be signed"
);
assert!(
requests[2].to_lowercase().contains("signature-input"),
"the upload-pack POST must be signed once the fetch needed auth"
);
}
/// Private fetch with no identity: the anonymous advertisement 404s and there is
/// no keypair to retry with, so the client surfaces the error after exactly one
/// request — no retry, no hang.
#[test]
fn fetch_without_keypair_does_not_retry_on_404() {
let (base_url, server) = serve_http(vec![TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-upload-pack",
status: "404 Not Found",
body: r#"{"message":"not found"}"#,
}]);
let repo_base = format!("{base_url}/zOwner/myrepo");
let mut stdin = std::io::Cursor::new(Vec::<u8>::new());
let err = handle_connect(&repo_base, "git-upload-pack", None, &mut stdin).unwrap_err();
assert!(err.to_string().contains("returned 404 Not Found"));
let requests = server.join().unwrap();
assert_eq!(requests.len(), 1, "no keypair means no signed retry");
assert!(!requests[0].to_lowercase().contains("signature-input"));
}
/// Private fetch where the SIGNED retry is also denied (caller is not a reader):
/// the client retries exactly once, then surfaces the 404 rather than looping.
/// Bounds the escalation to a single signed attempt.
#[test]
fn fetch_signed_retry_still_denied_surfaces_error_once() {
let kp = Keypair::generate();
let (base_url, server) = serve_http(vec![
TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-upload-pack",
status: "404 Not Found",
body: r#"{"message":"not found"}"#,
},
TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-upload-pack",
status: "404 Not Found",
body: r#"{"message":"not found"}"#,
},
]);
let repo_base = format!("{base_url}/zOwner/myrepo");
let mut stdin = std::io::Cursor::new(Vec::<u8>::new());
let err = handle_connect(&repo_base, "git-upload-pack", Some(&kp), &mut stdin).unwrap_err();
assert!(err.to_string().contains("returned 404 Not Found"));
let requests = server.join().unwrap();
assert_eq!(requests.len(), 2, "exactly one signed retry, then surface");
assert!(!requests[0].to_lowercase().contains("signature-input"));
assert!(requests[1].to_lowercase().contains("signature-input"));
}
/// Push (git-receive-pack) is unchanged: it signs from the FIRST request — you
/// cannot push anonymously — so there is no anonymous-first probe for push.
#[test]
fn push_advertisement_is_signed_from_the_first_request() {
let kp = Keypair::generate();
let (base_url, server) = serve_http(vec![
TestResponse {
request_line: "GET /zOwner/myrepo/info/refs?service=git-receive-pack",
status: "200 OK",
body: "0000",
},
TestResponse {
request_line: "POST /zOwner/myrepo/git-receive-pack",
status: "200 OK",
body: "",
},
]);
let repo_base = format!("{base_url}/zOwner/myrepo");
let mut stdin = std::io::Cursor::new(b"0000".to_vec());
handle_connect(&repo_base, "git-receive-pack", Some(&kp), &mut stdin)
.expect("push should succeed");
let requests = server.join().unwrap();
assert_eq!(requests.len(), 2);
assert!(
requests[0].to_lowercase().contains("signature-input"),
"push advertisement must be signed from the first request"
);
assert!(
requests[1].to_lowercase().contains("signature-input"),
"push POST must be signed"
);
}
/// Cross-crate seam: the signature the CLIENT actually emits (via the real
/// request builders) verifies under the SAME `gitlawb_core::http_sig`
/// primitives the node's `require_signature` uses, over the `@path` the client
/// genuinely transmits (derived from the built reqwest request, exactly as the
/// node derives it from `uri.path_and_query()`). A tampered `@path` must fail:
/// this is the byte-match the whole A1 client fix depends on, now executed end
/// to end (sign here, verify with the node's verifier), not reasoned.
#[test]
fn client_signature_verifies_under_node_verification_for_both_services() {
use gitlawb_core::http_sig::{build_signing_string, compute_content_digest, HttpSignature};
use gitlawb_core::identity::verify;
use std::collections::HashMap;
let kp = Keypair::generate();
let client = reqwest::blocking::Client::new();
let body = b"0009done\n".to_vec();
// Re-implements the node's require_signature verification (auth/mod.rs):
// parse headers, recompute content-digest from the body, rebuild the signing
// string over @method/@path/content-digest, Ed25519-verify. Ok iff the node
// would accept it.
let node_verifies = |method: &str,
path_and_query: &str,
body: &[u8],
sig_input: &str,
sig_header: &str,
content_digest: &str|
-> anyhow::Result<()> {
let sig = HttpSignature::parse(sig_input, sig_header)?;
sig.check_created()?;
assert!(
sig.missing_components().is_empty(),
"signature must cover all required components"
);
assert_eq!(sig.alg, "ed25519");
assert_eq!(
content_digest,
compute_content_digest(body),
"content-digest must match the body"
);
let vk = sig.key_id.to_verifying_key()?;
let mut values = HashMap::new();
values.insert("@method".to_string(), method.to_uppercase());
values.insert("@path".to_string(), path_and_query.to_string());
values.insert("content-digest".to_string(), content_digest.to_string());
let sig_params_value = sig_input.strip_prefix("sig1=").unwrap_or(sig_input);
let components: Vec<&str> = sig.components.iter().map(String::as_str).collect();
let signing_string = build_signing_string(&components, sig_params_value, &values)?;
let sig_array: [u8; 64] = sig.signature_bytes.as_slice().try_into()?;
verify(&vk, signing_string.as_bytes(), &sig_array)?;
Ok(())
};
// @path exactly as the node reconstructs it from the request it receives.
let path_and_query = |req: &reqwest::blocking::Request| match req.url().query() {
Some(q) => format!("{}?{}", req.url().path(), q),
None => req.url().path().to_string(),
};
let header = |req: &reqwest::blocking::Request, name: &str| {
req.headers()
.get(name)
.unwrap_or_else(|| panic!("missing {name}"))
.to_str()
.unwrap()
.to_string()
};
for service in ["git-upload-pack", "git-receive-pack"] {
// Phase-1 advertisement GET (empty body).
let refs_url = format!("http://node.example/zOwner/myrepo/info/refs?service={service}");
let req = build_advertisement_request(&client, &refs_url, Some(&kp))
.build()
.unwrap();
node_verifies(
"GET",
&path_and_query(&req),
b"",
&header(&req, "signature-input"),
&header(&req, "signature"),
&header(&req, "content-digest"),
)
.expect("client GET signature must verify under the node's verifier");
// Phase-2 pack POST (real body).
let post_url = format!("http://node.example/zOwner/myrepo/{service}");
let req = build_pack_post_request(&client, &post_url, service, &body, Some(&kp))
.body(body.clone())
.build()
.unwrap();
let pq = path_and_query(&req);
node_verifies(
"POST",
&pq,
&body,
&header(&req, "signature-input"),
&header(&req, "signature"),
&header(&req, "content-digest"),
)
.expect("client POST signature must verify under the node's verifier");
// Negative: a tampered @path (the A1 attack surface) must NOT verify.
let tampered = pq.replace(service, "git-evil");
assert!(
node_verifies(
"POST",
&tampered,
&body,
&header(&req, "signature-input"),
&header(&req, "signature"),
&header(&req, "content-digest"),
)
.is_err(),
"a tampered @path must fail verification"
);
}
}
#[test]
fn parse_standard_url() {
@@ -524,6 +1142,100 @@ mod tests {
);
}
#[test]
fn http_error_message_sanitizes_response_body_controls() {
let message = http_error_message(
"GET",
"/info/refs",
reqwest::StatusCode::BAD_GATEWAY,
"\u{1b}[2Jrepository\r\nis\tblocked\u{7}",
None,
);
assert!(message.contains("repository is blocked"));
assert!(!message.contains('\u{1b}'));
assert!(!message.contains('\u{7}'));
assert!(!message.contains('\r'));
assert!(!message.contains('\n'));
}
#[test]
fn http_error_message_truncates_long_response_body() {
let body = "x".repeat(MAX_ERROR_BODY_CHARS + 100);
let message = http_error_message(
"POST",
"/git-receive-pack",
reqwest::StatusCode::FORBIDDEN,
&body,
None,
);
let prefix = "POST /git-receive-pack returned 403 Forbidden: ";
assert_eq!(message.len(), prefix.len() + MAX_ERROR_BODY_CHARS);
assert!(message.starts_with(prefix));
}
#[test]
fn connect_get_error_includes_response_body() {
let (base_url, server) = serve_http(vec![TestResponse {
request_line: "GET /z6Mk/myrepo/info/refs?service=git-upload-pack HTTP/1.1",
status: "404 Not Found",
body: r#"{"message":"repository is not registered on this node"}"#,
}]);
let repo_base = format!("{base_url}/z6Mk/myrepo");
let mut stdin = std::io::Cursor::new(Vec::<u8>::new());
let err = handle_connect(&repo_base, "git-upload-pack", None, &mut stdin).unwrap_err();
let message = err.to_string();
assert!(message.contains("GET /info/refs returned 404 Not Found"));
assert!(message.contains("repository is not registered on this node"));
let requests = server.join().unwrap();
assert_eq!(requests.len(), 1);
}
#[test]
fn connect_post_error_includes_response_body() {
let (base_url, server) = serve_http(vec![
TestResponse {
request_line: "GET /z6Mk/myrepo/info/refs?service=git-receive-pack HTTP/1.1",
status: "200 OK",
body: "",
},
TestResponse {
request_line: "POST /z6Mk/myrepo/git-receive-pack HTTP/1.1",
status: "403 Forbidden",
body: r#"{"error":"forbidden","message":"push rejected: only the repo owner may push"}"#,
},
]);
let repo_base = format!("{base_url}/z6Mk/myrepo");
let mut stdin = std::io::Cursor::new(b"0000".to_vec());
let err = handle_connect(&repo_base, "git-receive-pack", None, &mut stdin).unwrap_err();
let message = err.to_string();
assert!(message.contains("POST /git-receive-pack returned 403 Forbidden"));
assert!(message.contains("push rejected: only the repo owner may push"));
let requests = server.join().unwrap();
assert_eq!(requests.len(), 2);
assert!(requests[1].contains("\r\n\r\n0000"));
}
#[test]
fn url_path_preserves_query_for_signed_advertisement() {
// The Phase-1 advertisement GET is signed over url_path(refs_url), and the
// node verifies the signature over its path_and_query, so the ?service=
// query MUST survive verbatim or the signatures will not byte-match.
assert_eq!(
url_path("http://127.0.0.1:7545/z6Mk/myrepo/info/refs?service=git-upload-pack"),
"/z6Mk/myrepo/info/refs?service=git-upload-pack"
);
assert_eq!(
url_path("http://127.0.0.1:7545/z6Mk/myrepo/info/refs?service=git-receive-pack"),
"/z6Mk/myrepo/info/refs?service=git-receive-pack"
);
}
fn args(items: &[&str]) -> Vec<String> {
items.iter().map(|s| s.to_string()).collect()
}
+1 -1
View File
@@ -1,7 +1,7 @@
[package]
name = "gitlawb-attest"
description = "External Attestation v1: pluggable provenance attachments for gitlawb ref-update certs"
version = "0.4.0" # x-release-please-version
version = "0.5.1" # x-release-please-version
edition.workspace = true
rust-version.workspace = true
license.workspace = true
+1 -1
View File
@@ -1,7 +1,7 @@
[package]
name = "gitlawb-core"
description = "Core cryptographic primitives for the gitlawb network: DIDs, CIDs, UCAN, HTTP Signatures, ref-update certificates"
version = "0.4.0" # x-release-please-version
version = "0.5.1" # x-release-please-version
edition.workspace = true
rust-version.workspace = true
license.workspace = true
+45
View File
@@ -64,6 +64,19 @@ impl Cid {
}
}
/// True when `s` parses as a CIDv1 with the raw codec — the exact shape
/// [`Cid::from_git_object_bytes`] produces and the `/ipfs` resolver looks up.
/// A legacy provider CID (Kubo dag-pb, Pinata CIDv0) parses to a different
/// version or codec and returns `false`, marking it an opportunistic-repair
/// candidate. Decidable from the string alone (no object bytes), so the pin path
/// can gate the byte-read/recompute cost on it and leave non-legacy rows at the
/// existing DB-only skip cost. An unparseable string is non-canonical (`false`).
pub fn is_raw_cidv1(s: &str) -> bool {
s.parse::<CidGeneric<64>>()
.map(|c| c.version() == cid::Version::V1 && c.codec() == RAW)
.unwrap_or(false)
}
impl fmt::Display for Cid {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.0)
@@ -177,6 +190,38 @@ mod tests {
assert!(result.is_err());
}
#[test]
fn is_raw_cidv1_classifies_codec_from_string() {
// The canonical resolver key: CIDv1 + raw codec → not a repair candidate.
let raw = Cid::from_git_object_bytes(b"blob 5\0hello");
assert!(
is_raw_cidv1(raw.as_str()),
"from_git_object_bytes output is CIDv1/raw"
);
// A CIDv0 (Pinata dag-pb legacy shape) → repair candidate.
assert!(
!is_raw_cidv1("QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG"),
"a CIDv0 dag-pb value is a legacy-repair candidate"
);
// A CIDv1 with the dag-pb codec (the Kubo above-block-size root) over the
// same multihash → still a repair candidate (codec, not just version).
let parsed = raw.as_str().parse::<CidGeneric<64>>().unwrap();
const DAG_PB: u64 = 0x70;
let dagpb = CidGeneric::<64>::new_v1(DAG_PB, *parsed.hash()).to_string();
assert!(
!is_raw_cidv1(&dagpb),
"a CIDv1 dag-pb value is a legacy-repair candidate"
);
// Garbage is non-canonical.
assert!(
!is_raw_cidv1("not-a-cid"),
"an unparseable string is non-canonical"
);
}
#[test]
fn sha256_hex_of_empty_input_is_well_known() {
// SHA-256("") is a fixed constant; verifies the hasher is wired correctly.
+1 -1
View File
@@ -1,7 +1,7 @@
[package]
name = "gitlawb-node"
description = "The gitlawb node daemon — git hosting over HTTP with DID auth"
version = "0.4.0" # x-release-please-version
version = "0.5.1" # x-release-please-version
edition.workspace = true
rust-version.workspace = true
license.workspace = true
+103 -16
View File
@@ -110,19 +110,28 @@ pub async fn create_bounty(
Ok((StatusCode::CREATED, Json(bounty)))
}
const MAX_BOUNTY_LIMIT: i64 = 200;
/// GET /api/v1/repos/{owner}/{repo}/bounties
pub async fn list_repo_bounties(
State(state): State<AppState>,
Path((owner, repo)): Path<(String, String)>,
Query(q): Query<ListBountiesQuery>,
auth: Option<Extension<AuthenticatedDid>>,
) -> Result<Json<serde_json::Value>> {
let caller = auth.as_ref().map(|e| e.0 .0.as_str());
crate::api::authorize_repo_read(&state, &owner, &repo, caller, "/").await?;
let limit = q.limit.unwrap_or(50).clamp(1, MAX_BOUNTY_LIMIT);
let bounties = state
.db
.list_bounties(
Some(&owner),
Some(&repo),
q.status.as_deref(),
q.limit.unwrap_or(50),
limit,
None,
None,
)
.await?;
@@ -133,25 +142,92 @@ pub async fn list_repo_bounties(
pub async fn list_all_bounties(
State(state): State<AppState>,
Query(q): Query<ListBountiesQuery>,
auth: Option<Extension<AuthenticatedDid>>,
) -> Result<Json<serde_json::Value>> {
let bounties = state
.db
.list_bounties(None, None, q.status.as_deref(), q.limit.unwrap_or(50))
.await?;
let raw_limit = q.limit.unwrap_or(50);
let limit = raw_limit.clamp(1, MAX_BOUNTY_LIMIT);
Ok(Json(serde_json::json!({ "bounties": bounties })))
let caller = auth.as_ref().map(|e| e.0 .0.as_str());
let mut memo = std::collections::HashMap::new();
let mut allowed: Vec<BountyRecord> = Vec::new();
let page_size = (limit * 5).clamp(1, 200);
let max_scanned = 10_000i64;
let mut total_scanned: i64 = 0;
let mut cursor: Option<(String, String)> = None;
while (allowed.len() as i64) < limit {
let after = cursor.as_ref().map(|(ts, id)| (ts.as_str(), id.as_str()));
let bounties = state
.db
.list_bounties(
None,
None,
q.status.as_deref(),
page_size,
after.map(|(ts, _)| ts),
after.map(|(_, id)| id),
)
.await?;
if bounties.is_empty() {
break;
}
total_scanned += bounties.len() as i64;
if total_scanned > max_scanned {
return Err(AppError::Incomplete(
"too many rows scanned; refine your filters".into(),
));
}
let last = bounties.last().unwrap();
for b in &bounties {
if (allowed.len() as i64) >= limit {
break;
}
let key = format!("{}/{}", b.repo_owner, b.repo_name);
let admitted = if let Some(&result) = memo.get(&key) {
result
} else {
let result = crate::api::authorize_repo_read(
&state,
&b.repo_owner,
&b.repo_name,
caller,
"/",
)
.await
.is_ok();
memo.insert(key, result);
result
};
if admitted {
allowed.push(b.clone());
}
}
cursor = Some((last.created_at.clone(), last.id.clone()));
}
Ok(Json(serde_json::json!({ "bounties": allowed })))
}
/// GET /api/v1/bounties/{id}
pub async fn get_bounty(
State(state): State<AppState>,
Path(id): Path<String>,
auth: Option<Extension<AuthenticatedDid>>,
) -> Result<Json<BountyRecord>> {
let bounty = state
.db
.get_bounty(&id)
.await?
.ok_or_else(|| AppError::NotFound(format!("bounty {id} not found")))?;
let not_found = || AppError::NotFound(format!("bounty {id} not found"));
let bounty = state.db.get_bounty(&id).await?.ok_or_else(not_found)?;
let caller = auth.as_ref().map(|e| e.0 .0.as_str());
if crate::api::authorize_repo_read(&state, &bounty.repo_owner, &bounty.repo_name, caller, "/")
.await
.is_err()
{
return Err(not_found());
}
Ok(Json(bounty))
}
@@ -162,11 +238,22 @@ pub async fn claim_bounty(
Path(id): Path<String>,
Json(req): Json<ClaimBountyRequest>,
) -> Result<Json<BountyRecord>> {
let bounty = state
.db
.get_bounty(&id)
.await?
.ok_or_else(|| AppError::NotFound(format!("bounty {id} not found")))?;
let not_found = || AppError::NotFound(format!("bounty {id} not found"));
let bounty = state.db.get_bounty(&id).await?.ok_or_else(not_found)?;
if crate::api::authorize_repo_read(
&state,
&bounty.repo_owner,
&bounty.repo_name,
Some(auth.0.as_str()),
"/",
)
.await
.is_err()
{
return Err(not_found());
}
if bounty.status != "open" {
return Err(AppError::BadRequest(format!(
+41 -15
View File
@@ -1,23 +1,44 @@
//! API handlers for ref certificates.
use axum::extract::{Path, State};
use std::collections::HashMap;
use axum::extract::{Extension, Path, Query, State};
use axum::Json;
use crate::auth::AuthenticatedDid;
use crate::error::{AppError, Result};
use crate::state::AppState;
/// GET /api/v1/repos/{owner}/{repo}/certs
/// GET /api/v1/repos/{owner}/{repo}/certs?limit=50
pub async fn list_certs(
State(state): State<AppState>,
Path((owner, name)): Path<(String, String)>,
Query(params): Query<HashMap<String, String>>,
auth: Option<Extension<AuthenticatedDid>>,
) -> Result<Json<serde_json::Value>> {
let record = state
.db
.get_repo(&owner, &name)
.await?
.ok_or_else(|| AppError::RepoNotFound(format!("{owner}/{name}")))?;
let limit = params
.get("limit")
.and_then(|v| v.parse::<i64>().ok())
.map(|v| v.max(1))
.unwrap_or(50)
.min(200);
let certs = state.db.list_ref_certificates(&record.id).await?;
let caller = auth.as_ref().map(|e| e.0 .0.as_str());
let (record, _rules) =
crate::api::authorize_repo_read(&state, &owner, &name, caller, "/").await?;
// When a prefix is given (short-ID resolution from the CLI) use a
// generous limit and delegate to the prefix-matched query so the
// caller can resolve IDs regardless of how many certs exist.
let prefix = params.get("prefix").filter(|p| !p.is_empty());
let certs = if let Some(prefix) = prefix {
state
.db
.list_ref_certificates_by_prefix(&record.id, prefix, 200)
.await?
} else {
state.db.list_ref_certificates(&record.id, limit).await?
};
let certs_json: Vec<serde_json::Value> = certs
.iter()
.map(|c| {
@@ -35,20 +56,21 @@ pub async fn list_certs(
})
.collect();
Ok(Json(serde_json::json!({ "certificates": certs_json })))
let count = certs_json.len();
Ok(Json(
serde_json::json!({ "certificates": certs_json, "count": count }),
))
}
/// GET /api/v1/repos/{owner}/{repo}/certs/{id}
pub async fn get_cert(
State(state): State<AppState>,
Path((owner, name, id)): Path<(String, String, String)>,
auth: Option<Extension<AuthenticatedDid>>,
) -> Result<Json<serde_json::Value>> {
// Verify the repo exists
let _record = state
.db
.get_repo(&owner, &name)
.await?
.ok_or_else(|| AppError::RepoNotFound(format!("{owner}/{name}")))?;
let caller = auth.as_ref().map(|e| e.0 .0.as_str());
let (record, _rules) =
crate::api::authorize_repo_read(&state, &owner, &name, caller, "/").await?;
let cert = state
.db
@@ -56,6 +78,10 @@ pub async fn get_cert(
.await?
.ok_or_else(|| AppError::NotFound(format!("certificate {id}")))?;
if cert.repo_id != record.id {
return Err(AppError::NotFound(format!("certificate {id}")));
}
Ok(Json(serde_json::json!({
"id": cert.id,
"repo_id": cert.repo_id,
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+203 -18
View File
@@ -61,11 +61,14 @@ pub async fn create_issue(
let json_str = serde_json::to_string(&issue)
.map_err(|e| AppError::BadRequest(format!("serialization error: {e}")))?;
// Shed 503 + Retry-After on an exhausted write-lock POOL instead of a generic
// git 500 (#173 F1). This path holds no admission permit, so it reaches the pool
// unthrottled; reuse the push handler's mapping so the two cannot drift.
let guard = state
.repo_store
.acquire_write(&record.owner_did, &record.name)
.await
.map_err(|e| AppError::Git(e.to_string()))?;
.map_err(|e| crate::api::repos::acquire_write_app_error(&e, &repo))?;
let disk_path = guard.path().to_path_buf();
let create_result = git_issues::create_issue(&disk_path, &issue_id, &json_str);
@@ -90,12 +93,11 @@ pub async fn create_issue(
pub async fn list_issues(
State(state): State<AppState>,
Path((owner, repo)): Path<(String, String)>,
auth: Option<Extension<AuthenticatedDid>>,
) -> Result<Json<serde_json::Value>> {
let record = state
.db
.get_repo(&owner, &repo)
.await?
.ok_or_else(|| AppError::RepoNotFound(format!("{owner}/{repo}")))?;
let caller = auth.as_ref().map(|e| e.0 .0.as_str());
let (record, _rules) =
crate::api::authorize_repo_read(&state, &owner, &repo, caller, "/").await?;
let disk_path = state
.repo_store
@@ -120,12 +122,11 @@ pub async fn list_issues(
pub async fn get_issue(
State(state): State<AppState>,
Path((owner, repo, issue_id)): Path<(String, String, String)>,
auth: Option<Extension<AuthenticatedDid>>,
) -> Result<Json<serde_json::Value>> {
let record = state
.db
.get_repo(&owner, &repo)
.await?
.ok_or_else(|| AppError::RepoNotFound(format!("{owner}/{repo}")))?;
let caller = auth.as_ref().map(|e| e.0 .0.as_str());
let (record, _rules) =
crate::api::authorize_repo_read(&state, &owner, &repo, caller, "/").await?;
let disk_path = state
.repo_store
@@ -191,14 +192,31 @@ pub async fn create_issue_comment(
pub async fn list_issue_comments(
State(state): State<AppState>,
Path((owner, repo, issue_id)): Path<(String, String, String)>,
auth: Option<Extension<AuthenticatedDid>>,
) -> Result<Json<serde_json::Value>> {
let _record = state
.db
.get_repo(&owner, &repo)
.await?
.ok_or_else(|| AppError::RepoNotFound(format!("{owner}/{repo}")))?;
let caller = auth.as_ref().map(|e| e.0 .0.as_str());
let (record, _rules) =
crate::api::authorize_repo_read(&state, &owner, &repo, caller, "/").await?;
let comments = state.db.list_issue_comments(&issue_id).await?;
let disk_path = state
.repo_store
.acquire(&record.owner_did, &record.name)
.await
.map_err(|e| AppError::Git(e.to_string()))?;
// Resolve the full issue ID (accepts 8-char prefix) so the DB fetch
// below uses the same canonical id as the git ref.
let full_id = match git_issues::resolve_issue_id(&disk_path, &issue_id)
.map_err(|e| AppError::Git(e.to_string()))?
{
Some(id) => id,
None => {
return Err(AppError::RepoNotFound(format!(
"issue {issue_id} not found"
)))
}
};
let comments = state.db.list_issue_comments(&full_id).await?;
Ok(Json(serde_json::json!({ "comments": comments })))
}
@@ -214,11 +232,13 @@ pub async fn close_issue(
.await?
.ok_or_else(|| AppError::RepoNotFound(format!("{owner}/{repo}")))?;
// Same capacity shed as create_issue above (#173 F1): an exhausted write-lock
// pool is a 503 + Retry-After, not a 500 git error.
let guard = state
.repo_store
.acquire_write(&record.owner_did, &record.name)
.await
.map_err(|e| AppError::Git(e.to_string()))?;
.map_err(|e| crate::api::repos::acquire_write_app_error(&e, &repo))?;
let disk_path = guard.path().to_path_buf();
// Owner OR issue author may close. The author lives in the issue's git-JSON
@@ -264,3 +284,168 @@ pub async fn close_issue(
Ok(Json(issue))
}
/// #173 F1 follow-up: the two issue write paths reach `acquire_write` holding NO
/// admission permit (unlike the push handler, which is capped by the git-push
/// semaphore), so they are the callers most likely to meet an exhausted write-lock
/// POOL under load. An exhausted pool is a capacity signal, so both must shed
/// 503 + Retry-After (`AppError::Overloaded`) the way the push handler does, not
/// report the generic 500 git error that says nothing about retrying.
#[cfg(test)]
mod lock_pool_shed_tests {
use super::*;
use axum::response::IntoResponse;
use sqlx::PgPool;
fn seed_repo(owner_did: &str, name: &str) -> crate::db::RepoRecord {
let now = Utc::now();
crate::db::RepoRecord {
id: Uuid::new_v4().to_string(),
name: name.to_string(),
owner_did: owner_did.to_string(),
description: None,
is_public: true,
default_branch: "main".to_string(),
created_at: now,
updated_at: now,
disk_path: format!("/tmp/{name}"),
forked_from: None,
machine_id: None,
}
}
/// State whose repo store draws write locks from a ONE-connection pool with a
/// short checkout timeout, so a single held guard exhausts it promptly rather
/// than at the pool default.
async fn one_connection_lock_pool_state(pool: &PgPool) -> AppState {
let mut state = crate::test_support::test_state(pool.clone()).await;
state.repo_store = crate::git::repo_store::RepoStore::new(
std::path::PathBuf::from("/tmp/gitlawb-issues-lockpool"),
None,
crate::git::repo_store::build_lock_pool(pool, 1, std::time::Duration::from_secs(1)),
);
state
}
/// The shed must be a real 503 carrying Retry-After, not just an internal enum
/// variant: assert on the rendered response so a remapping of `Overloaded` is
/// caught here too.
fn assert_sheds_503_with_retry_after(err: AppError, what: &str) {
let resp = err.into_response();
assert_eq!(
resp.status(),
StatusCode::SERVICE_UNAVAILABLE,
"{what}: an exhausted write-lock pool must shed 503, not a 500 git error"
);
assert_eq!(
resp.headers()
.get("retry-after")
.map(|v| v.to_str().unwrap()),
Some("1"),
"{what}: a capacity shed must tell the client when to retry"
);
}
/// RED-before/GREEN-after for `create_issue`. Both directions: the shed while the
/// only lock-pool connection is held by a guard on a DIFFERENT repo (so this is
/// pool capacity, not advisory-lock contention on this repo), and the must-not
/// case once that connection is back.
#[sqlx::test]
async fn create_issue_lock_pool_exhaustion_sheds_503_not_500(pool: PgPool) {
let owner = "did:key:zISSUECREATELOCKPOOLAAAAAAAAAAAAAAAAAAAA";
let state = one_connection_lock_pool_state(&pool).await;
state
.db
.create_repo(&seed_repo(owner, "lp-create"))
.await
.expect("seed repo");
let held = state
.repo_store
.acquire_write(owner, "other-repo")
.await
.expect("the first write takes the only lock-pool connection");
let shed = create_issue(
State(state.clone()),
Extension(AuthenticatedDid(owner.to_string())),
Path((owner.to_string(), "lp-create".to_string())),
Json(CreateIssueRequest {
title: "t".to_string(),
body: None,
signed_payload: None,
}),
)
.await;
let err = shed.expect_err("an exhausted lock pool must fail the call");
assert_sheds_503_with_retry_after(err, "create_issue");
// MUST-NOT: with the pool free again the call is not shed as capacity (it
// fails later on the nonexistent on-disk repo, which is a git 500).
held.release(false).await;
let admitted = create_issue(
State(state.clone()),
Extension(AuthenticatedDid(owner.to_string())),
Path((owner.to_string(), "lp-create".to_string())),
Json(CreateIssueRequest {
title: "t".to_string(),
body: None,
signed_payload: None,
}),
)
.await;
assert!(
!matches!(admitted, Err(AppError::Overloaded(_))),
"with the lock pool free, create_issue must not be shed as capacity; got {:?}",
admitted.err()
);
}
/// RED-before/GREEN-after for `close_issue`, same two directions.
#[sqlx::test]
async fn close_issue_lock_pool_exhaustion_sheds_503_not_500(pool: PgPool) {
let owner = "did:key:zISSUECLOSELOCKPOOLBBBBBBBBBBBBBBBBBBBBB";
let state = one_connection_lock_pool_state(&pool).await;
state
.db
.create_repo(&seed_repo(owner, "lp-close"))
.await
.expect("seed repo");
let held = state
.repo_store
.acquire_write(owner, "other-repo")
.await
.expect("the first write takes the only lock-pool connection");
let shed = close_issue(
State(state.clone()),
Extension(AuthenticatedDid(owner.to_string())),
Path((
owner.to_string(),
"lp-close".to_string(),
"deadbeef".to_string(),
)),
)
.await;
let err = shed.expect_err("an exhausted lock pool must fail the call");
assert_sheds_503_with_retry_after(err, "close_issue");
held.release(false).await;
let admitted = close_issue(
State(state.clone()),
Extension(AuthenticatedDid(owner.to_string())),
Path((
owner.to_string(),
"lp-close".to_string(),
"deadbeef".to_string(),
)),
)
.await;
assert!(
!matches!(admitted, Err(AppError::Overloaded(_))),
"with the lock pool free, close_issue must not be shed as capacity; got {:?}",
admitted.err()
);
}
}
+4 -5
View File
@@ -74,12 +74,11 @@ pub async fn remove_label(
pub async fn list_labels(
State(state): State<AppState>,
Path((owner, name)): Path<(String, String)>,
auth: Option<Extension<AuthenticatedDid>>,
) -> Result<Json<serde_json::Value>> {
let record = state
.db
.get_repo(&owner, &name)
.await?
.ok_or_else(|| AppError::RepoNotFound(format!("{owner}/{name}")))?;
let caller = auth.as_ref().map(|e| e.0 .0.as_str());
let (record, _rules) =
crate::api::authorize_repo_read(&state, &owner, &name, caller, "/").await?;
let labels = state.db.list_labels(&record.id).await?;
Ok(Json(serde_json::json!({ "labels": labels })))
+143 -23
View File
@@ -114,6 +114,12 @@ mod did_tests {
assert!(!did_matches("did:key:zABC", "did:key:zXYZ"));
assert!(!did_matches("zABC", "zXYZ"));
}
#[test]
fn empty_did_matches() {
assert!(did_matches("", ""));
assert!(did_matches("", "did:key:"));
}
}
/// Drift guard (plan 002 §Gate-type table, Step 5). Every in-scope mutation
@@ -194,9 +200,11 @@ mod authz_guard {
(issues, "create_issue", "authorize_repo_read("),
(bounties, "create_bounty", "authorize_repo_read("),
(repos, "fork_repo", "authorize_repo_read("),
// get_by_cid gates each iterated repo row directly via visibility_check
// (KTD2a: it must NOT route through authorize_repo_read's fuzzy re-resolve).
(ipfs, "get_by_cid", "visibility_check("),
// get_by_cid resolves each candidate (provenance path + legacy scan) through
// the shared `gate_and_serve` (#173 round 2); the gate markers themselves are
// asserted below. This row proves the delegation is real — the gate is
// actually reached, not dead code.
(ipfs, "get_by_cid", "gate_and_serve("),
// Bucket C — signer-self: the acting DID is matched/bound to auth.0
(tasks, "create_task", "did_matches("),
(tasks, "claim_task", "did_matches("),
@@ -230,6 +238,22 @@ mod authz_guard {
"visibility::require_owner must use did_matches for DID-safe owner matching"
);
// The CID read surface (#173) enforces its gate inside the shared
// `gate_and_serve`, which BOTH the provenance path and the legacy scan call, so
// the markers must live there (the get_by_cid row above only proves delegation).
// The repo's own "/" visibility check (KTD2a — never authorize_repo_read's fuzzy
// re-resolve) and the quarantine hard-drop BEFORE visibility (INV-11) are both
// load-bearing: removing either re-opens a leak on the provenance path.
let gate_body = fn_body(ipfs, "gate_and_serve");
assert!(
gate_body.contains("visibility_check("),
"gate_and_serve must gate the CID read surface via visibility_check (KTD2a)"
);
assert!(
gate_body.contains("if quarantined"),
"gate_and_serve must hard-drop a quarantined repo before the visibility gate (INV-11)"
);
for (src, func, marker) in rows {
let body = fn_body(src, func);
assert!(
@@ -324,6 +348,118 @@ mod authz_guard {
})
}
/// Collect `.rs` source files under `dir`. Recursive so the completeness scan
/// covers nested API modules (`api/<module>/mod.rs` and deeper), not only the
/// immediate `api/*.rs` children.
fn collect_rs_files(dir: &std::path::Path) -> Vec<std::path::PathBuf> {
let mut out = Vec::new();
for entry in std::fs::read_dir(dir).expect("read api dir") {
let path = entry.expect("dir entry").path();
if path.is_dir() {
out.extend(collect_rs_files(&path));
} else if path.extension().is_some_and(|e| e == "rs") {
out.push(path);
}
}
out
}
/// The `.rs` files under `api_root` (as paths RELATIVE to it) that the
/// completeness scan must inspect: everything except the top-level `mod.rs` (the
/// guard file itself) and the top-level files already covered by the per-handler
/// `sources` loop. A nested `api/<module>/<name>.rs` is a distinct source file
/// even when its basename matches a listed top-level file, so it stays in scope.
fn unlisted_source_files(
api_root: &std::path::Path,
listed: &std::collections::HashSet<&str>,
) -> Vec<String> {
collect_rs_files(api_root)
.iter()
.filter_map(|path| {
let rel = path
.strip_prefix(api_root)
.ok()?
.to_string_lossy()
.replace('\\', "/");
if rel == "mod.rs" || listed.contains(rel.as_str()) {
None
} else {
Some(rel)
}
})
.collect()
}
#[test]
fn collect_rs_files_recurses_subdirs() {
let tmp = tempfile::TempDir::new().unwrap();
let root = tmp.path();
std::fs::write(root.join("a.rs"), "").unwrap();
std::fs::write(root.join("note.txt"), "").unwrap();
std::fs::create_dir_all(root.join("sub/deep")).unwrap();
std::fs::write(root.join("sub/mod.rs"), "").unwrap();
std::fs::write(root.join("sub/deep/c.rs"), "").unwrap();
let names: std::collections::HashSet<String> = collect_rs_files(root)
.iter()
.map(|p| {
p.strip_prefix(root)
.unwrap()
.to_string_lossy()
.replace('\\', "/")
})
.collect();
assert!(names.contains("a.rs"));
assert!(
names.contains("sub/mod.rs"),
"nested module file must be collected"
);
assert!(
names.contains("sub/deep/c.rs"),
"deeply nested file must be collected"
);
assert!(
!names.iter().any(|n| n.ends_with(".txt")),
"non-rs files excluded"
);
assert_eq!(names.len(), 3);
}
// P3 (#119): the completeness scan must skip already-covered files by their path
// RELATIVE to api_root, not by basename. A nested api/<module>/repos.rs is a
// distinct source file from the covered top-level repos.rs and must still be
// scanned, or a new nested module could smuggle in an ungated repo-scoped handler
// behind a colliding filename.
#[test]
fn unlisted_source_files_scans_nested_file_with_colliding_basename() {
let tmp = tempfile::TempDir::new().unwrap();
let root = tmp.path();
std::fs::write(root.join("repos.rs"), "").unwrap();
std::fs::write(root.join("mod.rs"), "").unwrap();
std::fs::create_dir_all(root.join("sub")).unwrap();
std::fs::write(root.join("sub/repos.rs"), "").unwrap();
std::fs::write(root.join("sub/fresh.rs"), "").unwrap();
let listed: std::collections::HashSet<&str> = ["repos.rs"].into_iter().collect();
let unlisted = unlisted_source_files(root, &listed);
assert!(
unlisted.contains(&"sub/repos.rs".to_string()),
"a nested file whose basename matches a listed top-level file must still be scanned"
);
assert!(
unlisted.contains(&"sub/fresh.rs".to_string()),
"a nested file with a unique name must be scanned"
);
assert!(
!unlisted.contains(&"repos.rs".to_string()),
"the listed top-level file is covered by the per-handler loop"
);
assert!(
!unlisted.contains(&"mod.rs".to_string()),
"the top-level guard file is skipped"
);
}
/// Egress gate guard: every repo-scoped handler (`Path<(String, String..)>`)
/// must carry an authz marker — a read gate (`authorize_repo_read` /
/// `visibility_check`), or a write gate (`require_repo_owner` / `require_owner`
@@ -376,19 +512,15 @@ mod authz_guard {
"read-guard `sources` lists {f} but the file does not exist"
);
}
for entry in std::fs::read_dir(api_dir).expect("read src/api") {
let path = entry.expect("dir entry").path();
let fname = path.file_name().unwrap().to_string_lossy().into_owned();
if !fname.ends_with(".rs") || fname == "mod.rs" || listed.contains(fname.as_str()) {
continue;
}
let src = std::fs::read_to_string(&path).expect("read api file");
let api_root = std::path::Path::new(api_dir);
for rel in unlisted_source_files(api_root, &listed) {
let src = std::fs::read_to_string(api_root.join(&rel)).expect("read api file");
let has_repo_handler = handler_names(&src)
.iter()
.any(|n| is_repo_scoped(&fn_body(&src, n)));
assert!(
!has_repo_handler,
"api/{fname} declares a repo-scoped handler but is not in the egress \
"api/{rel} declares a repo-scoped handler but is not in the egress \
guard `sources` list — add it so its handlers are gate-checked"
);
}
@@ -396,18 +528,6 @@ mod authz_guard {
// Repo-scoped reads known to be ungated today, each tracked by an issue.
// Remove an entry the moment its gate lands (the staleness assert enforces it).
let known_ungated: &[(&str, &str)] = &[
// info/refs gates only git-upload-pack today; git-receive-pack
// advertisement is ungated until #119 makes the gate unconditional.
("git_info_refs", "#119"),
("list_certs", "#120"),
("get_cert", "#120"),
("list_issues", "#120"),
("get_issue", "#120"),
("list_issue_comments", "#120"),
("list_labels", "#120"),
("list_repo_bounties", "#120"),
("get_star_status", "#120"),
("list_repo_events", "#94 (PR #113)"),
("list_webhooks", "#94 (PR #113)"),
("list_replicas", "PR #113"),
("list_protected_branches", "PR #113"),
+456 -2
View File
@@ -296,8 +296,8 @@ pub async fn trigger_sync(State(state): State<AppState>) -> Result<Json<serde_js
repo.get("name").and_then(|v| v.as_str()),
) {
(Some(owner), Some(name)) => {
// Use short owner (last colon segment) matching DB convention
let short = owner.split(':').next_back().unwrap_or(owner);
// Use short owner (did:key-only normalization) matching DB convention
let short = crate::db::normalize_owner_key(owner);
format!("{short}/{name}")
}
_ => continue,
@@ -537,4 +537,458 @@ mod tests {
// pins the `o[0] == 0` check against an off-by-one into 1.x.
assert!(is_public_http_url("http://1.0.0.0/"));
}
// ── #82: /sync/trigger signature gate + per-IP brake on the peer-sync routes ──
//
// These drive the FULL production router (crate::server::build_router) so the
// route wiring, layer order, and config-mode branching are all under test.
// The positive-path and DID-agnostic cases mount the handler directly because
// signed_request_as injects only an AuthenticatedDid extension (not a real
// RFC-9421 signature), which require_signature rejects on the full router.
use crate::rate_limit::{IpRateLimiter, RateLimiter, TrustedProxy};
use crate::state::AppState;
use crate::test_support::{signed_request_as, test_state};
use axum::body::Body;
use axum::extract::ConnectInfo;
use axum::http::{header, Method, Request, StatusCode};
use axum::routing::post;
use axum::{middleware, Extension, Router};
use gitlawb_core::http_sig::sign_request;
use gitlawb_core::identity::Keypair;
use sqlx::PgPool;
use std::net::SocketAddr;
use std::sync::Arc;
use std::time::Duration;
use tower::ServiceExt;
fn unsigned_post(uri: &str, body: &str, peer: &str) -> Request<Body> {
let mut req = Request::builder()
.method(Method::POST)
.uri(uri)
.header(header::CONTENT_TYPE, "application/json")
.body(Body::from(body.to_string()))
.unwrap();
req.extensions_mut()
.insert(ConnectInfo(peer.parse::<SocketAddr>().unwrap()));
req
}
fn require_signed_peer_writes(state: &mut AppState) {
let mut cfg = (*state.config).clone();
cfg.require_signed_peer_writes = true;
state.config = Arc::new(cfg);
}
const NOTIFY_BODY: &str = r#"{"repo":"demo","ref_name":"refs/heads/main","new_sha":"0000000000000000000000000000000000000000","node_did":"PEER_DID"}"#;
// ── trigger: mandatory signature ──────────────────────────────────────────
#[sqlx::test]
async fn sync_trigger_rejects_unsigned_in_default_mode(pool: PgPool) {
// The hole: with require_signed_peer_writes=false (default), an anonymous
// caller reaches the fan-out. Must be 401 regardless of the flag.
let state = test_state(pool).await;
let router = crate::server::build_router(state);
let resp = router
.oneshot(unsigned_post(
"/api/v1/sync/trigger",
"{}",
"203.0.113.1:5000",
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
}
#[sqlx::test]
async fn sync_trigger_rejects_unsigned_in_signed_writes_mode(pool: PgPool) {
// The signature requirement must not depend on the flag.
let mut state = test_state(pool).await;
require_signed_peer_writes(&mut state);
let router = crate::server::build_router(state);
let resp = router
.oneshot(unsigned_post(
"/api/v1/sync/trigger",
"{}",
"203.0.113.2:5000",
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
}
#[sqlx::test]
async fn sync_trigger_admits_signed_caller(pool: PgPool) {
// must-not-over-throttle: a signed caller reaches the handler and (with no
// peers seeded) gets 200, not 429. Direct-mount because a real signature
// cannot be forged through the full router in tests.
let state = test_state(pool).await;
let did = Keypair::generate().did().to_string();
let app = Router::new()
.route("/api/v1/sync/trigger", post(super::trigger_sync))
.layer(middleware::from_fn(crate::rate_limit::rate_limit_by_ip))
.layer(Extension(IpRateLimiter {
limiter: RateLimiter::new(60, Duration::from_secs(3600)),
trust: TrustedProxy::None,
}))
.with_state(state);
let mut req =
signed_request_as(&did, Method::POST, "/api/v1/sync/trigger", Body::from("{}"));
req.extensions_mut().insert(ConnectInfo(
"203.0.113.3:5000".parse::<SocketAddr>().unwrap(),
));
let resp = app.oneshot(req).await.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
}
// ── trigger: IP brake ─────────────────────────────────────────────────────
#[sqlx::test]
async fn sync_trigger_ip_flood_is_throttled(pool: PgPool) {
// The brake is outermost, so an over-limit request 429s before it ever
// reaches require_signature — an unsigned request suffices to prove it.
let mut state = test_state(pool).await;
state.sync_trigger_rate_limiter = RateLimiter::new(1, Duration::from_secs(3600));
state.push_limiter_trust = TrustedProxy::None;
let peer = "203.0.113.4:5000";
// Exhaust the single-request budget up front.
assert!(
state
.sync_trigger_rate_limiter
.check(&peer.parse::<SocketAddr>().unwrap().ip().to_string())
.await
);
let router = crate::server::build_router(state);
let resp = router
.oneshot(unsigned_post("/api/v1/sync/trigger", "{}", peer))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::TOO_MANY_REQUESTS);
}
#[sqlx::test]
async fn sync_trigger_brake_is_did_agnostic(pool: PgPool) {
// Why per-IP, not per-DID: a DID farm (fresh did:key per request) does not
// bypass an IP-keyed brake. Two DISTINCT DIDs from one IP, brake limit 1 →
// the second is 429. A per-DID limiter would give each DID its own bucket.
let state = test_state(pool).await;
let app = Router::new()
.route("/api/v1/sync/trigger", post(super::trigger_sync))
.layer(middleware::from_fn(crate::rate_limit::rate_limit_by_ip))
.layer(Extension(IpRateLimiter {
limiter: RateLimiter::new(1, Duration::from_secs(3600)),
trust: TrustedProxy::None,
}))
.with_state(state);
let peer = "203.0.113.5:5000".parse::<SocketAddr>().unwrap();
let mk = |did: &str| {
let mut r =
signed_request_as(did, Method::POST, "/api/v1/sync/trigger", Body::from("{}"));
r.extensions_mut().insert(ConnectInfo(peer));
r
};
let first = app
.clone()
.oneshot(mk(&Keypair::generate().did().to_string()))
.await
.unwrap();
assert_eq!(first.status(), StatusCode::OK);
let second = app
.oneshot(mk(&Keypair::generate().did().to_string()))
.await
.unwrap();
assert_eq!(second.status(), StatusCode::TOO_MANY_REQUESTS);
}
#[sqlx::test]
async fn sync_trigger_forged_forwarded_header_cannot_bypass(pool: PgPool) {
// TrustedProxy::None keys on the socket peer, so rotating X-Forwarded-For
// from one socket peer does not refill the bucket. limit 1: first request
// consumes it (then 401 unsigned), second from the same socket → 429
// regardless of a different XFF value.
let mut state = test_state(pool).await;
state.sync_trigger_rate_limiter = RateLimiter::new(1, Duration::from_secs(3600));
state.push_limiter_trust = TrustedProxy::None;
let router = crate::server::build_router(state);
let mut a = unsigned_post("/api/v1/sync/trigger", "{}", "203.0.113.6:5000");
a.headers_mut()
.insert("x-forwarded-for", "1.1.1.1".parse().unwrap());
assert_eq!(
router.clone().oneshot(a).await.unwrap().status(),
StatusCode::UNAUTHORIZED
);
let mut b = unsigned_post("/api/v1/sync/trigger", "{}", "203.0.113.6:5000");
b.headers_mut()
.insert("x-forwarded-for", "2.2.2.2".parse().unwrap());
assert_eq!(
router.oneshot(b).await.unwrap().status(),
StatusCode::TOO_MANY_REQUESTS
);
}
#[sqlx::test]
async fn sync_trigger_rate_limit_zero_disables_but_signature_holds(pool: PgPool) {
// 0 disables the brake (RateLimiter::check early-returns), proving the two
// halves are independent: no 429 even under a flood, but the signature gate
// still 401s every unsigned request.
let mut state = test_state(pool).await;
state.sync_trigger_rate_limiter = RateLimiter::new(0, Duration::from_secs(3600));
state.push_limiter_trust = TrustedProxy::None;
let router = crate::server::build_router(state);
for _ in 0..3 {
let resp = router
.clone()
.oneshot(unsigned_post(
"/api/v1/sync/trigger",
"{}",
"203.0.113.7:5000",
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
}
}
// ── notify: braked, but signature behavior unchanged ──────────────────────
async fn seed_peer(state: &AppState) -> String {
let did = Keypair::generate().did().to_string();
state
.db
.upsert_peer(&did, "https://peer.example.com")
.await
.unwrap();
did
}
#[sqlx::test]
async fn sync_notify_unsigned_flood_is_throttled(pool: PgPool) {
// notify still accepts an unsigned known-peer notification (rolling-upgrade
// compat), but a flood from one IP is now braked. brake limit 1: first
// enqueues (200), second from the same IP → 429.
let mut state = test_state(pool).await;
state.peer_write_rate_limiter = RateLimiter::new(1, Duration::from_secs(3600));
state.push_limiter_trust = TrustedProxy::None;
let peer_did = seed_peer(&state).await;
let body = NOTIFY_BODY.replace("PEER_DID", &peer_did);
let router = crate::server::build_router(state);
let first = router
.clone()
.oneshot(unsigned_post(
"/api/v1/sync/notify",
&body,
"203.0.113.8:5000",
))
.await
.unwrap();
assert_eq!(first.status(), StatusCode::OK);
let second = router
.oneshot(unsigned_post(
"/api/v1/sync/notify",
&body,
"203.0.113.8:5000",
))
.await
.unwrap();
assert_eq!(second.status(), StatusCode::TOO_MANY_REQUESTS);
}
#[sqlx::test]
async fn sync_notify_distinct_ips_are_not_collaterally_throttled(pool: PgPool) {
// must-not-over-throttle: per-IP keying means one peer's volume does not
// throttle another. brake limit 1, two DIFFERENT source IPs → both 200.
let mut state = test_state(pool).await;
state.peer_write_rate_limiter = RateLimiter::new(1, Duration::from_secs(3600));
state.push_limiter_trust = TrustedProxy::None;
let peer_did = seed_peer(&state).await;
let body = NOTIFY_BODY.replace("PEER_DID", &peer_did);
let router = crate::server::build_router(state);
for ip in ["203.0.113.20:5000", "203.0.113.21:5000"] {
let resp = router
.clone()
.oneshot(unsigned_post("/api/v1/sync/notify", &body, ip))
.await
.unwrap();
assert_eq!(
resp.status(),
StatusCode::OK,
"distinct IP {ip} must not be throttled"
);
}
}
#[sqlx::test]
async fn notify_flood_does_not_exhaust_trigger_bucket(pool: PgPool) {
// Separate buckets: draining peer_write via an unsigned notify flood must
// NOT throttle the signed trigger caller from the same IP. After the notify
// bucket is spent, an unsigned trigger from that IP hits its own (unspent)
// bucket and is rejected by the SIGNATURE gate (401), not the brake (429).
let mut state = test_state(pool).await;
state.peer_write_rate_limiter = RateLimiter::new(1, Duration::from_secs(3600));
state.sync_trigger_rate_limiter = RateLimiter::new(60, Duration::from_secs(3600));
state.push_limiter_trust = TrustedProxy::None;
let peer_did = seed_peer(&state).await;
let body = NOTIFY_BODY.replace("PEER_DID", &peer_did);
let router = crate::server::build_router(state);
let ip = "203.0.113.9:5000";
assert_eq!(
router
.clone()
.oneshot(unsigned_post("/api/v1/sync/notify", &body, ip))
.await
.unwrap()
.status(),
StatusCode::OK
);
assert_eq!(
router
.clone()
.oneshot(unsigned_post("/api/v1/sync/notify", &body, ip))
.await
.unwrap()
.status(),
StatusCode::TOO_MANY_REQUESTS,
"peer_write bucket should now be exhausted for this IP"
);
let trigger = router
.oneshot(unsigned_post("/api/v1/sync/trigger", "{}", ip))
.await
.unwrap();
assert_eq!(
trigger.status(),
StatusCode::UNAUTHORIZED,
"trigger must hit its own bucket (401 from the sig gate), not the drained peer_write bucket (429)"
);
}
#[sqlx::test]
async fn announce_still_accepts_unsigned_in_default_mode(pool: PgPool) {
// must-not-over-reach: adding the brake must not tighten announce's
// rolling-upgrade behavior — an unsigned announce with a public URL still
// succeeds (it only gains the brake, which a single request is under).
let state = test_state(pool).await;
let did = Keypair::generate().did().to_string();
let body = format!(r#"{{"did":"{did}","http_url":"https://peer.example.com"}}"#);
let router = crate::server::build_router(state);
let resp = router
.oneshot(unsigned_post(
"/api/v1/peers/announce",
&body,
"203.0.113.10:5000",
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
}
// Build a request carrying a REAL RFC 9421 signature (not just an injected
// extension), so it passes require_signature on the full production router.
fn real_signed_trigger(peer: &str) -> Request<Body> {
let kp = Keypair::generate();
let body = b"{}";
let s = sign_request(&kp, "POST", "/api/v1/sync/trigger", body);
let mut req = Request::builder()
.method(Method::POST)
.uri("/api/v1/sync/trigger")
.header(header::CONTENT_TYPE, "application/json")
.header("content-digest", s.content_digest)
.header("signature-input", s.signature_input)
.header("signature", s.signature)
.body(Body::from(body.to_vec()))
.unwrap();
req.extensions_mut()
.insert(ConnectInfo(peer.parse::<SocketAddr>().unwrap()));
req
}
#[sqlx::test]
async fn sync_trigger_admits_real_signature_through_full_router(pool: PgPool) {
// The positive path through the REAL gate: a validly-signed trigger passes
// require_signature on the full router and reaches the handler (no peers
// seeded → 200), in BOTH config modes.
let state = test_state(pool.clone()).await;
let resp = crate::server::build_router(state)
.oneshot(real_signed_trigger("203.0.113.30:5000"))
.await
.unwrap();
assert_eq!(
resp.status(),
StatusCode::OK,
"a real signature must reach the handler (default mode)"
);
let mut state = test_state(pool).await;
require_signed_peer_writes(&mut state);
let resp = crate::server::build_router(state)
.oneshot(real_signed_trigger("203.0.113.30:5001"))
.await
.unwrap();
assert_eq!(
resp.status(),
StatusCode::OK,
"a real signature must reach the handler (signed-writes mode)"
);
}
#[sqlx::test]
async fn announce_flood_is_throttled(pool: PgPool) {
// The peer_write brake covers /peers/announce too (co-benefit): limit 1,
// first unsigned announce from an IP is 200, the second from that IP 429.
let mut state = test_state(pool).await;
state.peer_write_rate_limiter = RateLimiter::new(1, Duration::from_secs(3600));
state.push_limiter_trust = TrustedProxy::None;
let did = Keypair::generate().did().to_string();
let body = format!(r#"{{"did":"{did}","http_url":"https://peer.example.com"}}"#);
let router = crate::server::build_router(state);
let ip = "203.0.113.31:5000";
assert_eq!(
router
.clone()
.oneshot(unsigned_post("/api/v1/peers/announce", &body, ip))
.await
.unwrap()
.status(),
StatusCode::OK
);
assert_eq!(
router
.oneshot(unsigned_post("/api/v1/peers/announce", &body, ip))
.await
.unwrap()
.status(),
StatusCode::TOO_MANY_REQUESTS,
"announce must be braked by the peer_write limiter"
);
}
#[sqlx::test]
async fn sync_trigger_429_body_is_generic_not_push(pool: PgPool) {
// The shared 429 middleware now serves the sync routes, so its body must
// not claim "push" (RED before the message was genericized).
let mut state = test_state(pool).await;
state.sync_trigger_rate_limiter = RateLimiter::new(1, Duration::from_secs(3600));
state.push_limiter_trust = TrustedProxy::None;
let peer = "203.0.113.32:5000";
assert!(
state
.sync_trigger_rate_limiter
.check(&peer.parse::<SocketAddr>().unwrap().ip().to_string())
.await
);
let router = crate::server::build_router(state);
let resp = router
.oneshot(unsigned_post("/api/v1/sync/trigger", "{}", peer))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::TOO_MANY_REQUESTS);
let bytes = axum::body::to_bytes(resp.into_body(), usize::MAX)
.await
.unwrap();
let s = String::from_utf8_lossy(&bytes);
assert!(s.contains("rate limit exceeded"), "429 body: {s:?}");
assert!(
!s.contains("push"),
"a 429 on a sync route must not say 'push': {s:?}"
);
}
}
+1 -1
View File
@@ -98,7 +98,7 @@ pub async fn set_profile(
&state.http_client,
&state.config.pinata_upload_url,
&state.config.pinata_jwt,
&format!("profile-{}", &did),
&format!("profile-{}", did),
&profile_json,
)
.await
+117 -1
View File
@@ -209,11 +209,14 @@ pub async fn merge_pr(
return Err(AppError::BadRequest(format!("PR is already {}", pr.status)));
}
// Shed 503 + Retry-After on an exhausted write-lock POOL instead of a generic
// git 500 (#173 F1). Merging holds no admission permit, so it reaches the pool
// unthrottled; reuse the push handler's mapping so the two cannot drift.
let guard = state
.repo_store
.acquire_write(&record.owner_did, &record.name)
.await
.map_err(|e| AppError::Git(e.to_string()))?;
.map_err(|e| crate::api::repos::acquire_write_app_error(&e, &name))?;
let disk_path = guard.path().to_path_buf();
let merger_did = auth.0;
let merge_result = store::merge_branch(
@@ -424,3 +427,116 @@ pub async fn list_comments(
let comments = state.db.list_pr_comments(&pr.id).await?;
Ok(Json(serde_json::json!({ "comments": comments })))
}
/// #173 F1 follow-up: `merge_pr` reaches `acquire_write` holding NO admission permit
/// (unlike the push handler, which is capped by the git-push semaphore), so it is one
/// of the callers most likely to meet an exhausted write-lock POOL under load. An
/// exhausted pool is a capacity signal, so the merge must shed 503 + Retry-After
/// (`AppError::Overloaded`) the way the push handler does, not report the generic
/// 500 git error that says nothing about retrying.
#[cfg(test)]
mod lock_pool_shed_tests {
use super::*;
use axum::response::IntoResponse;
use sqlx::PgPool;
fn seed_repo(owner_did: &str, name: &str) -> crate::db::RepoRecord {
let now = Utc::now();
crate::db::RepoRecord {
id: Uuid::new_v4().to_string(),
name: name.to_string(),
owner_did: owner_did.to_string(),
description: None,
is_public: true,
default_branch: "main".to_string(),
created_at: now,
updated_at: now,
disk_path: format!("/tmp/{name}"),
forked_from: None,
machine_id: None,
}
}
/// RED-before/GREEN-after for `merge_pr`. Both directions: the shed while the only
/// lock-pool connection is held by a guard on a DIFFERENT repo (so this is pool
/// capacity, not advisory-lock contention on this repo), and the must-not case
/// once that connection is back.
#[sqlx::test]
async fn merge_pr_lock_pool_exhaustion_sheds_503_not_500(pool: PgPool) {
let owner = "did:key:zMERGELOCKPOOLOWNERAAAAAAAAAAAAAAAAAAAAA";
let mut state = crate::test_support::test_state(pool.clone()).await;
// One lock-pool connection with a short checkout timeout, so a single held
// guard exhausts it promptly rather than at the pool default.
state.repo_store = crate::git::repo_store::RepoStore::new(
std::path::PathBuf::from("/tmp/gitlawb-pulls-lockpool"),
None,
crate::git::repo_store::build_lock_pool(&pool, 1, std::time::Duration::from_secs(1)),
);
let repo = seed_repo(owner, "lp-merge");
let repo_id = repo.id.clone();
state.db.create_repo(&repo).await.expect("seed repo");
let now = Utc::now().to_rfc3339();
state
.db
.create_pr(&PullRequest {
id: Uuid::new_v4().to_string(),
repo_id: repo_id.clone(),
number: 1,
title: "lp".to_string(),
body: None,
author_did: owner.to_string(),
source_branch: "feature".to_string(),
target_branch: "main".to_string(),
status: "open".to_string(),
merged_by_did: None,
merged_at: None,
created_at: now.clone(),
updated_at: now,
})
.await
.expect("seed open PR");
let held = state
.repo_store
.acquire_write(owner, "other-repo")
.await
.expect("the first write takes the only lock-pool connection");
let shed = merge_pr(
State(state.clone()),
Extension(AuthenticatedDid(owner.to_string())),
Path((owner.to_string(), "lp-merge".to_string(), 1)),
)
.await;
let err = shed.expect_err("an exhausted lock pool must fail the call");
let resp = err.into_response();
assert_eq!(
resp.status(),
StatusCode::SERVICE_UNAVAILABLE,
"merge_pr: an exhausted write-lock pool must shed 503, not a 500 git error"
);
assert_eq!(
resp.headers()
.get("retry-after")
.map(|v| v.to_str().unwrap()),
Some("1"),
"merge_pr: a capacity shed must tell the client when to retry"
);
// MUST-NOT: with the pool free again the merge is not shed as capacity (it
// fails later on the nonexistent on-disk repo, which is a git 500).
held.release(false).await;
let admitted = merge_pr(
State(state.clone()),
Extension(AuthenticatedDid(owner.to_string())),
Path((owner.to_string(), "lp-merge".to_string(), 1)),
)
.await;
assert!(
!matches!(admitted, Err(AppError::Overloaded(_))),
"with the lock pool free, merge_pr must not be shed as capacity; got {:?}",
admitted.err()
);
}
}
File diff suppressed because it is too large. Load diff
+11 -18
View File
@@ -2,14 +2,14 @@
//!
//! Any authenticated agent can star or unstar a repo.
//! One agent = one star per repo (enforced by UNIQUE constraint in repo_stars).
//! Star count is public — no auth required on GET.
//! Star count is gated on repo read access.
use axum::extract::{Extension, Path, State};
use axum::http::StatusCode;
use axum::Json;
use crate::auth::AuthenticatedDid;
use crate::error::{AppError, Result};
use crate::error::Result;
use crate::state::AppState;
/// PUT /api/v1/repos/:owner/:repo/star
@@ -19,11 +19,8 @@ pub async fn star_repo(
Extension(auth): Extension<AuthenticatedDid>,
Path((owner, repo)): Path<(String, String)>,
) -> Result<(StatusCode, Json<serde_json::Value>)> {
let record = state
.db
.get_repo(&owner, &repo)
.await?
.ok_or_else(|| AppError::RepoNotFound(format!("{owner}/{repo}")))?;
let (record, _rules) =
crate::api::authorize_repo_read(&state, &owner, &repo, Some(auth.0.as_str()), "/").await?;
let caller = &auth.0;
let inserted = state.db.star_repo(&record.id, caller).await?;
@@ -54,11 +51,8 @@ pub async fn unstar_repo(
Extension(auth): Extension<AuthenticatedDid>,
Path((owner, repo)): Path<(String, String)>,
) -> Result<Json<serde_json::Value>> {
let record = state
.db
.get_repo(&owner, &repo)
.await?
.ok_or_else(|| AppError::RepoNotFound(format!("{owner}/{repo}")))?;
let (record, _rules) =
crate::api::authorize_repo_read(&state, &owner, &repo, Some(auth.0.as_str()), "/").await?;
let caller = &auth.0;
state.db.unstar_repo(&record.id, caller).await?;
@@ -74,16 +68,15 @@ pub async fn unstar_repo(
}
/// GET /api/v1/repos/:owner/:repo/star
/// Returns star count — unauthenticated.
/// Returns star count for callers who can read the repo.
pub async fn get_star_status(
State(state): State<AppState>,
Path((owner, repo)): Path<(String, String)>,
auth: Option<Extension<AuthenticatedDid>>,
) -> Result<Json<serde_json::Value>> {
let record = state
.db
.get_repo(&owner, &repo)
.await?
.ok_or_else(|| AppError::RepoNotFound(format!("{owner}/{repo}")))?;
let caller = auth.as_ref().map(|e| e.0 .0.as_str());
let (record, _rules) =
crate::api::authorize_repo_read(&state, &owner, &repo, caller, "/").await?;
let count = state.db.count_stars(&record.id).await?;
+23
View File
@@ -514,7 +514,30 @@ mod tests {
machine_id: None,
repo_store: crate::git::repo_store::RepoStore::for_testing(PathBuf::from("/tmp"), pool),
rate_limiter: RateLimiter::new(100, Duration::from_secs(60)),
create_ip_rate_limiter: RateLimiter::new(1000, Duration::from_secs(3600)),
push_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
ipfs_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
ipfs_work_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
ipfs_max_history_walks: crate::api::ipfs::MAX_HISTORY_WALKS_PER_REQUEST,
ipfs_max_legacy_probes: crate::api::ipfs::MAX_LEGACY_PROBES_PER_REQUEST,
ipfs_max_served_object_bytes: crate::api::ipfs::MAX_SERVED_OBJECT_BYTES,
push_limiter_trust: crate::rate_limit::TrustedProxy::None,
sync_trigger_rate_limiter: RateLimiter::new(60, Duration::from_secs(3600)),
peer_write_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
shutdown_tx: tokio::sync::watch::channel(false).0,
git_read_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_write_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_push_advert_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_encrypt_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
encrypt_inflight: crate::state::EncryptInflight::new(),
git_read_per_caller: crate::rate_limit::PerCallerConcurrency::with_default_max_keys(16),
git_push_advert_per_caller:
crate::rate_limit::PerCallerConcurrency::with_default_max_keys(8),
git_write_per_caller: crate::rate_limit::PerCallerConcurrency::with_default_max_keys(8),
git_ipfs_walk_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_ipfs_walk_per_caller:
crate::rate_limit::PerCallerConcurrency::with_default_max_keys(16),
git_bin: "git".to_string(),
}
}
+3 -2
View File
@@ -52,6 +52,7 @@ pub async fn issue_ref_certificate(
issued_at,
};
state.db.insert_ref_certificate(&cert).await?;
Ok(cert)
// Persist and return the row as it exists in the database (on a
// conflict the existing row survives when it is newer).
state.db.insert_ref_certificate(&cert).await
}
+603
View File
@@ -140,6 +140,20 @@ pub struct Config {
#[arg(long, env = "GITLAWB_MAX_PACK_BYTES", default_value_t = 2_147_483_648)]
pub max_pack_bytes: usize,
/// Per-client-IP rate limit for `POST /api/v1/sync/trigger`, in requests per
/// hour. `/sync/trigger` requires a signature and drives an O(peers) outbound
/// fan-out per call, so it gets a tight bucket. `0` disables. Default: 60.
#[arg(long, env = "GITLAWB_SYNC_TRIGGER_RATE_LIMIT", default_value_t = 60)]
pub sync_trigger_rate_limit: usize,
/// Per-client-IP rate limit for the peer-write routes (`/peers/announce`,
/// `/sync/notify`), in requests per hour. These accept unsigned requests from
/// known peers and run at higher frequency, so the bucket is generous. Keeping
/// it separate from the trigger bucket stops an unsigned notify flood from
/// draining the signed trigger caller's quota. `0` disables. Default: 600.
#[arg(long, env = "GITLAWB_PEER_WRITE_RATE_LIMIT", default_value_t = 600)]
pub peer_write_rate_limit: usize,
/// Optional address to bind a Prometheus `/metrics` exposition endpoint on.
/// Example: `127.0.0.1:9091`. Leave empty (default) to disable.
/// Bind to localhost or a private interface — the metrics endpoint is
@@ -152,6 +166,287 @@ pub struct Config {
/// in flight and exits. Default: 30s.
#[arg(long, env = "GITLAWB_SHUTDOWN_GRACE_SECS", default_value_t = 30)]
pub shutdown_grace_secs: u64,
/// Maximum wall-clock time a single served git operation (upload-pack /
/// receive-pack through `run_git_service`) may run before it is aborted and
/// its process group torn down, in seconds. Bounds a git that neither
/// finishes nor disconnects. Must be positive; set it very large to
/// effectively disable the bound. Default: 600s (10 min), generous for large
/// clones. Also bounds the ref advertisement (`info/refs`) and the withheld-blob
/// pack build (`upload_pack_excluding`'s pack-objects stage), which now share the
/// same timeout + process-group teardown (#174).
#[arg(
long,
env = "GITLAWB_GIT_SERVICE_TIMEOUT_SECS",
default_value_t = 600,
value_parser = clap::value_parser!(u64).range(1..)
)]
pub git_service_timeout_secs: u64,
/// Maximum wall-clock time the storage-acquisition phase of a served git
/// operation may run before the request is shed with a 503, in seconds. This
/// bounds `RepoStore::{acquire,acquire_fresh,acquire_write}` — the Tigris
/// HEAD/GET on a read/advert acquire and the advisory-lock retry loop (incl. a
/// per-iteration `pg_try_advisory_lock` that can block on a hung Postgres pool)
/// on a write acquire. A concurrency permit is taken BEFORE this phase, and
/// `git_service_timeout_secs` only starts once git spawns, so without this the
/// acquire phase is unbounded: a stalled backend pins the permit and drains the
/// pool until every later request 503s. On expiry the permit is released and a
/// bounded 503 + Retry-After is returned (fail-closed). Kept separate from
/// `git_service_timeout_secs` because acquisition and git execution are distinct
/// cost centers — one shared budget would let a slow acquire starve git. Must be
/// positive; set it very large to effectively disable the bound. Default: 30s.
#[arg(
long,
env = "GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS",
default_value_t = 30,
value_parser = clap::value_parser!(u64).range(1..)
)]
pub git_acquire_timeout_secs: u64,
/// Maximum connections in the PostgreSQL pool. This is a cap, not a floor
/// (connections open lazily). Size against the database server's
/// max_connections, remembering admin tooling opens its own pool.
#[arg(
long,
env = "GITLAWB_DB_MAX_CONNECTIONS",
default_value_t = 20,
value_parser = clap::value_parser!(u32).range(1..)
)]
pub db_max_connections: u32,
/// Maximum time a request waits for a pool connection before failing with
/// 503, in seconds. Bounds queueing when the database is slow or down.
#[arg(
long,
env = "GITLAWB_DB_ACQUIRE_TIMEOUT_SECS",
default_value_t = 5,
value_parser = clap::value_parser!(u64).range(1..)
)]
pub db_acquire_timeout_secs: u64,
/// Upper bound on each startup connect-and-migrate attempt, in seconds.
/// Migrations wait on a cross-instance advisory lock, so this must be
/// generous enough for a peer instance to finish migrating; on expiry the
/// attempt is retried (migrations are idempotent).
#[arg(
long,
env = "GITLAWB_DB_CONNECT_TIMEOUT_SECS",
default_value_t = 60,
value_parser = clap::value_parser!(u64).range(1..)
)]
pub db_connect_timeout_secs: u64,
/// First retry delay when the database is unavailable at startup, in
/// seconds. Doubles each attempt up to --db-retry-max-secs.
#[arg(
long,
env = "GITLAWB_DB_RETRY_INITIAL_SECS",
default_value_t = 5,
value_parser = clap::value_parser!(u64).range(1..)
)]
pub db_retry_initial_secs: u64,
/// Ceiling on the startup retry delay, in seconds.
#[arg(
long,
env = "GITLAWB_DB_RETRY_MAX_SECS",
default_value_t = 60,
value_parser = clap::value_parser!(u64).range(1..)
)]
pub db_retry_max_secs: u64,
/// Maximum number of served git operations (upload-pack / receive-pack /
/// info-refs) allowed to run concurrently. Beyond this the node sheds the
/// request with a clean 503 + Retry-After instead of spawning another git
/// subprocess and risking PID/thread exhaustion. Portable backstop: the
/// compose `pids_limit` is not present on Fly, whose connection-concurrency
/// cap is a different axis (500 connections each fan out to git +
/// pack-objects + threads). Size below the process budget with headroom.
///
/// This is the READ pool (`git_read_semaphore`): upload-pack and the UPLOAD-PACK
/// `info/refs` advertisement only. The authenticated push POST draws from a
/// separate write pool (`max_concurrent_git_pushes`) that anonymous reads can
/// never reach, and each read caller is additionally bounded by
/// `max_concurrent_reads_per_caller`, so an anonymous flood cannot shed the actual
/// push nor monopolize reads (#174). The anon-reachable RECEIVE-PACK `info/refs`
/// advertisement draws from its OWN dedicated pool (sized like the write pool but
/// disjoint), so an advertisement flood can never occupy a permit the
/// authenticated push POST needs at admission (#174).
///
/// A permit is held for the whole op. Every git subprocess that STREAMS is
/// duration-bounded and reaps its process group on disconnect: upload-pack,
/// receive-pack, and both info/refs advertisements run under
/// `git_service_timeout_secs` with `process_group(0)` teardown, and the
/// withheld-blob (`upload_pack_excluding`) pack-objects stage plus the push-side
/// candidate-discovery children (`rev-list` / `cat-file`) now run under the same
/// bounded runner with process-group teardown, so a stuck git child no longer
/// holds its slot indefinitely (#174 closed the duration/cancellation gaps this
/// comment previously tracked).
///
/// Default: 128. Must be between 1 and 1_048_576; the ceiling keeps the value
/// well under tokio's `Semaphore` permit limit so an oversized value is a
/// clean CLI error rather than a boot-time panic.
#[arg(
long,
env = "GITLAWB_MAX_CONCURRENT_GIT_OPS",
default_value_t = 128,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub max_concurrent_git_ops: usize,
/// Maximum number of concurrent `git-receive-pack` (push) operations. The
/// authenticated push POST draws from this dedicated pool, separate from
/// `max_concurrent_git_ops` (reads), so a flood of anonymous reads cannot shed an
/// authenticated push at admission (#174). The anon-reachable receive-pack
/// `info/refs` advertisement runs in a SEPARATE pool of the same size (derived
/// from this knob), disjoint from this one, so an advertisement flood cannot
/// occupy a POST's slot either (#174). Beyond this a push sheds a clean 503 +
/// Retry-After.
///
/// Default: 32. Must be between 1 and 1_048_576 (the ceiling keeps the value
/// under tokio's `Semaphore` permit limit so an oversized value is a clean CLI
/// error rather than a boot-time panic).
///
/// CONNECTION BUDGET. A push holds a Postgres connection from the node's separate
/// advisory-lock pool for the whole receive-pack, and that pool is sized from this
/// knob (this value + 8, clamped to 64 in `main.rs`). The node's total ceiling is
/// therefore `db_max_connections` (default 20) + the lock pool (default 40), i.e.
/// 60 by default, and at most `db_max_connections` + 64. Size BOTH against the
/// database server's `max_connections`: `db_max_connections`' own doc predates the
/// lock pool and no longer covers most of the node's connections. The +8 headroom
/// is shared with the three non-push `acquire_write` callers (`api/issues.rs` x2,
/// `api/pulls.rs`). Raising this knob past the clamp does NOT buy more lock-pool
/// connections; pushes beyond it wait briefly and then shed a 503 + Retry-After.
#[arg(
long,
env = "GITLAWB_MAX_CONCURRENT_GIT_PUSHES",
default_value_t = 32,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub max_concurrent_git_pushes: usize,
/// Maximum concurrent read operations (`upload-pack` and the upload-pack
/// `info/refs` advertisement) a single caller may hold at once, so one caller
/// cannot monopolize the `max_concurrent_git_ops` read pool (#174). Callers are
/// keyed on the RESOLVED SOURCE IP, never the DID — a signature does not move a
/// caller off this cap, so an authenticated client cannot mint DIDs to escape it.
/// IMPORTANT: the source-IP key is only as granular as `GITLAWB_TRUSTED_PROXY`.
/// Left unset (the default), a node behind an edge/NAT keys all callers on the
/// edge IP, so this cap collapses to a single global cap rather than per-client.
/// Set `GITLAWB_TRUSTED_PROXY` to key on the real client; a high-fanout caller (a
/// CI fleet behind one NAT) then needs the operator to raise this. Over-cap for a
/// caller sheds a clean 503 + Retry-After.
///
/// Default: 16. Must be between 1 and 1_048_576.
#[arg(
long,
env = "GITLAWB_MAX_CONCURRENT_READS_PER_CALLER",
default_value_t = 16,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub max_concurrent_reads_per_caller: usize,
/// Maximum number of concurrent `GET /ipfs/{cid}` requests that may run their
/// visibility walk at once. The publicly-reachable `/ipfs/{cid}` route runs
/// `allowed_blob_set_for_caller_bounded` in `spawn_blocking` — a full-history
/// git walk (up to `git_service_timeout_secs`) — for each candidate repo. It
/// draws from THIS pool, not any served-git pool: a distinct public cost center
/// on a distinct surface, so sharing a git pool would let anonymous /ipfs
/// traffic shed authenticated git ops (the auth-boundary trap). A permit is
/// held for the whole request (across the repo loop) so it reflects real
/// blocking-thread occupancy, not merely the tokio wait. Beyond this the request
/// sheds a clean 503 + Retry-After. Must be between 1 and 1_048_576; the ceiling
/// keeps the value under tokio's `Semaphore` permit limit so an oversized value
/// is a clean CLI error rather than a boot-time panic. Default: 32.
#[arg(
long,
env = "GITLAWB_MAX_CONCURRENT_IPFS_WALKS",
default_value_t = 32,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub max_concurrent_ipfs_walks: usize,
/// Maximum concurrent `/ipfs/{cid}` walk requests a single source may hold at
/// once, so one source cannot monopolize `max_concurrent_ipfs_walks` (#174).
/// Callers are keyed on the RESOLVED SOURCE IP (`client_key`/`GITLAWB_TRUSTED_PROXY`),
/// never the DID — `/ipfs` accepts any `did:key` via `optional_signature` with no
/// admission step, so keying on the DID would let one host mint disposable DIDs to
/// multiply its budget. A request with no resolvable key (no trusted header, no
/// peer) is bounded by the global pool only, never this sub-cap. Over-cap sheds a
/// clean 503 + Retry-After. Must be between 1 and 1_048_576. Default: 4.
#[arg(
long,
env = "GITLAWB_IPFS_WALK_PER_SOURCE",
default_value_t = 4,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub ipfs_walk_per_source: usize,
/// Per-request ceiling on the number of legacy (NULL-provenance) repos the
/// `/ipfs/{cid}` resolver's scan fallback will PROBE (`acquire` + `git cat-file
/// -t`) before giving up. The provenance path targets one repo; the legacy scan,
/// absent this bound, fans one anonymous request out to O(repos) subprocess spawns
/// and cold-cache fetches for a CID enumerable from the public pins index (#173,
/// INV-10). A truncated scan surfaces as a retryable 503, never a false 404. Wired
/// into `AppState::ipfs_max_legacy_probes` at construction; the history-walk ceiling
/// stays constant (`MAX_HISTORY_WALKS_PER_REQUEST`) and is NOT governed by this knob
/// (a smaller value would falsely 503 a provenanced request). Must be between 1 and
/// 1_048_576. Default: 256.
#[arg(
long,
env = "GITLAWB_IPFS_MAX_REPOS_WALKED",
default_value_t = crate::api::ipfs::MAX_LEGACY_PROBES_PER_REQUEST as usize,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub ipfs_max_repos_walked: usize,
/// Per-client-IP rate limit for `GET /ipfs/{cid}`, in requests per hour. The
/// route is publicly reachable (`optional_signature`) and each request can drive
/// a full-history git walk, so it carries a per-IP flood brake in addition to the
/// concurrency cap above (a rate limit bounds request *rate*, the semaphore
/// bounds concurrent slow holds — different axes). Keyed on the resolved client
/// IP via `GITLAWB_TRUSTED_PROXY`. `0` disables. Default: 600.
///
/// This is the pure once-per-request ROUTE brake. The resolver's internal
/// per-probe/per-walk WORK budget is a SEPARATE bucket whose capacity is DERIVED
/// from this value (`AppState::ipfs_work_budget`), not a knob of its own; `0` here
/// disables that derived bucket too.
#[arg(long, env = "GITLAWB_IPFS_RATE_LIMIT", default_value_t = 600)]
pub ipfs_rate_limit: usize,
/// Rows the legacy provider-CID repair sweep reads per batch (U4, #173).
///
/// The sweep walks every `pinned_cids` row on the node once, repairing rows that
/// releases before this branch keyed on a PROVIDER CID (Kubo dag-pb / Pinata CIDv0)
/// instead of the raw-content resolver key. This bounds one batch, so the sweep can
/// never turn into a single unbounded table scan competing with request traffic.
/// Conservative on purpose: paired with the inter-batch delay below the default is
/// ~64 rows per minute, which finishes a large pin set in hours of idle background
/// work rather than one expensive burst. Must be between 1 and 100_000.
#[arg(
long,
env = "GITLAWB_PIN_REPAIR_SWEEP_BATCH",
default_value_t = 64,
value_parser = clap::builder::RangedU64ValueParser::<i64>::new().range(1..=100_000)
)]
pub pin_repair_sweep_batch: i64,
/// Seconds the legacy provider-CID repair sweep sleeps between batches (U4, #173).
///
/// Each batch costs an indexed range scan plus, for the legacy rows in it, a
/// `git cat-file` per row. The delay is what keeps that off the DB's and the disk's
/// critical path: the sweep is repairing rows that have been unresolvable since the
/// upgrade, so finishing slowly is fine and finishing fast at the cost of live
/// traffic is not. `0` disables the pause (test and one-off operational use only).
/// Must be between 0 and 86_400.
#[arg(
long,
env = "GITLAWB_PIN_REPAIR_SWEEP_DELAY_SECS",
default_value_t = 60,
value_parser = clap::builder::RangedU64ValueParser::<u64>::new().range(0..=86_400)
)]
pub pin_repair_sweep_delay_secs: u64,
}
impl Config {
@@ -169,3 +464,311 @@ impl Config {
PathBuf::from(&self.key_path)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn git_service_timeout_defaults_to_600_and_rejects_zero() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).git_service_timeout_secs,
600
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--git-service-timeout-secs", "30"])
.git_service_timeout_secs,
30
);
// 0 is a footgun (immediate-504 on every request); clap must reject it.
assert!(
Config::try_parse_from(["gitlawb-node", "--git-service-timeout-secs", "0"]).is_err()
);
}
#[test]
fn max_concurrent_git_ops_defaults_and_rejects_out_of_range() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).max_concurrent_git_ops,
128
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-git-ops", "8"])
.max_concurrent_git_ops,
8
);
// 0 permits would shed every served-git request with a 503; clap must reject it.
assert!(Config::try_parse_from(["gitlawb-node", "--max-concurrent-git-ops", "0"]).is_err());
// Above the ceiling would panic tokio's Semaphore::new at boot (permits >
// usize::MAX >> 3); clap must reject it as a clean CLI error instead.
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-git-ops", "1048577"])
.is_err()
);
// The ceiling itself is accepted.
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-git-ops", "1048576"])
.max_concurrent_git_ops,
1_048_576
);
}
#[test]
fn max_concurrent_git_pushes_defaults_and_rejects_out_of_range() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).max_concurrent_git_pushes,
32
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-git-pushes", "8"])
.max_concurrent_git_pushes,
8
);
// 0 permits would shed every push with a 503; clap must reject it.
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-git-pushes", "0"]).is_err()
);
// Above the ceiling would panic tokio's Semaphore::new at boot; clap rejects it.
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-git-pushes", "1048577"])
.is_err()
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-git-pushes", "1048576"])
.max_concurrent_git_pushes,
1_048_576
);
}
#[test]
fn max_concurrent_ipfs_walks_defaults_and_rejects_out_of_range() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).max_concurrent_ipfs_walks,
32
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-ipfs-walks", "4"])
.max_concurrent_ipfs_walks,
4
);
// 0 permits would shed every /ipfs walk with a 503; clap must reject it.
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-ipfs-walks", "0"]).is_err()
);
// Above the ceiling would panic tokio's Semaphore::new at boot; clap rejects it.
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-ipfs-walks", "1048577"])
.is_err()
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-ipfs-walks", "1048576"])
.max_concurrent_ipfs_walks,
1_048_576
);
}
/// U4 (#173): the repair sweep's bounds are conservative by default and a batch of
/// 0 (a sweep that walks nothing and never terminates) is a CLI error, not a
/// runtime hang. The delay does accept 0, for tests and one-off operational runs.
#[test]
fn pin_repair_sweep_knobs_default_conservatively() {
let c = Config::parse_from(["gitlawb-node"]);
assert_eq!(c.pin_repair_sweep_batch, 64);
assert_eq!(c.pin_repair_sweep_delay_secs, 60);
assert!(Config::try_parse_from(["gitlawb-node", "--pin-repair-sweep-batch", "0"]).is_err());
assert!(
Config::try_parse_from(["gitlawb-node", "--pin-repair-sweep-batch", "100001"]).is_err()
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--pin-repair-sweep-batch", "8"])
.pin_repair_sweep_batch,
8
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--pin-repair-sweep-delay-secs", "0"])
.pin_repair_sweep_delay_secs,
0
);
assert!(
Config::try_parse_from(["gitlawb-node", "--pin-repair-sweep-delay-secs", "86401"])
.is_err()
);
}
#[test]
fn ipfs_walk_per_source_defaults_and_rejects_out_of_range() {
assert_eq!(Config::parse_from(["gitlawb-node"]).ipfs_walk_per_source, 4);
assert_eq!(
Config::parse_from(["gitlawb-node", "--ipfs-walk-per-source", "2"])
.ipfs_walk_per_source,
2
);
// 0 would shed every /ipfs walk from a keyed source; clap must reject it.
assert!(Config::try_parse_from(["gitlawb-node", "--ipfs-walk-per-source", "0"]).is_err());
assert!(
Config::try_parse_from(["gitlawb-node", "--ipfs-walk-per-source", "1048577"]).is_err()
);
}
#[test]
fn ipfs_max_repos_walked_defaults_and_rejects_out_of_range() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).ipfs_max_repos_walked,
256
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--ipfs-max-repos-walked", "8"])
.ipfs_max_repos_walked,
8
);
// 0 would walk no repos (serve nothing); clap must reject it.
assert!(Config::try_parse_from(["gitlawb-node", "--ipfs-max-repos-walked", "0"]).is_err());
assert!(
Config::try_parse_from(["gitlawb-node", "--ipfs-max-repos-walked", "1048577"]).is_err()
);
}
/// The `GITLAWB_IPFS_MAX_REPOS_WALKED` knob must actually reach the legacy-probe
/// budget it advertises (R5, KTD5): production seeds `ipfs_max_legacy_probes` from
/// this helper, so the knob is a no-op unless the helper reflects it. RED while the
/// helper returns the hardcoded `MAX_LEGACY_PROBES_PER_REQUEST` (256 regardless of
/// the knob), GREEN once it reads the knob.
#[test]
fn ipfs_max_repos_walked_wires_the_legacy_probe_budget() {
use crate::state::AppState;
// Knob set to 1 → a one-probe legacy budget.
let one = Config::parse_from(["gitlawb-node", "--ipfs-max-repos-walked", "1"]);
assert_eq!(
AppState::ipfs_legacy_probe_budget(&one),
1,
"the knob must control the legacy-probe budget, not be ignored"
);
// Unset knob preserves the shipped 256-probe behaviour.
let default = Config::parse_from(["gitlawb-node"]);
assert_eq!(
AppState::ipfs_legacy_probe_budget(&default),
256,
"the default knob keeps the shipped 256-probe budget"
);
assert_eq!(
AppState::ipfs_legacy_probe_budget(&default),
crate::api::ipfs::MAX_LEGACY_PROBES_PER_REQUEST,
"the default budget equals the constant it replaced"
);
// Ceiling guard: the knob never governs the history-walk ceiling, which must
// stay at MAX_PIN_SOURCES + 1 or a provenanced full source set false-503s.
assert!(
crate::api::ipfs::MAX_HISTORY_WALKS_PER_REQUEST > crate::db::MAX_PIN_SOURCES as u32,
"the history-walk ceiling is independent of the repos-walked knob"
);
}
/// The `/ipfs` work-budget capacity is DERIVED from the route limit (R6, KTD6), with
/// a hard floor of one full legacy search per window (the effective
/// `ipfs_max_legacy_probes`). This guards the derived default so a single
/// default-config deep search never self-throttles mid-scan and recreates the F6
/// admit-then-429 for a legitimate caller. A `RateLimiter` sized to the derived
/// budget must admit the whole probe budget back to back.
#[test]
fn ipfs_work_budget_derives_from_route_limit_and_clears_the_probe_floor() {
use crate::state::AppState;
// Default config: derived work budget = max(route 600, probe budget 256) = 600,
// comfortably above the 256-probe floor.
let default = Config::parse_from(["gitlawb-node"]);
let budget = AppState::ipfs_work_budget(&default);
assert_eq!(budget, 600, "default derives max(route 600, probe 256)");
assert!(
budget >= AppState::ipfs_legacy_probe_budget(&default) as usize,
"the work budget must clear one full legacy search per window"
);
// Tight route limit (1): the floor lifts the work budget to the probe budget
// (256), NOT down to 1 — a single deep search still completes its full scan.
let tight = Config::parse_from(["gitlawb-node", "--ipfs-rate-limit", "1"]);
assert_eq!(
AppState::ipfs_work_budget(&tight),
256,
"a tight route limit is floored at the 256-probe budget, not clamped to 1"
);
// Raised probe budget lifts the floor with it (the work budget tracks the
// effective probe budget, not the constant).
let raised = Config::parse_from([
"gitlawb-node",
"--ipfs-rate-limit",
"10",
"--ipfs-max-repos-walked",
"1000",
]);
assert_eq!(
AppState::ipfs_work_budget(&raised),
1000,
"the floor tracks the operator-raised legacy-probe budget"
);
// 0 route limit disables the derived bucket too (a 0-capacity limiter admits all).
let disabled = Config::parse_from(["gitlawb-node", "--ipfs-rate-limit", "0"]);
assert_eq!(
AppState::ipfs_work_budget(&disabled),
0,
"route limit 0 disables the derived work bucket alongside the route brake"
);
// Behavioral floor: a limiter sized to the derived (tight-route) budget admits
// the whole probe budget back to back for one source, then sheds the next.
let budget = AppState::ipfs_work_budget(&tight);
let limiter =
crate::rate_limit::RateLimiter::new(budget, std::time::Duration::from_secs(3600));
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
rt.block_on(async {
for i in 0..budget {
assert!(
limiter.check("1.2.3.4").await,
"probe {i} of one full default-config scan must be admitted (no mid-scan throttle)"
);
}
assert!(
!limiter.check("1.2.3.4").await,
"the probe past the derived budget is shed"
);
});
}
#[test]
fn max_concurrent_reads_per_caller_defaults_and_rejects_out_of_range() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).max_concurrent_reads_per_caller,
16
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-reads-per-caller", "4"])
.max_concurrent_reads_per_caller,
4
);
// 0 would shed every read from a keyed caller; clap must reject it.
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-reads-per-caller", "0"])
.is_err()
);
assert!(Config::try_parse_from([
"gitlawb-node",
"--max-concurrent-reads-per-caller",
"1048577"
])
.is_err());
assert_eq!(
Config::parse_from([
"gitlawb-node",
"--max-concurrent-reads-per-caller",
"1048576"
])
.max_concurrent_reads_per_caller,
1_048_576
);
}
}
File diff suppressed because it is too large. Load diff
+156 -12
View File
@@ -23,24 +23,104 @@ pub enum AppError {
#[allow(dead_code)]
Forbidden(String),
#[error("icaptcha proof required: {0}")]
IcaptchaProofRequired(String),
#[error("icaptcha proof required: {message}")]
IcaptchaProofRequired {
message: String,
/// iCaptcha service base URL the client should solve against.
url: String,
/// Minimum proof level this node requires.
level: u32,
},
#[error("invalid request: {0}")]
BadRequest(String),
#[error("too many requests: {0}")]
TooManyRequests(String),
#[error("incomplete: {0}")]
Incomplete(String),
#[error("search incomplete: {0}")]
SearchIncomplete(String),
#[error("git error: {0}")]
Git(String),
#[error("git service timed out: {0}")]
Timeout(String),
#[error("server overloaded: {0}")]
Overloaded(String),
#[error("database error: {0}")]
Db(#[from] sqlx::Error),
#[error("internal error: {0}")]
Internal(#[from] anyhow::Error),
Internal(anyhow::Error),
}
/// Shared error code/message for "the database is unreachable", also used by
/// the degraded startup server (main.rs) and the readiness probe (server.rs)
/// so clients see one vocabulary for the condition.
pub const DB_UNAVAILABLE_CODE: &str = "db_unavailable";
pub const DB_UNAVAILABLE_MESSAGE: &str = "database is temporarily unavailable";
/// Connection-level sqlx failures that mean the database is unreachable right
/// now (retryable, 503), as opposed to server-reported query errors.
fn db_unavailable(e: &sqlx::Error) -> bool {
matches!(
e,
sqlx::Error::PoolTimedOut
| sqlx::Error::PoolClosed
| sqlx::Error::Io(_)
| sqlx::Error::Tls(_)
)
}
/// The db layer returns `anyhow::Result`, so sqlx errors reach handlers inside
/// anyhow chains. Downcast them back out so the status mapping below can see
/// them — without this, every database outage surfaces as a 500 instead of a
/// 503. anyhow preserves downcastability through `.context()` layers.
impl From<anyhow::Error> for AppError {
fn from(err: anyhow::Error) -> Self {
match err.downcast::<sqlx::Error>() {
Ok(sql) => AppError::Db(sql),
Err(err) => AppError::Internal(err),
}
}
}
impl IntoResponse for AppError {
fn into_response(self) -> Response {
// iCaptcha challenges carry structured discovery so clients don't have to
// scrape the message: the service URL and required level are returned as
// both JSON fields and `x-icaptcha-url` / `x-icaptcha-level` headers
// (mirroring the header-bearing `human_detected` response in auth/mod.rs).
if let AppError::IcaptchaProofRequired {
message,
url,
level,
} = &self
{
use axum::http::HeaderValue;
let body = Json(json!({
"error": "icaptcha_proof_required",
"message": message,
"icaptcha_url": url,
"required_level": level,
}));
let mut resp = (StatusCode::FORBIDDEN, body).into_response();
let headers = resp.headers_mut();
if let Ok(v) = HeaderValue::from_str(url) {
headers.insert("x-icaptcha-url", v);
}
if let Ok(v) = HeaderValue::from_str(&level.to_string()) {
headers.insert("x-icaptcha-level", v);
}
return resp;
}
let (status, code, message) = match &self {
AppError::RepoNotFound(r) => (
StatusCode::NOT_FOUND,
@@ -55,17 +135,39 @@ impl IntoResponse for AppError {
AppError::NotFound(msg) => (StatusCode::NOT_FOUND, "not_found", msg.clone()),
AppError::Unauthorized(msg) => (StatusCode::UNAUTHORIZED, "not_an_agent", msg.clone()),
AppError::Forbidden(msg) => (StatusCode::FORBIDDEN, "forbidden", msg.clone()),
// 403, not 401: the caller IS an authenticated agent (credentials are
// valid) but is forbidden from this action without a valid, fresh
// iCaptcha proof. The distinct `icaptcha_proof_required` code — which
// clients branch on — keeps it separable from a plain `forbidden`.
AppError::IcaptchaProofRequired(msg) => (
StatusCode::FORBIDDEN,
"icaptcha_proof_required",
// IcaptchaProofRequired is handled above (it carries extra headers/fields).
AppError::IcaptchaProofRequired { .. } => unreachable!("handled before this match"),
AppError::BadRequest(msg) => (StatusCode::BAD_REQUEST, "bad_request", msg.clone()),
AppError::TooManyRequests(msg) => {
(StatusCode::TOO_MANY_REQUESTS, "rate_limited", msg.clone())
}
AppError::Incomplete(msg) => {
(StatusCode::UNPROCESSABLE_ENTITY, "incomplete", msg.clone())
}
// A bounded search that could not complete (the CID resolver hit its
// legacy-probe or walk ceiling), distinct from the 404 that asserts a
// definitive not-found: absence was NOT proven, so the caller should
// retry rather than treat it as gone (#173, F2). 503, retryable.
AppError::SearchIncomplete(msg) => (
StatusCode::SERVICE_UNAVAILABLE,
"search_incomplete",
msg.clone(),
),
AppError::BadRequest(msg) => (StatusCode::BAD_REQUEST, "bad_request", msg.clone()),
AppError::Git(msg) => (StatusCode::INTERNAL_SERVER_ERROR, "git_error", msg.clone()),
// 504, distinct from the 500 git_error and from the read-gate's 404 /
// the auth 401, so the client can tell a deadline from a failure.
AppError::Timeout(msg) => (StatusCode::GATEWAY_TIMEOUT, "git_timeout", msg.clone()),
AppError::Db(e) if db_unavailable(e) => (
StatusCode::SERVICE_UNAVAILABLE,
DB_UNAVAILABLE_CODE,
DB_UNAVAILABLE_MESSAGE.into(),
),
// 503 with a Retry-After (attached after this match — the shared tail
// can't carry per-variant headers). This is the single place Overloaded
// becomes a response, so it can never ship a 503 without the retry hint.
AppError::Overloaded(msg) => {
(StatusCode::SERVICE_UNAVAILABLE, "overloaded", msg.clone())
}
AppError::Db(e) => (StatusCode::INTERNAL_SERVER_ERROR, "db_error", e.to_string()),
AppError::Internal(e) => (
StatusCode::INTERNAL_SERVER_ERROR,
@@ -79,8 +181,50 @@ impl IntoResponse for AppError {
"message": message,
}));
(status, body).into_response()
let mut resp = (status, body).into_response();
// Both retryable 503s advertise when to retry: Overloaded (capacity shed) and
// SearchIncomplete (a bounded CID search cut short by a cap — retry may complete
// it). They ride the shared tail above for body/status, so the header is attached
// here rather than in bespoke early returns, keeping each variant handled once.
if matches!(
self,
AppError::Overloaded(_) | AppError::SearchIncomplete(_)
) {
resp.headers_mut().insert(
axum::http::header::RETRY_AFTER,
axum::http::HeaderValue::from_static("1"),
);
}
resp
}
}
pub type Result<T> = std::result::Result<T, AppError>;
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn timeout_maps_to_504_distinct_from_git_500() {
assert_eq!(
AppError::Timeout("x".into()).into_response().status(),
StatusCode::GATEWAY_TIMEOUT
);
// Guard against a swap with the generic git failure (500).
assert_eq!(
AppError::Git("x".into()).into_response().status(),
StatusCode::INTERNAL_SERVER_ERROR
);
}
#[test]
fn overloaded_maps_to_503_with_retry_after() {
let resp = AppError::Overloaded("x".into()).into_response();
assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE);
assert_eq!(
resp.headers().get("retry-after").unwrap().to_str().unwrap(),
"1"
);
}
}
+238 -96
View File
@@ -29,9 +29,9 @@
use std::collections::HashSet;
use std::path::{Path, PathBuf};
use std::process::Command;
use std::time::{Duration, Instant};
use anyhow::{bail, Context, Result};
use anyhow::Result;
/// Env var that forces the push path to always full-scan, bypassing the delta
/// optimization (KTD7 kill-switch). Reuses the already-tested fallback branch
@@ -64,11 +64,21 @@ pub enum PinCandidates {
/// Fail-closed: any condition where the introduced set cannot be safely
/// determined returns [`PinCandidates::FullScanRequired`] rather than a partial
/// set, so the caller full-scans instead of silently under-pinning.
pub fn resolve_push_delta(repo_path: &Path, new_tips: &[&str], old_tips: &[&str]) -> PinCandidates {
// KTD7 kill-switch: force the (already-tested) full-scan fallback. The env
// read is split out from the pure logic so the resolver stays unit-testable
// without touching process-global state.
resolve_push_delta_inner(repo_path, new_tips, old_tips, force_full_scan())
pub fn resolve_push_delta(
repo_path: &Path,
new_tips: &[&str],
old_tips: &[&str],
git_bin: &str,
deadline: Instant,
) -> PinCandidates {
resolve_push_delta_inner(
repo_path,
new_tips,
old_tips,
force_full_scan(),
git_bin,
deadline,
)
}
/// Whether the force-full-scan kill-switch env var is set.
@@ -83,6 +93,8 @@ fn resolve_push_delta_inner(
new_tips: &[&str],
old_tips: &[&str],
force_full_scan: bool,
git_bin: &str,
deadline: Instant,
) -> PinCandidates {
if force_full_scan {
tracing::debug!("{FORCE_FULL_SCAN_ENV} set — forcing full scan");
@@ -102,7 +114,7 @@ fn resolve_push_delta_inner(
// commit). Bare `cat-file -t` returns `tag` for an annotated tag, and
// `for-each-ref %(*objecttype)` peels only one level — neither is correct.
for tip in new_tips {
match peeled_object_type(repo_path, tip) {
match peeled_object_type(repo_path, tip, git_bin, deadline) {
Some(t) if t == "commit" => {}
other => {
tracing::debug!(
@@ -115,7 +127,7 @@ fn resolve_push_delta_inner(
}
}
match rev_list_delta(repo_path, new_tips, old_tips) {
match rev_list_delta(repo_path, new_tips, old_tips, git_bin, deadline) {
Ok(oids) => PinCandidates::Delta(oids),
Err(e) => {
tracing::debug!(err = %e, "push-delta rev-list failed — forcing full scan");
@@ -126,41 +138,43 @@ fn resolve_push_delta_inner(
/// Return the fully-peeled object type of `sha` (e.g. `commit`, `tree`,
/// `blob`), or `None` if the object is missing/unpeelable or git errored.
fn peeled_object_type(repo_path: &Path, sha: &str) -> Option<String> {
let output = Command::new("git")
.args(["cat-file", "-t", &format!("{sha}^{{}}")])
.current_dir(repo_path)
.output()
.ok()?;
if !output.status.success() {
return None;
}
Some(String::from_utf8_lossy(&output.stdout).trim().to_string())
fn peeled_object_type(
repo_path: &Path,
sha: &str,
git_bin: &str,
deadline: Instant,
) -> Option<String> {
let peel = format!("{sha}^{{}}");
let out = crate::git::visibility_pack::run_bounded_git(
git_bin,
&["cat-file", "-t", &peel],
repo_path,
b"",
deadline,
)
.ok()?;
Some(String::from_utf8_lossy(&out).trim().to_string())
}
/// Run `git rev-list --objects --no-object-names <new> --not <old>` and return
/// the bare OID set. Decides on `status.success()` *before* parsing stdout, so
/// a walk that prints a valid prefix then errors mid-walk is discarded.
fn rev_list_delta(repo_path: &Path, new_tips: &[&str], old_tips: &[&str]) -> Result<Vec<String>> {
fn rev_list_delta(
repo_path: &Path,
new_tips: &[&str],
old_tips: &[&str],
git_bin: &str,
deadline: Instant,
) -> Result<Vec<String>> {
let mut args: Vec<&str> = vec!["rev-list", "--objects", "--no-object-names"];
args.extend_from_slice(new_tips);
if !old_tips.is_empty() {
args.push("--not");
args.extend_from_slice(old_tips);
}
let output = Command::new("git")
.args(&args)
.current_dir(repo_path)
.output()
.context("failed to run git rev-list for push delta")?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
bail!("git rev-list failed: {stderr}");
}
let stdout = String::from_utf8_lossy(&output.stdout);
let out =
crate::git::visibility_pack::run_bounded_git(git_bin, &args, repo_path, b"", deadline)?;
let stdout = String::from_utf8_lossy(&out);
Ok(stdout
.lines()
.map(|l| l.trim().to_string())
@@ -175,23 +189,19 @@ fn rev_list_delta(repo_path: &Path, new_tips: &[&str], old_tips: &[&str]) -> Res
/// reconciliation sweep relies on. It returns *all* objects (including
/// unreachable/dangling ones), which is what the sweep needs to catch
/// stragglers — do not swap it for a reachability walk.
pub fn list_all_objects(repo_path: &Path) -> Result<Vec<String>> {
let output = Command::new("git")
.args([
pub fn list_all_objects(repo_path: &Path, git_bin: &str, deadline: Instant) -> Result<Vec<String>> {
let out = crate::git::visibility_pack::run_bounded_git(
git_bin,
&[
"cat-file",
"--batch-all-objects",
"--batch-check=%(objectname)",
])
.current_dir(repo_path)
.output()
.context("failed to run git cat-file")?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
bail!("git cat-file failed: {stderr}");
}
let stdout = String::from_utf8_lossy(&output.stdout);
],
repo_path,
b"",
deadline,
)?;
let stdout = String::from_utf8_lossy(&out);
Ok(stdout
.lines()
.map(|l| l.trim().to_string())
@@ -203,23 +213,23 @@ pub fn list_all_objects(repo_path: &Path) -> Result<Vec<String>> {
/// `--batch-check='%(objectname) %(objecttype)'`. The pin path's fail-closed
/// filter needs to tell blobs (content, withholdable) from commits/trees
/// (structural, never withheld) without typing the candidate list itself.
pub fn list_all_objects_with_type(repo_path: &Path) -> Result<Vec<(String, String)>> {
let output = Command::new("git")
.args([
pub fn list_all_objects_with_type(
repo_path: &Path,
git_bin: &str,
deadline: Instant,
) -> Result<Vec<(String, String)>> {
let out = crate::git::visibility_pack::run_bounded_git(
git_bin,
&[
"cat-file",
"--batch-all-objects",
"--batch-check=%(objectname) %(objecttype)",
])
.current_dir(repo_path)
.output()
.context("failed to run git cat-file")?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
bail!("git cat-file failed: {stderr}");
}
let stdout = String::from_utf8_lossy(&output.stdout);
],
repo_path,
b"",
deadline,
)?;
let stdout = String::from_utf8_lossy(&out);
Ok(stdout
.lines()
.filter_map(|l| {
@@ -235,8 +245,12 @@ pub fn list_all_objects_with_type(repo_path: &Path) -> Result<Vec<(String, Strin
/// fail-closed pin filter drops any candidate blob absent from the reachable,
/// visibility-allowed set; a dangling private blob is in this set but not the
/// allowed set, so it never replicates (#99).
pub fn all_blob_oids(repo_path: &Path) -> Result<HashSet<String>> {
Ok(list_all_objects_with_type(repo_path)?
pub fn all_blob_oids(
repo_path: &Path,
git_bin: &str,
deadline: Instant,
) -> Result<HashSet<String>> {
Ok(list_all_objects_with_type(repo_path, git_bin, deadline)?
.into_iter()
.filter(|(_, ty)| ty == "blob")
.map(|(oid, _)| oid)
@@ -272,18 +286,22 @@ pub async fn resolve_candidates_for_push(
repo_path: PathBuf,
new_tips: Vec<String>,
old_tips: Vec<String>,
git_bin: String,
timeout: Duration,
) -> PinCandidateSet {
tokio::task::spawn_blocking(move || {
// ONE shared deadline for the whole scan, per jatmn ("the same deadline").
let deadline = Instant::now() + timeout;
let new_refs: Vec<&str> = new_tips.iter().map(String::as_str).collect();
let old_refs: Vec<&str> = old_tips.iter().map(String::as_str).collect();
match resolve_push_delta(&repo_path, &new_refs, &old_refs) {
match resolve_push_delta(&repo_path, &new_refs, &old_refs, &git_bin, deadline) {
PinCandidates::Delta(objs) => {
tracing::info!(delta = objs.len(), repo = %repo_path.display(), "pin candidate set from push delta");
PinCandidateSet { candidates: objs, full_scan: false }
}
PinCandidates::FullScanRequired => {
tracing::warn!(repo = %repo_path.display(), "pin delta unavailable (non-commit tip, git error, or force-full-scan) — full-scan fallback");
match list_all_objects(&repo_path) {
match list_all_objects(&repo_path, &git_bin, deadline) {
Ok(objs) => PinCandidateSet { candidates: objs, full_scan: true },
Err(e) => {
tracing::warn!(repo = %repo_path.display(), err = %e, "full-scan fallback failed; pinning nothing this push (reconciliation sweep backstops)");
@@ -304,8 +322,13 @@ pub async fn resolve_candidates_for_push(
mod tests {
use super::*;
use std::collections::HashSet;
use std::process::Command;
use tempfile::TempDir;
fn td() -> std::time::Instant {
std::time::Instant::now() + std::time::Duration::from_secs(600)
}
/// Minimal git helper for building test repos.
struct Repo {
_td: TempDir,
@@ -363,9 +386,10 @@ mod tests {
let repo = Repo::new();
let c1 = repo.commit_file("a.txt", "one\n");
let c2 = repo.commit_file("b.txt", "two\n");
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&c2], &[&c1]))
.into_iter()
.collect();
let got: HashSet<String> =
delta(resolve_push_delta(&repo.path, &[&c2], &[&c1], "git", td()))
.into_iter()
.collect();
// The new blob b.txt and commit c2 are in the delta; the old blob a.txt
// and commit c1 are not.
let new_blob = repo.rev("HEAD:b.txt");
@@ -383,7 +407,7 @@ mod tests {
// genuinely new objects, never fewer.
let repo = Repo::new();
let c1 = repo.commit_file("a.txt", "one\n");
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&c1], &[]))
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&c1], &[], "git", td()))
.into_iter()
.collect();
assert!(got.contains(&c1));
@@ -399,9 +423,15 @@ mod tests {
// Rewrite history: reset to base, commit a different file.
repo.git(&["reset", "-q", "--hard", &base]);
let new_tip = repo.commit_file("c.txt", "three\n");
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&new_tip], &[&old_tip]))
.into_iter()
.collect();
let got: HashSet<String> = delta(resolve_push_delta(
&repo.path,
&[&new_tip],
&[&old_tip],
"git",
td(),
))
.into_iter()
.collect();
assert!(got.contains(&new_tip), "new tip in delta");
assert!(got.contains(&repo.rev(&format!("{new_tip}:c.txt"))));
// No error; force-push computes new-minus-old cleanly.
@@ -413,7 +443,7 @@ mod tests {
repo.commit_file("a.txt", "one\n");
// All updates were deletions => new_tips empty after the caller strips zeros.
assert_eq!(
resolve_push_delta(&repo.path, &[], &[ZERO]),
resolve_push_delta(&repo.path, &[], &[ZERO], "git", td()),
PinCandidates::Delta(Vec::new())
);
}
@@ -424,7 +454,7 @@ mod tests {
repo.commit_file("a.txt", "one\n");
let blob = repo.rev("HEAD:a.txt");
assert_eq!(
resolve_push_delta(&repo.path, &[&blob], &[]),
resolve_push_delta(&repo.path, &[&blob], &[], "git", td()),
PinCandidates::FullScanRequired,
"a blob tip must force full scan (rev-list would exit 0 and walk it)"
);
@@ -436,7 +466,7 @@ mod tests {
repo.commit_file("a.txt", "one\n");
let tree = repo.rev("HEAD^{tree}");
assert_eq!(
resolve_push_delta(&repo.path, &[&tree], &[]),
resolve_push_delta(&repo.path, &[&tree], &[], "git", td()),
PinCandidates::FullScanRequired
);
}
@@ -449,7 +479,7 @@ mod tests {
repo.git(&["tag", "-a", "treetag", "-m", "x", &tree]);
let tag = repo.rev("treetag");
assert_eq!(
resolve_push_delta(&repo.path, &[&tag], &[]),
resolve_push_delta(&repo.path, &[&tag], &[], "git", td()),
PinCandidates::FullScanRequired,
"annotated tag peeling to a tree must force full scan"
);
@@ -465,7 +495,7 @@ mod tests {
repo.git(&["tag", "-a", "t2", "-m", "x", &t1]);
let t2 = repo.rev("t2");
assert_eq!(
resolve_push_delta(&repo.path, &[&t2], &[]),
resolve_push_delta(&repo.path, &[&t2], &[], "git", td()),
PinCandidates::FullScanRequired
);
}
@@ -480,7 +510,7 @@ mod tests {
let c1 = repo.commit_file("a.txt", "one\n");
repo.git(&["tag", "-a", "rel", "-m", "release", &c1]);
let tag = repo.rev("rel");
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&tag], &[]))
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&tag], &[], "git", td()))
.into_iter()
.collect();
assert!(
@@ -501,7 +531,7 @@ mod tests {
let t1 = repo.rev("t1");
repo.git(&["tag", "-a", "t2", "-m", "x", &t1]);
let t2 = repo.rev("t2");
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&t2], &[]))
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&t2], &[], "git", td()))
.into_iter()
.collect();
assert!(
@@ -516,9 +546,14 @@ mod tests {
let repo = Repo::new();
let c1 = repo.commit_file("a.txt", "one\n");
let c2 = repo.commit_file("b.txt", "two\n");
let set =
resolve_candidates_for_push(repo.path.clone(), vec![c2.clone()], vec![c1.clone()])
.await;
let set = resolve_candidates_for_push(
repo.path.clone(),
vec![c2.clone()],
vec![c1.clone()],
"git".to_string(),
std::time::Duration::from_secs(600),
)
.await;
assert!(!set.full_scan, "happy-path delta is not a full scan");
let got: HashSet<String> = set.candidates.into_iter().collect();
let new_blob = repo.rev("HEAD:b.txt");
@@ -536,8 +571,18 @@ mod tests {
let repo = Repo::new();
repo.commit_file("a.txt", "one\n");
let blob = repo.rev("HEAD:a.txt");
let all: HashSet<String> = list_all_objects(&repo.path).unwrap().into_iter().collect();
let set = resolve_candidates_for_push(repo.path.clone(), vec![blob], vec![]).await;
let all: HashSet<String> = list_all_objects(&repo.path, "git", td())
.unwrap()
.into_iter()
.collect();
let set = resolve_candidates_for_push(
repo.path.clone(),
vec![blob],
vec![],
"git".to_string(),
std::time::Duration::from_secs(600),
)
.await;
assert!(set.full_scan, "non-commit tip is signalled as a full scan");
let got: HashSet<String> = set.candidates.into_iter().collect();
assert_eq!(got, all, "non-commit tip falls back to full repo scan");
@@ -549,7 +594,7 @@ mod tests {
repo.commit_file("a.txt", "one\n");
let bogus = "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef";
assert_eq!(
resolve_push_delta(&repo.path, &[bogus], &[]),
resolve_push_delta(&repo.path, &[bogus], &[], "git", td()),
PinCandidates::FullScanRequired,
"a missing/corrupt tip OID must force full scan"
);
@@ -564,7 +609,7 @@ mod tests {
let c1 = repo.commit_file("a.txt", "one\n");
let c2 = repo.commit_file("b.txt", "two\n");
let old_tree = repo.rev(&format!("{c1}^{{tree}}"));
let result = resolve_push_delta(&repo.path, &[&c2], &[&old_tree]);
let result = resolve_push_delta(&repo.path, &[&c2], &[&old_tree], "git", td());
match result {
PinCandidates::FullScanRequired => {} // safe: caller full-scans
PinCandidates::Delta(objs) => {
@@ -583,10 +628,15 @@ mod tests {
// branch2 from base advances independently
repo.git(&["checkout", "-q", "-b", "branch2", &base]);
let b2 = repo.commit_file("c.txt", "three\n");
let got: HashSet<String> =
delta(resolve_push_delta(&repo.path, &[&b1, &b2], &[&base, &base]))
.into_iter()
.collect();
let got: HashSet<String> = delta(resolve_push_delta(
&repo.path,
&[&b1, &b2],
&[&base, &base],
"git",
td(),
))
.into_iter()
.collect();
assert!(got.contains(&b1), "branch1 new commit");
assert!(got.contains(&b2), "branch2 new commit");
}
@@ -595,7 +645,7 @@ mod tests {
fn empty_repo_no_tips_yields_empty_delta() {
let repo = Repo::new();
assert_eq!(
resolve_push_delta(&repo.path, &[], &[]),
resolve_push_delta(&repo.path, &[], &[], "git", td()),
PinCandidates::Delta(Vec::new())
);
}
@@ -609,12 +659,12 @@ mod tests {
let repo = Repo::new();
let c1 = repo.commit_file("a.txt", "one\n");
assert_eq!(
resolve_push_delta_inner(&repo.path, &[&c1], &[], true),
resolve_push_delta_inner(&repo.path, &[&c1], &[], true, "git", td()),
PinCandidates::FullScanRequired
);
// And with the flag off, the same push yields a Delta.
assert!(matches!(
resolve_push_delta_inner(&repo.path, &[&c1], &[], false),
resolve_push_delta_inner(&repo.path, &[&c1], &[], false, "git", td()),
PinCandidates::Delta(_)
));
}
@@ -624,7 +674,7 @@ mod tests {
let repo = Repo::new();
repo.commit_file("a.txt", "one\n");
repo.commit_file("b.txt", "two\n");
let all = list_all_objects(&repo.path).unwrap();
let all = list_all_objects(&repo.path, "git", td()).unwrap();
// 2 commits + 2 trees + 2 blobs = 6 objects.
assert_eq!(all.len(), 6, "got: {all:?}");
}
@@ -642,7 +692,7 @@ mod tests {
std::fs::write(repo.path.join("orphan.bin"), b"dangling\n").unwrap();
let dangling = repo.git(&["hash-object", "-w", "orphan.bin"]);
let blobs = all_blob_oids(&repo.path).unwrap();
let blobs = all_blob_oids(&repo.path, "git", td()).unwrap();
assert!(blobs.contains(&reachable_blob), "reachable blob present");
assert!(
blobs.contains(&dangling),
@@ -652,7 +702,7 @@ mod tests {
assert!(!blobs.contains(&tree), "tree is not a blob");
// The typed lister tags each object; spot-check the dangling blob's type.
let typed = list_all_objects_with_type(&repo.path).unwrap();
let typed = list_all_objects_with_type(&repo.path, "git", td()).unwrap();
assert!(
typed
.iter()
@@ -660,4 +710,96 @@ mod tests {
"dangling object is typed as a blob"
);
}
#[cfg(unix)]
#[test]
fn list_all_objects_times_out_on_a_hung_git_instead_of_blocking() {
use std::os::unix::fs::PermissionsExt;
use std::time::{Duration, Instant};
let dir = tempfile::TempDir::new().unwrap();
// Fake git: hang on cat-file (bounded to 30s so a broken test can't wedge).
let fake = dir.path().join("fakegit");
std::fs::write(
&fake,
"#!/bin/sh\ncase \"$1\" in\n cat-file) i=0; while [ $i -lt 30 ]; do sleep 1; i=$((i+1)); done ;;\n *) : ;;\nesac\n",
)
.unwrap();
let mut perm = std::fs::metadata(&fake).unwrap().permissions();
perm.set_mode(0o755);
std::fs::set_permissions(&fake, perm).unwrap();
let git_bin = fake.to_str().unwrap().to_string();
let (tx, rx) = std::sync::mpsc::channel();
let path = dir.path().to_path_buf();
std::thread::spawn(move || {
let _ = tx.send(list_all_objects(
&path,
&git_bin,
Instant::now() + Duration::from_millis(150),
));
});
let res = rx.recv_timeout(Duration::from_secs(10)).expect(
"list_all_objects must return within the watchdog budget, not block on a hung git",
);
assert!(
res.is_err(),
"a hung git must make list_all_objects error out, not hang"
);
}
/// #174 finding 2: the DELTA-path children — `peeled_object_type` (cat-file),
/// `rev_list_delta` (rev-list) — and `list_all_objects_with_type` (cat-file) are
/// the remaining candidate-discovery execs (the common push path). Each must
/// return within the watchdog budget on a hung git rather than block and pin the
/// write permit. Revert any one to a bare `Command::output()` and its arm blocks
/// past the recv budget (RED).
#[cfg(unix)]
#[test]
fn delta_path_exec_fns_time_out_on_a_hung_git() {
use std::os::unix::fs::PermissionsExt;
use std::time::{Duration, Instant};
let dir = tempfile::TempDir::new().unwrap();
// Fake git hangs on BOTH cat-file and rev-list (bounded 30s so a broken
// test can't leak a permanent orphan or wedge the suite).
let fake = dir.path().join("fakegit");
std::fs::write(
&fake,
"#!/bin/sh\ncase \"$1\" in\n cat-file|rev-list) i=0; while [ $i -lt 30 ]; do sleep 1; i=$((i+1)); done ;;\n *) : ;;\nesac\n",
)
.unwrap();
let mut perm = std::fs::metadata(&fake).unwrap().permissions();
perm.set_mode(0o755);
std::fs::set_permissions(&fake, perm).unwrap();
let git_bin = fake.to_str().unwrap().to_string();
let path = dir.path().to_path_buf();
// Run `f` on a thread and require it to RETURN (not block) within 10s.
fn returns_within<T: Send + 'static>(f: impl FnOnce() -> T + Send + 'static) -> T {
let (tx, rx) = std::sync::mpsc::channel();
std::thread::spawn(move || {
let _ = tx.send(f());
});
rx.recv_timeout(Duration::from_secs(10)).expect(
"a delta-path exec fn must return within the watchdog budget, not block on a hung git",
)
}
let dl = || Instant::now() + Duration::from_millis(150);
let sha = "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef";
let (p, g, d) = (path.clone(), git_bin.clone(), dl());
assert!(
returns_within(move || peeled_object_type(&p, sha, &g, d)).is_none(),
"peeled_object_type must time out to None on a hung cat-file"
);
let (p, g, d) = (path.clone(), git_bin.clone(), dl());
assert!(
returns_within(move || rev_list_delta(&p, &[sha], &[], &g, d)).is_err(),
"rev_list_delta must error out on a hung rev-list"
);
let (p, g, d) = (path.clone(), git_bin.clone(), dl());
assert!(
returns_within(move || list_all_objects_with_type(&p, &g, d)).is_err(),
"list_all_objects_with_type must error out on a hung cat-file"
);
}
}
+734 -52
View File
@@ -11,9 +11,12 @@
use std::collections::HashSet;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::time::Duration;
use anyhow::{Context, Result};
use sqlx::PgPool;
use sqlx::pool::PoolConnection;
use sqlx::postgres::PgPoolOptions;
use sqlx::{PgPool, Postgres};
use tokio::sync::Mutex;
use tracing::{debug, info, warn};
@@ -25,30 +28,79 @@ use super::tigris::TigrisClient;
pub struct RepoStore {
repos_dir: PathBuf,
tigris: Option<TigrisClient>,
/// Shared Postgres pool for advisory locks.
pool: PgPool,
/// Dedicated Postgres pool for repo write advisory locks, built by
/// `build_lock_pool` (see there for why it is separate and why it carries an
/// `after_release` hook). Never use this for ordinary queries.
lock_pool: PgPool,
/// Tracks repos already confirmed to exist in Tigris — avoids redundant
/// HEAD checks and background uploads for repos we've already migrated.
migrated: Arc<Mutex<HashSet<String>>>,
/// Test-only stall injected at the head of `acquire_write`'s Tigris phase,
/// i.e. AFTER the advisory lock is taken and BEFORE the guard exists. That
/// window is exactly where the outer `tokio::time::timeout` in
/// `api/repos.rs` can drop the future (#173). `TigrisClient` takes its
/// endpoint from process-wide AWS env vars and has no injectable seam, so
/// this flag is the smallest way to hold a real `acquire_write` open in that
/// window and cancel it there.
#[cfg(test)]
tigris_stall: Option<std::time::Duration>,
/// Test-only counter of how many times a write guard from this store REACHED the
/// Tigris upload site in `release` (the point past the `success` check, where a
/// configured client would be uploaded to). It counts the decision, not a network
/// call: `TigrisClient` takes its endpoint from process-wide AWS env vars and has no
/// injectable seam, so every test runs with `tigris: None` and a counter inside the
/// `Some` arm could never move. Reaching the site is the property under test anyway:
/// an interrupted push must not publish a half-applied repo, and the disconnect path
/// must therefore never get here (#173 F2).
///
/// Per store rather than a process global, so cases running in parallel do not see
/// each other's uploads, and an `Arc` rather than a `thread_local` because the guard
/// is released from a detached task on another worker thread. Same test-only counter
/// idiom as `ipfs_pin::note_legacy_repair_read`.
#[cfg(test)]
upload_site_reached: Arc<std::sync::atomic::AtomicUsize>,
}
impl RepoStore {
/// Derives its own lock pool from `pool`, so callers that only have the main
/// pool (tests, `for_testing` sites in other modules) still get the
/// `after_release` semantics `acquire_write` depends on.
#[cfg(test)]
pub fn for_testing(repos_dir: PathBuf, pool: PgPool) -> Self {
Self {
Self::new(
repos_dir,
tigris: None,
pool,
migrated: Arc::new(tokio::sync::Mutex::new(std::collections::HashSet::new())),
}
None,
build_lock_pool(&pool, 8, Duration::from_secs(5)),
)
}
pub fn new(repos_dir: PathBuf, tigris: Option<TigrisClient>, pool: PgPool) -> Self {
/// Test-only: see `tigris_stall`.
#[cfg(test)]
pub fn with_tigris_stall(mut self, stall: Duration) -> Self {
self.tigris_stall = Some(stall);
self
}
/// Test-only: how many write guards from this store have reached the Tigris upload
/// site. See [`RepoStore::upload_site_reached`].
#[cfg(test)]
pub fn tigris_upload_site_reached(&self) -> usize {
self.upload_site_reached
.load(std::sync::atomic::Ordering::SeqCst)
}
/// `lock_pool` must come from `build_lock_pool`; a plain pool leaks advisory
/// locks on cancellation.
pub fn new(repos_dir: PathBuf, tigris: Option<TigrisClient>, lock_pool: PgPool) -> Self {
Self {
repos_dir,
tigris,
pool,
lock_pool,
migrated: Arc::new(Mutex::new(HashSet::new())),
#[cfg(test)]
tigris_stall: None,
#[cfg(test)]
upload_site_reached: Arc::new(std::sync::atomic::AtomicUsize::new(0)),
}
}
@@ -157,25 +209,64 @@ impl RepoStore {
let (owner_slug, local_path) = self.local_path(owner_did, repo_name)?;
let lock_key = advisory_lock_key(&owner_slug, repo_name);
// Acquire Postgres advisory lock with retry using pg_try_advisory_lock
// to avoid blocking indefinitely on stale locks from crashed connections.
let mut acquired = false;
// Acquire the Postgres advisory lock with retry, using pg_try_advisory_lock so a
// stale lock from a crashed connection can't block us indefinitely.
//
// The connection is checked out INSIDE the loop and RETURNED before each sleep.
// Only the connection that actually took the lock is retained. Two constraints
// pull in opposite directions here, and this is what satisfies both:
//
// * Session ownership. A session-level advisory lock belongs to the CONNECTION
// that took it, so the lock and its `pg_advisory_unlock` must run on the same
// one. Running them through the pool (`fetch_one(&self.pool)`) lets them land
// on different connections: the unlock silently returns false and the lock
// leaks, while a competing acquire that happens to draw the holding
// connection re-enters the lock and two pushes to one repo run concurrently.
// Hence: keep the connection that WON.
// * Occupancy. Holding a connection across the ~60 one-second sleeps would let
// one spinning acquire park a lock-pool connection for a minute. That is not
// just a push-path concern: `api/issues.rs` and `api/pulls.rs` reach
// acquire_write holding no concurrency permit at all, so a caller could park
// the whole pool and starve authenticated pushes on every repo (#173 F1).
// Hence: return the connection when we LOSE, before sleeping.
//
// Returning a losing connection is safe with respect to the cancellation design:
// `after_release` runs `pg_advisory_unlock_all()`, a no-op on a connection that
// took nothing, so it cannot disturb a lock held by any other connection
// (proven by `returning_an_unlocked_connection_does_not_clear_another_connections_lock`).
//
// Cancellation safety is unchanged: the future can only be dropped while a
// connection is checked out, and dropping it runs the same `after_release` hook,
// which clears whatever lock it had just taken (#173 U1).
let mut lock_conn = None;
for attempt in 0..60 {
let mut conn = self.lock_pool.acquire().await.map_err(|e| {
anyhow::Error::new(LockPoolBusy)
.context(format!("checking out a lock-pool connection: {e}"))
})?;
let row: (bool,) = sqlx::query_as("SELECT pg_try_advisory_lock($1)")
.bind(lock_key)
.fetch_one(&self.pool)
.fetch_one(&mut *conn)
.await
.context("trying advisory lock")?;
if row.0 {
acquired = true;
lock_conn = Some(conn);
break;
}
// Lost the race: give the connection back so a spinning acquire occupies
// nothing while it waits.
drop(conn);
if attempt < 59 {
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
}
}
if !acquired {
let Some(lock_conn) = lock_conn else {
anyhow::bail!("could not acquire advisory lock after 60s — possible stale lock for {owner_slug}/{repo_name}");
};
#[cfg(test)]
if let Some(stall) = self.tigris_stall {
tokio::time::sleep(stall).await;
}
// Always download the latest from Tigris before writing.
@@ -202,8 +293,10 @@ impl RepoStore {
repo_name: repo_name.to_string(),
local_path,
lock_key,
pool: self.pool.clone(),
lock_conn,
tigris: self.tigris.clone(),
#[cfg(test)]
upload_site_reached: Arc::clone(&self.upload_site_reached),
})
}
@@ -257,42 +350,63 @@ impl RepoStore {
/// segment is rejected. This is the CodeQL-recognised barrier
/// pattern for `rust/path-injection`.
fn local_path(&self, owner_did: &str, repo_name: &str) -> Result<(String, PathBuf)> {
validate_path_components(owner_did, repo_name)?;
let owner_slug = owner_did.replace([':', '/'], "_");
let local_path = self
.repos_dir
.join(&owner_slug)
.join(format!("{repo_name}.git"));
if !local_path.starts_with(&self.repos_dir) {
anyhow::bail!(
"computed repo path escaped repos_dir: {}",
local_path.display()
);
}
// Explicit component walk — sanitisation barrier that static analysers
// (CodeQL `rust/path-injection`) recognise. The path must be composed
// entirely of Normal segments after the root prefix; any ParentDir or
// CurDir component is a traversal attempt.
for component in local_path.components() {
use std::path::Component;
match component {
Component::Prefix(_) | Component::RootDir | Component::Normal(_) => {}
Component::ParentDir => {
anyhow::bail!("path contains parent-directory component");
}
Component::CurDir => {
anyhow::bail!("path contains current-directory component");
}
}
}
let local_path = validated_repo_disk_path(&self.repos_dir, owner_did, repo_name)?;
Ok((owner_slug, local_path))
}
}
/// The three-layer validated form of `store::repo_disk_path`, with NO Tigris fetch and
/// no `RepoStore` (#173 round 11, F3). Extracted from `RepoStore::local_path` so a
/// second caller that must not pull a cold repo, the U4 legacy provider-CID sweep, gets
/// the same barrier instead of the raw join. `local_path` is now a thin wrapper over
/// this, so the two cannot drift.
///
/// Three-layer defence against path traversal:
/// 1. Strict allowlist on `owner_did` and `repo_name` (no `..`, slashes,
/// null bytes, leading dots; length-bounded).
/// 2. The joined path must remain rooted at `repos_dir`.
/// 3. Every component of the joined path must be `Component::Normal`
/// (or the prefix/root from `repos_dir`); any `ParentDir`/`CurDir`
/// segment is rejected. This is the CodeQL-recognised barrier
/// pattern for `rust/path-injection`.
pub(crate) fn validated_repo_disk_path(
repos_dir: &Path,
owner_did: &str,
repo_name: &str,
) -> Result<PathBuf> {
validate_path_components(owner_did, repo_name)?;
let owner_slug = owner_did.replace([':', '/'], "_");
let local_path = repos_dir.join(&owner_slug).join(format!("{repo_name}.git"));
if !local_path.starts_with(repos_dir) {
anyhow::bail!(
"computed repo path escaped repos_dir: {}",
local_path.display()
);
}
// Explicit component walk — sanitisation barrier that static analysers
// (CodeQL `rust/path-injection`) recognise. The path must be composed
// entirely of Normal segments after the root prefix; any ParentDir or
// CurDir component is a traversal attempt.
for component in local_path.components() {
use std::path::Component;
match component {
Component::Prefix(_) | Component::RootDir | Component::Normal(_) => {}
Component::ParentDir => {
anyhow::bail!("path contains parent-directory component");
}
Component::CurDir => {
anyhow::bail!("path contains current-directory component");
}
}
}
Ok(local_path)
}
/// Strict allowlist validator for `owner_did` and `repo_name`.
///
/// Rejects any character that isn't explicitly safe, plus length and
@@ -347,6 +461,19 @@ fn validate_repo_name(repo_name: &str) -> Result<()> {
Ok(())
}
/// Error marker for "no lock-pool connection was available in time".
///
/// Carried through the `anyhow` chain (like [`smart_http::GitServiceTimeout`]) so the
/// HTTP handler can `downcast_ref` it and shed a 503 + Retry-After instead of the
/// generic 500 a git error maps to: an exhausted lock pool is a CAPACITY signal, and
/// telling the client to retry shortly is the same shed semantics the surrounding
/// admission code already uses (#173 F1).
///
/// [`smart_http::GitServiceTimeout`]: crate::git::smart_http::GitServiceTimeout
#[derive(Debug, thiserror::Error)]
#[error("no lock-pool connection available")]
pub struct LockPoolBusy;
/// Guard returned by `acquire_write()`. Holds the Postgres advisory lock and
/// uploads to Tigris + releases the lock on `release()`.
pub struct RepoWriteGuard {
@@ -354,8 +481,17 @@ pub struct RepoWriteGuard {
repo_name: String,
pub local_path: PathBuf,
lock_key: i64,
pool: PgPool,
/// The lock-pool connection that TOOK the advisory lock. It must be the one
/// that releases it (session locks are owned by their connection), and
/// holding it here is also what makes a guard dropped without `release`
/// safe: the drop returns the connection through `after_release`, which
/// clears the lock.
lock_conn: PoolConnection<Postgres>,
tigris: Option<TigrisClient>,
/// Shared with the store that handed this guard out; see
/// [`RepoStore::upload_site_reached`].
#[cfg(test)]
upload_site_reached: Arc<std::sync::atomic::AtomicUsize>,
}
impl RepoWriteGuard {
@@ -369,9 +505,16 @@ impl RepoWriteGuard {
/// half-applied or otherwise inconsistent repo would propagate corruption to
/// Tigris (and to every node that later downloads it). The lock is always
/// released regardless, to avoid stale locks blocking future writes.
pub async fn release(self, success: bool) {
pub async fn release(mut self, success: bool) {
// Upload to Tigris only on success.
if success {
// The upload site, recorded for tests before the client is consulted: with
// no injectable seam on `TigrisClient` a counter inside the arm below could
// never move, and it is reaching this point at all that an interrupted push
// must not do (#173 F2).
#[cfg(test)]
self.upload_site_reached
.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
if let Some(ref tigris) = self.tigris {
if let Err(e) = tigris
.upload(&self.owner_slug, &self.repo_name, &self.local_path)
@@ -384,14 +527,56 @@ impl RepoWriteGuard {
warn!(repo = %self.repo_name, "write failed — skipping tigris upload to avoid propagating an inconsistent repo");
}
// Release advisory lock
// Release the advisory lock on the connection that took it. Anything else
// (a fresh `&pool` checkout) is a no-op that returns false: Postgres
// scopes a session lock to its owning connection.
let _ = sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(self.lock_key)
.execute(&self.pool)
.execute(&mut *self.lock_conn)
.await;
// Dropping `self` returns the connection to the lock pool, where
// `after_release` sweeps anything the unlock above missed.
}
}
/// Build the dedicated advisory-lock pool a `RepoStore` runs its write locks on.
/// Connect options are cloned off an existing pool so callers need not re-parse
/// the database URL; the pool is lazy, so no connection is opened here.
///
/// Two properties, both load-bearing:
///
/// * The `after_release` hook runs `pg_advisory_unlock_all()` before a
/// connection goes back into the pool. sqlx's `PoolConnection::drop` spawns
/// `return_to_pool()`, which invokes this hook, so a connection dropped by
/// CANCELLATION still clears its locks. That is what keeps an `acquire_write`
/// killed mid-Tigris by the caller's `tokio::time::timeout` from leaking a
/// lock and wedging every later push to that repo (#173). Note the hook runs
/// from that spawned task, so the unlock is asynchronous with respect to the
/// drop: the lock clears shortly after the connection goes away, not
/// synchronously with it.
/// * It is a SEPARATE pool from the main query pool, not a slice of it. A push
/// holds its lock connection for the whole receive-pack, and
/// `db_max_connections` (default 20) is well below
/// `max_concurrent_git_pushes` (default 32), so drawing these from the main
/// pool would starve every other query during a push burst.
///
/// `acquire_timeout` bounds the wait when every lock-pool connection is busy, so
/// exhaustion surfaces as a clean error rather than an unbounded hang.
pub fn build_lock_pool(source: &PgPool, max_connections: u32, acquire_timeout: Duration) -> PgPool {
PgPoolOptions::new()
.max_connections(max_connections)
.acquire_timeout(acquire_timeout)
.after_release(|conn, _meta| {
Box::pin(async move {
sqlx::query("SELECT pg_advisory_unlock_all()")
.execute(&mut *conn)
.await?;
Ok(true)
})
})
.connect_lazy_with((*source.connect_options()).clone())
}
/// Compute a stable i64 hash for a Postgres advisory lock key.
fn advisory_lock_key(owner_slug: &str, repo_name: &str) -> i64 {
use std::hash::{Hash, Hasher};
@@ -404,6 +589,503 @@ fn advisory_lock_key(owner_slug: &str, repo_name: &str) -> i64 {
#[cfg(test)]
mod tests {
use super::*;
use std::time::Duration;
// ── advisory-lock test helpers (#173 U1) ───────────────────────────────
/// Postgres advisory locks live in a CLUSTER-wide space, not a per-database
/// one, so two `#[sqlx::test]` cases running against their own temporary
/// databases still share the key space. Every lock test therefore mints its
/// own key instead of reusing a fixed constant.
fn unique_lock_key() -> i64 {
use std::sync::atomic::{AtomicI64, Ordering};
static NEXT: AtomicI64 = AtomicI64::new(0);
let n = NEXT.fetch_add(1, Ordering::Relaxed);
((std::process::id() as i64) << 24) | (n & 0xff_ffff)
}
/// A plain pool (no `after_release` hook, no idle timeout) at the same
/// database as the `#[sqlx::test]` pool. Two separate reasons these tests
/// cannot just use the pool the harness hands them:
///
/// 1. Observing lock state has to happen from a session that is definitely
/// not the one under test. Session advisory locks are re-entrant, so
/// `pg_try_advisory_lock` on the very connection that already holds the key
/// returns true, and a same-pool probe silently reports a leaked lock free.
/// 2. The harness pool sets `idle_timeout(1s)`, so a connection returned to it
/// is closed about a second later and Postgres drops every lock that
/// session held. That would mask exactly the leak these tests exist to
/// catch, so the store under test runs on one of these too.
fn sibling_pool(pool: &PgPool, max_connections: u32) -> PgPool {
sqlx::postgres::PgPoolOptions::new()
.max_connections(max_connections)
.connect_lazy_with((*pool.connect_options()).clone())
}
/// Probe the lock from a connection that is NOT the one under test. Session
/// advisory locks are re-entrant within their own session, so a check from the
/// holding connection would pass vacuously and prove nothing.
async fn lock_is_free_elsewhere(pool: &PgPool, key: i64) -> bool {
let mut probe = pool.acquire().await.expect("probe connection");
let taken: (bool,) = sqlx::query_as("SELECT pg_try_advisory_lock($1)")
.bind(key)
.fetch_one(&mut *probe)
.await
.expect("probe try-lock");
if taken.0 {
sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(key)
.execute(&mut *probe)
.await
.expect("probe unlock");
}
taken.0
}
/// `after_release` runs from the task sqlx spawns in `PoolConnection::drop`,
/// so the unlock is ASYNCHRONOUS with respect to the drop. Callers must poll
/// rather than assume the lock is gone the instant the connection goes away.
async fn wait_until_free(pool: &PgPool, key: i64, within: Duration) -> bool {
let deadline = std::time::Instant::now() + within;
loop {
if lock_is_free_elsewhere(pool, key).await {
return true;
}
if std::time::Instant::now() >= deadline {
return false;
}
tokio::time::sleep(Duration::from_millis(25)).await;
}
}
// ── DESIGN GATE ────────────────────────────────────────────────────────
// The whole cancellation-safety design rests on one sqlx behaviour:
// `PoolConnection::drop` spawns `return_to_pool()`, which invokes the pool's
// `after_release` hook before the connection is reused. If that holds, a
// connection dropped by cancellation still runs `pg_advisory_unlock_all()`
// and the lock cannot leak. This test proves it by execution, through the
// production `build_lock_pool` so that stripping the hook there turns it red.
#[sqlx::test]
async fn dropped_pool_connection_runs_after_release_and_clears_locks(pool: PgPool) {
let key = unique_lock_key();
let lock_pool = build_lock_pool(&pool, 4, Duration::from_secs(5));
{
let mut conn = lock_pool.acquire().await.expect("lock-pool connection");
let taken: (bool,) = sqlx::query_as("SELECT pg_try_advisory_lock($1)")
.bind(key)
.fetch_one(&mut *conn)
.await
.expect("try-lock");
assert!(taken.0, "first try-lock must succeed");
assert!(
!lock_is_free_elsewhere(&pool, key).await,
"lock must be observably HELD from another session while the connection lives"
);
// Drop WITHOUT calling pg_advisory_unlock: this models cancellation.
}
assert!(
wait_until_free(&pool, key, Duration::from_secs(5)).await,
"after_release must clear the advisory lock of a dropped connection"
);
}
// ── acquire_write cancellation safety (#173 U1) ────────────────────────
/// The reviewer's named regression. `api/repos.rs` wraps `acquire_write` in a
/// `tokio::time::timeout`; when that fires during the Tigris phase the future
/// is dropped after the advisory lock was taken and before `RepoWriteGuard`
/// (the only thing that unlocks) exists. The lock then leaks and every later
/// push to the same repo spins the 60-attempt / 60s ceiling and fails.
#[sqlx::test]
async fn cancelled_acquire_write_mid_tigris_does_not_leak_the_lock(pool: PgPool) {
let repos_dir = PathBuf::from("/tmp/gitlawb-test-repos");
let owner = "did:key:z6MkCancelMidTigris";
let repo = "cancel-mid-tigris";
let store_pool = sibling_pool(&pool, 8);
let stalling = RepoStore::for_testing(repos_dir.clone(), store_pool.clone())
.with_tigris_stall(Duration::from_secs(30));
let cancelled = tokio::time::timeout(
Duration::from_millis(500),
stalling.acquire_write(owner, repo),
)
.await;
assert!(
cancelled.is_err(),
"the acquire must still be inside the Tigris phase when the timeout fires"
);
// Observed from an independent session, so the check cannot be satisfied
// by re-entrancy on whichever pooled connection happens to be handed back.
let probe = sibling_pool(&pool, 2);
let key = advisory_lock_key(&owner.replace([':', '/'], "_"), repo);
assert!(
wait_until_free(&probe, key, Duration::from_secs(5)).await,
"a cancelled acquire_write must leave no advisory lock held"
);
// A subsequent acquire for the SAME repo must succeed promptly. Before the
// fix it blocks on the leaked lock until the 60-attempt ceiling.
let store = RepoStore::for_testing(repos_dir, store_pool);
let guard = tokio::time::timeout(Duration::from_secs(5), store.acquire_write(owner, repo))
.await
.expect("second acquire_write must not block on a leaked lock")
.expect("second acquire_write must succeed");
guard.release(false).await;
}
/// Cancellation BEFORE the lock is taken must leave nothing behind: no lock,
/// and no lock-pool connection stranded. The lock pool here holds exactly one
/// connection, so a stranded one would make the follow-up acquire time out
/// waiting for a checkout.
#[sqlx::test]
async fn cancelled_acquire_write_before_the_lock_leaves_nothing_held(pool: PgPool) {
let probe = sibling_pool(&pool, 2);
let owner = "did:key:z6MkCancelEarly";
let repo = "cancel-early";
let key = advisory_lock_key(&owner.replace([':', '/'], "_"), repo);
let store = RepoStore::new(
PathBuf::from("/tmp/gitlawb-test-repos"),
None,
build_lock_pool(&pool, 1, Duration::from_secs(3)),
);
// A zero deadline polls the future once, which gets it no further than the
// first await (the pool checkout / the first try-lock round trip), so it is
// cancelled before any lock can be taken.
let cancelled =
tokio::time::timeout(Duration::ZERO, store.acquire_write(owner, repo)).await;
assert!(cancelled.is_err(), "the acquire must be cancelled");
assert!(
lock_is_free_elsewhere(&probe, key).await,
"no lock may be held when the acquire never got that far"
);
// The single lock-pool connection must be back: if cancellation stranded
// it, this checkout blocks until the 3s acquire timeout and fails.
let guard = tokio::time::timeout(Duration::from_secs(2), store.acquire_write(owner, repo))
.await
.expect("the lock-pool connection must have been returned")
.expect("acquire after cancellation");
guard.release(false).await;
}
/// Lock-pool exhaustion is a bounded wait and a clean error, never a panic and
/// never an unbounded hang.
#[sqlx::test]
async fn lock_pool_exhaustion_is_a_bounded_error(pool: PgPool) {
let owner = "did:key:z6MkExhaustion";
let store = RepoStore::new(
PathBuf::from("/tmp/gitlawb-test-repos"),
None,
build_lock_pool(&pool, 1, Duration::from_secs(2)),
);
let held = store
.acquire_write(owner, "exhaust-a")
.await
.expect("first acquire");
// Different repo, so this is not the advisory lock queueing: the only
// connection in the lock pool is checked out by `held`.
let started = std::time::Instant::now();
let err = tokio::time::timeout(
Duration::from_secs(10),
store.acquire_write(owner, "exhaust-b"),
)
.await
.expect("the wait must be bounded by the pool acquire timeout");
let err = match err {
Ok(_) => panic!("an exhausted lock pool must surface an error, not a guard"),
Err(e) => e,
};
assert!(
started.elapsed() < Duration::from_secs(6),
"the error must arrive on the acquire timeout, not after a long hang"
);
assert!(
err.to_string().contains("lock-pool connection"),
"the error must name the lock-pool checkout, got: {err}"
);
held.release(false).await;
}
/// #173 F1 (RED-before/GREEN-after). A contended `acquire_write` spins for up to
/// 60 one-second attempts. It must not OCCUPY a lock-pool connection for that whole
/// spin: `acquire_write` has non-push callers (`api/issues.rs`, `api/pulls.rs`) that
/// hold no concurrency permit, so any self-minted did:key could otherwise park a
/// connection per call and starve authenticated pushes on EVERY repo.
///
/// Lock pool of exactly 2, two spinners. Pre-fix (checkout hoisted above the retry
/// loop) they pin both connections for the full spin and an UNCONTENDED acquire on a
/// third repo dies on the pool acquire timeout. Post-fix each spinner returns its
/// connection before sleeping, so it occupies ~0 and the uncontended acquire sails
/// through.
#[sqlx::test]
async fn a_spinning_acquire_write_does_not_occupy_a_lock_pool_connection(pool: PgPool) {
let owner = "did:key:z6MkSpinOccupancy";
let owner_slug = owner.replace([':', '/'], "_");
let store = RepoStore::new(
PathBuf::from("/tmp/gitlawb-test-repos"),
None,
build_lock_pool(&pool, 2, Duration::from_secs(2)),
);
// An independent session holds both contended keys, so the spinners' try-locks
// return false on every iteration and they stay in the retry loop.
let holder = sibling_pool(&pool, 2);
let mut held_conn = holder.acquire().await.expect("holder connection");
for repo in ["spin-a", "spin-b"] {
let taken: (bool,) = sqlx::query_as("SELECT pg_try_advisory_lock($1)")
.bind(advisory_lock_key(&owner_slug, repo))
.fetch_one(&mut *held_conn)
.await
.expect("holder try-lock");
assert!(taken.0, "the holder must own {repo}'s key");
}
let mut spinners = Vec::new();
for repo in ["spin-a", "spin-b"] {
let store = store.clone();
spinners.push(tokio::spawn(async move {
store.acquire_write(owner, repo).await
}));
}
// Let both reach the spin (each has done at least one failed try-lock by now).
tokio::time::sleep(Duration::from_millis(500)).await;
let started = std::time::Instant::now();
let uncontended = tokio::time::timeout(
Duration::from_secs(10),
store.acquire_write(owner, "spin-free"),
)
.await
.expect("the uncontended acquire must return, not hang");
let elapsed = started.elapsed();
let free_guard = uncontended.unwrap_or_else(|e| {
panic!(
"an UNCONTENDED acquire_write on a DIFFERENT repo must not be starved by \
spinners holding the lock pool; got: {e}"
)
});
assert!(
elapsed < Duration::from_secs(2),
"the uncontended acquire must not queue behind the spinners for the pool \
acquire timeout; took {elapsed:?}"
);
free_guard.release(false).await;
// The drop-and-retake cycle must still END in a real, exclusive lock: free
// spin-a's key and the spinner that was cycling connections must take it.
sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(advisory_lock_key(&owner_slug, "spin-a"))
.execute(&mut *held_conn)
.await
.expect("release spin-a");
let winner = tokio::time::timeout(Duration::from_secs(15), spinners.remove(0))
.await
.expect("the spinner must finish once its key frees")
.expect("spinner task")
.expect("the spinner must acquire once the key frees");
let probe = sibling_pool(&pool, 2);
assert!(
!lock_is_free_elsewhere(&probe, advisory_lock_key(&owner_slug, "spin-a")).await,
"the lock a spinner finally took must be observably held from another session"
);
winner.release(false).await;
for s in spinners {
s.abort();
}
sqlx::query("SELECT pg_advisory_unlock_all()")
.execute(&mut *held_conn)
.await
.expect("release the remaining holder lock");
}
/// #173 F1, the property the fix rests on: returning a lock-pool connection that
/// holds NOTHING runs `after_release`'s `pg_advisory_unlock_all()`, which is a no-op
/// and must not disturb a lock held on a DIFFERENT connection of the same pool.
/// Session advisory locks are per connection, so this is by construction, but the
/// spin fix depends on it, so it is proven by execution rather than assumed.
#[sqlx::test]
async fn returning_an_unlocked_connection_does_not_clear_another_connections_lock(
pool: PgPool,
) {
let owner = "did:key:z6MkNoOpUnlockAll";
let repo = "noop-unlock";
let key = advisory_lock_key(&owner.replace([':', '/'], "_"), repo);
let probe = sibling_pool(&pool, 2);
let lock_pool = build_lock_pool(&pool, 4, Duration::from_secs(5));
let store = RepoStore::new(
PathBuf::from("/tmp/gitlawb-test-repos"),
None,
lock_pool.clone(),
);
let guard = store.acquire_write(owner, repo).await.expect("acquire");
// Churn the pool: check out and drop connections that hold no lock, exactly what
// a spinning acquire now does between attempts. Each return fires
// pg_advisory_unlock_all() on that connection.
for _ in 0..10 {
let mut conn = lock_pool.acquire().await.expect("churn checkout");
let _: (i32,) = sqlx::query_as("SELECT 1")
.fetch_one(&mut *conn)
.await
.expect("churn query");
drop(conn);
tokio::time::sleep(Duration::from_millis(10)).await;
}
assert!(
!lock_is_free_elsewhere(&probe, key).await,
"a held write lock must survive other lock-pool connections being returned"
);
guard.release(true).await;
assert!(
lock_is_free_elsewhere(&probe, key).await,
"release must still free the lock after the churn"
);
}
/// #173 F1: lock-pool exhaustion is a DISTINCT error the handler can shed as a 503,
/// not a generic git 500. Both directions: an exhausted pool downcasts to
/// [`LockPoolBusy`], and an unrelated failure (a rejected repo name) does not.
#[sqlx::test]
async fn lock_pool_exhaustion_is_a_distinct_downcastable_error(pool: PgPool) {
let owner = "did:key:z6MkBusyDowncast";
let store = RepoStore::new(
PathBuf::from("/tmp/gitlawb-test-repos"),
None,
build_lock_pool(&pool, 1, Duration::from_secs(1)),
);
let held = store
.acquire_write(owner, "busy-a")
.await
.expect("first acquire");
let err = match store.acquire_write(owner, "busy-b").await {
Ok(_) => panic!("an exhausted lock pool must error, not hand back a guard"),
Err(e) => e,
};
assert!(
err.downcast_ref::<LockPoolBusy>().is_some(),
"lock-pool exhaustion must be downcastable so the handler sheds 503, got: {err}"
);
// MUST-NOT: an ordinary rejection is not a capacity signal.
let other = match store.acquire_write(owner, "../escape").await {
Ok(_) => panic!("a traversal repo name must be rejected"),
Err(e) => e,
};
assert!(
other.downcast_ref::<LockPoolBusy>().is_none(),
"a validation failure must not masquerade as lock-pool capacity, got: {other}"
);
held.release(false).await;
}
/// Round trip: the lock is observably HELD between acquire and release, and
/// observably FREE after. Both checks run from an independent session; from
/// the holding session they would pass vacuously (session locks are
/// re-entrant) and would not notice an unlock that landed on the wrong
/// connection.
#[sqlx::test]
async fn acquire_write_holds_the_lock_until_release(pool: PgPool) {
let probe = sibling_pool(&pool, 2);
let owner = "did:key:z6MkRoundTrip";
let repo = "round-trip";
let key = advisory_lock_key(&owner.replace([':', '/'], "_"), repo);
let store = RepoStore::for_testing(PathBuf::from("/tmp/gitlawb-test-repos"), pool.clone());
let guard = store.acquire_write(owner, repo).await.expect("acquire");
assert!(
!lock_is_free_elsewhere(&probe, key).await,
"the lock must be held while the guard is alive"
);
// No polling here, deliberately. `release` must free the lock SYNCHRONOUSLY,
// which it can only do by unlocking on the connection that took it; a
// `pg_advisory_unlock` sent through the pool would land on some other
// session and return false. The `after_release` hook is a net for the
// cancellation path and fires from a spawned task well after this point, so
// it must not be what makes this assertion pass.
guard.release(true).await;
assert!(
lock_is_free_elsewhere(&probe, key).await,
"release must free the lock as seen from another session"
);
}
/// `release(false)` skips the Tigris upload but must still free the lock; a
/// failed write that kept the lock would wedge the repo.
#[sqlx::test]
async fn release_after_failed_write_still_frees_the_lock(pool: PgPool) {
let probe = sibling_pool(&pool, 2);
let owner = "did:key:z6MkFailedWrite";
let repo = "failed-write";
let key = advisory_lock_key(&owner.replace([':', '/'], "_"), repo);
let store = RepoStore::for_testing(PathBuf::from("/tmp/gitlawb-test-repos"), pool.clone());
let guard = store.acquire_write(owner, repo).await.expect("acquire");
guard.release(false).await;
assert!(
wait_until_free(&probe, key, Duration::from_secs(5)).await,
"release(success = false) must still free the lock"
);
}
/// The lock is per repo: a second acquire for the SAME repo waits for the
/// first to release, while a different repo proceeds straight through.
#[sqlx::test]
async fn same_repo_acquires_serialize_and_different_repos_do_not(pool: PgPool) {
let repos_dir = PathBuf::from("/tmp/gitlawb-test-repos");
let owner = "did:key:z6MkSerialize";
let store = RepoStore::for_testing(repos_dir, pool.clone());
let first = store
.acquire_write(owner, "serialize-a")
.await
.expect("first acquire");
// Different repo: unaffected by the held lock.
let other = tokio::time::timeout(
Duration::from_secs(2),
store.acquire_write(owner, "serialize-b"),
)
.await
.expect("a different repo must not wait on this lock")
.expect("acquire other repo");
other.release(false).await;
// Same repo: must not acquire while `first` is alive.
let contender = tokio::spawn({
let store = store.clone();
async move { store.acquire_write(owner, "serialize-a").await }
});
tokio::time::sleep(Duration::from_millis(1500)).await;
assert!(
!contender.is_finished(),
"a second acquire for the same repo must block while the first guard lives"
);
first.release(false).await;
let second = tokio::time::timeout(Duration::from_secs(10), contender)
.await
.expect("contender must finish once the lock is free")
.expect("contender task")
.expect("contender acquire");
second.release(false).await;
}
// ── repo_name validation ───────────────────────────────────────────────
File diff suppressed because it is too large. Load diff
+229 -9
View File
@@ -271,9 +271,8 @@ pub struct TreeEntry {
/// `/ipfs/<cid>` is computed from these same content bytes via
/// `gitlawb_core::cid::Cid::from_git_object_bytes`.
///
/// Returns `None` if the object does not exist in this repo.
pub fn read_object(repo_path: &Path, sha256_hex: &str) -> Result<Option<(String, Vec<u8>)>> {
// First check if the object exists and get its type
/// Get just the object type. Returns `None` if the object doesn't exist.
pub fn object_type(repo_path: &Path, sha256_hex: &str) -> Result<Option<String>> {
let type_output = Command::new("git")
.args(["cat-file", "-t", sha256_hex])
.current_dir(repo_path)
@@ -284,13 +283,17 @@ pub fn read_object(repo_path: &Path, sha256_hex: &str) -> Result<Option<(String,
return Ok(None);
}
let obj_type = String::from_utf8_lossy(&type_output.stdout)
.trim()
.to_string();
Ok(Some(
String::from_utf8_lossy(&type_output.stdout)
.trim()
.to_string(),
))
}
// Read the raw content bytes
/// Read an object's content if its type is already known.
pub fn read_object_content(repo_path: &Path, sha256_hex: &str, obj_type: &str) -> Result<Vec<u8>> {
let content_output = Command::new("git")
.args(["cat-file", &obj_type, sha256_hex])
.args(["cat-file", obj_type, sha256_hex])
.current_dir(repo_path)
.output()
.context("failed to run git cat-file <type>")?;
@@ -300,7 +303,144 @@ pub fn read_object(repo_path: &Path, sha256_hex: &str) -> Result<Option<(String,
bail!("git cat-file failed: {stderr}");
}
Ok(Some((obj_type, content_output.stdout)))
Ok(content_output.stdout)
}
/// Bounded twin of [`object_type`] for the `GET /ipfs/{cid}` serve path (#173
/// round-10, R1/KTD2). Runs `git cat-file -t` under
/// [`run_bounded_git`](crate::git::visibility_pack::run_bounded_git) so the child runs
/// in its own process group and a watchdog reaps it (SIGTERM -> grace -> SIGKILL) at
/// `timeout`. The bare [`object_type`] is a `spawn_blocking` `Command::output` that an
/// async timeout cannot cancel, so a wedged `cat-file` there pins the caller's held
/// /ipfs walk admission for the whole hang; this twin cannot. Semantics mirror
/// [`object_type`]: `Ok(Some(t))` for an existing object, `Ok(None)` when git exits
/// non-zero without timing out (the object is absent), and
/// `Err(`[`GitServiceTimeout`](crate::git::smart_http::GitServiceTimeout)`)` on the
/// deadline so the handler can mark the search truncated (retryable 503) rather than a
/// false not-found. Callers off the /ipfs path keep the bare helper.
pub fn object_type_bounded(
git_bin: &str,
repo_path: &Path,
sha256_hex: &str,
timeout: std::time::Duration,
) -> Result<Option<String>> {
let deadline = std::time::Instant::now() + timeout;
match crate::git::visibility_pack::run_bounded_git(
git_bin,
&["cat-file", "-t", sha256_hex],
repo_path,
b"",
deadline,
) {
Ok(out) => Ok(Some(String::from_utf8_lossy(&out).trim().to_string())),
Err(e) if e.is::<crate::git::smart_http::GitServiceTimeout>() => Err(e),
// A non-timeout failure is git reporting no such object (a non-zero exit),
// matching `object_type`'s `Ok(None)`.
Err(_) => Ok(None),
}
}
/// Bounded `git cat-file -s` size read for the `GET /ipfs/{cid}` serve path (#173
/// round-10, R1/KTD2): reads the object size WITHOUT its content (so an oversized object
/// is rejected before it is buffered, #173 F6), under
/// [`run_bounded_git`](crate::git::visibility_pack::run_bounded_git) so a wedged size
/// read is reaped at `timeout` instead of pinning the held /ipfs walk admission.
/// `Ok(Some(n))` on success, `Ok(None)` when the object is absent (a non-timeout
/// non-zero exit), `Err(GitServiceTimeout)` on the deadline.
pub fn object_size_bounded(
git_bin: &str,
repo_path: &Path,
sha256_hex: &str,
timeout: std::time::Duration,
) -> Result<Option<u64>> {
let deadline = std::time::Instant::now() + timeout;
match crate::git::visibility_pack::run_bounded_git(
git_bin,
&["cat-file", "-s", sha256_hex],
repo_path,
b"",
deadline,
) {
Ok(out) => Ok(String::from_utf8_lossy(&out).trim().parse::<u64>().ok()),
Err(e) if e.is::<crate::git::smart_http::GitServiceTimeout>() => Err(e),
Err(_) => Ok(None),
}
}
/// Bounded twin of [`read_object_content`] for the `GET /ipfs/{cid}` serve path (#173
/// round-10, R1/KTD2): `git cat-file <type>` under
/// [`run_bounded_git`](crate::git::visibility_pack::run_bounded_git) so a wedged content
/// read is reaped at `timeout` instead of pinning the held /ipfs walk admission. Returns
/// the raw object bytes on success and an error (including `GitServiceTimeout` on the
/// deadline) otherwise, mirroring [`read_object_content`].
pub fn read_object_content_bounded(
git_bin: &str,
repo_path: &Path,
sha256_hex: &str,
obj_type: &str,
timeout: std::time::Duration,
) -> Result<Vec<u8>> {
let deadline = std::time::Instant::now() + timeout;
crate::git::visibility_pack::run_bounded_git(
git_bin,
&["cat-file", obj_type, sha256_hex],
repo_path,
b"",
deadline,
)
}
/// Read a git object by its SHA-256 hex object ID.
///
/// Returns `(object_type, content_bytes)` where `content_bytes` is the raw
/// object content (without the git framing header). The CID served over
/// `/ipfs/<cid>` is computed from these same content bytes via
/// `gitlawb_core::cid::Cid::from_git_object_bytes`.
///
/// Returns `None` if the object does not exist in this repo.
pub fn read_object(repo_path: &Path, sha256_hex: &str) -> Result<Option<(String, Vec<u8>)>> {
let obj_type = match object_type(repo_path, sha256_hex)? {
Some(t) => t,
None => return Ok(None),
};
let content = read_object_content(repo_path, sha256_hex, &obj_type)?;
Ok(Some((obj_type, content)))
}
/// Bounded twin of [`read_object`] for write-side callers that must not hang on a
/// wedged `git cat-file` (the post-push pin path, #173): composes
/// [`object_type_bounded`] and [`read_object_content_bounded`] under ONE shared
/// deadline (mirroring `get_by_cid`'s read stage and `build_filtered_pack`), so a
/// single object read holds for at most `timeout` total, not one full timeout per
/// stage. `Ok(Some((type, bytes)))` on success, `Ok(None)` when the object is absent,
/// and `Err(`[`GitServiceTimeout`](crate::git::smart_http::GitServiceTimeout)`)` when
/// either stage overruns the deadline. The bare [`read_object`] is an unbounded
/// `Command::output` an async timeout cannot cancel, so a wedged `cat-file` there pins
/// the post-push coalescing key until process death; this twin cannot.
pub fn read_object_bounded(
git_bin: &str,
repo_path: &Path,
sha256_hex: &str,
timeout: std::time::Duration,
) -> Result<Option<(String, Vec<u8>)>> {
let deadline = std::time::Instant::now() + timeout;
let obj_type = match object_type_bounded(
git_bin,
repo_path,
sha256_hex,
deadline.saturating_duration_since(std::time::Instant::now()),
)? {
Some(t) => t,
None => return Ok(None),
};
let content = read_object_content_bounded(
git_bin,
repo_path,
sha256_hex,
&obj_type,
deadline.saturating_duration_since(std::time::Instant::now()),
)?;
Ok(Some((obj_type, content)))
}
/// Get the diff between two branches: changes on source_branch not in target_branch.
@@ -481,4 +621,84 @@ mod tests {
"unchanged file must not appear: {names:?}"
);
}
/// #173 round-10 (KTD2): `object_type_bounded` reaps a wedged `cat-file` child at its
/// deadline instead of blocking on it to natural exit, so a hung probe cannot pin the
/// /ipfs walk admission the owning task holds. A fake `git` records its pid and sleeps
/// far past the 1s deadline; the `run_bounded_git` watchdog (SIGTERM -> grace ->
/// SIGKILL of the process group) must kill it well before that natural exit, and the
/// call must surface `GitServiceTimeout`. REVERT PROOF (RED): swap the twin's
/// `run_bounded_git` for the bare `Command::output()` and the wedged child stays alive
/// past the deadline — the mid-flight liveness poll below reads it still running.
#[cfg(unix)]
#[test]
fn object_type_bounded_reaps_wedged_child_at_deadline() {
use std::time::Duration;
let tmp = tempfile::TempDir::new().unwrap();
let pidfile = tmp.path().join("catfile.pid");
// `cat-file` records its own pid then sleeps 8s (>> the 1s deadline) so the probe
// is genuinely wedged; the watchdog is what must end it.
let body = format!(
"#!/bin/sh\n\
case \"$1\" in\n\
cat-file) echo $$ > \"{}\"; sleep 8 ;;\n\
*) : ;;\n\
esac\n\
exit 0\n",
pidfile.display()
);
let git_path = tmp.path().join("fakegit");
std::fs::write(&git_path, &body).unwrap();
{
use std::os::unix::fs::PermissionsExt;
let mut perm = std::fs::metadata(&git_path).unwrap().permissions();
perm.set_mode(0o755);
std::fs::set_permissions(&git_path, perm).unwrap();
}
let repo = tmp.path().to_path_buf();
let git = git_path.to_str().unwrap().to_string();
let alive = |pid: i32| unsafe { libc::kill(pid, 0) == 0 };
// The bounded probe blocks until the watchdog tears the child down, so run it on
// a worker thread and poll for the reap from here.
let handle = std::thread::spawn(move || {
super::object_type_bounded(&git, &repo, "deadbeef", Duration::from_secs(1))
});
let mut pid = None;
for _ in 0..500 {
if let Some(p) = std::fs::read_to_string(&pidfile)
.ok()
.and_then(|s| s.trim().parse::<i32>().ok())
{
pid = Some(p);
break;
}
std::thread::sleep(Duration::from_millis(10));
}
let pid = pid.expect("the fake cat-file must have spawned and recorded its pid");
// Past the 1s deadline + SIGTERM grace but well before the 8s natural exit: the
// watchdog must already have reaped the wedged group. A bare, unbounded read would
// leave it running here — the load-bearing RED.
std::thread::sleep(Duration::from_secs(3));
let reaped = !alive(pid);
// Defensive reap so a RED run leaks no orphan.
unsafe {
libc::kill(pid, libc::SIGKILL);
}
assert!(
reaped,
"object_type_bounded must reap the wedged cat-file child at the deadline, \
not leave it running to its natural exit"
);
let res = handle.join().expect("probe thread joins");
let err = res.expect_err("a deadline overrun must be an error, not a value");
assert!(
err.is::<crate::git::smart_http::GitServiceTimeout>(),
"a deadline overrun must surface GitServiceTimeout, got: {err:?}"
);
}
}
File diff suppressed because it is too large. Load diff
+307 -5
View File
@@ -49,13 +49,30 @@ impl QueryRoot {
&self,
ctx: &Context<'_>,
repo: Option<String>,
#[graphql(default = 20)] limit: i64,
#[graphql(
default = 20,
desc = "Max 200; larger requests return the newest 200 rows (no continuation cursor)."
)]
limit: i64,
) -> Result<Vec<RefUpdateType>> {
let db = ctx.data_unchecked::<Arc<Db>>();
let updates = db
.list_ref_updates_filtered(repo.as_deref(), limit)
.await
.map_err(|e| async_graphql::Error::new(e.to_string()))?;
// Gate each row on the same "/" visibility decision the repos resolver
// uses, so anonymous callers get no row for a local repo they can't read
// (#112). The shared collector applies the fail-closed gate *before* the
// limit (paging past dropped private rows) so a small limit still returns
// the latest visible events, and keeps this surface byte-identical to the
// REST feed (#114). The row slug is peer-supplied, so the pure filter
// treats it as untrusted input; remote (no local match) rows pass.
let caller = ctx
.data::<crate::auth::AuthenticatedDid>()
.ok()
.map(|d| d.0.as_str());
let updates =
crate::api::events::collect_visible_ref_updates(db, repo.as_deref(), limit, caller)
.await
.map_err(|e| async_graphql::Error::new(e.to_string()))?;
Ok(updates
.into_iter()
.map(|u| RefUpdateType {
@@ -94,3 +111,288 @@ impl QueryRoot {
Ok(t.map(AgentTaskType::from))
}
}
#[cfg(test)]
mod tests {
use crate::db::{Db, ReceivedRefUpdate, RepoRecord};
use chrono::Utc;
use sqlx::PgPool;
use std::sync::Arc;
const OWNER: &str = "did:key:z6MkOwner";
async fn db(pool: PgPool) -> Arc<Db> {
let db = Db::for_testing(pool);
db.run_migrations().await.unwrap();
Arc::new(db)
}
fn schema(db: Arc<Db>) -> super::super::GitlawbSchema {
let (ref_tx, _) = tokio::sync::broadcast::channel(16);
let (task_tx, _) = tokio::sync::broadcast::channel(16);
super::super::build_schema(db, ref_tx, task_tx)
}
fn repo(id: &str, owner_did: &str, name: &str, is_public: bool) -> RepoRecord {
RepoRecord {
id: id.into(),
name: name.into(),
owner_did: owner_did.into(),
description: None,
is_public,
default_branch: "main".into(),
created_at: Utc::now(),
updated_at: Utc::now(),
disk_path: format!("/srv/{id}"),
forked_from: None,
machine_id: None,
}
}
fn ref_row(id: &str, slug: &str) -> ReceivedRefUpdate {
ReceivedRefUpdate {
id: id.into(),
node_did: "did:key:z6MkNode".into(),
pusher_did: "did:key:z6MkPusher".into(),
repo: slug.into(),
ref_name: "refs/heads/main".into(),
old_sha: "0".repeat(40),
new_sha: "a".repeat(40),
timestamp: Utc::now().to_rfc3339(),
cert_id: None,
received_at: Utc::now().to_rfc3339(),
from_peer: "peer1".into(),
}
}
/// Count `refUpdates` rows in a GraphQL response.
fn count(resp: &async_graphql::Response) -> usize {
assert!(resp.errors.is_empty(), "graphql errors: {:?}", resp.errors);
let async_graphql::Value::Object(obj) = &resp.data else {
panic!("data not an object: {:?}", resp.data);
};
let async_graphql::Value::List(rows) = obj.get("refUpdates").expect("refUpdates key")
else {
panic!("refUpdates not a list");
};
rows.len()
}
async fn anon(schema: &super::super::GitlawbSchema, query: &str) -> async_graphql::Response {
schema.execute(async_graphql::Request::new(query)).await
}
async fn authed(
schema: &super::super::GitlawbSchema,
query: &str,
did: &str,
) -> async_graphql::Response {
schema
.execute(
async_graphql::Request::new(query)
.data(crate::auth::AuthenticatedDid(did.to_string())),
)
.await
}
// Scenario 1 — anon must not get a private local repo's row on the
// repo:Some branch. This is the load-bearing RED→GREEN case.
#[sqlx::test]
async fn ref_updates_private_repo_dropped_for_anon(pool: PgPool) {
let db = db(pool).await;
db.create_repo(&repo("r1", OWNER, "widget", false))
.await
.unwrap();
db.insert_ref_update(&ref_row("u1", "z6MkOwner/widget"))
.await
.unwrap();
let schema = schema(db);
// The GraphQL `repo` arg is the raw slug DB filter, so it must equal the
// stored slug to select the row at all — this is the exact leak path.
let q = r#"{ refUpdates(repo: "z6MkOwner/widget") { refName newSha pusherDid } }"#;
assert_eq!(count(&anon(&schema, q).await), 0);
}
// Scenario 2 — owner still sees their own private repo's row.
#[sqlx::test]
async fn ref_updates_private_repo_kept_for_owner(pool: PgPool) {
let db = db(pool).await;
db.create_repo(&repo("r1", OWNER, "widget", false))
.await
.unwrap();
db.insert_ref_update(&ref_row("u1", "z6MkOwner/widget"))
.await
.unwrap();
let schema = schema(db);
let q = r#"{ refUpdates(repo: "z6MkOwner/widget") { refName } }"#;
assert_eq!(count(&authed(&schema, q, OWNER).await), 1);
}
// Scenario 3 — unfiltered (repo:None): anon gets only the public row.
#[sqlx::test]
async fn ref_updates_unfiltered_anon_gets_only_public(pool: PgPool) {
let db = db(pool).await;
db.create_repo(&repo("pub", OWNER, "openrepo", true))
.await
.unwrap();
db.create_repo(&repo("priv", OWNER, "secret", false))
.await
.unwrap();
db.insert_ref_update(&ref_row("u_pub", "z6MkOwner/openrepo"))
.await
.unwrap();
db.insert_ref_update(&ref_row("u_priv", "z6MkOwner/secret"))
.await
.unwrap();
let schema = schema(db);
let q = r#"{ refUpdates { repo refName } }"#;
let resp = anon(&schema, q).await;
assert_eq!(count(&resp), 1);
// The one row returned must be the public repo's.
let async_graphql::Value::Object(obj) = &resp.data else {
unreachable!()
};
let async_graphql::Value::List(rows) = obj.get("refUpdates").unwrap() else {
unreachable!()
};
let async_graphql::Value::Object(row) = &rows[0] else {
unreachable!()
};
assert_eq!(
row.get("repo").unwrap(),
&async_graphql::Value::from("z6MkOwner/openrepo")
);
}
// Scenario 4 — alias fail-closed: private repo's row stored full-DID form.
#[sqlx::test]
async fn ref_updates_full_did_slug_dropped_for_anon(pool: PgPool) {
let db = db(pool).await;
db.create_repo(&repo("r1", "did:key:zABC", "widget", false))
.await
.unwrap();
db.insert_ref_update(&ref_row("u1", "did:key:zABC/widget"))
.await
.unwrap();
let schema = schema(db);
// repo:None so the slug is not the DB filter key (which is verbatim);
// the gate must still drop it.
let q = r#"{ refUpdates { repo } }"#;
assert_eq!(count(&anon(&schema, q).await), 0);
}
// Scenario 5 — truncated-key fail-closed: 8-char-prefix owner form.
#[sqlx::test]
async fn ref_updates_truncated_key_slug_dropped_for_anon(pool: PgPool) {
let db = db(pool).await;
db.create_repo(&repo("r1", "did:key:zABCDEFGH", "widget", false))
.await
.unwrap();
db.insert_ref_update(&ref_row("u1", "zABCDEF/widget"))
.await
.unwrap();
let schema = schema(db);
let q = r#"{ refUpdates { repo } }"#;
assert_eq!(count(&anon(&schema, q).await), 0);
}
// Scenario 6 — remote slug (no local match) is returned to anon.
#[sqlx::test]
async fn ref_updates_remote_slug_kept_for_anon(pool: PgPool) {
let db = db(pool).await;
db.create_repo(&repo("r1", OWNER, "widget", false))
.await
.unwrap();
// Row whose slug matches no local repo (different owner + name).
db.insert_ref_update(&ref_row("u1", "zZZZOTHER/gadget"))
.await
.unwrap();
let schema = schema(db);
let q = r#"{ refUpdates { repo } }"#;
assert_eq!(count(&anon(&schema, q).await), 1);
}
// Scenario 7 (#114 P2) — a small limit must page past the newest rows when
// they are private, so the older public rows are still returned. Before the
// gate moved ahead of the limit this returned 0 (the newest `limit` rows were
// all private and got filtered after the SQL LIMIT). RED→GREEN.
#[sqlx::test]
async fn ref_updates_small_limit_pages_past_newest_private(pool: PgPool) {
let db = db(pool).await;
db.create_repo(&repo("pub", OWNER, "openrepo", true))
.await
.unwrap();
db.create_repo(&repo("priv", OWNER, "secret", false))
.await
.unwrap();
// 3 older PUBLIC rows …
for i in 0..3 {
let mut r = ref_row(&format!("pub{i}"), "z6MkOwner/openrepo");
r.timestamp = format!("2026-07-01T10:00:0{i}+00:00");
db.insert_ref_update(&r).await.unwrap();
}
// … then 5 NEWER PRIVATE rows (the newest in the feed).
for i in 0..5 {
let mut r = ref_row(&format!("priv{i}"), "z6MkOwner/secret");
r.timestamp = format!("2026-07-01T10:00:1{i}+00:00");
db.insert_ref_update(&r).await.unwrap();
}
let schema = schema(db);
// limit 3 < the 5 newest (all private): anon must still get 3 public rows.
let q = r#"{ refUpdates(limit: 3) { repo } }"#;
let resp = anon(&schema, q).await;
assert_eq!(count(&resp), 3, "paging must reach the older public rows");
let async_graphql::Value::Object(obj) = &resp.data else {
unreachable!()
};
let async_graphql::Value::List(rows) = obj.get("refUpdates").unwrap() else {
unreachable!()
};
for row in rows {
let async_graphql::Value::Object(r) = row else {
unreachable!()
};
assert_eq!(
r.get("repo").unwrap(),
&async_graphql::Value::from("z6MkOwner/openrepo"),
"every returned row must be the public repo's"
);
}
}
// Scenario 8 — a quarantined mirror is withheld on the GraphQL surface too.
// Guards that the resolver keeps delegating to the shared collector (where the
// quarantine fold lives); a REST-only test would miss a resolver that stopped.
#[sqlx::test]
async fn ref_updates_quarantined_mirror_dropped_for_anon(pool: PgPool) {
let db = db(pool).await;
db.upsert_mirror_repo("z6MkQuar", "secret", "/tmp/q", None, true)
.await
.unwrap();
db.insert_ref_update(&ref_row("u1", "z6MkQuar/secret"))
.await
.unwrap();
let schema = schema(db);
let q = r#"{ refUpdates { repo } }"#;
assert_eq!(count(&anon(&schema, q).await), 0);
}
// Scenario 8b — the GraphQL surface also withholds a quarantined repo from an
// authenticated OWNER, not just anon. Without the collector's quarantine drop
// the owner short-circuit in visibility_check keeps the row on this surface
// too, so the REST owner test alone would not guard the resolver.
#[sqlx::test]
async fn ref_updates_quarantined_repo_dropped_for_owner(pool: PgPool) {
let db = db(pool).await;
db.create_repo(&repo("q1", "did:key:z6MkQuar", "secret", false))
.await
.unwrap();
db.set_repo_quarantine("q1", true).await.unwrap();
db.insert_ref_update(&ref_row("u1", "z6MkQuar/secret"))
.await
.unwrap();
let schema = schema(db);
let q = r#"{ refUpdates { repo } }"#;
assert_eq!(count(&authed(&schema, q, "did:key:z6MkQuar").await), 0);
}
}
@@ -11,6 +11,15 @@ pub struct SubscriptionRoot;
#[Subscription]
impl SubscriptionRoot {
/// Live ref-update stream. `/graphql/ws` is mounted outside the
/// `optional_signature` layer, so this resolver has NO caller identity and
/// cannot gate per-subscriber — it relays whatever enters the broadcast
/// channel to any anonymous client. Its visibility safety therefore rests
/// entirely on the WRITE side: the push handler only sends a
/// `RefUpdateBroadcast` for repos the anonymous public may read (inside its
/// `if announce` block, `api/repos.rs`). This is a single-point invariant —
/// any new sender to `ref_update_tx` MUST be `announce`-gated, or private-repo
/// ref metadata leaks here to unauthenticated subscribers (#112/#114 class).
async fn ref_updates(
&self,
ctx: &Context<'_>,
+24 -7
View File
@@ -237,9 +237,13 @@ impl ProofGuard {
match self.0 {
Some(p) => {
if !db.consume_proof_jti(&p.jti, p.exp).await? {
return Err(AppError::IcaptchaProofRequired(
"iCaptcha proof already used (replay); solve a fresh challenge".to_string(),
));
let (url, level) = url_and_level();
return Err(AppError::IcaptchaProofRequired {
message: "iCaptcha proof already used (replay); solve a fresh challenge"
.to_string(),
url,
level,
});
}
Ok(Some(p))
}
@@ -265,10 +269,23 @@ pub fn verify_request(headers: &HeaderMap, did: &str) -> Result<ProofGuard, AppE
}
fn reject_error(v: &Verifier, reason: &str) -> AppError {
AppError::IcaptchaProofRequired(format!(
"iCaptcha proof required ({reason}). Solve a challenge at {} for level >= {} and resend with the {} header.",
v.url, v.required_level, PROOF_HEADER
))
AppError::IcaptchaProofRequired {
message: format!(
"iCaptcha proof required ({reason}). Solve a challenge at {} for level >= {} and resend with the {} header.",
v.url, v.required_level, PROOF_HEADER
),
url: v.url.clone(),
level: v.required_level,
}
}
/// iCaptcha service url + required level for error responses, read from the
/// initialized verifier (falls back to defaults if somehow uninitialized).
fn url_and_level() -> (String, u32) {
VERIFIER
.get()
.map(|v| (v.url.clone(), v.required_level))
.unwrap_or_else(|| ("https://icaptcha.gitlawb.com".to_string(), 3))
}
/// Mode-aware decision. Pure and IO-free (no DB; clock injected via `now`) so it
+544 -13
View File
@@ -9,6 +9,372 @@
use anyhow::Result;
use gitlawb_core::cid::Cid;
use std::time::Duration;
/// Attempts (including the first) for a transient DB-record retry.
const PIN_RECORD_ATTEMPTS: u32 = 3;
/// Backoff between DB-record retry attempts.
const PIN_RECORD_BACKOFF: Duration = Duration::from_millis(50);
/// Run an idempotent DB-record operation with a bounded retry so a sub-second
/// transient error does not silently leave the pin-source set permanently
/// incomplete. The resolver treats a nonempty below-cap source set as complete,
/// so a dropped `record_pin_source`/`record_pinned_cid` makes `GET /ipfs/{cid}`
/// 404 a valid public copy. Every wrapped insert is idempotent (`ON CONFLICT DO
/// NOTHING` / provenance-preserving upsert), so re-running is safe. On exhausted
/// attempts the last error is returned and the caller records the durable
/// `pin_sources_incomplete` marker (U3, #173), which is what keeps the resolver's
/// bounded scan fallback available for that object instead of 404ing a public copy.
/// Shared with the `pinata.rs` twin so both pin paths retry identically. Runs
/// inside the already-detached post-push task, so the backoff adds no push latency.
pub(crate) async fn retry_db_record<F, Fut>(mut op: F) -> Result<()>
where
F: FnMut() -> Fut,
Fut: std::future::Future<Output = Result<()>>,
{
let mut attempt = 1;
loop {
match op().await {
Ok(()) => return Ok(()),
Err(e) => {
if attempt >= PIN_RECORD_ATTEMPTS {
return Err(e);
}
tokio::time::sleep(PIN_RECORD_BACKOFF).await;
attempt += 1;
}
}
}
}
/// Opportunistically repair a legacy provider-CID row on the already-pinned skip
/// path (#173 R8, KTD8). Releases before this branch stored the PROVIDER CID
/// (Kubo dag-pb / Pinata CIDv0) in `pinned_cids.cid`; the `/ipfs` resolver
/// recomputes the raw CID from object bytes and 404s any row whose key does not
/// match, yet `list_pinned_cids` still advertises the stored key — so a client
/// gets a CID the resolver deliberately withholds. When a re-push carries the
/// object again, rewrite the key to the raw CID and stash the old provider value
/// in `legacy_provider_cid`.
///
/// COST GATE: candidacy is decided from the stored key's codec alone — a
/// CIDv1/raw key is already the resolver key and reads NO bytes, keeping the
/// steady-state skip cost DB-only. Only a legacy-codec row reads the object to
/// recompute. A row whose bytes are gone stays withheld (no destructive rewrite).
async fn repair_legacy_provider_cid(
repo_path: &std::path::Path,
git_bin: &str,
git_timeout: Duration,
sha: &str,
db: &crate::db::Db,
) -> Result<RepairOutcome> {
let stored = match db.cid_for_oid(sha).await? {
Some(c) => c,
None => return Ok(RepairOutcome::Settled),
};
// Cost gate: a canonical raw CIDv1 key is already correct — never read bytes.
if gitlawb_core::cid::is_raw_cidv1(&stored) {
return Ok(RepairOutcome::Settled);
}
// Legacy-codec row: read the object to recompute. Counted so a test can prove
// the gate above spares non-legacy rows this read.
#[cfg(test)]
note_legacy_repair_read();
// `read_object_bounded` is SYNCHRONOUS `git cat-file`, and its budget is
// `git_service_timeout_secs` (600 by default), so running it inline parks a tokio
// worker for as long as git takes: one wedged read on the sweep's first pass at boot
// holds a worker for ten minutes, per legacy row. Push it to the blocking pool, the
// same shape `replication_withheld_set` uses in api/repos.rs (#173 round 11, F4).
// Both callers of this function are async, so neither changes shape. The read-counter
// increment above stays on THIS thread so the thread_local cost-gate assertion holds.
let read = {
let repo_path = repo_path.to_path_buf();
let git_bin = git_bin.to_string();
let sha = sha.to_string();
tokio::task::spawn_blocking(move || {
crate::git::store::read_object_bounded(&git_bin, &repo_path, &sha, git_timeout)
})
.await
};
let data = match read {
Ok(Ok(Some((_ty, bytes)))) => bytes,
// Bytes gone: the row stays withheld, never destructively rewritten. Nothing a
// later pass changes, so this is a TERMINAL outcome for the sweep's re-walk gate.
Ok(Ok(None)) => return Ok(RepairOutcome::Settled),
// A wedged/D-state `git cat-file` (timeout/infra): the repair is opportunistic
// and best-effort, so skip it and return Ok so the pin task PROCEEDS to
// requeue_or_release rather than hanging the coalescing key until process death
// (grok F2, #173). A later re-push or the deferred sweep retries the repair.
Ok(Err(e)) => {
tracing::warn!(sha = %sha, err = %e, "skipping legacy provider-CID repair: bounded object read failed");
return Ok(RepairOutcome::Retryable);
}
// The blocking task panicked or was cancelled: same best-effort treatment, and
// worth another walk because it says nothing about the row itself.
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "skipping legacy provider-CID repair: object read task failed");
return Ok(RepairOutcome::Retryable);
}
};
let raw = Cid::from_git_object_bytes(&data).to_string();
if raw == stored {
return Ok(RepairOutcome::Settled);
}
db.repair_legacy_provider_cid(sha, &raw, &stored).await?;
Ok(RepairOutcome::Repaired)
}
/// What one opportunistic repair did with a row, so the sweep can tell a skip a later
/// run could fix from one nothing will (U4 re-walk, #173 round 11). The push skip path
/// ignores the value: it repairs whatever the push happens to carry and a failure there
/// is already warn-only.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum RepairOutcome {
/// Nothing to do, or nothing a re-walk would change: the stored key is already the
/// raw resolver key, the recomputed key matches it, or the object's bytes are gone.
Settled,
/// The bounded object read failed (a wedged `git cat-file`, an unreadable repo).
/// The bytes may be readable later, so the row is worth walking again.
Retryable,
/// The row's key was rewritten to the raw-content CID.
Repaired,
}
/// What one sweep pass (or a whole sweep run) did. `scanned` counts `pinned_cids`
/// rows READ, which is the quantity the batch size bounds; `repaired` counts rows
/// whose key was actually rewritten to the raw CID.
#[derive(Debug, Default, PartialEq, Eq)]
pub(crate) struct SweepStats {
pub scanned: usize,
pub repaired: usize,
pub passes: usize,
/// Rows left unrepaired for a reason a LATER run could fix (the source repo is not
/// on this node's local disk, a DB read failed, a bounded object read failed). A
/// nonzero count is what makes the run rewind its cursor instead of parking it at
/// the end of the table forever. Rows that are unrepairable in principle (no
/// provenance, the repo row is gone, the bytes are gone) are NOT counted here.
pub retryable_skips: usize,
}
/// One bounded pass of the U4 sweep: read at most `batch` `pinned_cids` rows after
/// the persisted cursor, repair the legacy ones, and persist the new cursor.
///
/// The batch is what bounds the pass. It caps rows READ, not rows repaired, because
/// the legacy predicate is a codec decode SQL cannot express; a table of raw rows
/// therefore costs one indexed range scan per pass and nothing else.
///
/// The cursor advances to the LAST row read whatever happened to each row, including
/// rows that were skipped as unrepairable. A cursor that only advanced on success
/// would re-read the same unrepairable row on every pass and the sweep would never
/// reach the rows behind it.
async fn sweep_pass(
repos_dir: &std::path::Path,
git_bin: &str,
git_timeout: Duration,
batch: i64,
db: &crate::db::Db,
) -> Result<SweepStats> {
let cursor = db.pin_repair_cursor().await?;
let rows = db.pinned_cids_after(&cursor, batch).await?;
let scanned = rows.len();
let mut repaired = 0usize;
let mut retryable_skips = 0usize;
let mut last = cursor;
for (sha, stored) in rows {
// Advance FIRST: every path below this line may skip the row, and none of them
// may wedge the walk (scenario 7).
last = sha.clone();
// Same cost gate as the skip-path repair: a canonical raw CIDv1 key is already
// the resolver key, so it reads no bytes and resolves no repo.
if gitlawb_core::cid::is_raw_cidv1(&stored) {
continue;
}
// Resolve the row's repo from its recorded provenance (first-pinner plus the
// bounded additional source set). An empty set is a pin recorded before
// provenance existed: nothing to read the bytes from, so skip it.
let sources = match db.pin_sources_for_oid(&sha).await {
Ok(s) => s,
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "sweep: failed to read pin sources");
// A DB read error says nothing about the row, so a later run retries it.
retryable_skips += 1;
continue;
}
};
// Whether this row ended the source walk repaired, and whether anything it hit
// along the way was a transient obstacle rather than a permanent one.
let mut row_repaired = false;
let mut row_retryable = false;
for repo_id in sources {
let repo = match db.get_repo_by_id(&repo_id).await {
Ok(Some(r)) => r,
// The repo row is gone: a later source may still hold the bytes. A
// deleted repo does not come back, so this is not a retryable skip.
Ok(None) => continue,
Err(e) => {
tracing::warn!(repo_id = %repo_id, err = %e, "sweep: failed to read repo");
row_retryable = true;
continue;
}
};
// Derive the LOCAL disk path rather than going through `repo_store.acquire`.
// The sweep is opportunistic background maintenance over every pinned row on
// the node, so it must never pull a cold repo back from remote storage: that
// would turn a repair pass into a bulk restore. A repo that is not on local
// disk simply reads no bytes here and stays withheld, but it IS a retryable
// skip: on a Tigris-backed node the repo is cold now and warm later, and
// without the re-walk that row would never be repaired by anything.
// The path goes through the repo store's VALIDATED resolver (allowlisted
// components, rooted at repos_dir, no ParentDir/CurDir segment), not the raw
// join: the sweep is a second caller of that path logic and gets the same
// barrier the acquire path has (#173 round 11, F3). It is the non-fetching
// variant, so the no-cold-pull property above is untouched.
let repo_path = match crate::git::repo_store::validated_repo_disk_path(
repos_dir,
&repo.owner_did,
&repo.name,
) {
Ok(p) => p,
// An unsafe name is not something a later run fixes, so it is terminal.
Err(e) => {
tracing::warn!(repo_id = %repo_id, err = %e, "sweep: rejected unsafe repo path");
continue;
}
};
if !repo_path.is_dir() {
row_retryable = true;
continue;
}
match repair_legacy_provider_cid(&repo_path, git_bin, git_timeout, &sha, db).await {
Ok(RepairOutcome::Repaired) => {
repaired += 1;
row_repaired = true;
break;
}
// The bytes could not be read from this source right now: try the next
// source, and if none of them works, walk the row again on a later run.
Ok(RepairOutcome::Retryable) => row_retryable = true,
// Nothing to repair from this source and nothing a re-walk changes.
Ok(RepairOutcome::Settled) => {}
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "sweep: legacy provider-CID repair failed");
row_retryable = true;
}
}
}
if !row_repaired && row_retryable {
retryable_skips += 1;
}
}
db.set_pin_repair_cursor(&last).await?;
Ok(SweepStats {
scanned,
repaired,
passes: 1,
retryable_skips,
})
}
/// Test seam for a single bounded pass (scenarios 4 and 5 drive passes by hand to
/// observe the batch bound and the restart-resumes-from-cursor behavior).
#[cfg_attr(not(test), allow(dead_code))]
pub(crate) async fn sweep_legacy_provider_cids_once(
repos_dir: &std::path::Path,
git_bin: &str,
git_timeout: Duration,
batch: i64,
db: &crate::db::Db,
) -> Result<SweepStats> {
sweep_pass(repos_dir, git_bin, git_timeout, batch, db).await
}
/// U4 (#173): the one-shot legacy provider-CID migration sweep.
///
/// Releases before this branch stored the PROVIDER CID (Kubo dag-pb / Pinata CIDv0)
/// in `pinned_cids.cid`. This branch's `/ipfs/{cid}` resolver recomputes the raw
/// content CID and withholds any row whose key does not match, so those rows are
/// unresolvable. The opportunistic repair on the already-pinned skip path only fires
/// when a later push re-carries the object, and normal git negotiation omits objects
/// the node already has, so on an upgraded node that push generally never comes. This
/// walks the table instead.
///
/// Runs until a pass comes back short of a full batch, which is the end of the table.
/// Sleeps `delay` between full batches so it cannot monopolize the DB, and persists
/// its cursor every pass so a restart continues instead of rewinding. Errors reading
/// or repairing an individual row are warn-and-skip; only a failure of the batch query
/// or the cursor write ends the run, and a later run picks up from the stored cursor.
///
/// A run that skipped at least one RETRYABLE row rewinds the cursor to the start of the
/// table on its way out (#173 round 11). Without that the cursor parked at the maximum
/// `sha256_hex` for good: every later boot read zero rows, so a row skipped for a
/// transient reason (its repo cold on a Tigris-backed node, a DB or object read error)
/// was skipped permanently, unadvertised and unresolvable with nothing left to fix it.
/// The rewind is a per-RUN decision made after the walk has already finished, never
/// mid-walk, so it cannot spin: the cost is one extra ordered scan on the next run, and
/// a row that is unrepairable in principle (bytes gone, provenance gone) does not count
/// as retryable, so a node holding one does not re-walk on every boot forever.
pub(crate) async fn sweep_legacy_provider_cids(
repos_dir: &std::path::Path,
git_bin: &str,
git_timeout: Duration,
batch: i64,
delay: Duration,
db: &crate::db::Db,
) -> SweepStats {
let mut totals = SweepStats::default();
loop {
let pass = match sweep_pass(repos_dir, git_bin, git_timeout, batch, db).await {
Ok(p) => p,
Err(e) => {
tracing::warn!(err = %e, "legacy provider-CID sweep pass failed; stopping");
break;
}
};
totals.scanned += pass.scanned;
totals.repaired += pass.repaired;
totals.retryable_skips += pass.retryable_skips;
totals.passes += 1;
// A short batch means the ordered walk reached the end of the table. Stop here
// rather than after an extra empty pass, and do NOT sleep on the way out.
if (pass.scanned as i64) < batch {
break;
}
tokio::time::sleep(delay).await;
}
if totals.retryable_skips > 0 {
if let Err(e) = db.set_pin_repair_cursor("").await {
tracing::warn!(err = %e, "failed to rewind the legacy provider-CID sweep cursor");
}
}
totals
}
// Test-only cost-gate counter (R8, U7): how many times the opportunistic repair
// read an object's bytes on the skip path. The codec gate must spare a CIDv1/raw
// row this read; the counter is the both-ways guard (removing the gate reads the
// raw row and increments it). Same thread_local discipline as the serve-path
// oversize counter — the pin tests await `pin_new_objects` on a current-thread
// runtime, so the increment and the assertion share one thread.
#[cfg(test)]
thread_local! {
static LEGACY_REPAIR_READS: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
}
#[cfg(test)]
pub(crate) fn reset_legacy_repair_reads() {
LEGACY_REPAIR_READS.with(|c| c.set(0));
}
#[cfg(test)]
pub(crate) fn legacy_repair_reads() -> usize {
LEGACY_REPAIR_READS.with(|c| c.get())
}
#[cfg(test)]
fn note_legacy_repair_read() {
LEGACY_REPAIR_READS.with(|c| c.set(c.get() + 1));
}
/// Pin a single git object to the local IPFS/Kubo node.
///
@@ -91,14 +457,18 @@ pub async fn cat(ipfs_api: &str, cid: &str) -> Result<Vec<u8>> {
/// `..._fail_closed` filter on the full-scan path before calling, so this
/// function never sees a withheld blob. `repo_path` is still needed to read each
/// object's bytes. The twin in `pinata.rs` mirrors this shape — change both in
/// lockstep.
/// lockstep. `repo_id` records the pin's provenance so `GET /ipfs/{cid}` resolves
/// straight to this repo instead of scanning every repo (#173).
///
/// Returns a list of `(sha256_hex, cid)` pairs for objects pinned this call.
pub async fn pin_new_objects(
ipfs_api: &str,
repo_path: &std::path::Path,
git_bin: &str,
git_timeout: Duration,
object_list: Vec<String>,
db: &crate::db::Db,
repo_id: &str,
) -> Vec<(String, String)> {
if ipfs_api.is_empty() {
return vec![];
@@ -107,9 +477,54 @@ pub async fn pin_new_objects(
let mut pinned = Vec::new();
for sha in object_list {
// Skip if already pinned
// Skip if already pinned — but first backfill provenance if the existing
// pin has none. A legacy pin (recorded before repo_id existed, #173, jatmn)
// is skipped here before record_pinned_cid ever runs, so its NULL provenance
// would never resolve to one repo and known CIDs keep hitting the scan. The
// backfill only sets repo_id (AND repo_id IS NULL guard preserves
// first-pinner-owns) and never re-pins the bytes — the object is already on IPFS.
match db.is_pinned(&sha).await {
Ok(true) => continue,
Ok(true) => {
match db.provenance_for_oid(&sha).await {
Ok(None) => {
if let Err(e) = db.backfill_pin_provenance(&sha, repo_id).await {
tracing::warn!(sha = %sha, err = %e, "failed to backfill pin provenance");
}
}
Ok(Some(_)) => {}
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "DB error reading pin provenance");
}
}
// F1 (#173 round 8): record this repo as an ADDITIONAL source for the
// already-pinned object. This is the load-bearing skip-branch insert —
// a later repo pushing a shared object hits this path (already pinned),
// and without it `GET /ipfs/{cid}` only ever knows the first pinner, so a
// shared object first pinned from a private/quarantined repo 404s even
// when this repo would serve it. Bounded per object (MAX_PIN_SOURCES).
if let Err(e) = retry_db_record(|| db.record_pin_source(&sha, repo_id)).await {
tracing::warn!(sha = %sha, err = %e, "failed to record pin source");
// U3 (#173): the retries are spent and this repo is NOT in the source
// set, so the set is known incomplete. Persist that, or the resolver
// reads a non-empty below-cap set as COMPLETE and 404s an object this
// repo would serve. Warn-only in turn: if the marker write also fails
// the object degrades to the pre-U3 behavior, never worse.
if let Err(e) = db.mark_pin_sources_incomplete(&sha).await {
tracing::warn!(sha = %sha, err = %e, "failed to mark pin sources incomplete");
}
}
// R8 (#173 round 10): opportunistically repair a legacy provider-CID
// row (Kubo dag-pb / Pinata) to the raw-content resolver key on this
// re-push. Cost-gated on the stored key's codec — a non-legacy row
// reads no bytes. Warn-only: a failure leaves the row as-is for a
// later re-push or the deferred one-shot sweep.
if let Err(e) =
repair_legacy_provider_cid(repo_path, git_bin, git_timeout, &sha, db).await
{
tracing::warn!(sha = %sha, err = %e, "failed to repair legacy provider CID");
}
continue;
}
Ok(false) => {}
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "DB error checking pinned status");
@@ -117,22 +532,49 @@ pub async fn pin_new_objects(
}
}
// Read raw object content
let data = match crate::git::store::read_object(repo_path, &sha) {
Ok(Some((_obj_type, bytes))) => bytes,
Ok(None) => continue,
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "failed to read git object for pinning");
continue;
}
};
// Read raw object content under a bounded read so a wedged/D-state `git
// cat-file` (stuck NFS/Tigris backend) is reaped at `git_timeout` instead of
// hanging pin_new_objects forever — which would pin the post-push coalescing
// key until process death (grok F2, #173). On Err the object is simply not
// pinned this pass; a later pass/push retries.
let data =
match crate::git::store::read_object_bounded(git_bin, repo_path, &sha, git_timeout) {
Ok(Some((_obj_type, bytes))) => bytes,
Ok(None) => continue,
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "failed to read git object for pinning");
continue;
}
};
// Pin to IPFS
match pin_git_object(ipfs_api, &sha, &data).await {
Ok(cid) if !cid.is_empty() => {
if let Err(e) = db.record_pinned_cid(&sha, &cid).await {
// The resolver key (`pinned_cids.cid`) must be the locally-computed
// raw-content CID, never the provider Hash: Kubo returns a dag-pb/UnixFS
// root for objects above its block size, which does not hash the raw
// content, so `GET /ipfs/{provider_cid}` would resolve then fail the F2
// integrity check (list-then-404). The serve path reads bytes from git and
// verifies them against the requested CID, so the raw CID is the correct
// key. Mirrors the pinata twin, which already records the raw CID.
let raw_cid = gitlawb_core::cid::Cid::from_git_object_bytes(&data).to_string();
// F1 (#173 round 8): the first pinner is recorded in pin_repo_sources too,
// so every source (first and subsequent) is tried uniformly by the
// resolver. U3 (#173): the pin and its source go down in ONE transaction.
// As two independent best-effort calls this path could land the pin while
// dropping its own source, producing a source set silently missing its
// first pinner; atomically there is no such window, and a total failure
// leaves the object unpinned so the next push retries the whole thing.
if let Err(e) =
retry_db_record(|| db.record_pinned_cid_with_source(&sha, &raw_cid, repo_id))
.await
{
tracing::warn!(sha = %sha, err = %e, "failed to record pinned CID in DB");
}
// Return the provider Hash (not the resolver key), mirroring the pinata
// twin's contract: the DB `cid` is the raw resolver key (recorded above),
// the returned value is the provider CID. Here the return is consumed only
// for logging, but keeping the twins structurally identical avoids drift.
pinned.push((sha, cid));
}
Ok(_) => {}
@@ -144,3 +586,92 @@ pub async fn pin_new_objects(
pinned
}
#[cfg(test)]
mod tests {
use super::*;
use std::cell::Cell;
// The retry helper is the load-bearing unit: it converts a sub-second
// transient DB error at the three warn-only record sites into a landed row,
// instead of a permanently incomplete pin-source set. These drive the helper
// directly against a controlled closure (the record sites take a concrete
// `&Db` over a `PgPool`, so a failing-first wrapper cannot slot in without
// changing signatures — see U6 seam note).
#[tokio::test]
async fn retry_lands_after_transient_failures() {
let calls = Cell::new(0u32);
let result = retry_db_record(|| {
let n = calls.get() + 1;
calls.set(n);
async move {
if n < PIN_RECORD_ATTEMPTS {
Err(anyhow::anyhow!("transient failure on attempt {n}"))
} else {
Ok(())
}
}
})
.await;
assert!(
result.is_ok(),
"retry lands the row after transient failures"
);
assert_eq!(
calls.get(),
PIN_RECORD_ATTEMPTS,
"op is retried until it succeeds"
);
}
#[tokio::test]
async fn retry_returns_last_err_after_exhaustion() {
let calls = Cell::new(0u32);
let result = retry_db_record(|| {
let n = calls.get() + 1;
calls.set(n);
async move { Err::<(), _>(anyhow::anyhow!("attempt {n} failed")) }
})
.await;
let err = result.expect_err("all attempts fail so the last error surfaces");
assert_eq!(
calls.get(),
PIN_RECORD_ATTEMPTS,
"attempts are bounded to the cap"
);
assert_eq!(
err.to_string(),
"attempt 3 failed",
"the LAST error is returned, not the first"
);
}
// Happy path against a real DB: a single-attempt success lands the row, and a
// redundant call is idempotent (`ON CONFLICT DO NOTHING`), so the source set
// holds exactly one row.
#[sqlx::test]
async fn retry_records_pin_source_once(pool: sqlx::PgPool) {
let db = crate::db::Db::for_testing(pool);
db.run_migrations().await.unwrap();
let sha = "a".repeat(64);
let repo_id = "repo-retry-1";
retry_db_record(|| db.record_pin_source(&sha, repo_id))
.await
.expect("happy-path record succeeds in one attempt");
retry_db_record(|| db.record_pin_source(&sha, repo_id))
.await
.expect("a redundant record is idempotent");
let sources = db.pin_sources_for_oid(&sha).await.unwrap();
assert_eq!(
sources,
vec![repo_id.to_string()],
"exactly one source row lands under ON CONFLICT DO NOTHING"
);
}
}
File diff suppressed because it is too large. Load diff
+62 -4
View File
@@ -74,8 +74,10 @@ pub async fn pin_object(
/// `..._fail_closed` filter on the full-scan path before calling. `repo_path` is
/// still needed to read each object's bytes. The twin in `ipfs_pin.rs` mirrors
/// this shape — change both in lockstep. Objects already recorded with a
/// `pinata_cid` are skipped. Returns `(sha_hex, cid)` pairs for each newly
/// pinned object.
/// `pinata_cid` are skipped. `repo_id` records the pin's provenance (#173).
/// Returns `(sha_hex, provider_cid)` pairs for each newly pinned object: the
/// provider CID is the Pinata gateway CID (used for branch→CID recording and
/// ref-update gossip), NOT the raw resolver-key CID stored in `pinned_cids.cid`.
pub async fn pin_new_objects(
client: &reqwest::Client,
upload_url: &str,
@@ -83,6 +85,7 @@ pub async fn pin_new_objects(
repo_path: &std::path::Path,
object_list: Vec<String>,
db: &crate::db::Db,
repo_id: &str,
) -> Vec<(String, String)> {
if jwt.is_empty() {
return vec![];
@@ -92,7 +95,39 @@ pub async fn pin_new_objects(
for sha in object_list {
match db.has_pinata_cid(&sha).await {
Ok(true) => continue,
Ok(true) => {
// Backfill NULL first-pinner provenance from a known source, in lockstep
// with the ipfs_pin skip branch: a pinata-only node otherwise leaves
// pre-provenance rows' `pinned_cids.repo_id` NULL forever (grok P2-D). The
// resolver still finds the object via the pin_repo_sources union below, so
// this is a consistency backfill, not a correctness fix.
match db.provenance_for_oid(&sha).await {
Ok(None) => {
if let Err(e) = db.backfill_pin_provenance(&sha, repo_id).await {
tracing::warn!(sha = %sha, err = %e, "failed to backfill pin provenance");
}
}
Ok(Some(_)) => {}
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "DB error reading pin provenance");
}
}
// F1 (#173 round 8): record this repo as an additional source for the
// already-pinned object (mirrors the ipfs_pin skip-branch insert) so the
// resolver can serve a shared object from any pin-path source. U3 (#173):
// retried through the SHARED helper (this was a bare call, so a single
// transient error dropped the source outright) and, on exhaustion, marked
// durably so the resolver keeps the bounded scan fallback for the object.
if let Err(e) =
crate::ipfs_pin::retry_db_record(|| db.record_pin_source(&sha, repo_id)).await
{
tracing::warn!(sha = %sha, err = %e, "failed to record pin source");
if let Err(e) = db.mark_pin_sources_incomplete(&sha).await {
tracing::warn!(sha = %sha, err = %e, "failed to mark pin sources incomplete");
}
}
continue;
}
Ok(false) => {}
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "DB error checking pinata_cid");
@@ -111,9 +146,32 @@ pub async fn pin_new_objects(
match pin_object(client, upload_url, jwt, &sha, &data).await {
Ok(cid) if !cid.is_empty() => {
if let Err(e) = db.record_pinata_cid(&sha, &cid).await {
// The resolver key (`pinned_cids.cid`) must be the locally-computed
// raw-content CID, never the provider CID: Pinata wraps the bytes in
// dag-pb/UnixFS, so its returned CID does not hash the raw content and
// must not become an alias `/ipfs/{cid}` serves raw git bytes for (#173).
let raw_cid = gitlawb_core::cid::Cid::from_git_object_bytes(&data).to_string();
// U3 (#173): both records go through the shared retry helper, at parity
// with the ipfs_pin twin. These were bare calls, so one transient DB error
// permanently dropped a pin source.
if let Err(e) = crate::ipfs_pin::retry_db_record(|| {
db.record_pinata_cid(&sha, &raw_cid, &cid, Some(repo_id))
})
.await
{
tracing::warn!(sha = %sha, err = %e, "failed to record pinata_cid in DB");
}
// F1 (#173 round 8): also record the first pinner in pin_repo_sources.
// U3: an exhausted retry marks the set incomplete so the resolver keeps
// the scan fallback rather than 404ing a copy it could serve.
if let Err(e) =
crate::ipfs_pin::retry_db_record(|| db.record_pin_source(&sha, repo_id)).await
{
tracing::warn!(sha = %sha, err = %e, "failed to record pin source");
if let Err(e) = db.mark_pin_sources_incomplete(&sha).await {
tracing::warn!(sha = %sha, err = %e, "failed to mark pin sources incomplete");
}
}
pinned.push((sha, cid));
}
Ok(_) => {}
+666 -19
View File
@@ -1,15 +1,21 @@
use std::collections::HashMap;
use std::net::SocketAddr;
use std::sync::Arc;
use std::time::{Duration, Instant};
use axum::extract::Request;
use axum::http::StatusCode;
use axum::extract::{ConnectInfo, Request};
use axum::http::{HeaderMap, StatusCode};
use axum::middleware::Next;
use axum::response::{IntoResponse, Response};
use tokio::sync::Mutex;
use crate::auth::AuthenticatedDid;
/// Default ceiling on the number of distinct keys a limiter tracks. Bounds the
/// limiter's own memory so a caller that can vary the key (many DIDs, or a
/// spoofable IP header) cannot grow the map without limit.
const DEFAULT_MAX_KEYS: usize = 100_000;
#[derive(Clone)]
struct Window {
timestamps: Vec<Instant>,
@@ -20,39 +26,121 @@ pub struct RateLimiter {
state: Arc<Mutex<HashMap<String, Window>>>,
max_requests: usize,
window: Duration,
/// Hard cap on tracked keys. When full, expired keys are evicted (at most
/// once per [`sweep_interval`](Self::sweep_interval), see below); if still
/// full, a request under a *new* key is rejected rather than inserted — so
/// the map can never exceed this bound and a rejected request never
/// allocates a new entry.
max_keys: usize,
/// Timestamp of the last inline capacity sweep. The eviction scan is
/// O(max_keys), so under a distinct-key flood (when the map sits at the
/// cap) running it on every miss would serialize all traffic behind a full
/// scan. Gating it to at most once per interval amortizes that cost; the
/// background [`cleanup`](Self::cleanup) loop still reclaims on its own.
last_sweep: Arc<Mutex<Instant>>,
}
impl RateLimiter {
// Retained for tests and callers that don't need a tight key bound;
// production limiters use `new_bounded`.
#[allow(dead_code)]
pub fn new(max_requests: usize, window: Duration) -> Self {
Self::new_bounded(max_requests, window, DEFAULT_MAX_KEYS)
}
/// Like [`new`] but with an explicit cap on the number of distinct keys.
/// Production limiters keyed on client-influenced values (per-DID, per-IP)
/// set this so the limiter's own state cannot be turned into a
/// memory-exhaustion vector.
pub fn new_bounded(max_requests: usize, window: Duration, max_keys: usize) -> Self {
Self {
state: Arc::new(Mutex::new(HashMap::new())),
max_requests,
window,
max_keys: max_keys.max(1),
last_sweep: Arc::new(Mutex::new(Instant::now())),
}
}
async fn check(&self, key: &str) -> bool {
/// How often the inline capacity sweep may run: at most once per second, or
/// once per window when the window is shorter. Bounds post-flood staleness
/// (a full map self-heals within one interval) without paying the O(max_keys)
/// scan on every capacity miss.
fn sweep_interval(&self) -> Duration {
self.window.min(Duration::from_secs(1))
}
pub(crate) async fn check(&self, key: &str) -> bool {
// max_requests == 0 means the limiter is disabled, not "block all".
if self.max_requests == 0 {
return true;
}
let now = Instant::now();
let mut state = self.state.lock().await;
// Look up before inserting so the common case (key already tracked)
// doesn't allocate a String per request.
if !state.contains_key(key) {
state.insert(
key.to_string(),
Window {
timestamps: Vec::new(),
},
);
// Fast path: an already-tracked key never allocates and never grows the
// map, so it is unaffected by the key cap.
if let Some(window) = state.get_mut(key) {
window
.timestamps
.retain(|t| now.duration_since(*t) < self.window);
if window.timestamps.len() >= self.max_requests {
return false;
}
window.timestamps.push(now);
return true;
}
let window = state.get_mut(key).expect("window just ensured");
window
.timestamps
.retain(|t| now.duration_since(*t) < self.window);
if window.timestamps.len() >= self.max_requests {
// New key. Enforce the cap BEFORE inserting so a flood of distinct keys
// cannot grow the map, and a rejected request never allocates an entry.
if state.len() >= self.max_keys {
// Reclaim expired keys, but at most once per sweep interval — the
// scan is O(max_keys) and the map sits at the cap precisely during a
// distinct-key flood, so sweeping per miss would serialize traffic
// behind a full scan. Between sweeps a new key is simply rejected.
let mut last_sweep = self.last_sweep.lock().await;
if now.duration_since(*last_sweep) >= self.sweep_interval() {
state.retain(|_, w| {
w.timestamps
.retain(|t| now.duration_since(*t) < self.window);
!w.timestamps.is_empty()
});
*last_sweep = now;
}
drop(last_sweep);
if state.len() >= self.max_keys {
return false;
}
}
state.insert(
key.to_string(),
Window {
timestamps: vec![now],
},
);
true
}
/// Non-consuming check: is this key ALREADY at its limit for the current window?
/// Unlike [`check`], it records nothing and never inserts a new key — used to shed
/// expensive preparatory work (e.g. the `/ipfs/{cid}` legacy scan's O(repos) DB
/// preload) BEFORE it runs, without perturbing the per-unit budget the consuming
/// `check` maintains (#173, F3). An unknown key or a disabled limiter is not
/// throttled. Prunes the key's expired timestamps as a side effect (keeps state
/// tidy) but adds none, so it cannot itself fill or grow the map.
pub(crate) async fn is_throttled(&self, key: &str) -> bool {
if self.max_requests == 0 {
return false;
}
window.timestamps.push(now);
true
let now = Instant::now();
let mut state = self.state.lock().await;
if let Some(window) = state.get_mut(key) {
window
.timestamps
.retain(|t| now.duration_since(*t) < self.window);
return window.timestamps.len() >= self.max_requests;
}
false
}
pub async fn cleanup(&self) {
@@ -64,6 +152,103 @@ impl RateLimiter {
!w.timestamps.is_empty()
});
}
/// Number of distinct keys currently tracked. Test-only introspection so a
/// cross-module test can assert that a sweep actually evicted expired entries.
#[cfg(test)]
pub(crate) async fn tracked_keys(&self) -> usize {
self.state.lock().await.len()
}
}
/// A bounded per-caller CONCURRENCY limiter — distinct from [`RateLimiter`], which
/// caps request RATE. Each caller key may hold at most `per_caller` in-flight
/// permits at once; beyond that [`try_acquire`](Self::try_acquire) returns `None`
/// and the caller sheds. Used to stop one caller (a single anonymous source-IP or
/// DID) monopolizing the served-git read pool (#174).
///
/// The key map is self-bounding: a key is removed the instant its in-flight count
/// reaches zero, so it never holds more keys than there are concurrently-active
/// callers (itself bounded by the read semaphore). A `max_keys` reject-before-insert
/// backstop guarantees a key farm can never grow the map even if that invariant
/// weakened — a NEW key at the cap is rejected WITHOUT allocating an entry (INV-15).
///
/// Uses a `std::sync::Mutex` (not the file's `tokio::sync::Mutex`) because the
/// permit's `Drop` must release synchronously; the critical section holds no await.
#[derive(Clone)]
pub struct PerCallerConcurrency {
state: Arc<std::sync::Mutex<HashMap<String, usize>>>,
per_caller: usize,
max_keys: usize,
}
/// RAII permit from [`PerCallerConcurrency::try_acquire`]. On drop it decrements
/// the caller's in-flight count and removes the key when it reaches zero.
pub struct PerCallerPermit {
state: Arc<std::sync::Mutex<HashMap<String, usize>>>,
key: String,
}
impl PerCallerConcurrency {
pub fn new(per_caller: usize, max_keys: usize) -> Self {
Self {
state: Arc::new(std::sync::Mutex::new(HashMap::new())),
per_caller: per_caller.max(1),
max_keys: max_keys.max(1),
}
}
/// Convenience constructor with the default key bound.
pub fn with_default_max_keys(per_caller: usize) -> Self {
Self::new(per_caller, DEFAULT_MAX_KEYS)
}
/// `Some(permit)` when the caller is under its cap and the map has room;
/// `None` (shed) otherwise. Reject-before-insert: a new key at `max_keys` is
/// rejected without allocating.
pub fn try_acquire(&self, key: &str) -> Option<PerCallerPermit> {
// Recover from a poisoned lock rather than panicking: the critical section
// is pure counter arithmetic and cannot itself panic, so a poisoned mutex
// would only ever come from an unrelated abort, and a slightly-off count
// self-heals as permits drop. A panic here would instead brick the limiter
// for every caller (each subsequent lock re-panics).
let mut map = self.state.lock().unwrap_or_else(|e| e.into_inner());
match map.get_mut(key) {
Some(count) => {
if *count >= self.per_caller {
return None;
}
*count += 1;
}
None => {
if map.len() >= self.max_keys {
return None;
}
map.insert(key.to_string(), 1);
}
}
Some(PerCallerPermit {
state: self.state.clone(),
key: key.to_string(),
})
}
#[cfg(test)]
pub fn tracked_keys(&self) -> usize {
self.state.lock().unwrap().len()
}
}
impl Drop for PerCallerPermit {
fn drop(&mut self) {
let mut map = self.state.lock().unwrap_or_else(|e| e.into_inner());
if let Some(count) = map.get_mut(&self.key) {
*count -= 1;
if *count == 0 {
map.remove(&self.key);
}
}
}
}
pub async fn rate_limit_by_did(request: Request, next: Next) -> Response {
@@ -88,10 +273,197 @@ pub async fn rate_limit_by_did(request: Request, next: Next) -> Response {
next.run(request).await
}
/// Which forwarded header (if any) the operator's edge is trusted to set. Only
/// a proxy the operator controls may be believed; a raw client can put any
/// value in `Fly-Client-IP` / `X-Forwarded-For`, so trusting them unconditionally
/// lets a flooder rotate the header and never fill a bucket. Configured via
/// `GITLAWB_TRUSTED_PROXY`.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum TrustedProxy {
/// No trusted proxy: ignore forwarded headers, key on the socket peer IP.
/// Safe default for direct/self-hosted nodes.
None,
/// Behind Fly's edge, which sets (and overwrites any client-supplied)
/// `Fly-Client-IP`.
Fly,
/// Behind a single reverse proxy (e.g. Caddy on the AWS image) that appends
/// the real client as the rightmost `X-Forwarded-For` hop.
XForwardedFor,
}
impl TrustedProxy {
/// Parse `GITLAWB_TRUSTED_PROXY`. Unknown/empty → `None` (trust nothing).
pub fn from_env_value(s: &str) -> Self {
match s.trim().to_ascii_lowercase().as_str() {
"fly" | "fly-client-ip" => TrustedProxy::Fly,
"xff" | "x-forwarded-for" | "caddy" => TrustedProxy::XForwardedFor,
_ => TrustedProxy::None,
}
}
}
fn trimmed_nonempty(v: &str) -> Option<String> {
let t = v.trim();
(!t.is_empty()).then(|| t.to_string())
}
/// Resolve the rate-limit key for a request. In a trusted-proxy mode the
/// operator's edge header is preferred; when that header is missing or empty we
/// fall back to the socket peer address rather than skipping the limiter (a
/// malformed header must never disable the brake). With no trusted proxy the
/// socket peer is always used and forwarded headers are ignored entirely.
/// Returns `None` only when neither a trusted header nor a peer address is
/// available (e.g. a synthetic test request with no `ConnectInfo`).
pub fn client_key(
headers: &HeaderMap,
peer: Option<SocketAddr>,
trust: TrustedProxy,
) -> Option<String> {
let from_header = match trust {
TrustedProxy::None => None,
TrustedProxy::Fly => headers
.get("fly-client-ip")
.and_then(|v| v.to_str().ok())
.and_then(trimmed_nonempty),
// Rightmost hop = the value appended by the immediately-upstream trusted
// proxy. The leftmost hop is client-controlled and must not be trusted.
TrustedProxy::XForwardedFor => headers
.get("x-forwarded-for")
.and_then(|v| v.to_str().ok())
.and_then(|v| v.rsplit(',').next())
.and_then(trimmed_nonempty),
};
from_header.or_else(|| peer.map(|p| p.ip().to_string()))
}
/// Per-client-IP limiter for the git push path, carrying the trusted-proxy
/// policy. A newtype so it can live in request extensions (keyed by type)
/// alongside the per-DID [`RateLimiter`]. Per-DID limits are useless against a
/// push flood from a DID farm (one throwaway DID per repo), so the push path
/// throttles on the resolved client IP instead.
#[derive(Clone)]
pub struct IpRateLimiter {
pub limiter: RateLimiter,
pub trust: TrustedProxy,
}
/// Infallible extractor for the socket peer address from `ConnectInfo`. Yields
/// `None` when the server was started without connect-info (e.g. `oneshot` in
/// tests), so a handler never 500s on its absence — the limiter simply falls
/// back per [`client_key`].
pub struct PeerAddr(pub Option<SocketAddr>);
impl<S: Send + Sync> axum::extract::FromRequestParts<S> for PeerAddr {
type Rejection = std::convert::Infallible;
async fn from_request_parts(
parts: &mut axum::http::request::Parts,
_state: &S,
) -> Result<Self, Self::Rejection> {
Ok(PeerAddr(
parts
.extensions
.get::<ConnectInfo<SocketAddr>>()
.map(|c| c.0),
))
}
}
/// The shared 429 response for the per-IP flood brakes. Route-agnostic: this
/// middleware now serves the push path AND the peer-sync routes, so the message
/// stays generic (the offending path is recorded in the warn log below).
pub fn too_many_requests() -> Response {
(
StatusCode::TOO_MANY_REQUESTS,
[("retry-after", "60")],
"rate limit exceeded — try again later",
)
.into_response()
}
/// Throttle the git push path by resolved client IP. The socket peer address is
/// read from `ConnectInfo` (see `into_make_service_with_connect_info` in
/// `main`). Only skips the limiter when no key can be resolved at all.
pub async fn rate_limit_by_ip(request: Request, next: Next) -> Response {
let limiter = request.extensions().get::<IpRateLimiter>().cloned();
let peer = request
.extensions()
.get::<ConnectInfo<SocketAddr>>()
.map(|c| c.0);
if let Some(limiter) = limiter {
if let Some(key) = client_key(request.headers(), peer, limiter.trust) {
if !limiter.limiter.check(&key).await {
tracing::warn!(key = %key, path = %request.uri().path(), "per-IP rate limit exceeded");
return too_many_requests();
}
}
}
next.run(request).await
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn per_caller_concurrency_caps_one_caller_and_frees_on_drop() {
let lim = PerCallerConcurrency::new(2, 100);
let p1 = lim.try_acquire("did:key:zA").expect("first under cap");
let p2 = lim.try_acquire("did:key:zA").expect("second under cap");
assert!(
lim.try_acquire("did:key:zA").is_none(),
"a third in-flight op for the same caller sheds (over the per-caller cap)"
);
// A DIFFERENT caller is unaffected — the cap is per-caller, not global.
let _other = lim
.try_acquire("did:key:zB")
.expect("a different caller has its own budget");
drop(p1);
assert!(
lim.try_acquire("did:key:zA").is_some(),
"freeing one in-flight slot lets the same caller back in"
);
drop(p2);
}
#[test]
fn per_caller_concurrency_map_is_self_bounding_and_reject_before_insert() {
// Self-bounding: acquire+drop many distinct keys — the map never grows
// because a key is removed the instant its in-flight count hits zero.
let lim = PerCallerConcurrency::new(4, 3);
for i in 0..50 {
let _p = lim.try_acquire(&format!("k{i}"));
}
assert_eq!(
lim.tracked_keys(),
0,
"keys with zero in-flight ops are removed, so an acquire+drop flood leaves the map empty"
);
// Reject-before-insert: HOLD max_keys distinct keys, then a new key sheds
// WITHOUT growing the map past the cap (INV-15 — a rejected request never
// allocates an entry).
let held: Vec<_> = (0..3)
.map(|i| lim.try_acquire(&format!("h{i}")).unwrap())
.collect();
assert_eq!(
lim.tracked_keys(),
3,
"three distinct callers held concurrently"
);
assert!(
lim.try_acquire("h3").is_none(),
"a new key at max_keys is rejected"
);
assert_eq!(
lim.tracked_keys(),
3,
"the rejected new key did not allocate an entry (reject-before-insert)"
);
drop(held);
}
#[tokio::test]
async fn allows_within_limit() {
let limiter = RateLimiter::new(3, Duration::from_secs(60));
@@ -134,4 +506,279 @@ mod tests {
let state = limiter.state.lock().await;
assert!(state.is_empty());
}
#[tokio::test]
async fn zero_limit_disables() {
let limiter = RateLimiter::new(0, Duration::from_secs(60));
for _ in 0..1000 {
assert!(limiter.check("k").await);
}
}
// ── key-cap / memory-bound (P2) ──────────────────────────────────────
#[tokio::test]
async fn caps_tracked_keys_and_rejects_new_ones_when_full() {
// Cap of 2 distinct keys; generous request budget so rejection is due to
// the key cap, not the per-key limit.
let limiter = RateLimiter::new_bounded(100, Duration::from_secs(60), 2);
assert!(limiter.check("a").await);
assert!(limiter.check("b").await);
// Third distinct key would grow the map past the cap → rejected.
assert!(!limiter.check("c").await);
// The map never exceeded the cap, and the rejected key was NOT inserted.
let state = limiter.state.lock().await;
assert_eq!(state.len(), 2);
assert!(!state.contains_key("c"));
}
#[tokio::test]
async fn known_key_unaffected_by_cap() {
let limiter = RateLimiter::new_bounded(100, Duration::from_secs(60), 1);
assert!(limiter.check("a").await); // fills the single slot
assert!(limiter.check("a").await); // same key still served
assert!(!limiter.check("b").await); // new key rejected — cap full
}
#[tokio::test]
async fn expired_keys_evicted_to_admit_new_when_full() {
let limiter = RateLimiter::new_bounded(100, Duration::from_millis(40), 1);
assert!(limiter.check("old").await);
tokio::time::sleep(Duration::from_millis(55)).await;
// "old" is now expired; a new key triggers inline eviction and is admitted.
assert!(limiter.check("new").await);
let state = limiter.state.lock().await;
assert!(state.contains_key("new"));
assert!(!state.contains_key("old"));
}
#[tokio::test]
async fn capacity_sweep_is_amortized_within_interval() {
// The O(max_keys) eviction scan must not run on every capacity miss.
// With a 1s sweep interval, the resident (unexpired) key is never
// evicted by a burst of misses, so repeated new keys are all rejected
// without disturbing existing state.
let limiter = RateLimiter::new_bounded(100, Duration::from_secs(1), 1);
assert!(limiter.check("resident").await);
for i in 0..100 {
assert!(
!limiter.check(&format!("flood-{i}")).await,
"new key must be rejected while the single slot is held"
);
}
let state = limiter.state.lock().await;
assert_eq!(state.len(), 1, "no flood key was ever inserted");
assert!(state.contains_key("resident"));
}
// ── client_key / trusted-proxy resolution (P1 + P2) ─────────────────
fn headers(pairs: &[(&str, &str)]) -> HeaderMap {
let mut h = HeaderMap::new();
for (k, v) in pairs {
h.insert(
axum::http::HeaderName::from_bytes(k.as_bytes()).unwrap(),
v.parse().unwrap(),
);
}
h
}
fn peer(s: &str) -> Option<SocketAddr> {
Some(s.parse().unwrap())
}
#[test]
fn none_mode_ignores_headers_and_uses_peer() {
// Even a well-formed Fly header is ignored without a trusted proxy.
let h = headers(&[("fly-client-ip", "203.0.113.7")]);
assert_eq!(
client_key(&h, peer("198.51.100.9:4000"), TrustedProxy::None).as_deref(),
Some("198.51.100.9")
);
}
#[test]
fn fly_mode_trusts_fly_header() {
let h = headers(&[
("fly-client-ip", "203.0.113.7"),
("x-forwarded-for", "1.2.3.4"),
]);
assert_eq!(
client_key(&h, peer("10.0.0.1:1"), TrustedProxy::Fly).as_deref(),
Some("203.0.113.7")
);
}
#[test]
fn fly_mode_empty_header_falls_back_to_peer_not_skip() {
// Empty Fly-Client-IP must NOT collapse traffic onto Some("") nor skip
// the limiter — it falls back to the real peer.
let h = headers(&[("fly-client-ip", "")]);
assert_eq!(
client_key(&h, peer("198.51.100.9:4000"), TrustedProxy::Fly).as_deref(),
Some("198.51.100.9")
);
}
#[test]
fn xff_mode_uses_rightmost_hop_not_client_controlled_first() {
// Client prepends spoofed hops; only the rightmost (proxy-appended) is trusted.
let h = headers(&[("x-forwarded-for", "9.9.9.9, 8.8.8.8, 198.51.100.9")]);
assert_eq!(
client_key(&h, peer("10.0.0.1:1"), TrustedProxy::XForwardedFor).as_deref(),
Some("198.51.100.9")
);
}
#[test]
fn xff_mode_empty_leading_hop_does_not_disable_brake() {
// "X-Forwarded-For: ,1.2.3.4" — rightmost hop is used; never None-skips.
let h = headers(&[("x-forwarded-for", ",1.2.3.4")]);
assert_eq!(
client_key(&h, peer("10.0.0.1:1"), TrustedProxy::XForwardedFor).as_deref(),
Some("1.2.3.4")
);
}
#[test]
fn malformed_xff_falls_back_to_peer() {
let h = headers(&[("x-forwarded-for", " , ")]);
assert_eq!(
client_key(&h, peer("198.51.100.9:4000"), TrustedProxy::XForwardedFor).as_deref(),
Some("198.51.100.9")
);
}
#[test]
fn trusted_proxy_parsing() {
assert_eq!(TrustedProxy::from_env_value("fly"), TrustedProxy::Fly);
assert_eq!(
TrustedProxy::from_env_value("X-Forwarded-For"),
TrustedProxy::XForwardedFor
);
assert_eq!(
TrustedProxy::from_env_value("caddy"),
TrustedProxy::XForwardedFor
);
assert_eq!(TrustedProxy::from_env_value(""), TrustedProxy::None);
assert_eq!(TrustedProxy::from_env_value("garbage"), TrustedProxy::None);
}
// ── middleware 429 path ─────────────────────────────────────────────
/// A minimal router with the push limiter layered over an OK handler,
/// driven via `oneshot`. `ConnectInfo` is attached to each request directly
/// (as the production make-service does) so the middleware resolves a peer.
fn ip_limited_router(limiter: IpRateLimiter) -> axum::Router {
axum::Router::new()
.route(
"/o/r/git-receive-pack",
axum::routing::post(|| async { StatusCode::OK }),
)
.layer(axum::middleware::from_fn(rate_limit_by_ip))
.layer(axum::Extension(limiter))
}
async fn post_from(router: &axum::Router, peer: SocketAddr) -> StatusCode {
use tower::ServiceExt;
let mut req = axum::http::Request::builder()
.method(axum::http::Method::POST)
.uri("/o/r/git-receive-pack")
.body(axum::body::Body::empty())
.unwrap();
req.extensions_mut().insert(ConnectInfo(peer));
router.clone().oneshot(req).await.unwrap().status()
}
#[tokio::test]
async fn middleware_returns_429_over_limit() {
let router = ip_limited_router(IpRateLimiter {
limiter: RateLimiter::new(2, Duration::from_secs(60)),
trust: TrustedProxy::None,
});
let peer: SocketAddr = "203.0.113.7:5000".parse().unwrap();
// Two requests inside the budget, the third over it (shared Arc state).
assert_eq!(post_from(&router, peer).await, StatusCode::OK);
assert_eq!(post_from(&router, peer).await, StatusCode::OK);
assert_eq!(
post_from(&router, peer).await,
StatusCode::TOO_MANY_REQUESTS
);
}
#[tokio::test]
async fn middleware_isolates_distinct_peers() {
let router = ip_limited_router(IpRateLimiter {
limiter: RateLimiter::new(1, Duration::from_secs(60)),
trust: TrustedProxy::None,
});
let a: SocketAddr = "203.0.113.1:1".parse().unwrap();
let b: SocketAddr = "203.0.113.2:1".parse().unwrap();
assert_eq!(post_from(&router, a).await, StatusCode::OK);
assert_eq!(post_from(&router, b).await, StatusCode::OK); // independent bucket
assert_eq!(post_from(&router, a).await, StatusCode::TOO_MANY_REQUESTS);
}
// ── creation-route layering: IP brake runs BEFORE auth ──────────────
/// A minimal stand-in for the creation route group: an inner "auth" layer
/// that rejects every request with 401 (as signature verification would for
/// an unsigned caller), wrapped by the per-IP creation limiter as the
/// outermost layer — exactly the ordering `server::build_router` applies to
/// `/api/v1/repos` et al. Lets us assert the security-relevant property
/// (flood traffic is braked before auth burns CPU) without a database, which
/// the full `#[sqlx::test]` route test in `api::repos` cannot cover in the
/// plain unit-test pass.
fn ip_limited_over_auth_router(limiter: IpRateLimiter) -> axum::Router {
async fn always_unauthorized(_req: Request, _next: Next) -> Response {
StatusCode::UNAUTHORIZED.into_response()
}
axum::Router::new()
.route(
"/api/v1/repos",
axum::routing::post(|| async { StatusCode::CREATED }),
)
.layer(axum::middleware::from_fn(always_unauthorized))
.layer(axum::middleware::from_fn(rate_limit_by_ip))
.layer(axum::Extension(limiter))
}
async fn post_repos_from(router: &axum::Router, peer: SocketAddr) -> StatusCode {
use tower::ServiceExt;
let mut req = axum::http::Request::builder()
.method(axum::http::Method::POST)
.uri("/api/v1/repos")
.body(axum::body::Body::empty())
.unwrap();
req.extensions_mut().insert(ConnectInfo(peer));
router.clone().oneshot(req).await.unwrap().status()
}
#[tokio::test]
async fn creation_ip_brake_rejects_before_auth() {
// A DID farm mints a fresh signature per repo, so the per-DID limiter
// never trips; the per-IP brake is what stops the flood. It must run
// outermost, ahead of auth, so a flood is rejected with 429 before
// signature verification — otherwise every request would reach (and pay
// for) auth and merely 401.
let router = ip_limited_over_auth_router(IpRateLimiter {
limiter: RateLimiter::new(1, Duration::from_secs(60)),
trust: TrustedProxy::None,
});
let peer: SocketAddr = "203.0.113.42:7000".parse().unwrap();
// First request passes the IP brake and reaches the (failing) auth layer.
assert_eq!(
post_repos_from(&router, peer).await,
StatusCode::UNAUTHORIZED
);
// Budget now exhausted: the IP brake short-circuits with 429 *before*
// auth — proving the limiter is the outermost layer.
assert_eq!(
post_repos_from(&router, peer).await,
StatusCode::TOO_MANY_REQUESTS,
"an exhausted IP must be braked with 429 before auth runs, not leak to 401"
);
}
}
+88 -5
View File
@@ -81,8 +81,19 @@ pub fn build_router(state: AppState) -> Router {
.route("/api/v1/tasks", get(tasks::list_tasks))
.route("/api/v1/tasks/{id}", get(tasks::get_task));
// ── Rate-limited creation routes — require HTTP Signature + per-DID throttle
// ── Rate-limited creation routes — require HTTP Signature, plus a per-DID
// throttle AND a per-IP flood brake. The per-DID limiter (inner) caps a
// single identity; the per-IP limiter (outer) caps a DID farm that mints a
// fresh throwaway did:key per repo to slip past both the per-DID limit and
// the iCaptcha gate — the mechanism behind the recurring spam-repo floods.
// The per-IP layer wraps the auth layer (outermost = runs first) so flood
// traffic is rejected before signature verification burns CPU, matching the
// push path.
let limiter = state.rate_limiter.clone();
let create_ip_limiter = rate_limit::IpRateLimiter {
limiter: state.create_ip_rate_limiter.clone(),
trust: state.push_limiter_trust,
};
let creation_routes = add_auth_layers(
Router::new()
.route("/api/v1/repos", post(repos::create_repo))
@@ -96,7 +107,9 @@ pub fn build_router(state: AppState) -> Router {
.layer(middleware::from_fn(rate_limit::rate_limit_by_did))
.layer(axum::Extension(limiter)),
state.clone(),
);
)
.layer(middleware::from_fn(rate_limit::rate_limit_by_ip))
.layer(axum::Extension(create_ip_limiter));
// ── Write routes — require HTTP Signature (no rate limit) ─────────────
let write_routes = add_auth_layers(
@@ -175,6 +188,14 @@ pub fn build_router(state: AppState) -> Router {
// HTTP Signature is enforced on receive-pack (push) — the git-remote-gitlawb
// helper signs requests with RFC 9421 signatures using the agent's keypair.
let pack_limit = state.config.max_pack_bytes;
// Per-IP throttle wraps the auth layer (outermost = runs first): flood
// traffic is rejected before signature verification burns CPU. Per-DID
// limiting is deliberately NOT used here — a DID farm (one throwaway
// identity per repo, as in the June 2026 push flood) never trips it.
let push_limiter = rate_limit::IpRateLimiter {
limiter: state.push_rate_limiter.clone(),
trust: state.push_limiter_trust,
};
let git_write_routes = add_auth_layers(
Router::new()
.route(
@@ -184,16 +205,29 @@ pub fn build_router(state: AppState) -> Router {
.layer(DefaultBodyLimit::disable())
.layer(RequestBodyLimitLayer::new(pack_limit)),
state.clone(),
);
)
.layer(middleware::from_fn(rate_limit::rate_limit_by_ip))
.layer(axum::Extension(push_limiter));
// ── IPFS content-addressed retrieval and pin listing ──────────────────
// `/ipfs/{cid}` carries `optional_signature` so `get_by_cid` sees the caller
// identity and can apply per-repo visibility (#110); anonymous callers stay
// anonymous and still read genuinely public content. `/api/v1/ipfs/pins`
// stays unsigned — gating the pin index is tracked separately (#121).
// `/ipfs/{cid}` also carries a per-IP flood brake: it is anon-reachable and each
// request can drive a full-history git walk, so the per-IP rate limiter is the
// outermost layer (rejects a flood before the walk-admission work), mirroring the
// push/create routers. The extension MUST be attached or rate_limit_by_ip is a
// silent no-op. `/api/v1/ipfs/pins` (no walk) is merged in unbraked, as before.
let ipfs_limiter = rate_limit::IpRateLimiter {
limiter: state.ipfs_rate_limiter.clone(),
trust: state.push_limiter_trust,
};
let ipfs_routes = Router::new()
.route("/ipfs/{cid}", get(ipfs::get_by_cid))
.layer(middleware::from_fn(auth::optional_signature))
.layer(middleware::from_fn(rate_limit::rate_limit_by_ip))
.layer(axum::Extension(ipfs_limiter))
.merge(Router::new().route("/api/v1/ipfs/pins", get(ipfs::list_pins)));
// ── Arweave permanent anchors ──────────────────────────────────────────
@@ -238,7 +272,8 @@ pub fn build_router(state: AppState) -> Router {
.route(
"/api/v1/agents/{did}/bounties",
get(bounties::agent_bounty_stats),
);
)
.layer(middleware::from_fn(auth::optional_signature));
// ── Profile routes (write — require HTTP Signature) ─────────────────
let profile_write_routes = add_auth_layers(
@@ -267,15 +302,41 @@ pub fn build_router(state: AppState) -> Router {
.route("/api/v1/peers", get(peers::list_peers))
.route("/api/v1/peers/{did}/ping", get(peers::ping_peer));
// /sync/trigger drives an O(peers) outbound fan-out + per-repo enqueue, so it
// ALWAYS requires a signature (both config modes) and carries a tight per-IP
// brake. A signature alone does not cap cost — a did:key farm self-registers
// (INV-10) — so the IP brake is a separate, load-bearing half. The brake is
// outermost (runs before signature verification burns CPU) and is keyed on the
// client IP before any DID is read, so DID rotation cannot bypass it.
let sync_trigger_routes = add_auth_layers(
Router::new().route("/api/v1/sync/trigger", post(peers::trigger_sync)),
state.clone(),
)
.layer(middleware::from_fn(rate_limit::rate_limit_by_ip))
.layer(axum::Extension(rate_limit::IpRateLimiter {
limiter: state.sync_trigger_rate_limiter.clone(),
trust: state.push_limiter_trust,
}));
// announce + notify keep their rolling-upgrade signature behavior (unsigned
// accepted until all peers upgrade), but both reach peer-write side effects —
// notify hits the same enqueue_sync sink as trigger — so they carry a per-IP
// brake too, on a SEPARATE bucket from trigger's, so an unsigned notify flood
// cannot drain the signed trigger caller's quota.
let mut peer_write_routes = Router::new()
.route("/api/v1/peers/announce", post(peers::announce))
.route("/api/v1/sync/trigger", post(peers::trigger_sync))
.route("/api/v1/sync/notify", post(peers::notify_sync));
peer_write_routes = if state.config.require_signed_peer_writes {
add_auth_layers(peer_write_routes, state.clone())
} else {
peer_write_routes.layer(middleware::from_fn(auth::optional_signature))
};
let peer_write_routes = peer_write_routes
.layer(middleware::from_fn(rate_limit::rate_limit_by_ip))
.layer(axum::Extension(rate_limit::IpRateLimiter {
limiter: state.peer_write_rate_limiter.clone(),
trust: state.push_limiter_trust,
}));
// ── Read routes — open for public repos ───────────────────────────────
let read_routes = Router::new()
@@ -405,6 +466,7 @@ pub fn build_router(state: AppState) -> Router {
let meta_routes = Router::new()
.route("/", get(node_info))
.route("/health", get(health))
.route("/ready", get(ready))
.route("/api/v1/p2p/info", get(p2p_info))
.route("/api/v1/stats", get(stats))
.route("/api/v1/contracts", get(contracts_info));
@@ -424,6 +486,7 @@ pub fn build_router(state: AppState) -> Router {
.merge(read_routes)
.merge(peer_read_routes)
.merge(peer_write_routes)
.merge(sync_trigger_routes)
.merge(ipfs_routes)
.merge(arweave_routes)
.merge(meta_routes)
@@ -446,6 +509,26 @@ async fn health() -> Json<serde_json::Value> {
Json(json!({ "status": "ok" }))
}
/// Readiness = "this node can serve real traffic", and every real endpoint
/// depends on the database, so probe it rather than reporting a constant.
/// The probe gets its own short bound so a wedged pool can't hang the health
/// check for the full acquire timeout.
async fn ready(State(state): State<AppState>) -> axum::response::Response {
let probe = tokio::time::timeout(std::time::Duration::from_secs(2), state.db.ping()).await;
match probe {
Ok(Ok(())) => Json(json!({ "status": "ready" })).into_response(),
_ => (
axum::http::StatusCode::SERVICE_UNAVAILABLE,
Json(json!({
"status": "degraded",
"error": crate::error::DB_UNAVAILABLE_CODE,
"message": crate::error::DB_UNAVAILABLE_MESSAGE,
})),
)
.into_response(),
}
}
async fn node_info(State(state): State<AppState>) -> Json<serde_json::Value> {
let p2p_peer_id = state.p2p.as_ref().map(|h| h.local_peer_id.to_string());
Json(json!({
+340
View File
@@ -51,6 +51,74 @@ pub struct AppState {
pub repo_store: RepoStore,
/// Per-DID rate limiter for creation endpoints (repos, issues, PRs)
pub rate_limiter: RateLimiter,
/// Per-client-IP rate limiter for the same creation endpoints. The per-DID
/// limiter above cannot brake a creation flood from a DID farm — one
/// throwaway `did:key` per repo means each DID makes a single create call
/// and never trips its own bucket. A valid iCaptcha proof does not cap this
/// either: the enforced level draws only machine-solvable deterministic
/// challenges (and the caller can pin the easy type), so a bot mints a fresh
/// DID, solves a proof, and creates a repo unthrottled. Braking on the
/// resolved client IP is what actually stops a single-source flood (same
/// rationale as `push_rate_limiter`). Keyed by `push_limiter_trust`.
pub create_ip_rate_limiter: RateLimiter,
/// Per-client-IP rate limiter for git-receive-pack. Per-DID limits cannot
/// brake a push flood from a DID farm (one throwaway DID per repo), so the
/// push path throttles on the resolved client IP instead.
pub push_rate_limiter: RateLimiter,
/// Per-client-IP ROUTE brake for `GET /ipfs/{cid}`: charged ONCE per request by the
/// `rate_limit_by_ip` middleware (server.rs), never inside the handler. It bounds
/// request RATE (the "requests per hour" contract of `GITLAWB_IPFS_RATE_LIMIT`) on
/// the non-farmable source IP, so an anonymous flood of the public route is capped.
/// The per-probe/per-walk WORK accounting the resolver does WITHIN a request draws
/// from the SEPARATE `ipfs_work_rate_limiter` below — the two cannot share one bucket
/// or a single request that spends a route token and then its own probe token off the
/// same bucket is admitted at the route and falsely shed mid-request (#173 round-10,
/// R6). Keyed by `push_limiter_trust`.
pub ipfs_rate_limiter: RateLimiter,
/// Per-client-IP WORK-budget limiter for the `GET /ipfs/{cid}` resolver's internal
/// fan-out: charged per legacy (NULL-provenance) PROBE (`acquire` + `cat-file`) and
/// per provenance-path WALK, and peeked non-consuming before the O(repos) legacy
/// preload. A legacy CID from the public pins index otherwise lets one request drive
/// O(repos) subprocess spawns and cold Tigris fetches, and repeat requests amplify
/// that across requests with zero limiter contact (INV-10, F3). Charging the work to
/// the non-farmable source IP bounds it. A bucket DISTINCT from the route brake above:
/// one request legitimately spends many work tokens (a full legacy scan is up to
/// `ipfs_max_legacy_probes` probes), so it must not double as the once-per-request
/// route bucket. Capacity is DERIVED from the route limit (`AppState::ipfs_work_budget`,
/// no separate operator knob), floored at the legacy-probe budget so a single default-
/// config deep search never self-throttles mid-scan. Keyed by `push_limiter_trust`.
pub ipfs_work_rate_limiter: RateLimiter,
/// Per-request ceiling on full-history reachability walks the CID resolver
/// may spawn (default `api::ipfs::MAX_HISTORY_WALKS_PER_REQUEST`). A field,
/// not a bare const, so tests can shrink it to exercise the cap cheaply;
/// production keeps the const default.
pub ipfs_max_history_walks: u32,
/// Per-request ceiling on legacy (NULL-provenance) repo probes in the CID
/// resolver's scan fallback (default `api::ipfs::MAX_LEGACY_PROBES_PER_REQUEST`).
/// Bounds the anonymous `acquire` + `cat-file` fan-out across the node (#173,
/// INV-10); a field for the same test-seam reason as `ipfs_max_history_walks`.
pub ipfs_max_legacy_probes: u32,
/// Hard ceiling on the byte size of an object `GET /ipfs/{cid}` will buffer and
/// serve (default `api::ipfs::MAX_SERVED_OBJECT_BYTES`). The serve reads via a
/// blocking `git cat-file` and buffers the whole object; without a bound a large
/// public blob could exhaust memory or block a runtime worker (#173, F6, INV-10).
/// A field for the same test-seam reason as the sibling caps.
pub ipfs_max_served_object_bytes: u64,
/// Which forwarded header (if any) the edge is trusted to set, for
/// resolving the push limiter's client-IP key. See `GITLAWB_TRUSTED_PROXY`.
/// Node-wide; also keys the two peer-sync limiters below.
pub push_limiter_trust: crate::rate_limit::TrustedProxy,
/// Per-client-IP limiter for `POST /api/v1/sync/trigger` (tight). The route
/// requires a signature, but a signature does not cap cost (a did:key farm
/// self-registers), and its per-call cost is an O(peers) fan-out, so the IP
/// brake is a separate, load-bearing half. Its own bucket so an unsigned
/// `/sync/notify` flood cannot drain the signed trigger caller's quota.
pub sync_trigger_rate_limiter: RateLimiter,
/// Per-client-IP limiter for the peer-write routes (`/peers/announce`,
/// `/sync/notify`) (generous). `/sync/notify` reaches the same `enqueue_sync`
/// sink as trigger and accepts unsigned requests from known peers, so it is
/// braked too; each peer's distinct IP gets its own bucket.
pub peer_write_rate_limiter: RateLimiter,
/// Process-wide graceful-shutdown signal. Sending `true` causes every
/// task that holds a `watch::Receiver` to exit at its next await point.
/// Used by:
@@ -59,6 +127,94 @@ pub struct AppState {
/// * the libp2p swarm task
/// * the gossip, sync, operator heartbeat, and rate-limit cleanup loops
pub shutdown_tx: tokio::sync::watch::Sender<bool>,
/// Bounds concurrent served git READ operations (upload-pack + both info/refs
/// advertisements). A read handler acquires a permit before spawning git and
/// holds it for the op; when none are free the request is shed with a 503.
/// Writes draw from `git_write_semaphore` so a read flood cannot shed an
/// authenticated push at admission (#174).
pub git_read_semaphore: Arc<tokio::sync::Semaphore>,
/// Bounds concurrent `git-receive-pack` (push) operations, a pool separate
/// from `git_read_semaphore` so an anonymous READ flood can never shed an
/// authenticated push (#174). Sized by `max_concurrent_git_pushes`. Drawn from
/// by the `git-receive-pack` POST (owner-gated) ONLY. The anon-reachable
/// receive-pack `info/refs` advertisement draws from the SEPARATE
/// `git_push_advert_semaphore` below, never this pool, so a multi-source flood
/// of push-handshake advertisements can never occupy a permit an authenticated
/// POST needs at admission (#174).
pub git_write_semaphore: Arc<tokio::sync::Semaphore>,
/// Bounds concurrent anon-reachable `git-receive-pack` `info/refs`
/// advertisements — a pool SEPARATE from `git_write_semaphore` so adverts (which
/// hold a permit across `acquire_fresh` + `info/refs`) can never consume a slot
/// the authenticated POST relies on. A per-source flood can at worst exhaust this
/// advert pool (each source also capped by `git_push_advert_per_caller` and the
/// per-IP push rate limiter), and the reserved POST pool is untouched (#174).
pub git_push_advert_semaphore: Arc<tokio::sync::Semaphore>,
/// Bounds concurrent post-push encrypt-then-pin history walks. Each successful
/// path-scoped push releases its handler write permit and then runs a DETACHED
/// full-history walk (`withheld_blob_recipients_bounded`) to seal withheld blobs;
/// without a cap, N fast pushes spawn N concurrent full-history git walks past
/// `max_concurrent_git_pushes` (which only bounds the in-handler phase). The walk
/// acquires a permit here and DEFERS (blocks) when the pool is full rather than
/// shedding — the work is background and dropping it would lose the recovery copy
/// (#174 P1-e). A pool of its own, not `git_write_semaphore`: a long background
/// walk must not hold a foreground write slot, and a handler already holding a
/// write permit that needed a second would self-deadlock at pool size 1.
pub git_encrypt_semaphore: Arc<tokio::sync::Semaphore>,
/// Bounds the OUTSTANDING post-push encryption-task set by per-repo coalescing
/// (#174 P2-2). `git_encrypt_semaphore` caps *active* walks; this caps how many
/// detached tasks *spawn and park* on that semaphore's `acquire_owned().await`.
/// Before spawning a per-push encryption task, the receive-pack handler consults
/// this set: if the repo already has a task in flight it coalesces (skips the
/// duplicate spawn) rather than parking a new unbounded waiter. Coalescing only
/// delays a duplicate walk — it never drops the withheld-blob recovery copy, which
/// `2a54c15` deliberately kept fail-closed (there is no reconciliation sweep to
/// re-derive a dropped copy). See [`EncryptInflight`].
pub encrypt_inflight: EncryptInflight,
/// Per-caller concurrency sub-cap on the read pool: each caller (keyed on the
/// resolved source IP, #174 U1) may hold at most `max_concurrent_reads_per_caller`
/// in-flight read ops, so one caller cannot monopolize `git_read_semaphore`
/// (#174). Applied by `git_upload_pack` and the upload-pack `info/refs`
/// advertisement.
pub git_read_per_caller: crate::rate_limit::PerCallerConcurrency,
/// Per-source concurrency sub-cap on the anon-reachable receive-pack `info/refs`
/// advertisement: each source IP may hold at most a small share of the write
/// pool, so a multi-source flood of push-handshake advertisements cannot
/// saturate `git_write_semaphore` and shed authenticated pushes (#174). Sized as
/// a fraction of `max_concurrent_git_pushes`, so filling the write pool takes many
/// distinct source IPs (each also braked by the per-IP push rate limiter).
pub git_push_advert_per_caller: crate::rate_limit::PerCallerConcurrency,
/// Per-source concurrency sub-cap on the authenticated `git-receive-pack` POST:
/// each source IP may hold at most a small share of `git_write_semaphore`, so one
/// host minting disposable `did:key` identities cannot open enough slow pushes to
/// monopolize the write pool and 503 every other source's push (#174 P1-d). Keyed
/// on the resolved source IP (never the DID — a DID farm defeats a DID key). Sized
/// like `git_push_advert_per_caller`, a fraction of `max_concurrent_git_pushes`.
pub git_write_per_caller: crate::rate_limit::PerCallerConcurrency,
/// Bounds concurrent `GET /ipfs/{cid}` visibility-walk requests. The public
/// `/ipfs/{cid}` route runs `allowed_blob_set_for_caller_bounded` in
/// `spawn_blocking` (a full-history git walk) with NO served-git admission of its
/// own; without this a permissionless caller fans out concurrent walks past every
/// git pool, exhausting the blocking pool + PIDs (#174 P1-3). A request acquires a
/// permit before the repo loop and holds it for the whole request (across every
/// `spawn_blocking` walk), so the slot reflects real thread occupancy — a tokio
/// walk-timeout cannot free it while the blocking work still runs. A pool of its
/// own (`max_concurrent_ipfs_walks`), NOT a git pool: distinct cost center + public
/// surface, so anonymous /ipfs traffic can never shed an authenticated git op.
pub git_ipfs_walk_semaphore: Arc<tokio::sync::Semaphore>,
/// Per-source concurrency sub-cap on the `/ipfs/{cid}` walk pool: each source
/// (keyed on the resolved source IP, never the DID — `/ipfs` admits any `did:key`
/// unthrottled, so a DID key would be free to mint around) may hold at most
/// `ipfs_walk_per_source` in-flight walk slots, so one source cannot monopolize
/// `git_ipfs_walk_semaphore` (#174 P1-3). A request with no resolvable key is
/// bounded by the global pool only, never this sub-cap. The key map is bounded
/// (`with_default_max_keys`, reject-before-insert) so a source-key farm cannot grow
/// it (INV-15).
pub git_ipfs_walk_per_caller: crate::rate_limit::PerCallerConcurrency,
/// The `git` executable the served-git withheld-blob walk spawns. Production is
/// `"git"` (resolved via PATH); injectable so a fake `git` can drive the walk's
/// process-group teardown in handler tests without mutating the process-global
/// PATH (#174).
pub git_bin: String,
}
impl AppState {
@@ -68,6 +224,21 @@ impl AppState {
self.shutdown_tx.subscribe()
}
/// Sweep expired entries from every per-IP/DID rate limiter. Driven by the
/// periodic cleanup task so a bounded limiter's key map sheds stale entries
/// instead of sitting near its cap until an inline capacity sweep reclaims
/// them. Every limiter on the state is swept here; adding a new limiter means
/// adding it to this list.
pub(crate) async fn sweep_rate_limiters(&self) {
self.rate_limiter.cleanup().await;
self.create_ip_rate_limiter.cleanup().await;
self.push_rate_limiter.cleanup().await;
self.ipfs_rate_limiter.cleanup().await;
self.ipfs_work_rate_limiter.cleanup().await;
self.sync_trigger_rate_limiter.cleanup().await;
self.peer_write_rate_limiter.cleanup().await;
}
/// Trigger graceful shutdown. Idempotent — calling more than once
/// has no effect. Returns `true` if this call was the one that
/// flipped the signal.
@@ -88,4 +259,173 @@ impl AppState {
pub fn is_shutting_down(&self) -> bool {
*self.shutdown_tx.borrow()
}
/// Legacy-probe budget wired from the `GITLAWB_IPFS_MAX_REPOS_WALKED` operator
/// knob (R5, KTD5). The knob seeds `ipfs_max_legacy_probes` at construction so it
/// controls the per-request legacy (NULL-provenance) probe fan-out it advertises.
/// It deliberately does NOT feed `ipfs_max_history_walks`: that ceiling must stay
/// at `MAX_HISTORY_WALKS_PER_REQUEST` (`MAX_PIN_SOURCES + 1`) or a provenanced
/// request with a full source set is truncated into a false 503, and the knob's
/// range starts at 1. The knob is `usize`, the field `u32`; the range cap
/// (1_048_576) keeps the cast lossless.
pub(crate) fn ipfs_legacy_probe_budget(config: &crate::config::Config) -> u32 {
config.ipfs_max_repos_walked as u32
}
/// Work-budget capacity for [`ipfs_work_rate_limiter`](Self#structfield.ipfs_work_rate_limiter)
/// (R6, KTD6), DERIVED from the route limit rather than a new operator knob. The route
/// limiter (`ipfs_rate_limiter`) charges once per request; this separate bucket absorbs
/// the resolver's per-probe/per-walk work charges so both the route "requests per hour"
/// contract and the amplification bound hold. Floor: at least one full legacy search per
/// window — the effective `ipfs_max_legacy_probes` (the `GITLAWB_IPFS_MAX_REPOS_WALKED`
/// knob, U4) — so a single default-config deep search cannot self-throttle mid-scan and
/// recreate the F6 admit-then-429 for a legitimate caller. `GITLAWB_IPFS_RATE_LIMIT=0`
/// disables the route brake and this derived bucket alike (a 0-capacity limiter admits
/// everything).
pub(crate) fn ipfs_work_budget(config: &crate::config::Config) -> usize {
if config.ipfs_rate_limit == 0 {
return 0;
}
config.ipfs_rate_limit.max(config.ipfs_max_repos_walked)
}
}
/// Bounds the OUTSTANDING post-push encryption-task set by per-repo coalescing
/// (#174 P2-2). Each successful path-scoped push `tokio::spawn`s a DETACHED task that
/// parks on `git_encrypt_semaphore.acquire_owned().await` (which DEFERS when the pool
/// is full rather than shedding — `2a54c15` kept it fail-closed so the withheld-blob
/// recovery copy is never dropped). The semaphore caps *active* walks, but nothing
/// capped how many detached tasks *spawn and park* on that await: N rapid pushes to a
/// repo spawn N parked tasks, each holding cloned object lists/rules/paths/keys — an
/// unbounded outstanding set.
///
/// This tracks the repo keys with an in-flight encryption task, each carrying a
/// DIRTY flag. Before spawning, the handler calls [`try_begin`](Self::try_begin): if
/// a task for the repo is already in-flight it MARKS THE REPO DIRTY and returns `None`
/// so the handler SKIPS spawning a duplicate (coalesce). The in-flight task consults
/// the flag at its tail via [`requeue_or_release`](EncryptInflightGuard::requeue_or_release):
/// a set flag makes it run ONE MORE pass (re-reading repo state) before it exits, so a
/// push coalesced during the in-flight window is REQUEUED, never dropped. This bounds
/// the outstanding set to <=1 task per repo without losing work: there is no
/// reconciliation sweep, so a *dropped* job would be lost forever.
///
/// The returned [`EncryptInflightGuard`] is moved into the detached task. The normal
/// exit path is `requeue_or_release`, which removes the repo key ATOMICALLY with the
/// "clean" decision — no push can land in the gap between "checked clean" and "task
/// exits". `Drop` is a panic backstop only: if the task panics (or returns without
/// calling `requeue_or_release`) it still removes the key so a crashed walk can never
/// permanently lock a repo out of future recovery copies.
#[derive(Clone, Default)]
pub struct EncryptInflight {
// std::sync::Mutex: only ever held for O(1) HashMap ops in a sync context (before
// `tokio::spawn`, at the task tail, and in Drop) — never across an await, so a std
// Mutex is correct and cheaper than a tokio one. The value is the DIRTY flag: a
// coalesced push flips it true so the in-flight task requeues one more pass.
repos: Arc<std::sync::Mutex<std::collections::HashMap<String, bool>>>,
}
impl EncryptInflight {
pub fn new() -> Self {
Self::default()
}
/// Try to begin an encryption task for `repo_id`. Returns `Some(guard)` if no task
/// for the repo was in-flight (the caller should spawn). If one already is, MARKS
/// the repo dirty and returns `None` (the caller COALESCES — skips the duplicate
/// spawn; the in-flight task requeues one more pass to cover this push).
pub fn try_begin(&self, repo_id: &str) -> Option<EncryptInflightGuard> {
let mut map = self.repos.lock().expect("encrypt_inflight mutex poisoned");
match map.get_mut(repo_id) {
Some(dirty) => {
*dirty = true;
None
}
None => {
map.insert(repo_id.to_string(), false);
Some(EncryptInflightGuard {
repos: Arc::clone(&self.repos),
repo_id: repo_id.to_string(),
released: false,
})
}
}
}
/// Number of repos with an in-flight encryption task. Test/metrics observability;
/// the bound under saturation is `len() <= number of distinct repos`, i.e. at most
/// one task per repo.
#[allow(dead_code)]
pub fn len(&self) -> usize {
self.repos
.lock()
.expect("encrypt_inflight mutex poisoned")
.len()
}
#[allow(dead_code)]
pub fn is_empty(&self) -> bool {
self.len() == 0
}
/// Test-only: read the dirty flag for `repo_id` (`None` if no task is in-flight).
#[cfg(test)]
pub fn dirty(&self, repo_id: &str) -> Option<bool> {
self.repos
.lock()
.expect("encrypt_inflight mutex poisoned")
.get(repo_id)
.copied()
}
}
/// RAII guard for one in-flight encryption task's repo key. The task drives it at its
/// tail with [`requeue_or_release`](Self::requeue_or_release); `Drop` is a panic
/// backstop. Move-only — cloning would double-release.
pub struct EncryptInflightGuard {
repos: Arc<std::sync::Mutex<std::collections::HashMap<String, bool>>>,
repo_id: String,
released: bool,
}
impl EncryptInflightGuard {
/// TASK-TAIL check-and-clear, atomic with the release decision. If a push
/// coalesced since the last pass (dirty), clear the flag and return `true` (the
/// task loops one more pass). Otherwise remove the key and return `false` (the task
/// exits). Atomic under the mutex: a concurrent push either sets the flag BEFORE
/// this reads it (-> requeue covers it) or arrives AFTER the key is removed (-> a
/// fresh `try_begin` spawns a new task) — there is no window where the key is
/// present-but-clean while the task exits.
pub fn requeue_or_release(&mut self) -> bool {
let mut map = self.repos.lock().expect("encrypt_inflight mutex poisoned");
match map.get_mut(&self.repo_id) {
Some(dirty) if *dirty => {
*dirty = false;
true
}
_ => {
map.remove(&self.repo_id);
self.released = true;
false
}
}
}
}
impl Drop for EncryptInflightGuard {
fn drop(&mut self) {
// Panic backstop ONLY. The normal exit is `requeue_or_release` (which removed
// the key atomically and set `released`), so this does nothing then. If the
// task panicked or returned without releasing, remove the key so one crashed
// walk never permanently locks the repo out.
//
// Accepted residual: a panic with the dirty flag still set drops the requeued
// work — Drop cannot loop — the same loss class as the on-panic behavior before
// this change. No reconciliation sweep re-derives it.
if self.released {
return;
}
if let Ok(mut map) = self.repos.lock() {
map.remove(&self.repo_id);
}
}
}
File diff suppressed because it is too large. Load diff
+439 -5
View File
@@ -4,7 +4,8 @@
//! based on the repo's visibility rules with a fallback to the legacy
//! `is_public` flag. It performs no I/O so it is exhaustively unit tested.
use crate::db::VisibilityRule;
use crate::db::{RepoRecord, VisibilityRule};
use std::collections::HashMap;
use unicode_normalization::UnicodeNormalization;
#[derive(Debug, PartialEq, Eq)]
@@ -24,11 +25,12 @@ fn nfc(s: &str) -> String {
s.nfc().collect()
}
/// True if `caller` is the repo owner (matches full did:key or its short form),
/// mirroring the owner-match idiom in `api/protect.rs`.
/// True if `caller` is the repo owner. Uses [`crate::api::did_matches`] so the
/// owner check is representation-agnostic within `did:key` (full `did:key:z...`
/// matches bare `z...` stored by mirror rows) while still denying cross-method
/// collisions (`did:gitlawb:z...` vs bare `z...`).
fn is_owner(owner_did: &str, caller: &str) -> bool {
let owner_short = owner_did.split(':').next_back().unwrap_or(owner_did);
caller == owner_did || caller == owner_short
crate::api::did_matches(owner_did, caller)
}
/// The match prefix for a glob: "/" stays "/", "/secret/**" becomes "/secret".
@@ -127,6 +129,105 @@ pub fn listable_at_root(
visibility_check(rules, is_public, owner_did, caller, "/") == Decision::Allow
}
/// Whether a single `received_ref_updates` row (identified by its peer-supplied
/// `row_repo` slug) should be shown to `caller` (None = anonymous) on the
/// cross-repo ref-updates feeds (#112 GraphQL, #114 REST).
///
/// Pure and I/O-free: both call sites load the deduped local repo set and its
/// visibility rules once per request and pass them in, so the gate logic lives
/// here and visibility.rs keeps its "no I/O" property.
///
/// The slug is written verbatim from the inbound gossip/notify message, so it is
/// untrusted input, not a canonical key. The decision is fail-closed by
/// construction: the only KEEP paths are (a) a slug with no `/` (cannot name a
/// local `owner/name` pair, so remote by definition), (b) all matched local
/// records are readable, and (c) no local record matches (remote/gossip-only).
/// Any local match the caller cannot read at root DROPs the row. There is no
/// catch-all keep on unexpected state.
///
/// Slug/record owner keys are matched prefix-tolerantly (one is a prefix of the
/// other), covering the exact short-key, the full `did:key:` form, and the
/// URL-truncated 8-char form. Prefix over-match can only over-drop a genuinely
/// remote row (fail-safe), never over-serve a private local one.
///
/// The live call sites are the #112 (GraphQL) and #114 (REST) feed handlers,
/// added in the following units; exercised by the unit tests below meanwhile.
pub fn ref_update_row_visible(
deduped: &[RepoRecord],
rules_by_repo: &HashMap<String, Vec<VisibilityRule>>,
caller: Option<&str>,
row_repo: &str,
) -> bool {
// A slug with no '/' cannot name a local owner/name pair — remote by
// definition (same branch as "matches nothing local") → KEEP.
if row_repo.rsplit_once('/').is_none() {
return true;
}
// Match each local record with the shared slug predicate (one matcher, so
// this gate and the collector's quarantine drop cannot disagree about which
// rows a repo owns), then fail closed on any matched record the caller
// cannot read at root.
for record in deduped {
if !ref_update_row_names_repo(record, row_repo) {
continue;
}
let rules = rules_by_repo
.get(&record.id)
.map(Vec::as_slice)
.unwrap_or(&[]);
if !listable_at_root(rules, record.is_public, &record.owner_did, caller) {
return false;
}
}
// Reached only if every matched local record is readable, or nothing local
// matched (remote/gossip-only). Both are the KEEP paths; there is no
// default-keep on unexpected state — an unreadable match already returned.
true
}
/// Whether `row_repo`'s peer-supplied slug names the local `record`. The single
/// match predicate shared by the feed gate ([`ref_update_row_visible`]) and the
/// quarantine hard-drop in the ref-update collector, so the two cannot diverge
/// about which rows a repo owns — a second, drifting matcher is exactly the #134
/// slug-collision class.
///
/// A record matches when the slug's `name` half is equal and one owner key is a
/// prefix of the other (prefix-tolerant: exact short key, full `did:key:` form,
/// and the URL-truncated 8-char form). Both the slug owner and the record owner
/// are reduced to their last ':'-delimited segment before comparing. The node's
/// own emitter builds the wire slug via `normalize_owner_key` (bare short key for
/// a single-segment `did:key:`, the full DID for every other method; see
/// api/repos.rs), but the stored slug is UNTRUSTED: a peer can broadcast a row
/// under any owner form, including a method-stripped `user/name`. Reducing both
/// sides to the trailing segment matches a repo's own rows whichever form keyed
/// them, and still catches an attacker-planted short slug.
///
/// This intentionally diverges from `did_matches` / `DEDUP_CTE`, which strip only
/// bare `did:key:` and keep other DID methods whole: those compare trusted,
/// canonical DIDs, while this slug is untrusted. Applying the keep-whole rule here
/// would fail open: a hostile `user/name` slug would not prefix-match a private
/// `did:web:host:user` record and would leak. The price is a fail-SAFE over-match
/// when a remote owner shares both a trailing segment and a repo name with a local
/// private repo (negligible for full did:key ids; only did:web / truncated forms
/// collide): it can hide a genuinely remote row, never serve a private one.
pub fn ref_update_row_names_repo(record: &RepoRecord, row_repo: &str) -> bool {
let Some((owner_part, name)) = row_repo.rsplit_once('/') else {
return false;
};
if record.name != name {
return false;
}
let row_key = owner_part.split(':').next_back().unwrap_or(owner_part);
let record_key = record
.owner_did
.split(':')
.next_back()
.unwrap_or(&record.owner_did);
record_key.starts_with(row_key) || row_key.starts_with(record_key)
}
/// The subtree path globs that `caller` (None = anonymous) may NOT read, given
/// the repo's rules. Whole-repo ("/") rules are excluded: a denied whole-repo
/// read is handled by the 404 gate before a clone ever starts. Each remaining
@@ -299,6 +400,85 @@ mod tests {
);
}
#[test]
fn non_key_owner_bare_short_does_not_match() {
// A non-key DID owner (did:gitlawb:z6MkFoo) probed with the bare last
// segment (z6MkFoo) must NOT match — the old is_owner would return true
// via split(':').next_back(), which this PR tightens.
let rules = [rule("/", VisibilityMode::A, &[])];
assert_eq!(
visibility_check(&rules, false, "did:gitlawb:z6MkFoo", Some("z6MkFoo"), "/"),
Decision::Deny,
"bare short must not match a non-key DID owner"
);
// The full non-key DID still matches itself.
assert_eq!(
visibility_check(
&rules,
false,
"did:gitlawb:z6MkFoo",
Some("did:gitlawb:z6MkFoo"),
"/"
),
Decision::Allow,
"full non-key DID still matches itself"
);
}
// #153 forward guard: is_owner must match across bare-key / full-did:key
// representations should someone revert to single-side normalize_owner_key.
#[test]
fn bare_owner_full_caller_allows_owner() {
let rules = [rule("/", VisibilityMode::A, &[])];
assert_eq!(
visibility_check(&rules, false, "z6MkOwner", Some("did:key:z6MkOwner"), "/"),
Decision::Allow,
"bare owner + full did:key caller must match (guard against single-side revert)"
);
}
// #135 T1: a Mode-B rule on `/secret/**` must DENY the withheld directory's
// OWN path `/secret` (the `path == prefix` arm), not just strict descendants —
// otherwise get_by_cid's tree gate would serve the /secret tree object and leak
// its children. Pins parity with get_tree, which denies the /secret path.
#[test]
fn subtree_rule_denies_the_withheld_directory_itself() {
let reader = "did:key:z6MkReader";
let rules = [rule("/secret/**", VisibilityMode::B, &[reader])];
assert_eq!(
visibility_check(&rules, true, OWNER, None, "/secret"),
Decision::Deny,
"anon denied at the withheld directory's OWN path /secret"
);
assert_eq!(
visibility_check(&rules, true, OWNER, None, "/public"),
Decision::Allow,
"anon allowed at a sibling path outside the withheld subtree"
);
assert_eq!(
visibility_check(&rules, true, OWNER, Some(reader), "/secret"),
Decision::Allow,
"listed reader allowed at the withheld directory (caller-aware)"
);
}
// #153 regression: cross-method DID must still be denied even when the
// trailing segment collides with a bare owner key.
#[test]
fn cross_method_did_denied_with_bare_owner() {
let rules = [rule("/", VisibilityMode::A, &[])];
assert_eq!(
visibility_check(&rules, false, "z6MkFoo", Some("did:gitlawb:z6MkFoo"), "/"),
Decision::Deny,
"cross-method DID must not match a bare did:key owner"
);
assert_eq!(
visibility_check(&rules, false, "z6MkFoo", Some("did:web:z6MkFoo"), "/"),
Decision::Deny,
"did:web must not match a bare did:key owner"
);
}
#[test]
fn root_rule_allows_listed_reader() {
let rules = [rule("/", VisibilityMode::A, &["did:key:z6MkFriend"])];
@@ -403,6 +583,260 @@ mod tests {
));
}
// ── ref_update_row_visible (feed gate) ──────────────────────────────────
fn rec(id: &str, owner_did: &str, name: &str, is_public: bool) -> RepoRecord {
RepoRecord {
id: id.into(),
name: name.into(),
owner_did: owner_did.into(),
description: None,
is_public,
default_branch: "main".into(),
created_at: Utc::now(),
updated_at: Utc::now(),
disk_path: format!("/srv/{id}"),
forked_from: None,
machine_id: None,
}
}
fn rules_for(entries: &[(&str, &[VisibilityRule])]) -> HashMap<String, Vec<VisibilityRule>> {
entries
.iter()
.map(|(id, rs)| (id.to_string(), rs.to_vec()))
.collect()
}
#[test]
fn feed_public_local_repo_kept_for_anon() {
let deduped = [rec("r1", "did:key:z6MkOwner", "widget", true)];
let rules = HashMap::new();
assert!(ref_update_row_visible(
&deduped,
&rules,
None,
"z6MkOwner/widget"
));
}
#[test]
fn feed_private_local_repo_dropped_for_anon_kept_for_owner() {
let deduped = [rec("r1", "did:key:z6MkOwner", "widget", false)];
let rules = HashMap::new();
// Anonymous → drop.
assert!(!ref_update_row_visible(
&deduped,
&rules,
None,
"z6MkOwner/widget"
));
// Owner (full DID) → keep.
assert!(ref_update_row_visible(
&deduped,
&rules,
Some("did:key:z6MkOwner"),
"z6MkOwner/widget"
));
}
#[test]
fn feed_root_rule_reader_reincluded() {
// Private repo (is_public=false) with a root rule granting a named
// reader. Delegates to listable_at_root: anon and non-reader denied,
// named reader allowed.
let deduped = [rec("r1", OWNER, "widget", false)];
let root = [rule("/", VisibilityMode::A, &["did:key:z6MkFriend"])];
let rules = rules_for(&[("r1", &root)]);
assert!(!ref_update_row_visible(
&deduped,
&rules,
None,
"z6MkOwner/widget"
));
assert!(!ref_update_row_visible(
&deduped,
&rules,
Some("did:key:z6MkStranger"),
"z6MkOwner/widget"
));
assert!(ref_update_row_visible(
&deduped,
&rules,
Some("did:key:z6MkFriend"),
"z6MkOwner/widget"
));
}
#[test]
fn feed_alias_full_did_slug_dropped_for_anon() {
// Owner stored full-DID; slug also carries the full-DID form. Still
// matches (row_key normalizes to the short key) → drop. Round 1's
// string-match would have leaked this.
let deduped = [rec("r1", "did:key:zABC", "widget", false)];
let rules = HashMap::new();
assert!(!ref_update_row_visible(
&deduped,
&rules,
None,
"did:key:zABC/widget"
));
}
#[test]
fn feed_truncated_key_slug_dropped_for_anon() {
// Slug carries an 8-char URL-truncated prefix of the owner key; still
// matches via prefix tolerance → drop. Round 2's get_repo path leaked.
let deduped = [rec("r1", "did:key:zABCDEFGH", "widget", false)];
let rules = HashMap::new();
assert!(!ref_update_row_visible(
&deduped,
&rules,
None,
"zABCDEF/widget"
));
}
#[test]
fn feed_mirror_coexistence_private_canonical_dropped_for_anon() {
// Pure-level mirror-coexistence: the deduped set contains only the
// private canonical record for (owner,name). A matching slug drops for
// anon. (DB-level dedup survivor property is pinned separately.)
let deduped = [rec("uuid-canonical", "did:key:z6Mkwbud", "nipmod", false)];
let rules = HashMap::new();
assert!(!ref_update_row_visible(
&deduped,
&rules,
None,
"z6Mkwbud/nipmod"
));
}
#[test]
fn feed_empty_owner_slug_matches_and_drops() {
// Slug "/name": empty owner_part → row_key "" → starts_with("") matches
// every same-named record. Fail-safe pin for a private repo.
let deduped = [rec("r1", "did:key:z6MkOwner", "widget", false)];
let rules = HashMap::new();
assert!(!ref_update_row_visible(&deduped, &rules, None, "/widget"));
}
#[test]
fn feed_one_char_owner_slug_matches_and_drops() {
// 1-char owner prefix that the private repo's key starts with → match.
let deduped = [rec("r1", "did:key:z6MkOwner", "widget", false)];
let rules = HashMap::new();
assert!(!ref_update_row_visible(&deduped, &rules, None, "z/widget"));
}
#[test]
fn feed_remote_slug_no_match_kept() {
// Different owner key, no prefix relation → no local match → keep.
let deduped = [rec("r1", "did:key:z6MkOwner", "widget", false)];
let rules = HashMap::new();
assert!(ref_update_row_visible(
&deduped,
&rules,
None,
"zZZZOTHER/widget"
));
}
#[test]
fn feed_private_didweb_short_slug_dropped_for_anon() {
// A private did:web repo must drop its row for anon when the slug arrives
// in the short `{last-segment}/{name}` form. Post-#141 the emitter no longer
// produces this form for did:web (normalize_owner_key keeps the full DID),
// so this is the untrusted/crafted variant a peer can broadcast; the full-DID
// variant the emitter now sends is covered by the next test. A did:key-aware
// (keep-whole) rule would fail to match `alice` against `did:web:host:alice`
// and leak.
let deduped = [rec("r1", "did:web:host:alice", "widget", false)];
let rules = HashMap::new();
assert!(!ref_update_row_visible(
&deduped,
&rules,
None,
"alice/widget"
));
}
#[test]
fn feed_multi_segment_did_slug_dropped_for_anon() {
// A private repo owned by a multi-segment DID must also fail closed under
// the full-DID slug form. Post-#141 this IS the form the emitter broadcasts
// (normalize_owner_key keeps non-did:key methods whole), and a peer can craft
// it too. The gate drops it: both sides reduce to the last ':' segment
// ("user"), so they match. Fail-safe regardless of which form keyed the row.
let deduped = [rec("r1", "did:web:host:user", "widget", false)];
let rules = HashMap::new();
assert!(!ref_update_row_visible(
&deduped,
&rules,
None,
"did:web:host:user/widget"
));
}
#[test]
fn feed_multi_segment_did_slug_kept_for_public() {
// Symmetric keep-side: a PUBLIC multi-segment-DID repo's row must still be
// returned to anon after the last-':'-segment normalization — guards
// against a regression that over-drops legitimate did:web rows.
let deduped = [rec("r1", "did:web:host:user", "widget", true)];
let rules = HashMap::new();
assert!(ref_update_row_visible(
&deduped,
&rules,
None,
"did:web:host:user/widget"
));
}
#[test]
fn feed_malformed_slug_no_slash_kept_no_panic() {
let deduped = [rec("r1", "did:key:z6MkOwner", "widget", false)];
let rules = HashMap::new();
assert!(ref_update_row_visible(&deduped, &rules, None, "noslug"));
}
#[test]
fn feed_empty_deduped_set_keeps_any_slug() {
let deduped: [RepoRecord; 0] = [];
let rules = HashMap::new();
assert!(ref_update_row_visible(
&deduped,
&rules,
None,
"z6MkOwner/widget"
));
assert!(ref_update_row_visible(&deduped, &rules, None, "anything"));
}
// The shared slug matcher underpinning both the feed gate and the collector's
// quarantine drop: it must recognize a repo's own row across every owner-DID
// form (bare short key, full did:key, URL-truncated prefix) and must not
// over-match a different name or an unrelated owner. The quarantine drop calls
// this directly, so its match contract is load-bearing for withholding a
// quarantined mirror from a caller who matches the mirror's owner_did.
#[test]
fn names_repo_matches_owner_forms_and_rejects_mismatches() {
// Bare short owner key (the form upsert_mirror_repo stores).
let bare = rec("m", "z6MkQuar", "secret", false);
assert!(ref_update_row_names_repo(&bare, "z6MkQuar/secret"));
// Full did:key owner; the short-segment slug still matches via last-segment.
let full = rec("c", "did:key:z6MkQuar", "secret", false);
assert!(ref_update_row_names_repo(&full, "z6MkQuar/secret"));
assert!(ref_update_row_names_repo(&full, "did:key:z6MkQuar/secret"));
// URL-truncated 8-char prefix slug → prefix-tolerant match.
let long = rec("l", "did:key:z6MkQuarLONGKEY", "secret", false);
assert!(ref_update_row_names_repo(&long, "z6MkQuar/secret"));
// Different name, different owner, and no-slash → no match.
assert!(!ref_update_row_names_repo(&bare, "z6MkQuar/other"));
assert!(!ref_update_row_names_repo(&bare, "z6MkOther/secret"));
assert!(!ref_update_row_names_repo(&bare, "noslash"));
}
// #101: a deny rule must withhold a path that denotes the same characters in
// a different Unicode normalization form. Without NFC normalization at the
// matcher seam, an NFC-authored rule byte-compares unequal to an NFD-stored
+139
View File
@@ -0,0 +1,139 @@
//! #174 U6 — INV-22 completeness guard (rung-raising).
//!
//! INV-22: a permit held per op recovers only if every path that holds it is also
//! duration-bounded and reaps the process group before releasing admission, and every
//! detached git/blocking task carries its own admission. PR #174 fixed five paths
//! (U1-U5) that violated this. Each fix has a per-unit RED/GREEN regression; together
//! those form the five-revert matrix. This guard adds the missing piece: a source-scan
//! tripwire that fails when a NEW site reintroduces the class, or when one of the five
//! gates is removed.
//!
//! It lives in `tests/` (a separate crate) on purpose: a guard that scanned the same
//! file it lives in would match its own identifier literals and pass vacuously. Here
//! the scanned `src/` files never contain this file's literals, so each check is
//! load-bearing — reverting the named gate turns the assertion red.
//!
//! These are deliberately coarse structural checks, not a parser. They cannot prove a
//! gate is *correct* (the per-unit tests do that); they prove a gate is *present and
//! not bypassed*, which is what stops the class from silently regressing.
use std::path::Path;
fn src(rel: &str) -> String {
let p = Path::new(env!("CARGO_MANIFEST_DIR")).join("src").join(rel);
std::fs::read_to_string(&p).unwrap_or_else(|e| panic!("read {}: {e}", p.display()))
}
#[test]
fn inv22_concurrency_gates_present_and_not_bypassed() {
let repos = src("api/repos.rs");
let smart_http = src("git/smart_http.rs");
let vis = src("git/visibility_pack.rs");
let ipfs = src("api/ipfs.rs");
// U1 / P1-a: run_bounded_git stands the watchdog down only after confirming the
// child actually terminated (WNOWAIT), not on the raw stdout-drain EOF — otherwise
// a child that closes stdout then hangs pins the permit past the deadline. The
// probe is defined and called, so >= 2 occurrences; reverting the fix removes both.
assert!(
vis.matches("child_terminated_without_reaping").count() >= 2,
"U1/P1-a gate missing: run_bounded_git must confirm child exit via \
child_terminated_without_reaping before signalling the watchdog"
);
// U2 / P1-c: on client disconnect KillGroupOnDrop must launch a detached reaper
// that runs the full TERM/grace/KILL/reap, not a lone SIGTERM. The reaper is spawned
// via a runtime handle in Drop; `Handle::try_current` is unique to that launch (the
// timeout path already has an async context and never calls it).
assert!(
smart_http.contains("Handle::try_current"),
"U2/P1-c gate missing: KillGroupOnDrop::drop must launch the full \
TERM/grace/KILL reaper on disconnect (Handle::try_current), not a lone SIGTERM"
);
// U4 / P1-d: git_receive_pack must acquire the per-source write sub-cap before the
// global write permit. The acquire reads `state.git_write_per_caller`; comments name
// the field without the `state.` prefix, so this targets the real acquire site.
assert!(
repos.contains("state.git_write_per_caller"),
"U4/P1-d gate missing: git_receive_pack must acquire the per-source write cap \
(state.git_write_per_caller) before the global write permit"
);
// U5 / P1-e: the detached post-push encryption walk must run through the
// admission-gated helper, which is wired to the shared encrypt pool.
assert!(
repos.contains("fn withheld_recipients_gated")
&& repos.contains("state.git_encrypt_semaphore"),
"U5/P1-e gate missing: the encryption walk must run through \
withheld_recipients_gated, which acquires git_encrypt_semaphore"
);
// U1 / R2 (#173 round-10): the path-scoped filtered-pack serve must thread the
// caller's AdmissionGuard through BOTH git stages so read + per-caller admission is
// held until the pack-objects group is reaped on disconnect, closing the cap bypass
// the plain upload_pack path already fixed. Two load-bearing markers: rev-list hands
// the disarmed guard back (its tuple return type), and build_filtered_pack forwards
// that guard into the pack-objects stage (the `admission` arg after the
// "pack-objects" label). Reverting either — dropping the guard between stages, or
// passing `None` to pack-objects — trips this.
assert!(
smart_http.contains("Result<(Vec<String>, Option<AdmissionGuard>)>")
&& smart_http.contains("\"pack-objects\",\n admission,"),
"U1/R2 gate missing: build_filtered_pack must thread the AdmissionGuard through \
rev-list -> pack-objects so the permits are held until the pack-objects group \
is reaped on disconnect (the path-scoped half of #174 P1-a)"
);
// U2 / R1 (#173 round-10): the `GET /ipfs/{cid}` serve pipeline must run in a
// DETACHED tokio task that OWNS the AdmissionGuard built from BOTH walk permits, so a
// cancelled or timed-out request releases admission only after the spawned (bounded)
// work completes — not the instant the handler future drops (the /ipfs half of #174
// P1-a). Two load-bearing markers: the guard is constructed from the two named
// permits, and the whole pipeline is moved into a `tokio::spawn`. Reverting to
// handler-local permits (dropping the guard/spawn) trips this.
assert!(
ipfs.contains("AdmissionGuard::new(ipfs_walk_permit, ipfs_caller_permit)")
&& ipfs.contains(
"let serve: tokio::task::JoinHandle<Result<Response>> = tokio::spawn(async move {"
),
"U2/R1 gate missing: get_by_cid must move both /ipfs admission permits into an \
AdmissionGuard owned by a detached tokio::spawn task so admission is released \
only after the spawned serve work completes (the /ipfs half of #174 P1-a)"
);
// U2 / KTD2 (#173 round-10): the probe/read children on the /ipfs path must be the
// duration-bounded twins (process-group teardown via run_bounded_git), not the bare
// `store::object_type` / `read_object_content` (or an unbounded `cat-file -s`) a tokio
// timeout cannot cancel — otherwise a wedged cat-file lingers and pins the held
// admission past the deadline. Reverting any twin call site back to a bare read trips
// this.
assert!(
ipfs.contains("object_type_bounded(")
&& ipfs.contains("object_size_bounded(")
&& ipfs.contains("read_object_content_bounded("),
"U2/KTD2 gate missing: the /ipfs probe+read must call the run_bounded_git-backed \
*_bounded twins so a wedged cat-file is reaped at the deadline, not left to pin \
the held /ipfs walk admission"
);
// P1-e non-bypass tripwire: the bounded recipients walk is spawn_blocking'd nowhere
// but inside withheld_recipients_gated. A second call site (count > 1) is a new
// detached git walk that skips the admission gate — exactly the class U5 closed.
assert_eq!(
repos.matches("withheld_blob_recipients_bounded").count(),
1,
"P1-e bypass: the bounded recipients walk must be invoked only inside \
withheld_recipients_gated; a new call site bypasses the encrypt-walk admission cap"
);
// U4 / P2-2: the detached post-push encryption task must be gated by the per-repo
// coalescing set (`encrypt_inflight.try_begin`) so the OUTSTANDING parked-task set is
// bounded to <=1 per repo. Removing the gate lets N rapid pushes spawn N parked
// waiters (the unbounded set U4 closed); the semaphore only caps active walks.
assert!(
repos.contains("state.encrypt_inflight.try_begin"),
"U4/P2-2 gate missing: the detached post-push encryption spawn must consult \
encrypt_inflight.try_begin to coalesce per repo (bound the outstanding-task set)"
);
}
+2 -1
View File
@@ -1,7 +1,7 @@
[package]
name = "gl"
description = "The gitlawb CLI — identity management, node control, MCP server"
version = "0.4.0" # x-release-please-version
version = "0.5.1" # x-release-please-version
edition.workspace = true
rust-version.workspace = true
license.workspace = true
@@ -12,6 +12,7 @@ path = "src/main.rs"
[dependencies]
gitlawb-core = { path = "../gitlawb-core" }
icaptcha-client = { path = "../icaptcha-client" }
tokio = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
+28 -10
View File
@@ -8,6 +8,10 @@ use std::path::PathBuf;
use crate::http::NodeClient;
use crate::identity::load_keypair_from_dir;
fn signed_client(node: &str, dir: Option<&std::path::Path>) -> NodeClient {
NodeClient::new(node, load_keypair_from_dir(dir).ok())
}
#[derive(Args)]
pub struct BountyArgs {
#[command(subcommand)]
@@ -49,6 +53,8 @@ pub enum BountyCmd {
status: Option<String>,
#[arg(long, default_value = "https://node.gitlawb.com", env = "GITLAWB_NODE")]
node: String,
#[arg(long)]
dir: Option<PathBuf>,
},
/// Show details of a specific bounty
Show {
@@ -56,6 +62,8 @@ pub enum BountyCmd {
id: String,
#[arg(long, default_value = "https://node.gitlawb.com", env = "GITLAWB_NODE")]
node: String,
#[arg(long)]
dir: Option<PathBuf>,
},
/// Claim an open bounty
Claim {
@@ -121,8 +129,13 @@ pub async fn run(args: BountyArgs) -> Result<()> {
node,
dir,
} => cmd_create(repo, title, amount, issue, tx_hash, deadline, node, dir).await,
BountyCmd::List { repo, status, node } => cmd_list(repo, status, node).await,
BountyCmd::Show { id, node } => cmd_show(id, node).await,
BountyCmd::List {
repo,
status,
node,
dir,
} => cmd_list(repo, status, node, dir).await,
BountyCmd::Show { id, node, dir } => cmd_show(id, node, dir).await,
BountyCmd::Claim {
id,
wallet,
@@ -192,8 +205,13 @@ async fn cmd_create(
Ok(())
}
async fn cmd_list(repo: Option<String>, status: Option<String>, node: String) -> Result<()> {
let client = NodeClient::new(&node, None);
async fn cmd_list(
repo: Option<String>,
status: Option<String>,
node: String,
dir: Option<PathBuf>,
) -> Result<()> {
let client = signed_client(&node, dir.as_deref());
let url = if let Some(ref repo) = repo {
let (owner, name) = repo
@@ -214,7 +232,7 @@ async fn cmd_list(repo: Option<String>, status: Option<String>, node: String) ->
};
let resp = client
.get(&url)
.get_authed(&url)
.await
.context("failed to connect to node")?;
let body: Value = resp.json().await.unwrap_or_default();
@@ -237,9 +255,9 @@ async fn cmd_list(repo: Option<String>, status: Option<String>, node: String) ->
Ok(())
}
async fn cmd_show(id: String, node: String) -> Result<()> {
let client = NodeClient::new(&node, None);
let resp = client.get(&format!("/api/v1/bounties/{id}")).await?;
async fn cmd_show(id: String, node: String, dir: Option<PathBuf>) -> Result<()> {
let client = signed_client(&node, dir.as_deref());
let resp = client.get_authed(&format!("/api/v1/bounties/{id}")).await?;
let status = resp.status();
let body: Value = resp.json().await.unwrap_or_default();
@@ -462,7 +480,7 @@ mod tests {
.create_async()
.await;
cmd_list(None, None, server.url()).await.unwrap();
cmd_list(None, None, server.url(), None).await.unwrap();
}
#[tokio::test]
@@ -506,7 +524,7 @@ mod tests {
.create_async()
.await;
let err = cmd_show("nonexistent".to_string(), server.url())
let err = cmd_show("nonexistent".to_string(), server.url(), None)
.await
.unwrap_err();
assert!(err.to_string().contains("not found"));
+34 -17
View File
@@ -5,10 +5,15 @@
use anyhow::{Context, Result};
use clap::{Args, Subcommand};
use serde_json::Value;
use std::path::PathBuf;
use crate::http::NodeClient;
use crate::identity::load_keypair_from_dir;
fn signed_client(node: &str, dir: Option<&std::path::Path>) -> NodeClient {
NodeClient::new(node, load_keypair_from_dir(dir).ok())
}
#[derive(Args)]
pub struct CertArgs {
#[command(subcommand)]
@@ -23,6 +28,8 @@ pub enum CertCmd {
repo: String,
#[arg(long, default_value = "https://node.gitlawb.com", env = "GITLAWB_NODE")]
node: String,
#[arg(long)]
dir: Option<PathBuf>,
},
/// Show a specific ref certificate and verify its signature
Show {
@@ -32,28 +39,39 @@ pub enum CertCmd {
id: String,
#[arg(long, default_value = "https://node.gitlawb.com", env = "GITLAWB_NODE")]
node: String,
#[arg(long)]
dir: Option<PathBuf>,
},
}
pub async fn run(args: CertArgs) -> Result<()> {
match args.cmd {
CertCmd::List { repo, node } => cmd_list(repo, node).await,
CertCmd::Show { repo, id, node } => cmd_show(repo, id, node).await,
CertCmd::List { repo, node, dir } => cmd_list(repo, node, dir).await,
CertCmd::Show {
repo,
id,
node,
dir,
} => cmd_show(repo, id, node, dir).await,
}
}
/// Resolve "repo" into (owner, name) using the caller's DID when no slash is given.
async fn resolve_repo(repo: &str, node: &str) -> Result<(String, String)> {
async fn resolve_repo(
repo: &str,
node: &str,
dir: Option<&std::path::Path>,
) -> Result<(String, String)> {
if let Some((owner, name)) = repo.split_once('/') {
Ok((owner.to_string(), name.to_string()))
} else {
let short = if let Ok(kp) = load_keypair_from_dir(None) {
let short = if let Ok(kp) = load_keypair_from_dir(dir) {
let did = kp.did().to_string();
did.split(':').next_back().unwrap_or(&did).to_string()
} else {
let client = NodeClient::new(node, None);
let client = signed_client(node, dir);
let info: Value = client
.get("/")
.get_authed("/")
.await?
.json()
.await
@@ -65,13 +83,13 @@ async fn resolve_repo(repo: &str, node: &str) -> Result<(String, String)> {
}
}
async fn cmd_list(repo: String, node: String) -> Result<()> {
let (owner, name) = resolve_repo(&repo, &node).await?;
async fn cmd_list(repo: String, node: String, dir: Option<PathBuf>) -> Result<()> {
let (owner, name) = resolve_repo(&repo, &node, dir.as_deref()).await?;
let client = NodeClient::new(&node, None);
let client = signed_client(&node, dir.as_deref());
let path = format!("/api/v1/repos/{owner}/{name}/certs");
let resp: Value = client
.get(&path)
.get_authed(&path)
.await?
.json()
.await
@@ -96,16 +114,16 @@ async fn cmd_list(repo: String, node: String) -> Result<()> {
Ok(())
}
async fn cmd_show(repo: String, id: String, node: String) -> Result<()> {
let (owner, name) = resolve_repo(&repo, &node).await?;
async fn cmd_show(repo: String, id: String, node: String, dir: Option<PathBuf>) -> Result<()> {
let (owner, name) = resolve_repo(&repo, &node, dir.as_deref()).await?;
let client = NodeClient::new(&node, None);
let client = signed_client(&node, dir.as_deref());
let id = resolve_cert_id(&client, &owner, &name, &id).await?;
// Fetch the certificate
let path = format!("/api/v1/repos/{owner}/{name}/certs/{id}");
let resp = client
.get(&path)
.get_authed(&path)
.await?
.error_for_status()
.context("certificate not found")?;
@@ -168,9 +186,9 @@ async fn resolve_cert_id(client: &NodeClient, owner: &str, name: &str, id: &str)
return Ok(id.to_string());
}
let path = format!("/api/v1/repos/{owner}/{name}/certs");
let path = format!("/api/v1/repos/{owner}/{name}/certs?prefix={id}");
let resp: Value = client
.get(&path)
.get_authed(&path)
.await?
.error_for_status()
.context("failed to list certificates")?
@@ -182,7 +200,6 @@ async fn resolve_cert_id(client: &NodeClient, owner: &str, name: &str, id: &str)
let matches: Vec<String> = certs
.iter()
.filter_map(|cert| cert["id"].as_str())
.filter(|cert_id| cert_id.starts_with(id))
.map(ToString::to_string)
.collect();
+38
View File
@@ -174,6 +174,19 @@ pub async fn run(args: DoctorArgs) -> Result<()> {
"node",
format!("{} — v{version} ({short_did}…)", args.node),
));
// Capability drift: a node newer than this CLI may require features
// (RFC 9421 signing, iCaptcha) the CLI doesn't speak.
let gl_ver = env!("CARGO_PKG_VERSION");
if version != "?" && is_newer(version, gl_ver) {
checks.push(Check::warn(
"gl version",
format!(
"node is v{version} but gl is v{gl_ver} — your CLI may be missing \
features (signing / iCaptcha) this node requires"
),
"upgrade gl: curl -sSf https://gitlawb.com/install.sh | sh",
));
}
}
Ok(resp) => {
checks.push(Check::fail(
@@ -191,6 +204,31 @@ pub async fn run(args: DoctorArgs) -> Result<()> {
}
}
// ── 4b. iCaptcha capability ───────────────────────────────────────────
// Gated writes (repo create / register / fork) auto-solve a challenge at the
// iCaptcha service; check it's reachable so the failure mode is obvious.
let icaptcha_url = std::env::var("GITLAWB_ICAPTCHA_URL")
.unwrap_or_else(|_| icaptcha_client::DEFAULT_URL.to_string());
match NodeClient::new(&icaptcha_url, None).get("/v1/pubkey").await {
Ok(resp) if resp.status().is_success() => {
checks.push(Check::pass("iCaptcha", format!("{icaptcha_url} reachable")));
}
Ok(resp) => {
checks.push(Check::warn(
"iCaptcha",
format!("{icaptcha_url} returned HTTP {}", resp.status()),
"gated writes (repo create / register) may fail until iCaptcha is reachable",
));
}
Err(e) => {
checks.push(Check::warn(
"iCaptcha",
format!("{icaptcha_url} unreachable: {e}"),
"set GITLAWB_ICAPTCHA_URL or check connectivity — repo create / register solve a challenge there",
));
}
}
// ── 5. git-remote-gitlawb helper ──────────────────────────────────────
// Use PATH lookup only — invoking the binary directly triggers git internals
// that error with "fatal: not a git repository" outside of a git repo.
+497 -38
View File
@@ -1,8 +1,19 @@
//! Signed HTTP client for gitlawb API calls (async).
//!
//! Writes are signed with RFC 9421 HTTP Signatures. When the node gates a write
//! behind iCaptcha (HTTP 403 `icaptcha_proof_required`, advertised via the
//! `x-icaptcha-url` / `x-icaptcha-level` headers), the client transparently
//! solves the challenge and retries the same signed request with the
//! `x-icaptcha-proof` header — see `crates/icaptcha-client`.
use anyhow::{Context, Result};
use gitlawb_core::http_sig::sign_request;
use gitlawb_core::identity::Keypair;
use icaptcha_client::IcaptchaCfg;
/// Max times we'll fetch a fresh proof and retry a 403-iCaptcha response
/// (absorbs proof expiry / first-seen replay).
const MAX_ICAPTCHA_RETRIES: usize = 2;
pub struct NodeClient {
inner: reqwest::Client,
@@ -34,6 +45,15 @@ impl NodeClient {
.with_context(|| format!("GET {url}"))
}
/// GET that signs when a keypair is available; falls back to unsigned for public repos.
pub async fn get_authed(&self, path: &str) -> Result<reqwest::Response> {
if self.keypair.is_some() {
self.get_signed(path).await
} else {
self.get(path).await
}
}
/// GET with RFC 9421 HTTP Signature auth, for owner-only read endpoints.
/// Signs over the empty body (same shape the node verifies for signed reads).
pub async fn get_signed(&self, path: &str) -> Result<reqwest::Response> {
@@ -52,63 +72,502 @@ impl NodeClient {
req.send().await.with_context(|| format!("GET {url}"))
}
/// POST with JSON body + RFC 9421 HTTP Signature auth.
/// POST with JSON body + RFC 9421 signing + transparent iCaptcha solve/retry.
pub async fn post(&self, path: &str, body: &[u8]) -> Result<reqwest::Response> {
let url = format!("{}{}", self.node_url, path);
let mut req = self
.inner
.post(&url)
.header("Content-Type", "application/json")
.body(body.to_vec());
if let Some(kp) = &self.keypair {
let signed = sign_request(kp, "POST", path, body);
req = req
.header("Content-Digest", signed.content_digest)
.header("Signature-Input", signed.signature_input)
.header("Signature", signed.signature);
}
req.send().await.with_context(|| format!("POST {url}"))
self.send_signed("POST", path, body).await
}
/// PUT with RFC 9421 HTTP Signature auth (idempotent write).
/// PUT with RFC 9421 signing + transparent iCaptcha solve/retry.
pub async fn put(&self, path: &str, body: &[u8]) -> Result<reqwest::Response> {
let url = format!("{}{}", self.node_url, path);
let mut req = self
.inner
.put(&url)
.header("Content-Type", "application/json")
.body(body.to_vec());
if let Some(kp) = &self.keypair {
let signed = sign_request(kp, "PUT", path, body);
req = req
.header("Content-Digest", signed.content_digest)
.header("Signature-Input", signed.signature_input)
.header("Signature", signed.signature);
}
req.send().await.with_context(|| format!("PUT {url}"))
self.send_signed("PUT", path, body).await
}
/// DELETE with RFC 9421 HTTP Signature auth.
/// DELETE with RFC 9421 signing + transparent iCaptcha solve/retry.
pub async fn delete(&self, path: &str, body: &[u8]) -> Result<reqwest::Response> {
self.send_signed("DELETE", path, body).await
}
/// Sign + send a write. On a 403 iCaptcha challenge (detected via the
/// `x-icaptcha-*` headers) solve it and retry the same signed request with
/// the proof header, up to [`MAX_ICAPTCHA_RETRIES`]. Emits an actionable
/// hint on a 401 "not an agent" (the old-CLI / unregistered failure mode).
async fn send_signed(
&self,
method: &str,
path: &str,
body: &[u8],
) -> Result<reqwest::Response> {
let mut proof: Option<String> = None;
let mut attempts = 0;
loop {
let resp = self.send_once(method, path, body, proof.as_deref()).await?;
let status = resp.status();
if status == reqwest::StatusCode::UNAUTHORIZED
&& resp
.headers()
.get("x-gitlawb-error")
.and_then(|v| v.to_str().ok())
== Some("human_detected")
{
eprintln!(
"note: this node requires signed requests (RFC 9421). If writes keep \
failing, your `gl` may be too old — upgrade it — or you're not registered: \
run `gl register`."
);
}
if status == reqwest::StatusCode::FORBIDDEN && attempts < MAX_ICAPTCHA_RETRIES {
if let Some(cfg) = self.icaptcha_cfg(resp.headers())? {
attempts += 1;
proof = Some(obtain_proof(cfg).await?);
continue;
}
}
return Ok(resp);
}
}
/// Build, sign, and send one request, optionally attaching a proof header.
async fn send_once(
&self,
method: &str,
path: &str,
body: &[u8],
proof: Option<&str>,
) -> Result<reqwest::Response> {
let url = format!("{}{}", self.node_url, path);
let mut req = self
.inner
.delete(&url)
.request(method.parse().expect("valid HTTP method"), &url)
.header("Content-Type", "application/json")
.body(body.to_vec());
if let Some(kp) = &self.keypair {
let signed = sign_request(kp, "DELETE", path, body);
let signed = sign_request(kp, method, path, body);
req = req
.header("Content-Digest", signed.content_digest)
.header("Signature-Input", signed.signature_input)
.header("Signature", signed.signature);
}
if let Some(p) = proof {
req = req.header(icaptcha_client::PROOF_HEADER, p);
}
req.send().await.with_context(|| format!("DELETE {url}"))
req.send().await.with_context(|| format!("{method} {url}"))
}
/// If `headers` describe an iCaptcha 403, build the solve config (binding the
/// proof's `sub` to our DID). Returns `None` for a non-iCaptcha 403.
fn icaptcha_cfg(&self, headers: &reqwest::header::HeaderMap) -> Result<Option<IcaptchaCfg>> {
let url = headers.get("x-icaptcha-url").and_then(|v| v.to_str().ok());
let level = headers
.get("x-icaptcha-level")
.and_then(|v| v.to_str().ok());
if url.is_none() && level.is_none() {
return Ok(None); // not an iCaptcha challenge
}
let kp = self
.keypair
.as_ref()
.context("iCaptcha challenge requires an identity keypair (run `gl identity new`)")?;
Ok(Some(IcaptchaCfg::new(
kp.did().to_string(),
url.map(str::to_string),
level.and_then(|l| l.parse().ok()),
)))
}
}
/// Run the (blocking) iCaptcha solve loop off the async runtime.
async fn obtain_proof(cfg: IcaptchaCfg) -> Result<String> {
tokio::task::spawn_blocking(move || icaptcha_client::obtain_proof(&cfg, None))
.await
.context("iCaptcha solver task panicked")?
}
#[cfg(test)]
mod tests {
use super::*;
use gitlawb_core::identity::Keypair;
use mockito::Server;
use std::ffi::OsString;
use std::sync::{Mutex, MutexGuard};
/// Serializes the two integration tests that touch the process-global
/// `GITLAWB_ICAPTCHA_URL` / `GITLAWB_ICAPTCHA_INSECURE` env vars so they
/// never race.
static ICAPTCHA_ENV_LOCK: Mutex<()> = Mutex::new(());
fn test_keypair() -> Keypair {
Keypair::generate()
}
fn headers_from_pairs(pairs: &[(&str, &str)]) -> reqwest::header::HeaderMap {
let mut h = reqwest::header::HeaderMap::new();
for (k, v) in pairs {
h.insert(
k.parse::<reqwest::header::HeaderName>().unwrap(),
v.parse::<reqwest::header::HeaderValue>().unwrap(),
);
}
h
}
// ── icaptcha_cfg ────────────────────────────────────────────────────
#[test]
fn icaptcha_cfg_returns_some_when_both_headers_present() {
let kp = test_keypair();
let client = NodeClient::new("http://localhost", Some(kp.clone()));
let headers = headers_from_pairs(&[
("x-icaptcha-url", "https://icaptcha.gitlawb.com"),
("x-icaptcha-level", "3"),
]);
let cfg = client.icaptcha_cfg(&headers).unwrap().unwrap();
assert_eq!(cfg.did, kp.did().to_string());
assert_eq!(cfg.level, 3);
}
#[test]
fn icaptcha_cfg_defaults_level_when_only_url_present() {
let kp = test_keypair();
let client = NodeClient::new("http://localhost", Some(kp));
let headers = headers_from_pairs(&[("x-icaptcha-url", "https://icaptcha.gitlawb.com")]);
let cfg = client.icaptcha_cfg(&headers).unwrap().unwrap();
assert_eq!(cfg.level, icaptcha_client::DEFAULT_LEVEL);
}
#[test]
fn icaptcha_cfg_defaults_url_when_only_level_present() {
let kp = test_keypair();
let client = NodeClient::new("http://localhost", Some(kp));
let headers = headers_from_pairs(&[("x-icaptcha-level", "5")]);
let cfg = client.icaptcha_cfg(&headers).unwrap().unwrap();
assert_eq!(cfg.level, 5);
}
#[test]
fn icaptcha_cfg_returns_none_without_icaptcha_headers() {
let client = NodeClient::new("http://localhost", Some(test_keypair()));
let headers = reqwest::header::HeaderMap::new();
assert!(client.icaptcha_cfg(&headers).unwrap().is_none());
}
#[test]
fn icaptcha_cfg_returns_none_with_unrelated_headers() {
let client = NodeClient::new("http://localhost", Some(test_keypair()));
let headers = headers_from_pairs(&[("content-type", "application/json")]);
assert!(client.icaptcha_cfg(&headers).unwrap().is_none());
}
#[test]
fn icaptcha_cfg_errors_when_no_keypair() {
let client = NodeClient::new("http://localhost", None);
let headers = headers_from_pairs(&[("x-icaptcha-level", "3")]);
let err = client.icaptcha_cfg(&headers).unwrap_err();
assert!(err.to_string().contains("identity keypair"));
}
#[test]
fn icaptcha_cfg_ignores_unparseable_level() {
let client = NodeClient::new("http://localhost", Some(test_keypair()));
let headers = headers_from_pairs(&[
("x-icaptcha-url", "https://icaptcha.gitlawb.com"),
("x-icaptcha-level", "not-a-number"),
]);
let cfg = client.icaptcha_cfg(&headers).unwrap().unwrap();
assert_eq!(cfg.level, icaptcha_client::DEFAULT_LEVEL);
}
// ── send_once ───────────────────────────────────────────────────────
#[tokio::test]
async fn send_once_attaches_proof_header_when_provided() {
let mut server = Server::new_async().await;
let m = server
.mock("POST", "/api/test")
.match_header("x-icaptcha-proof", "test.proof.token")
.with_status(200)
.with_body("ok")
.create_async()
.await;
let client = NodeClient::new(server.url(), None);
let resp = client
.send_once("POST", "/api/test", b"{}", Some("test.proof.token"))
.await
.unwrap();
assert_eq!(resp.status(), 200);
m.assert();
}
#[tokio::test]
async fn send_once_omits_proof_header_when_not_provided() {
let mut server = Server::new_async().await;
let m = server
.mock("POST", "/api/test")
.match_header("x-icaptcha-proof", mockito::Matcher::Missing)
.with_status(200)
.with_body("ok")
.create_async()
.await;
let client = NodeClient::new(server.url(), None);
let resp = client
.send_once("POST", "/api/test", b"{}", None)
.await
.unwrap();
assert_eq!(resp.status(), 200);
m.assert();
}
#[tokio::test]
async fn send_once_signs_request_when_keypair_present() {
let mut server = Server::new_async().await;
let m = server
.mock("POST", "/api/test")
.match_header("Signature", mockito::Matcher::Any)
.match_header("Signature-Input", mockito::Matcher::Any)
.match_header("Content-Digest", mockito::Matcher::Any)
.with_status(200)
.with_body("ok")
.create_async()
.await;
let client = NodeClient::new(server.url(), Some(test_keypair()));
let resp = client
.send_once("POST", "/api/test", b"{}", None)
.await
.unwrap();
assert_eq!(resp.status(), 200);
m.assert();
}
#[tokio::test]
async fn send_once_does_not_sign_when_no_keypair() {
let mut server = Server::new_async().await;
let m = server
.mock("POST", "/api/test")
.match_header("Signature", mockito::Matcher::Missing)
.match_header("Signature-Input", mockito::Matcher::Missing)
.match_header("Content-Digest", mockito::Matcher::Missing)
.with_status(200)
.with_body("ok")
.create_async()
.await;
let client = NodeClient::new(server.url(), None);
let resp = client
.send_once("POST", "/api/test", b"{}", None)
.await
.unwrap();
assert_eq!(resp.status(), 200);
m.assert();
}
// ── send_signed ─────────────────────────────────────────────────────
#[tokio::test]
async fn send_signed_returns_non_icaptcha_403_without_retry() {
let mut server = Server::new_async().await;
let m = server
.mock("POST", "/api/register")
.with_status(403)
.with_header("content-type", "application/json")
.with_body(r#"{"error":"forbidden"}"#)
.create_async()
.await;
let client = NodeClient::new(server.url(), Some(test_keypair()));
let resp = client
.send_signed("POST", "/api/register", b"{}")
.await
.unwrap();
assert_eq!(resp.status(), 403);
m.assert();
}
#[tokio::test]
async fn send_signed_returns_first_response_on_success() {
let mut server = Server::new_async().await;
let m = server
.mock("POST", "/api/register")
.with_status(201)
.with_header("content-type", "application/json")
.with_body(r#"{"status":"created"}"#)
.create_async()
.await;
let client = NodeClient::new(server.url(), Some(test_keypair()));
let resp = client
.send_signed("POST", "/api/register", b"{}")
.await
.unwrap();
assert_eq!(resp.status(), 201);
m.assert();
}
#[tokio::test]
async fn send_signed_handles_405_not_icaptcha() {
let mut server = Server::new_async().await;
let m = server
.mock("POST", "/api/register")
.with_status(405)
.with_body(r#"{"error":"method not allowed"}"#)
.create_async()
.await;
let client = NodeClient::new(server.url(), Some(test_keypair()));
let resp = client
.send_signed("POST", "/api/register", b"{}")
.await
.unwrap();
assert_eq!(resp.status(), 405);
m.assert();
}
// ── send_signed iCaptcha retry (full integration) ────────────────────
/// Set GITLAWB_ICAPTCHA_URL and GITLAWB_ICAPTCHA_INSECURE so the iCaptcha
/// client trusts a local mockito HTTP server, restoring any prior values on
/// drop so a test run launched with those variables keeps working.
/// Holds [`ICAPTCHA_ENV_LOCK`] for its lifetime so concurrent tests don't
/// race on the process-global env vars.
struct IcaptchaEnv {
_lock: MutexGuard<'static, ()>,
prev_url: Option<OsString>,
prev_insecure: Option<OsString>,
}
impl IcaptchaEnv {
fn new(url: &str) -> Self {
let lock = ICAPTCHA_ENV_LOCK.lock().unwrap();
let prev_url = std::env::var_os("GITLAWB_ICAPTCHA_URL");
let prev_insecure = std::env::var_os("GITLAWB_ICAPTCHA_INSECURE");
std::env::set_var("GITLAWB_ICAPTCHA_URL", url);
std::env::set_var("GITLAWB_ICAPTCHA_INSECURE", "1");
IcaptchaEnv {
_lock: lock,
prev_url,
prev_insecure,
}
}
}
impl Drop for IcaptchaEnv {
fn drop(&mut self) {
match self.prev_url.take() {
Some(v) => std::env::set_var("GITLAWB_ICAPTCHA_URL", v),
None => std::env::remove_var("GITLAWB_ICAPTCHA_URL"),
}
match self.prev_insecure.take() {
Some(v) => std::env::set_var("GITLAWB_ICAPTCHA_INSECURE", v),
None => std::env::remove_var("GITLAWB_ICAPTCHA_INSECURE"),
}
}
}
/// Set up a mock iCaptcha server that responds to challenge + answer.
/// `hits` sets the expected call count for both endpoints so the test can
/// verify the solve loop was entered the correct number of times.
struct MockIcaptcha {
challenge: mockito::Mock,
answer: mockito::Mock,
_guard: IcaptchaEnv,
url: String,
}
impl MockIcaptcha {
async fn new(server: &mut mockito::ServerGuard, hits: usize) -> Self {
let url = server.url();
let guard = IcaptchaEnv::new(&url);
let challenge = server
.mock("POST", "/v1/challenge")
.with_status(200)
.with_header("content-type", "application/json")
.with_body(
r#"{"challengeId":"c1","type":"arithmetic","difficulty":1,"prompt":"What is 1 + 1?","token":"tk1"}"#,
)
.expect(hits)
.create_async()
.await;
let answer = server
.mock("POST", "/v1/answer")
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"status":"passed","proof":"mock.proof"}"#)
.expect(hits)
.create_async()
.await;
Self {
challenge,
answer,
_guard: guard,
url,
}
}
}
#[tokio::test]
async fn send_signed_solves_icaptcha_and_retries_to_success() {
let mut node = Server::new_async().await;
let mut icaptcha = Server::new_async().await;
let ic = MockIcaptcha::new(&mut icaptcha, 1).await;
let n1 = node
.mock("POST", "/api/register")
.with_status(403)
.with_header("content-type", "application/json")
.with_header("x-icaptcha-url", &ic.url)
.with_header("x-icaptcha-level", "3")
.with_body(r#"{"error":"icaptcha_proof_required"}"#)
.expect(1)
.create_async()
.await;
let n2 = node
.mock("POST", "/api/register")
.match_header("x-icaptcha-proof", "mock.proof")
.with_status(201)
.with_header("content-type", "application/json")
.with_body(r#"{"status":"created"}"#)
.expect(1)
.create_async()
.await;
let client = NodeClient::new(node.url(), Some(test_keypair()));
let resp = client
.send_signed("POST", "/api/register", b"{}")
.await
.unwrap();
assert_eq!(resp.status(), 201);
n1.assert();
n2.assert();
ic.challenge.assert();
ic.answer.assert();
}
#[tokio::test]
async fn send_signed_returns_403_after_icaptcha_retries_exhausted() {
let mut node = Server::new_async().await;
let mut icaptcha = Server::new_async().await;
// MAX_ICAPTCHA_RETRIES = 2, so with every call returning 403 with
// iCaptcha headers the solve loop runs twice (2 challenge + 2 answer).
let ic = MockIcaptcha::new(&mut icaptcha, 2).await;
// The original + 2 retries = 3 node calls before the loop gives up.
let n = node
.mock("POST", "/api/register")
.with_status(403)
.with_header("content-type", "application/json")
.with_header("x-icaptcha-url", &ic.url)
.with_header("x-icaptcha-level", "3")
.with_body(r#"{"error":"icaptcha_proof_required"}"#)
.expect(3)
.create_async()
.await;
let client = NodeClient::new(node.url(), Some(test_keypair()));
let resp = client
.send_signed("POST", "/api/register", b"{}")
.await
.unwrap();
assert_eq!(resp.status(), 403);
n.assert();
ic.challenge.assert();
ic.answer.assert();
}
}
+334 -14
View File
@@ -3,6 +3,8 @@
//! Communicates with the gitlawb node to list pinned CIDs and retrieve git
//! objects by their content-addressed CID.
use std::path::PathBuf;
use anyhow::{Context, Result};
use clap::{Args, Subcommand};
use serde_json::Value;
@@ -21,6 +23,9 @@ pub enum IpfsCmd {
List {
#[arg(long, default_value = "https://node.gitlawb.com", env = "GITLAWB_NODE")]
node: String,
/// Identity directory (default: ~/.gitlawb)
#[arg(long)]
dir: Option<PathBuf>,
},
/// Retrieve and display a git object from the node by its CIDv1
Get {
@@ -28,24 +33,32 @@ pub enum IpfsCmd {
cid: String,
#[arg(long, default_value = "https://node.gitlawb.com", env = "GITLAWB_NODE")]
node: String,
/// Identity directory (default: ~/.gitlawb)
#[arg(long)]
dir: Option<PathBuf>,
},
}
pub async fn run(args: IpfsArgs) -> Result<()> {
match args.cmd {
IpfsCmd::List { node } => cmd_list(node).await,
IpfsCmd::Get { cid, node } => cmd_get(cid, node).await,
IpfsCmd::List { node, dir } => cmd_list(node, dir).await,
IpfsCmd::Get { cid, node, dir } => cmd_get(cid, node, dir).await,
}
}
async fn cmd_list(node: String) -> Result<()> {
let client = NodeClient::new(&node, None);
let resp: Value = client
.get("/api/v1/ipfs/pins")
.await?
.json()
.await
.context("failed to parse pins response")?;
async fn cmd_list(node: String, dir: Option<PathBuf>) -> Result<()> {
// #134 gates /api/v1/ipfs/pins behind auth: sign the request with the
// caller's identity. On no identity, propagate load_keypair_from_dir's
// error (it already names `gl identity new`) rather than a bare 401.
let keypair = crate::identity::load_keypair_from_dir(dir.as_deref())?;
let client = NodeClient::new(&node, Some(keypair));
let resp = client.get_signed("/api/v1/ipfs/pins").await?;
if !resp.status().is_success() {
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
anyhow::bail!("node returned {status} for pins listing: {body}");
}
let resp: Value = resp.json().await.context("failed to parse pins response")?;
let pins = resp["pins"].as_array().cloned().unwrap_or_default();
let count = resp["count"].as_u64().unwrap_or(pins.len() as u64);
@@ -76,11 +89,32 @@ async fn cmd_list(node: String) -> Result<()> {
Ok(())
}
async fn cmd_get(cid: String, node: String) -> Result<()> {
let client = NodeClient::new(&node, None);
let path = format!("/ipfs/{cid}");
async fn cmd_get(cid: String, node: String, dir: Option<PathBuf>) -> Result<()> {
// #173 (F5): the resolver now serves path-scoped objects to authorized readers,
// so sign with an available identity like `gl ipfs list` — otherwise an owner or
// listed reader gets the opaque anonymous 404 for content they can read.
// `get_authed` signs when a keypair is present and falls back to unsigned.
//
// An explicit `--dir` is a request to use THAT identity: propagate a
// missing/corrupt-keystore error (like `list`) instead of silently sending an
// anonymous request the authorized reader would see as the node's opaque 404
// (#173 review). Only the default (no `--dir`) keeps the best-effort unsigned
// fallback, so `get` stays usable for genuinely public content.
let keypair = match dir.as_deref() {
Some(dir) => Some(crate::identity::load_keypair_from_dir(Some(dir))?),
None => crate::identity::load_keypair_from_dir(None).ok(),
};
let client = NodeClient::new(&node, keypair);
// #173 review (F1): the node now accepts equivalent multibase spellings,
// including base64 CIDs (prefix 'm'), whose alphabet contains '/', '+', '='.
// Interpolating the CID raw would make the client request (and sign)
// `/ipfs/<prefix>/<suffix>`, which neither matches the single-segment Axum
// route nor points at the intended target. Percent-encode the CID as exactly
// one path segment so the signed and sent target agree and the server's
// `Path` extractor decodes it back to the original CID.
let path = format!("/ipfs/{}", encode_cid_segment(&cid));
let resp = client
.get(&path)
.get_authed(&path)
.await
.with_context(|| format!("failed to fetch CID {cid} from {node}"))?;
@@ -108,3 +142,289 @@ async fn cmd_get(cid: String, node: String) -> Result<()> {
Ok(())
}
/// Percent-encode a CID so it occupies exactly one path segment of `/ipfs/<cid>`.
/// `urlencoding::encode` escapes every byte outside the RFC 3986 unreserved set
/// (ALPHA / DIGIT / `-._~`), so the base64-CID characters that would otherwise
/// break the single-segment route — `/`, `+`, `=` — are all escaped, and the
/// server's `Path` extractor decodes the result back to the original CID (#173
/// review, F1).
fn encode_cid_segment(cid: &str) -> String {
urlencoding::encode(cid).into_owned()
}
#[cfg(test)]
mod tests {
use super::*;
/// Seed a keypair into a temp dir the way `load_keypair_from_dir` expects,
/// then return the dir handle (keeps it alive for the test's duration).
fn seed_keystore() -> tempfile::TempDir {
let dir = tempfile::TempDir::new().unwrap();
let kp = gitlawb_core::identity::Keypair::generate();
std::fs::write(
dir.path().join("identity.pem"),
kp.to_pem().unwrap().as_bytes(),
)
.unwrap();
dir
}
#[tokio::test]
async fn test_cmd_list_signs_request_and_renders_pins() {
let mut server = mockito::Server::new_async().await;
let keystore = seed_keystore();
// Happy path: signed GET to /api/v1/ipfs/pins carrying the RFC 9421
// signature headers, node returns a populated pins body.
let m = server
.mock("GET", "/api/v1/ipfs/pins")
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.match_header("content-digest", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(
r#"{"pins":[{"cid":"bafyone","sha256_hex":"abc123","pinned_at":"2026-07-02T12:00:00.123456Z"}],"count":1}"#,
)
.create_async()
.await;
cmd_list(server.url(), Some(keystore.path().to_path_buf()))
.await
.unwrap();
m.assert_async().await;
}
#[tokio::test]
async fn test_cmd_list_empty_pins() {
let mut server = mockito::Server::new_async().await;
let keystore = seed_keystore();
let m = server
.mock("GET", "/api/v1/ipfs/pins")
.match_header("signature", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"pins":[],"count":0}"#)
.create_async()
.await;
cmd_list(server.url(), Some(keystore.path().to_path_buf()))
.await
.unwrap();
m.assert_async().await;
}
#[tokio::test]
async fn test_cmd_list_no_identity_errors_without_request() {
let mut server = mockito::Server::new_async().await;
// Empty keystore dir: no identity.pem present.
let empty = tempfile::TempDir::new().unwrap();
// The endpoint must never be hit when there is no identity.
let m = server
.mock("GET", "/api/v1/ipfs/pins")
.expect(0)
.create_async()
.await;
let err = cmd_list(server.url(), Some(empty.path().to_path_buf()))
.await
.expect_err("no identity should be an error");
assert!(
err.to_string().contains("gl identity new")
|| err.to_string().contains("no identity found"),
"error should name `gl identity new`, got: {err}"
);
m.assert_async().await;
}
#[tokio::test]
async fn test_cmd_list_non_success_status_is_error_not_empty() {
let mut server = mockito::Server::new_async().await;
let keystore = seed_keystore();
// A signed request the node rejects (401) must surface as an error,
// not be silently parsed into an empty pin list.
let m = server
.mock("GET", "/api/v1/ipfs/pins")
.match_header("signature", mockito::Matcher::Any)
.with_status(401)
.with_header("content-type", "application/json")
.with_body(r#"{"error":"unauthorized"}"#)
.create_async()
.await;
let err = cmd_list(server.url(), Some(keystore.path().to_path_buf()))
.await
.expect_err("non-2xx status should be an error");
assert!(
err.to_string().contains("401"),
"error should mention the status, got: {err}"
);
m.assert_async().await;
}
/// #173 (F5): `gl ipfs get` must SIGN with an available identity, like
/// `gl ipfs list`, so an owner/reader can retrieve a path-scoped object the node
/// now resolves by CID. RED before the fix: cmd_get ignores the identity dir and
/// sends an unsigned request, so the signature-matching mock is never hit
/// (cmd_get errors on the unmatched 501, and m.assert fails). GREEN after: the
/// signed request carries the RFC 9421 headers and is served 200.
#[tokio::test]
async fn test_cmd_get_signs_when_identity_present() {
let mut server = mockito::Server::new_async().await;
let keystore = seed_keystore();
let m = server
.mock("GET", "/ipfs/bafkreitestcid")
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/octet-stream")
.with_header("x-git-hash", "abc123")
.with_body("object bytes")
.create_async()
.await;
cmd_get(
"bafkreitestcid".to_string(),
server.url(),
Some(keystore.path().to_path_buf()),
)
.await
.expect("signed get of a resolvable object should succeed");
m.assert_async().await;
}
/// #173 (F5) must-not: a genuine anonymous denial must surface as an error, not
/// be masked as success. With no identity dir the request is unsigned; a 404
/// from the node must produce an Err mentioning the status.
#[tokio::test]
async fn test_cmd_get_anonymous_denial_is_error() {
let mut server = mockito::Server::new_async().await;
let m = server
.mock("GET", "/ipfs/bafkreidenied")
.with_status(404)
.with_header("content-type", "text/plain")
.with_body("no git object found")
.create_async()
.await;
let err = cmd_get("bafkreidenied".to_string(), server.url(), None)
.await
.expect_err("a 404 denial must be an error, not masked success");
assert!(
err.to_string().contains("404"),
"error should mention the status, got: {err}"
);
m.assert_async().await;
}
/// #173 (INV-8) must-not: the node's new 503 "search incomplete" (the legacy CID
/// scan hit its bound and could not prove absence) must surface as an actionable
/// Err naming the status, NOT be rendered as an empty/"not found" success — a
/// retryable outcome the caller has to see. Mirrors the 404 denial case for the
/// bounded-search response the resolver now emits.
#[tokio::test]
async fn test_cmd_get_search_incomplete_503_is_error() {
let mut server = mockito::Server::new_async().await;
let m = server
.mock("GET", "/ipfs/bafkreiincomplete")
.with_status(503)
.with_header("content-type", "application/json")
.with_body(r#"{"error":"search_incomplete","message":"CID search incomplete — retry"}"#)
.create_async()
.await;
let err = cmd_get("bafkreiincomplete".to_string(), server.url(), None)
.await
.expect_err("a 503 incomplete-search must be an error, not masked as not-found");
assert!(
err.to_string().contains("503"),
"error should mention the status, got: {err}"
);
m.assert_async().await;
}
/// #173 review (F1): a base64 CID (multibase prefix 'm') can contain '/', '+',
/// and '='. The client must percent-encode it into ONE path segment before
/// building and signing `/ipfs/<cid>`; otherwise the '/' splits the target so
/// it misses the single-segment Axum route and the signature covers the wrong
/// path. Assert the encoded segment carries no raw '/', '+', or '=', and that
/// it decodes back to the original CID (the server's `Path` extractor performs
/// that same decode). RED with the old raw `format!("/ipfs/{cid}")`: the
/// segment still contains '/'.
#[test]
fn test_encode_cid_segment_escapes_base64_alphabet() {
let cid = "mFoo/Bar+baz==";
let encoded = encode_cid_segment(cid);
assert!(
!encoded.contains('/'),
"encoded CID must be a single path segment (no raw '/'), got: {encoded}"
);
assert!(
!encoded.contains('+'),
"encoded CID must escape '+', got: {encoded}"
);
assert!(
!encoded.contains('='),
"encoded CID must escape '=', got: {encoded}"
);
let decoded = urlencoding::decode(&encoded).expect("encoded CID must decode");
assert_eq!(
decoded, cid,
"encoding must round-trip back to the original CID"
);
}
/// #173 review: `gl ipfs get --dir <path>` must PROPAGATE a missing/corrupt
/// identity-load error like `gl ipfs list`, not silently fall back to an anonymous
/// request — otherwise an authorized reader pointing `--dir` at a broken keystore
/// gets the node's opaque 404 instead of the actionable key-load error. The
/// unsigned fallback is preserved only when NO `--dir` is given (covered by
/// `test_cmd_get_anonymous_denial_is_error`). RED before the fix (`.ok()` swallows
/// the error, an anonymous request is sent, and the `.expect(0)` mock is hit),
/// GREEN after.
#[tokio::test]
async fn test_cmd_get_explicit_dir_no_identity_errors_without_request() {
let mut server = mockito::Server::new_async().await;
// Empty keystore dir passed explicitly via --dir: no identity.pem present.
let empty = tempfile::TempDir::new().unwrap();
// The endpoint must never be hit when an explicit --dir fails to load.
let m = server
.mock("GET", "/ipfs/bafkreitestcid")
.expect(0)
.create_async()
.await;
let err = cmd_get(
"bafkreitestcid".to_string(),
server.url(),
Some(empty.path().to_path_buf()),
)
.await
.expect_err("an explicit --dir that fails to load must be an error");
assert!(
err.to_string().contains("gl identity new")
|| err.to_string().contains("no identity found")
|| err.to_string().contains("failed to load keypair"),
"error should name the key-load failure, got: {err}"
);
m.assert_async().await;
}
}
+10 -6
View File
@@ -10,6 +10,10 @@ use uuid::Uuid;
use crate::http::NodeClient;
use crate::identity::load_keypair_from_dir;
fn signed_client(node: &str, dir: Option<&std::path::Path>) -> NodeClient {
NodeClient::new(node, load_keypair_from_dir(dir).ok())
}
#[derive(Args)]
pub struct IssueArgs {
#[command(subcommand)]
@@ -215,10 +219,10 @@ async fn cmd_create(
async fn cmd_list(repo: String, node: String, dir: Option<PathBuf>) -> Result<()> {
let (owner, name) = resolve_repo(&repo, &node, dir.as_deref()).await?;
let client = NodeClient::new(&node, None);
let client = signed_client(&node, dir.as_deref());
let path = format!("/api/v1/repos/{owner}/{name}/issues");
let resp: Value = client
.get(&path)
.get_authed(&path)
.await?
.json()
.await
@@ -254,10 +258,10 @@ async fn cmd_list(repo: String, node: String, dir: Option<PathBuf>) -> Result<()
async fn cmd_show(repo: String, id: String, node: String, dir: Option<PathBuf>) -> Result<()> {
let (owner, name) = resolve_repo(&repo, &node, dir.as_deref()).await?;
let client = NodeClient::new(&node, None);
let client = signed_client(&node, dir.as_deref());
let path = format!("/api/v1/repos/{owner}/{name}/issues/{id}");
let resp = client
.get(&path)
.get_authed(&path)
.await
.context("failed to connect to node")?;
let status = resp.status();
@@ -347,10 +351,10 @@ async fn cmd_issue_comments(
dir: Option<PathBuf>,
) -> Result<()> {
let (owner, name) = resolve_repo(&repo, &node, dir.as_deref()).await?;
let client = NodeClient::new(&node, None);
let client = signed_client(&node, dir.as_deref());
let resp: Value = client
.get(&format!(
.get_authed(&format!(
"/api/v1/repos/{owner}/{name}/issues/{id}/comments"
))
.await?
+3 -3
View File
@@ -970,14 +970,14 @@ async fn call_tool(
}
u
};
let resp: Value = client.get(&url).await?.json().await?;
let resp: Value = client.get_authed(&url).await?.json().await?;
Ok(serde_json::to_string_pretty(&resp)?)
}
"bounty_show" => {
let id = args["id"].as_str().context("missing 'id'")?;
let resp: Value = client
.get(&format!("/api/v1/bounties/{id}"))
.get_authed(&format!("/api/v1/bounties/{id}"))
.await?
.json()
.await?;
@@ -1165,7 +1165,7 @@ async fn call_tool(
(owner, repo.to_string())
};
let resp: Value = client
.get(&format!("/api/v1/repos/{owner}/{name}/issues"))
.get_authed(&format!("/api/v1/repos/{owner}/{name}/issues"))
.await?
.json()
.await?;
+379 -12
View File
@@ -2,10 +2,12 @@
use anyhow::Result;
use clap::{Args, Subcommand};
use gitlawb_core::identity::Keypair;
use serde_json::Value;
use std::path::PathBuf;
use crate::http::NodeClient;
use crate::identity::load_keypair_from_dir;
use crate::node_stake;
#[derive(Args)]
@@ -20,6 +22,9 @@ pub enum NodeCmd {
Status {
#[arg(long, default_value = "https://node.gitlawb.com", env = "GITLAWB_NODE")]
node: String,
/// Identity directory (default: ~/.gitlawb)
#[arg(long)]
dir: Option<PathBuf>,
},
/// Check trust score for a DID
Trust {
@@ -121,7 +126,7 @@ pub enum NodeCmd {
pub async fn run(args: NodeArgs) -> Result<()> {
match args.cmd {
NodeCmd::Status { node } => cmd_status(node).await,
NodeCmd::Status { node, dir } => cmd_status(node, dir).await,
NodeCmd::Trust { did, node } => cmd_trust(did, node).await,
NodeCmd::Resolve { did, node } => cmd_resolve(did, node).await,
NodeCmd::Register {
@@ -177,7 +182,101 @@ async fn try_get_json(client: &NodeClient, path: &str) -> Option<Value> {
resp.json::<Value>().await.ok()
}
async fn cmd_status(node: String) -> Result<()> {
/// Outcome of fetching the IPFS pins panel for `gl node status`.
///
/// #134 gates `/api/v1/ipfs/pins` behind auth, so this panel signs its request
/// when an identity is available and otherwise reports that the caller must
/// sign in. A pins failure never aborts the dashboard.
#[derive(Debug)]
enum PinsPanel {
/// Signed read succeeded and returned pins (carries the resolved count).
Pins(u64),
/// Signed read succeeded but the node has no pins recorded.
Empty,
/// Signed read was rejected (401/other) or errored.
Unavailable,
/// No identity available; no request was issued.
NeedsIdentity,
/// An explicit `--dir` was given but held no usable identity; carries the dir
/// so the panel can name it instead of the misleading "sign in" prompt.
IdentityError(PathBuf),
}
/// Identity resolved for the signed pins read, decided before the request so the
/// dashboard can distinguish "no identity at all" from "an explicit `--dir` that
/// could not be loaded".
enum PinsAuth {
/// A usable keypair; the read will be signed.
Keyed(Keypair),
/// No identity requested and none in the default keystore.
Anonymous,
/// An explicit `--dir` was given but held no usable identity (carries the dir).
DirUnusable(PathBuf),
}
/// Resolve the pins-panel identity from an optional `--dir`. Mirrors `gl ipfs
/// list`'s identity selection: an explicit `--dir` that fails to load is reported
/// as `DirUnusable` (so the panel names the bad path) rather than collapsing to
/// the misleading "sign in to view"; a missing default keystore (no `--dir`)
/// degrades quietly to `Anonymous`.
fn resolve_pins_auth(dir: Option<&std::path::Path>) -> PinsAuth {
match load_keypair_from_dir(dir) {
Ok(kp) => PinsAuth::Keyed(kp),
Err(_) => match dir {
Some(d) => PinsAuth::DirUnusable(d.to_path_buf()),
None => PinsAuth::Anonymous,
},
}
}
/// Fetch the pins panel state. With a keypair, signs the `/api/v1/ipfs/pins`
/// read and maps the outcome; with no identity it returns `NeedsIdentity`, and
/// with an unusable explicit `--dir` it returns `IdentityError` — both without
/// issuing a request. Injectable (node URL + resolved auth) so tests drive it
/// with a mock server and never touch the default keystore.
async fn fetch_pins(node: &str, auth: PinsAuth) -> PinsPanel {
let kp = match auth {
PinsAuth::Keyed(kp) => kp,
PinsAuth::Anonymous => return PinsPanel::NeedsIdentity,
PinsAuth::DirUnusable(dir) => return PinsPanel::IdentityError(dir),
};
let client = NodeClient::new(node, Some(kp));
let resp = match client.get_signed("/api/v1/ipfs/pins").await {
Ok(r) => r,
Err(_) => return PinsPanel::Unavailable,
};
if !resp.status().is_success() {
return PinsPanel::Unavailable;
}
let Ok(body) = resp.json::<Value>().await else {
return PinsPanel::Unavailable;
};
let count = body["count"]
.as_u64()
.unwrap_or_else(|| body["pins"].as_array().map(|a| a.len() as u64).unwrap_or(0));
if count == 0 {
PinsPanel::Empty
} else {
PinsPanel::Pins(count)
}
}
/// Render the one-line pins-panel status for the `gl node status` dashboard.
fn pins_status_line(panel: &PinsPanel) -> String {
match panel {
PinsPanel::Pins(count) => format!(" Pinned CIDs: {count}"),
PinsPanel::Empty => " Pinned CIDs: 0".to_string(),
PinsPanel::Unavailable => " IPFS pins: unavailable".to_string(),
PinsPanel::NeedsIdentity => {
" IPFS pins: sign in to view (run `gl identity new`)".to_string()
}
PinsPanel::IdentityError(dir) => {
format!(" IPFS pins: no usable identity in {}", dir.display())
}
}
}
async fn cmd_status(node: String, dir: Option<PathBuf>) -> Result<()> {
let client = NodeClient::new(&node, None);
// ── Fetch node info (required — bail if unreachable) ──────────────────
@@ -194,13 +293,21 @@ async fn cmd_status(node: String) -> Result<()> {
let version = info["version"].as_str().unwrap_or("unknown");
let network = info["network"].as_str().unwrap_or("unknown");
// The pins panel signs its read (#134 gates /api/v1/ipfs/pins behind auth).
// `--dir` selects the same identity directory as `gl ipfs list` (#146); an
// explicit --dir that can't be loaded is surfaced in the panel rather than
// masquerading as "no identity", while a missing default keystore degrades
// quietly. A pins failure never aborts the dashboard.
let pins_auth = resolve_pins_auth(dir.as_deref());
// ── Fetch remaining endpoints in parallel ─────────────────────────────
let (peers_val, repos_val, p2p_val, events_val, pins_val) = tokio::join!(
// Peers/repos/p2p/events stay anonymous; only pins is signed.
let (peers_val, repos_val, p2p_val, events_val, pins_panel) = tokio::join!(
try_get_json(&client, "/api/v1/peers"),
try_get_json(&client, "/api/v1/repos"),
try_get_json(&client, "/api/v1/p2p/info"),
try_get_json(&client, "/api/v1/events/ref-updates?limit=5"),
try_get_json(&client, "/api/v1/ipfs/pins"),
fetch_pins(&node, pins_auth),
);
// ── Render dashboard ──────────────────────────────────────────────────
@@ -307,14 +414,7 @@ async fn cmd_status(node: String) -> Result<()> {
// Pins
println!("Pins");
if let Some(ref pins) = pins_val {
let count = pins["count"]
.as_u64()
.unwrap_or_else(|| pins["pins"].as_array().map(|a| a.len() as u64).unwrap_or(0));
println!(" Pinned CIDs: {count}");
} else {
println!(" IPFS not configured");
}
println!("{}", pins_status_line(&pins_panel));
println!();
Ok(())
@@ -409,3 +509,270 @@ async fn cmd_resolve(did: String, node: String) -> Result<()> {
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use gitlawb_core::identity::Keypair;
#[tokio::test]
async fn test_fetch_pins_keyed_happy_signs_and_returns_pins() {
let mut server = mockito::Server::new_async().await;
let kp = Keypair::generate();
// A keyed fetch must sign the request (RFC 9421 headers) and, on a
// populated 200 body, land in the Pins state carrying the pins.
let m = server
.mock("GET", "/api/v1/ipfs/pins")
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.match_header("content-digest", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(
r#"{"pins":[{"cid":"bafyone","sha256_hex":"abc123","pinned_at":"2026-07-02T12:00:00Z"}],"count":1}"#,
)
.create_async()
.await;
let panel = fetch_pins(&server.url(), PinsAuth::Keyed(kp)).await;
match panel {
PinsPanel::Pins(count) => assert_eq!(count, 1),
other => panic!("expected Pins, got {other:?}"),
}
m.assert_async().await;
}
#[tokio::test]
async fn test_fetch_pins_keyed_empty_returns_empty() {
let mut server = mockito::Server::new_async().await;
let kp = Keypair::generate();
let m = server
.mock("GET", "/api/v1/ipfs/pins")
.match_header("signature", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"pins":[],"count":0}"#)
.create_async()
.await;
let panel = fetch_pins(&server.url(), PinsAuth::Keyed(kp)).await;
assert!(
matches!(panel, PinsPanel::Empty),
"expected Empty, got {panel:?}"
);
m.assert_async().await;
}
#[tokio::test]
async fn test_fetch_pins_keyed_rejected_returns_unavailable() {
let mut server = mockito::Server::new_async().await;
let kp = Keypair::generate();
// Node rejects the signed read (401): the panel must degrade to
// Unavailable without panicking, so cmd_status still completes.
let m = server
.mock("GET", "/api/v1/ipfs/pins")
.match_header("signature", mockito::Matcher::Any)
.with_status(401)
.with_header("content-type", "application/json")
.with_body(r#"{"error":"unauthorized"}"#)
.create_async()
.await;
let panel = fetch_pins(&server.url(), PinsAuth::Keyed(kp)).await;
assert!(
matches!(panel, PinsPanel::Unavailable),
"expected Unavailable, got {panel:?}"
);
m.assert_async().await;
}
#[tokio::test]
async fn test_fetch_pins_unkeyed_needs_identity_without_request() {
let mut server = mockito::Server::new_async().await;
// With no keypair the endpoint must never be hit.
let m = server
.mock("GET", "/api/v1/ipfs/pins")
.expect(0)
.create_async()
.await;
let panel = fetch_pins(&server.url(), PinsAuth::Anonymous).await;
assert!(
matches!(panel, PinsPanel::NeedsIdentity),
"expected NeedsIdentity, got {panel:?}"
);
m.assert_async().await;
}
#[tokio::test]
async fn test_fetch_pins_malformed_body_returns_unavailable() {
let mut server = mockito::Server::new_async().await;
let kp = Keypair::generate();
// 2xx but the body is not valid JSON: must degrade to Unavailable,
// never panic.
let m = server
.mock("GET", "/api/v1/ipfs/pins")
.match_header("signature", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body("not json{{{")
.create_async()
.await;
let panel = fetch_pins(&server.url(), PinsAuth::Keyed(kp)).await;
assert!(
matches!(panel, PinsPanel::Unavailable),
"malformed body -> Unavailable, got {panel:?}"
);
m.assert_async().await;
}
#[tokio::test]
async fn test_fetch_pins_transport_error_returns_unavailable() {
// Bind then drop to obtain a definitely-closed port -> connection
// refused -> get_signed Err -> Unavailable (no panic).
let port = {
let l = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
l.local_addr().unwrap().port()
};
let kp = Keypair::generate();
let panel = fetch_pins(&format!("http://127.0.0.1:{port}"), PinsAuth::Keyed(kp)).await;
assert!(
matches!(panel, PinsPanel::Unavailable),
"transport error -> Unavailable, got {panel:?}"
);
}
#[test]
fn test_pins_status_line_renders_each_state() {
assert_eq!(pins_status_line(&PinsPanel::Pins(3)), " Pinned CIDs: 3");
assert_eq!(pins_status_line(&PinsPanel::Empty), " Pinned CIDs: 0");
assert_eq!(
pins_status_line(&PinsPanel::Unavailable),
" IPFS pins: unavailable"
);
assert_eq!(
pins_status_line(&PinsPanel::NeedsIdentity),
" IPFS pins: sign in to view (run `gl identity new`)"
);
assert_eq!(
pins_status_line(&PinsPanel::IdentityError(PathBuf::from("/tmp/id"))),
" IPFS pins: no usable identity in /tmp/id"
);
}
// Adversarial follow-up to jatmn #146 P2: an explicit `--dir` that holds no
// usable identity must resolve to DirUnusable (which names the path), NOT the
// Anonymous "sign in to view / run `gl identity new`" case — otherwise a user
// who did supply an identity dir is misdirected to create one they may have.
#[test]
fn resolve_pins_auth_reports_explicit_bad_dir_as_unusable() {
let bad = PathBuf::from("/nonexistent/gl-id-xyz");
match resolve_pins_auth(Some(&bad)) {
PinsAuth::DirUnusable(d) => assert_eq!(d, bad),
_ => panic!("explicit unusable --dir must be DirUnusable, not Anonymous"),
}
}
// Mirror of the above for the absent-`--dir` case: `None` must never resolve to
// DirUnusable — it degrades to Anonymous (no default keystore) or Keyed (one
// present). Env-independent: only the None->DirUnusable mapping is guarded, so it
// holds whether or not the runner has a default `~/.gitlawb` identity.
#[test]
fn resolve_pins_auth_none_is_never_dir_unusable() {
assert!(
!matches!(resolve_pins_auth(None), PinsAuth::DirUnusable(_)),
"absent --dir must degrade to Anonymous (or Keyed), never DirUnusable"
);
}
// The success wiring for #146: an explicit --dir with a valid identity must
// load that key (dir -> load_keypair_from_dir -> Keyed), so the pins read is
// signed with the selected identity rather than the default keystore.
#[test]
fn resolve_pins_auth_loads_keyed_identity_from_explicit_dir() {
let dir = tempfile::TempDir::new().unwrap();
let kp = Keypair::generate();
std::fs::write(
dir.path().join("identity.pem"),
kp.to_pem().unwrap().as_bytes(),
)
.unwrap();
let PinsAuth::Keyed(loaded) = resolve_pins_auth(Some(dir.path())) else {
panic!("an explicit --dir with a valid identity must resolve to Keyed");
};
// A bare `Keyed(_)` match would also pass if load fell through to the default
// keystore; assert the loaded identity is the one written to --dir so the pins
// read signs with the selected key, never the default.
assert_eq!(
loaded.did(),
kp.did(),
"must sign with the identity from --dir, not the default keystore"
);
}
#[tokio::test]
async fn test_fetch_pins_dir_unusable_reports_identity_error_without_request() {
let mut server = mockito::Server::new_async().await;
// An unusable explicit --dir must render the identity error and never hit
// the endpoint (no identity to sign with).
let m = server
.mock("GET", "/api/v1/ipfs/pins")
.expect(0)
.create_async()
.await;
let bad = PathBuf::from("/nonexistent/gl-id-xyz");
let panel = fetch_pins(&server.url(), PinsAuth::DirUnusable(bad.clone())).await;
match panel {
PinsPanel::IdentityError(d) => assert_eq!(d, bad),
other => panic!("expected IdentityError, got {other:?}"),
}
m.assert_async().await;
}
// jatmn #146 P2: `gl node status` must accept the same `--dir` identity
// selector as `gl ipfs list`, so a user who selected an identity via --dir can
// authenticate the status pins panel instead of seeing "sign in to view".
#[test]
fn status_accepts_dir_identity_selector() {
use clap::Parser;
#[derive(Parser)]
struct TestCli {
#[command(flatten)]
args: NodeArgs,
}
let cli = TestCli::try_parse_from([
"gl",
"status",
"--node",
"http://example",
"--dir",
"/tmp/id",
])
.expect("`node status` must accept --dir");
let NodeCmd::Status { dir, .. } = cli.args.cmd else {
panic!("expected the Status subcommand");
};
assert_eq!(dir, Some(PathBuf::from("/tmp/id")));
// Absent --dir must stay None so the default keystore path is unchanged.
let cli = TestCli::try_parse_from(["gl", "status"]).expect("status parses without --dir");
let NodeCmd::Status { dir, .. } = cli.args.cmd else {
panic!("expected the Status subcommand");
};
assert_eq!(dir, None);
}
}
+43 -32
View File
@@ -208,20 +208,16 @@ pub async fn run(args: RepoArgs) -> Result<()> {
}
/// Derive the short DID key segment from a keypair, or fall back to the node's DID.
async fn resolve_owner_did(node: &str, dir: Option<&std::path::Path>) -> Result<String> {
if let Ok(kp) = load_keypair_from_dir(dir) {
let did = kp.did().to_string();
return Ok(did.split(':').next_back().unwrap_or(&did).to_string());
}
let client = NodeClient::new(node, None);
let info: Value = client
.get("/")
.await?
.json()
.await
.context("failed to fetch node info")?;
let did = info["did"].as_str().context("node missing DID")?;
Ok(did.split(':').next_back().unwrap_or(did).to_string())
/// Resolve the owner short-DID for a bare repo name from the LOCAL identity.
/// Never falls back to the node's own DID (that produced bogus "owned by the
/// node" results for repos that don't exist) — if there's no local identity the
/// caller must pass an explicit `owner/name`.
async fn resolve_owner_did(_node: &str, dir: Option<&std::path::Path>) -> Result<String> {
let kp = load_keypair_from_dir(dir).context(
"no local identity to resolve the repo owner — pass `owner/name`, or run `gl identity new`",
)?;
let did = kp.did().to_string();
Ok(did.split(':').next_back().unwrap_or(&did).to_string())
}
async fn cmd_create(
@@ -306,25 +302,28 @@ async fn cmd_list(node: String, dir: Option<PathBuf>) -> Result<()> {
}
async fn cmd_clone(name: String, node: String, dir: Option<PathBuf>) -> Result<()> {
let did = if let Ok(kp) = load_keypair_from_dir(dir.as_deref()) {
kp.did().to_string()
// Owner is taken from an explicit `owner/name`, else the LOCAL identity —
// never the node's own DID. A missing repo then surfaces as the helper's
// clear 404 rather than a clone under an invented owner.
let (did, repo_name) = if let Some((owner, rest)) = name.split_once('/') {
(owner.to_string(), rest.to_string())
} else {
let client = NodeClient::new(&node, None);
let info: Value = client.get("/").await?.json().await?;
info["did"]
.as_str()
.context("node missing DID")?
.to_string()
// Bare name: derive the SHORT owner key via the same helper the other
// commands use, so the URL is `gitlawb://z6Mk.../name` — not the full
// `did:key:z6Mk...` form, whose colons in the authority break the helper.
(resolve_owner_did(&node, dir.as_deref()).await?, name)
};
let url = format!("gitlawb://{did}/{name}");
let url = format!("gitlawb://{did}/{repo_name}");
println!(" cloning {url}");
let status = std::process::Command::new("git")
.arg("clone")
.arg(&url)
// Point the remote helper at the same node the user selected.
.env("GITLAWB_NODE", &node)
.status()
.context("failed to run git clone — is git installed?")?;
if !status.success() {
anyhow::bail!("git clone failed");
anyhow::bail!("git clone failed — does the repo exist on this node?");
}
Ok(())
}
@@ -340,12 +339,24 @@ async fn cmd_info(repo: String, node: String, dir: Option<PathBuf>) -> Result<()
(short, repo)
};
let r: Value = client
.get(&format!("/api/v1/repos/{owner}/{name}"))
.await?
.json()
.await
.context("repo not found")?;
let resp = client.get(&format!("/api/v1/repos/{owner}/{name}")).await?;
// A non-existent (or unreadable/quarantined) repo is a real 404 from the
// node — surface it plainly instead of printing a stub card with `?` fields
// and a placeholder owner DID.
if !resp.status().is_success() {
if resp.status().as_u16() == 404 {
anyhow::bail!("repository '{owner}/{name}' not found");
}
let status = resp.status();
let msg = resp
.json::<Value>()
.await
.ok()
.and_then(|v| v["message"].as_str().map(String::from))
.unwrap_or_else(|| "request failed".to_string());
anyhow::bail!("repo info failed ({status}): {msg}");
}
let r: Value = resp.json().await.context("parse repo info")?;
let owner_did = r["owner_did"].as_str().unwrap_or(&owner);
let gitlawb_url = format!("gitlawb://{owner_did}/{name}");
@@ -647,10 +658,10 @@ async fn cmd_label_remove(
async fn cmd_label_list(repo: String, node: String, dir: Option<PathBuf>) -> Result<()> {
let (owner, name) = resolve_owner_repo_pair(&repo, &node, dir.as_deref()).await?;
let client = NodeClient::new(&node, None);
let client = NodeClient::new(&node, load_keypair_from_dir(dir.as_deref()).ok());
let resp: Value = client
.get(&format!("/api/v1/repos/{owner}/{name}/labels"))
.get_authed(&format!("/api/v1/repos/{owner}/{name}/labels"))
.await?
.json()
.await
+15 -9
View File
@@ -34,12 +34,14 @@ pub enum StarCmd {
#[arg(long)]
dir: Option<PathBuf>,
},
/// Show star count for a repository (no auth required)
/// Show star count for a repository
Count {
/// Repository in owner/repo format
repo: String,
#[arg(long, default_value = "https://node.gitlawb.com", env = "GITLAWB_NODE")]
node: String,
#[arg(long)]
dir: Option<PathBuf>,
},
}
@@ -47,7 +49,7 @@ pub async fn run(args: StarArgs) -> Result<()> {
match args.cmd {
StarCmd::Add { repo, node, dir } => cmd_add(repo, node, dir).await,
StarCmd::Remove { repo, node, dir } => cmd_remove(repo, node, dir).await,
StarCmd::Count { repo, node } => cmd_count(repo, node).await,
StarCmd::Count { repo, node, dir } => cmd_count(repo, node, dir).await,
}
}
@@ -110,15 +112,15 @@ async fn cmd_remove(repo: String, node: String, dir: Option<PathBuf>) -> Result<
Ok(())
}
async fn cmd_count(repo: String, node: String) -> Result<()> {
async fn cmd_count(repo: String, node: String, dir: Option<PathBuf>) -> Result<()> {
let (owner, name) = repo
.split_once('/')
.map(|(o, n)| (o.to_string(), n.to_string()))
.context("use owner/repo format for count (e.g. alice/myrepo)")?;
let client = NodeClient::new(&node, None);
let client = NodeClient::new(&node, load_keypair_from_dir(dir.as_deref()).ok());
let resp = client
.get(&format!("/api/v1/repos/{owner}/{name}/star"))
.get_authed(&format!("/api/v1/repos/{owner}/{name}/star"))
.await
.context("failed to connect to node")?;
@@ -310,16 +312,20 @@ mod tests {
.create_async()
.await;
cmd_count("alice/myrepo".to_string(), server.url())
cmd_count("alice/myrepo".to_string(), server.url(), None)
.await
.unwrap();
}
#[tokio::test]
async fn test_cmd_count_requires_slash() {
let err = cmd_count("noslash".to_string(), "http://127.0.0.1:1".to_string())
.await
.unwrap_err();
let err = cmd_count(
"noslash".to_string(),
"http://127.0.0.1:1".to_string(),
None,
)
.await
.unwrap_err();
assert!(err.to_string().contains("owner/repo format"));
}
+285 -10
View File
@@ -1,4 +1,4 @@
use anyhow::Result;
use anyhow::{Context, Result};
use clap::{Args, Subcommand};
use std::path::PathBuf;
@@ -30,15 +30,39 @@ pub enum SyncCmd {
pub async fn run(args: SyncArgs) -> Result<()> {
match args.cmd {
SyncCmd::Trigger => {
let keypair = load_keypair_from_dir(args.dir.as_deref()).ok();
let client = NodeClient::new(&args.node, keypair);
let resp: serde_json::Value = client
.post("/api/v1/sync/trigger", b"{}")
.await?
.json()
.await?;
let peers = resp["peers_reached"].as_u64().unwrap_or(0);
let enqueued = resp["repos_enqueued"].as_u64().unwrap_or(0);
// /api/v1/sync/trigger always requires a signature, so a missing or
// unreadable identity must fail here, locally, rather than sending an
// unsigned request that can only 401 remotely (matches the other
// signed CLI writes).
let keypair = load_keypair_from_dir(args.dir.as_deref())
.context("identity not found — run `gl identity new` first")?;
let client = NodeClient::new(&args.node, Some(keypair));
let resp = client.post("/api/v1/sync/trigger", b"{}").await?;
// The node now requires a signature on this route and rate-limits it,
// so a denial (401/429/…) is expected. Check the status BEFORE parsing:
// otherwise a JSON-ish error body deserializes into a zero-count struct
// and prints a fabricated "✓ sync triggered / 0 peers" success.
let status = resp.status();
if !status.is_success() {
// Bound the read: a hostile or broken node must not force an
// unbounded allocation just to surface a denial (INV-6, read half).
let raw = read_body_capped(resp, 8 * 1024).await;
let msg = serde_json::from_str::<serde_json::Value>(&raw)
.ok()
.and_then(|v| {
v.get("message")
.or_else(|| v.get("error"))
.and_then(|m| m.as_str())
.map(str::to_string)
})
.unwrap_or(raw);
anyhow::bail!(
"sync trigger failed ({status}): {}",
sanitize_node_msg(&msg)
);
}
let resp: serde_json::Value = resp.json().await?;
let (peers, enqueued) = trigger_counts(&resp);
println!("✓ sync triggered");
println!(" peers reached: {peers}");
println!(" repos enqueued: {enqueued}");
@@ -69,3 +93,254 @@ pub async fn run(args: SyncArgs) -> Result<()> {
}
Ok(())
}
/// Extract `(peers_reached, repos_enqueued)` from a successful sync-trigger
/// response. Split out so the extraction is unit-testable (missing or malformed
/// fields default to 0 rather than panicking).
fn trigger_counts(resp: &serde_json::Value) -> (u64, u64) {
(
resp["peers_reached"].as_u64().unwrap_or(0),
resp["repos_enqueued"].as_u64().unwrap_or(0),
)
}
/// Read at most `cap` bytes of a response body. Bounds the allocation from a
/// hostile or broken node returning a huge error body — the display is capped
/// separately, but the read itself must not be unbounded (INV-6, read half).
async fn read_body_capped(mut resp: reqwest::Response, cap: usize) -> String {
let mut buf: Vec<u8> = Vec::new();
while buf.len() < cap {
match resp.chunk().await {
Ok(Some(chunk)) => {
let take = (cap - buf.len()).min(chunk.len());
buf.extend_from_slice(&chunk[..take]);
if take < chunk.len() {
break; // hit the cap mid-chunk
}
}
_ => break, // end of body or read error — return what we have
}
}
String::from_utf8_lossy(&buf).into_owned()
}
/// Strip terminal-dangerous characters from (and cap the length of) a
/// node-supplied error string before surfacing it. The node a caller talks to
/// could be hostile and embed escape sequences in its error body; those must not
/// reach the terminal verbatim (INV-6). We drop the C0/C1 control bytes (which
/// defangs ANSI/OSC escapes) AND the Unicode bidi/format controls (which
/// `char::is_control` does not cover — they can reorder the displayed line).
fn sanitize_node_msg(s: &str) -> String {
s.chars()
.filter(|c| !c.is_control() && !is_bidi_format(*c))
.take(200)
.collect()
}
/// Unicode bidirectional and directional-isolate format characters (category
/// `Cf`). These are not `char::is_control()` (that is category `Cc` only), but a
/// right-to-left override or isolate can visually reorder a terminal line to
/// spoof the error text, so they are stripped alongside the control bytes.
fn is_bidi_format(c: char) -> bool {
matches!(c,
'\u{200E}' | '\u{200F}' | '\u{061C}' // LRM, RLM, ALM
| '\u{202A}'..='\u{202E}' // LRE, RLE, PDF, LRO, RLO
| '\u{2066}'..='\u{2069}' // LRI, RLI, FSI, PDI
)
}
#[cfg(test)]
mod tests {
use super::*;
fn trigger_args(node: String) -> (SyncArgs, tempfile::TempDir) {
// Seed a real identity so `run` gets past the mandatory-keypair check and
// reaches the status-handling path. The mocks below return a fixed status
// regardless of the signature, so these tests exercise the client's
// status-check-before-parse, not signature verification (that is proved
// server-side). Return the TempDir so the caller keeps it alive.
let dir = tempfile::TempDir::new().unwrap();
let kp = gitlawb_core::identity::Keypair::generate();
std::fs::write(
dir.path().join("identity.pem"),
kp.to_pem().unwrap().as_bytes(),
)
.unwrap();
let args = SyncArgs {
cmd: SyncCmd::Trigger,
node,
dir: Some(dir.path().to_path_buf()),
};
(args, dir)
}
#[tokio::test]
async fn trigger_requires_identity_fails_before_request() {
// Empty identity dir → no keypair. `sync trigger` must fail locally with
// a clear identity error BEFORE issuing any request. The node URL points
// at an unreachable port, so a request attempt would surface a different
// (connection) error; getting the identity error proves we never dialed.
let dir = tempfile::TempDir::new().unwrap();
let args = SyncArgs {
cmd: SyncCmd::Trigger,
node: "http://127.0.0.1:1".to_string(),
dir: Some(dir.path().to_path_buf()),
};
let err = run(args).await.unwrap_err();
assert!(
err.to_string().contains("identity not found"),
"expected a local identity error before any request, got: {err}"
);
}
#[tokio::test]
async fn trigger_surfaces_401_as_error_not_fake_success() {
let mut server = mockito::Server::new_async().await;
let _m = server
.mock("POST", "/api/v1/sync/trigger")
.with_status(401)
.with_header("content-type", "application/json")
// Valid JSON: the parse-without-status-check bug deserializes this
// into a zero-count success struct and prints "✓ sync triggered".
.with_body(r#"{"message":"unauthorized"}"#)
.create_async()
.await;
let (args, _dir) = trigger_args(server.url());
let err = run(args).await.unwrap_err();
assert!(
err.to_string().contains("401"),
"expected 401 surfaced, got: {err}"
);
}
#[tokio::test]
async fn trigger_surfaces_429_as_error() {
let mut server = mockito::Server::new_async().await;
let _m = server
.mock("POST", "/api/v1/sync/trigger")
.with_status(429)
.with_header("content-type", "application/json")
.with_body(r#"{"message":"slow down"}"#)
.create_async()
.await;
let (args, _dir) = trigger_args(server.url());
let err = run(args).await.unwrap_err();
assert!(
err.to_string().contains("429"),
"expected 429 surfaced, got: {err}"
);
}
#[tokio::test]
async fn trigger_sanitizes_control_chars_in_node_error() {
// A hostile node embeds an ANSI color escape (ESC) and a bell (BEL) in
// the JSON message field. The surfaced error must contain neither raw
// control byte, while keeping the printable text.
let mut server = mockito::Server::new_async().await;
let _m = server
.mock("POST", "/api/v1/sync/trigger")
.with_status(401)
.with_header("content-type", "application/json")
// Valid JSON whose message carries JSON-escaped ESC (\u001b) and
// BEL (\u0007); serde decodes them to real control bytes a naive
// client would print. (The status-check bug fake-successes here.)
.with_body("{\"message\":\"pwned\\u001b[31m\\u0007bad\"}")
.create_async()
.await;
let (args, _dir) = trigger_args(server.url());
let err = run(args).await.unwrap_err();
let s = err.to_string();
assert!(!s.contains('\u{1b}'), "ESC leaked to terminal: {s:?}");
assert!(!s.contains('\u{07}'), "BEL leaked to terminal: {s:?}");
assert!(
s.contains("pwned") && s.contains("bad"),
"message text dropped: {s:?}"
);
}
#[tokio::test]
async fn trigger_ok_prints_counts() {
let mut server = mockito::Server::new_async().await;
let _m = server
.mock("POST", "/api/v1/sync/trigger")
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"peers_reached":2,"repos_enqueued":5}"#)
.create_async()
.await;
let (args, _dir) = trigger_args(server.url());
run(args).await.unwrap();
}
#[test]
fn sanitize_strips_controls_bidi_and_caps_length() {
// C0 (ESC/BEL) and the Cf bidi override (U+202E) are both removed; the
// printable text survives. (Note: a stripped ESC leaves any following
// "[31m" as inert literal text — that is the point, so the input here
// avoids that residue to keep the expectation unambiguous.)
let out = sanitize_node_msg("a\u{1b}\u{07}b\u{202e}c");
assert!(
!out.chars().any(|c| c.is_control()),
"control char leaked: {out:?}"
);
assert!(
!out.contains('\u{202e}'),
"RLO bidi override leaked: {out:?}"
);
assert_eq!(out, "abc");
// Length is capped at 200 chars regardless of input size.
let long = "x".repeat(250);
assert_eq!(sanitize_node_msg(&long).chars().count(), 200);
}
#[tokio::test]
async fn trigger_handles_oversized_error_body_without_unbounded_output() {
// A hostile/broken node returns a 2 MB error body. The command must still
// surface the denial with a bounded message, not hang or dump the body.
let mut server = mockito::Server::new_async().await;
let _m = server
.mock("POST", "/api/v1/sync/trigger")
.with_status(401)
.with_body("A".repeat(2_000_000))
.create_async()
.await;
let (args, _dir) = trigger_args(server.url());
let err = run(args).await.unwrap_err();
let s = err.to_string();
assert!(s.contains("401"), "denial not surfaced: {s:.80?}");
assert!(
s.len() < 500,
"error message not bounded: {} chars",
s.len()
);
}
#[tokio::test]
async fn read_body_capped_bounds_the_read() {
// The read must stop at the cap — a 2 MB body yields at most `cap` bytes,
// not the whole thing (which resp.text() would return).
let mut server = mockito::Server::new_async().await;
let _m = server
.mock("GET", "/big")
.with_status(200)
.with_body("A".repeat(2_000_000))
.create_async()
.await;
let resp = reqwest::get(format!("{}/big", server.url())).await.unwrap();
let out = read_body_capped(resp, 8192).await;
assert!(out.len() <= 8192, "read not bounded: {} bytes", out.len());
assert!(!out.is_empty(), "expected some body");
}
#[test]
fn trigger_counts_extracts_both_values() {
let v = serde_json::json!({"peers_reached": 2, "repos_enqueued": 5});
assert_eq!(trigger_counts(&v), (2, 5));
// Missing/malformed fields default to 0, never panic.
assert_eq!(trigger_counts(&serde_json::json!({})), (0, 0));
assert_eq!(
trigger_counts(&serde_json::json!({"peers_reached": "x"})),
(0, 0)
);
}
}
+18
View File
@@ -0,0 +1,18 @@
[package]
name = "icaptcha-client"
version = "0.4.0" # x-release-please-version
edition.workspace = true
rust-version.workspace = true
license.workspace = true
authors.workspace = true
repository.workspace = true
[dependencies]
gitlawb-core = { path = "../gitlawb-core" }
reqwest = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
anyhow = { workspace = true }
thiserror = { workspace = true }
tracing = { workspace = true }
sha2 = { workspace = true }
+507
View File
@@ -0,0 +1,507 @@
//! Client for the iCaptcha proof-of-intelligence service.
//!
//! The node gates spam-prone writes (repo create / fork / register) behind an
//! iCaptcha proof. This crate implements the *sanctioned client flow*: on a
//! `403 icaptcha_proof_required`, request a challenge for the required level,
//! solve the deterministic computational types locally, obtain the signed
//! proof, and hand it back so the caller can retry the original signed request
//! with the `x-icaptcha-proof` header.
//!
//! `requesterId` is always the caller's DID, so the proof's `sub` claim matches
//! the authenticated signer (the node enforces `sub == authenticated DID`).
//!
//! Blocking HTTP (reqwest::blocking) so the git remote helper can use it
//! directly; `gl` (async) calls it via `tokio::task::spawn_blocking`.
use std::io::Read;
use anyhow::{anyhow, bail, Context, Result};
use serde::Deserialize;
use serde_json::json;
pub mod pow;
pub mod solvers;
/// Default iCaptcha service base URL (used when the node doesn't advertise one).
pub const DEFAULT_URL: &str = "https://icaptcha.gitlawb.com";
/// Default required level (the node's default floor).
pub const DEFAULT_LEVEL: u32 = 3;
/// Header the gated write must echo the proof back in.
pub const PROOF_HEADER: &str = "x-icaptcha-proof";
/// Cap on bytes read from an error response, and characters kept in the excerpt.
/// The iCaptcha origin is only as trusted as the node that advertised it (a MITM
/// on the default transport qualifies), so its error bodies are attacker-
/// influenceable bytes that reach the TTY — they must be bounded and stripped of
/// control characters before printing (CWE-150, same treatment as #137).
const MAX_ERR_BYTES: u64 = 4096;
const MAX_ERR_CHARS: usize = 500;
/// Read a bounded error body and return a control-char-stripped, length-capped
/// single-line excerpt safe to print to a terminal.
fn error_excerpt(resp: reqwest::blocking::Response) -> String {
let mut buf = Vec::new();
if resp.take(MAX_ERR_BYTES).read_to_end(&mut buf).is_err() {
return String::new();
}
sanitize_excerpt(&String::from_utf8_lossy(&buf))
}
/// Strip C0/C1 control characters, collapse whitespace runs to single spaces,
/// and cap the length — so a hostile response body cannot inject terminal
/// escapes or flood the TTY.
fn sanitize_excerpt(s: &str) -> String {
let mut out = String::new();
let mut pending_space = false;
let mut kept = 0;
for ch in s.trim().chars() {
if kept >= MAX_ERR_CHARS {
break;
}
if ch.is_control() {
if matches!(ch, '\n' | '\r' | '\t') {
pending_space = true;
}
continue;
}
if ch.is_whitespace() {
pending_space = true;
continue;
}
if pending_space && !out.is_empty() {
out.push(' ');
kept += 1;
if kept >= MAX_ERR_CHARS {
break;
}
}
out.push(ch);
kept += 1;
pending_space = false;
}
out
}
/// Whether `u` parses as a trusted URL.
///
/// Production trusts only `https`. When `GITLAWB_ICAPTCHA_INSECURE` is set (a
/// runtime escape hatch for integration tests) the function also trusts
/// `http://127.0.0.1` and `http://localhost` so the full iCaptcha retry path
/// can be exercised against a local mockito server.
fn is_https(u: &str) -> bool {
let Ok(parsed) = reqwest::Url::parse(u) else {
return false;
};
if parsed.scheme() == "https" {
return true;
}
if std::env::var_os("GITLAWB_ICAPTCHA_INSECURE").is_some()
&& parsed.scheme() == "http"
&& matches!(parsed.host_str(), Some("127.0.0.1" | "localhost"))
{
return true;
}
false
}
/// Lowercased origin string (`scheme://host[:port]`) of a URL.
///
/// Includes the effective port so that `http://localhost:3000` and
/// `http://localhost:9000` are treated as distinct origins — a hostile node
/// cannot redirect the iCaptcha solve to a different loopback port.
fn origin_key(u: &str) -> Option<String> {
let parsed = reqwest::Url::parse(u).ok()?;
let scheme = parsed.scheme();
let host = parsed.host_str()?.to_ascii_lowercase();
let port = parsed
.port_or_known_default()
.map(|p| format!(":{p}"))
.unwrap_or_default();
Some(format!("{scheme}://{host}{port}"))
}
/// Decide which iCaptcha origin to actually talk to, and whether it is trusted
/// enough to receive the operator's API key.
///
/// The node-advertised `x-icaptcha-url` is untrusted: a hostile node could point
/// it at an attacker origin (client-side SSRF) or capture a bearer token on the
/// first hop. So an advertised URL is honored only when it is `https` AND its
/// host is allowlisted — the public default host, or the operator's own
/// `GITLAWB_ICAPTCHA_URL`. Otherwise we fall back to the operator/default origin.
///
/// The API key is attached only when the resolved origin is the operator's
/// explicitly-configured host (never a node-discovered one), so a key is never
/// exfiltrated to an origin the operator did not choose.
fn resolve_solver_url(advertised: Option<&str>, operator: Option<&str>) -> (String, bool) {
let operator = operator.filter(|u| is_https(u));
let operator_origin = operator.as_ref().and_then(|u| origin_key(u));
let default_origin = origin_key(DEFAULT_URL);
let allowed = |origin: &Option<String>| -> bool {
origin
.as_ref()
.map(|o| Some(o) == default_origin.as_ref() || Some(o) == operator_origin.as_ref())
.unwrap_or(false)
};
let fallback = || {
operator
.map(String::from)
.unwrap_or_else(|| DEFAULT_URL.to_string())
};
let chosen = match advertised {
Some(a) if is_https(a) && allowed(&origin_key(a)) => a.to_string(),
Some(a) => {
tracing::warn!(
advertised = %a,
"ignoring iCaptcha URL advertised by node (must be https with an allowlisted host); using trusted origin"
);
fallback()
}
None => fallback(),
};
// Key goes only to the operator's own configured origin.
let key_trusted = operator_origin.is_some() && origin_key(&chosen) == operator_origin;
(chosen, key_trusted)
}
/// Computational challenge types this client solves locally. Restricting the
/// request to these avoids dictionary (anagram/logic) and LLM (wordproblem/
/// riddle) types, which can't be auto-solved.
const SOLVABLE_TYPES: [&str; 3] = ["arithmetic", "algebra", "sequence"];
/// Bound on challenge/answer rounds (the service escalates difficulty on a miss;
/// correct solvers shouldn't escalate, but cap it regardless).
const MAX_ROUNDS: usize = 8;
/// Where + at what level to solve. `did` becomes the proof's `sub`.
#[derive(Debug, Clone)]
pub struct IcaptchaCfg {
pub url: String,
pub did: String,
pub level: u32,
/// Optional bearer token for an API-key-protected iCaptcha deployment.
pub api_key: Option<String>,
}
impl IcaptchaCfg {
/// Build config from the caller DID plus an optionally node-advertised
/// url/level (e.g. the `x-icaptcha-url` / `x-icaptcha-level` headers) and
/// optional level.
///
/// The advertised URL is validated and allowlisted (see [`resolve_solver_url`])
/// so a hostile node cannot redirect the solve to an attacker origin. The
/// `GITLAWB_ICAPTCHA_API_KEY` bearer token is attached only when the resolved
/// origin is the operator's own `GITLAWB_ICAPTCHA_URL`, never a
/// node-discovered one.
pub fn new(did: impl Into<String>, url: Option<String>, level: Option<u32>) -> Self {
let operator = std::env::var("GITLAWB_ICAPTCHA_URL")
.ok()
.filter(|s| !s.is_empty());
let (resolved_url, key_trusted) = resolve_solver_url(url.as_deref(), operator.as_deref());
let api_key = if key_trusted {
std::env::var("GITLAWB_ICAPTCHA_API_KEY")
.ok()
.filter(|s| !s.is_empty())
} else {
None
};
Self {
url: resolved_url,
did: did.into(),
level: level.unwrap_or(DEFAULT_LEVEL),
api_key,
}
}
}
/// A challenge handed back by the service (mirrors `icaptcha` `Challenge`).
#[derive(Debug, Clone, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct Challenge {
pub challenge_id: String,
#[serde(rename = "type")]
pub kind: String,
pub difficulty: u32,
pub prompt: String,
pub token: String,
/// Proof-of-work to solve and echo back as `powNonce` on the answer. Present
/// only when the service has PoW enabled; absent/None means no PoW required.
#[serde(default)]
pub pow: Option<pow::PowChallenge>,
}
#[derive(Debug, Deserialize)]
#[serde(tag = "status", rename_all = "lowercase")]
enum AnswerResult {
Passed { proof: String },
Continue { challenge: Challenge },
Failed { reason: String },
}
/// Solver callback for types this crate can't solve deterministically.
pub type Solver<'a> = dyn Fn(&Challenge) -> Option<String> + 'a;
/// Run the full challenge → solve → answer loop and return a fresh proof token.
///
/// `solver` is consulted for challenge types the built-in solvers don't handle
/// (anagram/logic/LLM); pass `None` to fall back to an interactive stdin prompt.
pub fn obtain_proof(cfg: &IcaptchaCfg, solver: Option<&Solver>) -> Result<String> {
let client = reqwest::blocking::Client::builder()
.timeout(std::time::Duration::from_secs(30))
.build()
.context("build iCaptcha http client")?;
let mut challenge = request_challenge(&client, cfg)?;
for _ in 0..MAX_ROUNDS {
let answer = solvers::solve(&challenge.kind, &challenge.prompt)
.or_else(|| solver.and_then(|s| s(&challenge)))
.or_else(|| interactive_prompt(&challenge))
.ok_or_else(|| {
anyhow!(
"cannot solve iCaptcha challenge type '{}' automatically; \
set GITLAWB_ICAPTCHA_API_KEY/solver or solve interactively",
challenge.kind
)
})?;
// Solve the proof-of-work bound to this challenge, if the service
// requires one. A required-but-unsolvable PoW (unknown algorithm or a
// difficulty above our cap) is a hard error — submitting without it
// would just be rejected.
let pow_nonce = match &challenge.pow {
Some(p) => Some(pow::solve(p).ok_or_else(|| {
anyhow!(
"cannot solve iCaptcha proof-of-work (algorithm '{}', difficulty {})",
p.algorithm,
p.difficulty
)
})?),
None => None,
};
match submit_answer(
&client,
cfg,
&challenge.token,
&answer,
pow_nonce.as_deref(),
)? {
AnswerResult::Passed { proof } => return Ok(proof),
AnswerResult::Continue { challenge: next } => challenge = next,
AnswerResult::Failed { reason } => {
bail!("iCaptcha challenge failed: {}", sanitize_excerpt(&reason))
}
}
}
bail!("iCaptcha not solved within {MAX_ROUNDS} rounds")
}
fn request_challenge(client: &reqwest::blocking::Client, cfg: &IcaptchaCfg) -> Result<Challenge> {
let url = format!("{}/v1/challenge", cfg.url.trim_end_matches('/'));
let body = json!({
"requesterId": cfg.did,
"requiredLevel": cfg.level,
"types": SOLVABLE_TYPES,
});
let mut req = client.post(&url).json(&body);
if let Some(key) = &cfg.api_key {
req = req.bearer_auth(key);
}
let resp = req.send().with_context(|| format!("POST {url}"))?;
if !resp.status().is_success() {
let status = resp.status();
let text = error_excerpt(resp);
bail!("iCaptcha challenge request failed ({status}): {text}");
}
resp.json::<Challenge>().context("parse iCaptcha challenge")
}
fn submit_answer(
client: &reqwest::blocking::Client,
cfg: &IcaptchaCfg,
token: &str,
answer: &str,
pow_nonce: Option<&str>,
) -> Result<AnswerResult> {
let url = format!("{}/v1/answer", cfg.url.trim_end_matches('/'));
let mut body = json!({ "token": token, "answer": answer });
if let Some(nonce) = pow_nonce {
body["powNonce"] = json!(nonce);
}
let mut req = client.post(&url).json(&body);
if let Some(key) = &cfg.api_key {
req = req.bearer_auth(key);
}
let resp = req.send().with_context(|| format!("POST {url}"))?;
if !resp.status().is_success() {
let status = resp.status();
let text = error_excerpt(resp);
bail!("iCaptcha answer request failed ({status}): {text}");
}
resp.json::<AnswerResult>()
.context("parse iCaptcha answer result")
}
/// Last-resort fallback: show the prompt and read an answer from the terminal.
/// Returns `None` when stdin isn't a usable interactive source (e.g. an agent),
/// so the caller surfaces a clear "couldn't auto-solve" error instead.
fn interactive_prompt(challenge: &Challenge) -> Option<String> {
use std::io::{stderr, stdin, Write};
let mut err = stderr();
let _ = writeln!(
err,
"iCaptcha challenge ({}, level {}): {}\nAnswer: ",
challenge.kind, challenge.difficulty, challenge.prompt
);
let _ = err.flush();
let mut line = String::new();
match stdin().read_line(&mut line) {
Ok(0) | Err(_) => None,
Ok(_) => {
let a = line.trim().to_string();
if a.is_empty() {
None
} else {
Some(a)
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::ffi::OsString;
use std::sync::{Mutex, MutexGuard};
/// Serializes tests that touch the process-global
/// `GITLAWB_ICAPTCHA_INSECURE` env var so they never race.
static ICAPTCHA_ENV_LOCK: Mutex<()> = Mutex::new(());
/// Set `GITLAWB_ICAPTCHA_INSECURE` for the test lifetime, restoring any
/// prior value on drop.
struct InsecureEnv {
_lock: MutexGuard<'static, ()>,
prev: Option<OsString>,
}
impl InsecureEnv {
fn new() -> Self {
let lock = ICAPTCHA_ENV_LOCK.lock().unwrap();
let prev = std::env::var_os("GITLAWB_ICAPTCHA_INSECURE");
std::env::set_var("GITLAWB_ICAPTCHA_INSECURE", "1");
InsecureEnv { _lock: lock, prev }
}
}
impl Drop for InsecureEnv {
fn drop(&mut self) {
match self.prev.take() {
Some(v) => std::env::set_var("GITLAWB_ICAPTCHA_INSECURE", v),
None => std::env::remove_var("GITLAWB_ICAPTCHA_INSECURE"),
}
}
}
// ── P1: hostile error bodies must not reach the terminal raw ──────────
#[test]
fn sanitize_strips_control_chars_and_escapes() {
// ESC-based terminal escape + NUL + bell must all be removed.
let hostile = "\x1b[2J\x1b[31mowned\x07\0 done";
let clean = sanitize_excerpt(hostile);
assert!(!clean.contains('\x1b'), "ESC survived: {clean:?}");
assert!(
!clean.chars().any(|c| c.is_control()),
"control survived: {clean:?}"
);
assert!(clean.contains("owned") && clean.contains("done"));
}
#[test]
fn sanitize_collapses_whitespace_and_caps_length() {
let collapsed = sanitize_excerpt("a\n\n\t b");
assert_eq!(collapsed, "a b");
let long = "x".repeat(MAX_ERR_CHARS + 500);
assert_eq!(sanitize_excerpt(&long).chars().count(), MAX_ERR_CHARS);
}
// ── P2: never talk to (or hand a key to) an untrusted node-chosen URL ──
#[test]
fn rejects_non_https_advertised_url() {
// A plaintext node-advertised URL is ignored; we fall back to default.
let (url, key_trusted) = resolve_solver_url(Some("http://evil.example/x"), None);
assert_eq!(url, DEFAULT_URL);
assert!(!key_trusted);
}
#[test]
fn rejects_https_url_with_non_allowlisted_host() {
let (url, key_trusted) = resolve_solver_url(Some("https://evil.example"), None);
assert_eq!(url, DEFAULT_URL, "attacker host must not be honored");
assert!(!key_trusted);
}
#[test]
fn honors_advertised_default_host_but_withholds_key_without_operator() {
// Default public host is allowlisted for talking, but with no operator
// origin configured the API key has no trusted destination.
let (url, key_trusted) = resolve_solver_url(Some("https://icaptcha.gitlawb.com/v2"), None);
assert_eq!(url, "https://icaptcha.gitlawb.com/v2");
assert!(!key_trusted);
}
#[test]
fn key_trusted_only_for_operator_configured_origin() {
let op = "https://icap.mynode.example";
// Node advertises the operator's own origin → talk there, key allowed.
let (url, key_trusted) =
resolve_solver_url(Some("https://icap.mynode.example/v1"), Some(op));
assert_eq!(url, "https://icap.mynode.example/v1");
assert!(key_trusted);
// Node advertises an attacker origin while an operator is configured →
// ignore the advert, fall back to the operator origin, key stays with it.
let (url, key_trusted) = resolve_solver_url(Some("https://evil.example"), Some(op));
assert_eq!(url, op);
assert!(key_trusted);
}
#[test]
fn no_advert_uses_operator_or_default() {
let op = "https://icap.mynode.example";
assert_eq!(resolve_solver_url(None, Some(op)), (op.to_string(), true));
assert_eq!(
resolve_solver_url(None, None),
(DEFAULT_URL.to_string(), false)
);
}
#[test]
fn non_https_operator_is_not_trusted() {
// A misconfigured plaintext operator URL is not a valid key destination.
let (url, key_trusted) = resolve_solver_url(None, Some("http://icap.mynode.example"));
assert_eq!(url, DEFAULT_URL);
assert!(!key_trusted);
}
#[test]
fn rejects_insecure_advertised_url_on_different_port() {
// With GITLAWB_ICAPTCHA_INSECURE=1, two loopback URLs on different
// ports must be treated as distinct origins so a hostile node cannot
// redirect the bearer key to a different listener on localhost.
let _env = InsecureEnv::new();
let op = "http://localhost:3000";
let (url, key_trusted) = resolve_solver_url(Some("http://localhost:9000"), Some(op));
assert_eq!(
url, op,
"must fall back to operator origin, not use different port"
);
assert!(key_trusted, "key stays trusted with operator origin");
}
}
+130
View File
@@ -0,0 +1,130 @@
//! Proof-of-work solver for the iCaptcha answer step.
//!
//! iCaptcha requires a small proof-of-work with each answer so that minting a
//! proof costs CPU — a per-proof cost a distributed abuser cannot dodge by
//! rotating IPs. The work is bound to the per-challenge id: find a `nonce` such
//! that `sha256("{challenge}:{nonce}")` has at least `difficulty` leading zero
//! bits. This mirrors `icaptcha/src/pow.ts`.
//!
//! At the service default (~20 bits) this is well under a second for a single
//! solve; we still cap the search so a hostile/misconfigured difficulty can
//! never hang the client.
use sha2::{Digest, Sha256};
/// PoW parameters advertised by the service in a challenge (mirrors the
/// service's `PowChallenge`).
#[derive(Debug, Clone, serde::Deserialize)]
pub struct PowChallenge {
/// Only `sha256-leading-zero-bits` is understood; other algorithms are
/// rejected by [`solve`] rather than silently mis-solved.
pub algorithm: String,
/// The string to hash against (the per-challenge id).
pub challenge: String,
/// Required leading zero bits.
pub difficulty: u32,
}
/// Hard cap on nonce search iterations. 2^26 ≈ 67M keeps a ~20-bit target (~1M
/// expected hashes) comfortably solvable while bounding worst-case work if the
/// service ever advertises an unexpectedly high difficulty.
const MAX_ITERS: u64 = 1 << 26;
const ALGORITHM: &str = "sha256-leading-zero-bits";
/// Leading zero bits of a byte slice.
fn leading_zero_bits(bytes: &[u8]) -> u32 {
let mut bits = 0;
for &b in bytes {
if b == 0 {
bits += 8;
} else {
bits += b.leading_zeros();
break;
}
}
bits
}
/// sha256(`{challenge}:{nonce}`).
fn pow_hash(challenge: &str, nonce: &str) -> [u8; 32] {
let mut hasher = Sha256::new();
hasher.update(challenge.as_bytes());
hasher.update(b":");
hasher.update(nonce.as_bytes());
hasher.finalize().into()
}
/// Solve the PoW, returning a nonce whose hash meets the difficulty. Returns
/// `None` for an unknown algorithm or if no nonce is found within the cap (a
/// difficulty far above what the service uses in practice).
pub fn solve(pow: &PowChallenge) -> Option<String> {
if pow.algorithm != ALGORITHM {
tracing::warn!(algorithm = %pow.algorithm, "unknown iCaptcha PoW algorithm; cannot solve");
return None;
}
if pow.difficulty == 0 {
return Some("0".to_string());
}
for i in 0..MAX_ITERS {
let nonce = format!("{i:x}");
if leading_zero_bits(&pow_hash(&pow.challenge, &nonce)) >= pow.difficulty {
return Some(nonce);
}
}
tracing::warn!(
difficulty = pow.difficulty,
max_iters = MAX_ITERS,
"iCaptcha PoW not solved within iteration cap"
);
None
}
#[cfg(test)]
mod tests {
use super::*;
fn pow(challenge: &str, difficulty: u32) -> PowChallenge {
PowChallenge {
algorithm: ALGORITHM.to_string(),
challenge: challenge.to_string(),
difficulty,
}
}
#[test]
fn leading_zero_bits_counts_across_bytes() {
assert_eq!(leading_zero_bits(&[0xff]), 0);
assert_eq!(leading_zero_bits(&[0x7f]), 1);
assert_eq!(leading_zero_bits(&[0x01]), 7);
assert_eq!(leading_zero_bits(&[0x00, 0xff]), 8);
assert_eq!(leading_zero_bits(&[0x00, 0x01]), 15);
}
#[test]
fn solves_and_the_solution_verifies() {
let p = pow("challenge-abc", 12);
let nonce = solve(&p).expect("should solve a 12-bit target");
assert!(leading_zero_bits(&pow_hash(&p.challenge, &nonce)) >= 12);
}
#[test]
fn solution_is_bound_to_the_challenge() {
// A nonce solving one challenge should (overwhelmingly) not solve another
// at the same difficulty — the work is challenge-specific.
let nonce = solve(&pow("challenge-A", 12)).unwrap();
assert!(leading_zero_bits(&pow_hash("challenge-B", &nonce)) < 12);
}
#[test]
fn rejects_unknown_algorithm() {
let mut p = pow("c", 8);
p.algorithm = "scrypt".to_string();
assert!(solve(&p).is_none());
}
#[test]
fn zero_difficulty_is_trivially_solved() {
assert_eq!(solve(&pow("c", 0)).as_deref(), Some("0"));
}
}
+270
View File
@@ -0,0 +1,270 @@
//! Deterministic solvers for the computational iCaptcha challenge types.
//!
//! Prompt/answer formats mirror the iCaptcha service generators
//! (`icaptcha/src/generators/{arithmetic,algebra,sequence}.ts`). The service
//! grades numerics by value (`Number(a) === Number(b)`), so returning the plain
//! integer string is sufficient. Anagram/logic (need a dictionary/parser) and
//! the LLM types are intentionally NOT solved here — the client requests only
//! these three types and falls back to a hook/interactive prompt otherwise.
/// Solve a challenge of the given `type` from its prompt. Returns `None` for
/// types we don't solve locally or when the prompt can't be parsed.
pub fn solve(challenge_type: &str, prompt: &str) -> Option<String> {
match challenge_type {
"arithmetic" => solve_arithmetic(prompt).map(|n| n.to_string()),
"algebra" => solve_algebra(prompt).map(|n| n.to_string()),
"sequence" => solve_sequence(prompt).map(|n| n.to_string()),
_ => None,
}
}
/// `What is 12 + 7 - 3?` -> evaluate the additive chain left to right.
fn solve_arithmetic(prompt: &str) -> Option<i64> {
let expr = prompt
.trim()
.strip_prefix("What is ")?
.trim_end_matches('?')
.trim();
let mut tokens = expr.split_whitespace();
let mut acc: i64 = tokens.next()?.parse().ok()?;
while let Some(op) = tokens.next() {
let n: i64 = tokens.next()?.parse().ok()?;
match op {
"+" => acc += n,
"-" => acc -= n,
_ => return None,
}
}
Some(acc)
}
/// `Solve for x: 3x + 4 = 19` (and the `=`-both-sides and `a(x + b)` variants).
/// Parses each side into `coeff*x + const`, then x = (cR - cL) / (aL - aR).
fn solve_algebra(prompt: &str) -> Option<i64> {
let eq = prompt.trim().strip_prefix("Solve for x:")?.trim();
let (lhs, rhs) = eq.split_once('=')?;
let (al, cl) = parse_linear(lhs.trim())?;
let (ar, cr) = parse_linear(rhs.trim())?;
let denom = al - ar;
if denom == 0 {
return None;
}
let num = cr - cl;
if num % denom != 0 {
return None;
}
Some(num / denom)
}
/// Parse a linear expression in `x` into `(coeff_of_x, constant)`.
/// Handles `Nx`, `x`, integer constants, and a single `N(x ± M)` product,
/// with `+`/`-` separators (the formats the generator emits).
fn parse_linear(s: &str) -> Option<(i64, i64)> {
let s = s.trim();
// Parenthesized product: a(x ± b) -> coeff a, const a*(±b).
if let Some(open) = s.find('(') {
let a: i64 = s[..open].trim().parse().ok()?;
let close = s.find(')')?;
let inner = &s[open + 1..close]; // "x + 4" / "x - 4"
let mut it = inner.split_whitespace();
if it.next()? != "x" {
return None;
}
let (coeff_inner, const_inner) = match it.next() {
None => (1, 0),
Some(op) => {
let m: i64 = it.next()?.parse().ok()?;
match op {
"+" => (1, m),
"-" => (1, -m),
_ => return None,
}
}
};
return Some((a * coeff_inner, a * const_inner));
}
// Sum of `±`-separated terms.
let mut coeff = 0i64;
let mut konst = 0i64;
let mut sign = 1i64;
for tok in s.split_whitespace() {
match tok {
"+" => sign = 1,
"-" => sign = -1,
t => {
if let Some(cpart) = t.strip_suffix('x') {
let c: i64 = match cpart {
"" | "+" => 1,
"-" => -1,
_ => cpart.parse().ok()?,
};
coeff += sign * c;
} else {
let n: i64 = t.parse().ok()?;
konst += sign * n;
}
sign = 1;
}
}
}
Some((coeff, konst))
}
/// `What is the next number in this sequence? 2, 4, 6, 8, 10, ?`
fn solve_sequence(prompt: &str) -> Option<i64> {
let tail = prompt.split_once("sequence?")?.1;
let nums: Vec<i64> = tail
.split(',')
.filter_map(|t| t.trim().parse::<i64>().ok())
.collect();
if nums.len() < 3 {
return None;
}
next_in_sequence(&nums)
}
fn next_in_sequence(n: &[i64]) -> Option<i64> {
let last = *n.last()?;
// Arithmetic: constant first difference.
let d = n[1] - n[0];
if n.windows(2).all(|w| w[1] - w[0] == d) {
return Some(last + d);
}
// Geometric: constant integer ratio.
if n.iter().all(|&v| v != 0) && n[0] != 0 && n[1] % n[0] == 0 {
let r = n[1] / n[0];
if r != 0 && n.windows(2).all(|w| w[1] == w[0] * r) {
return Some(last * r);
}
}
// Fibonacci-like: each term is the sum of the two before it.
if n.len() >= 3 && (2..n.len()).all(|i| n[i] == n[i - 1] + n[i - 2]) {
return Some(n[n.len() - 1] + n[n.len() - 2]);
}
// Squares: all perfect squares with consecutive roots.
let roots: Option<Vec<i64>> = n.iter().map(|&v| isqrt_exact(v)).collect();
if let Some(roots) = roots {
if roots.windows(2).all(|w| w[1] == w[0] + 1) {
let nr = roots[roots.len() - 1] + 1;
return Some(nr * nr);
}
}
// Alternating sign over an arithmetic magnitude (generator starts positive).
let signs_alternate = n
.iter()
.enumerate()
.all(|(i, &v)| if i % 2 == 0 { v >= 0 } else { v < 0 });
let mags: Vec<i64> = n.iter().map(|v| v.abs()).collect();
let md = mags[1] - mags[0];
if signs_alternate && mags.windows(2).all(|w| w[1] - w[0] == md) {
let next_mag = mags[mags.len() - 1] + md;
let next_sign = if last >= 0 { -1 } else { 1 };
return Some(next_sign * next_mag);
}
None
}
/// Exact integer square root, or `None` if `v` isn't a perfect square.
fn isqrt_exact(v: i64) -> Option<i64> {
if v < 0 {
return None;
}
let r = (v as f64).sqrt().round() as i64;
[r - 1, r, r + 1]
.into_iter()
.find(|&cand| cand >= 0 && cand * cand == v)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn arithmetic_chains() {
assert_eq!(
solve("arithmetic", "What is 12 + 7 - 3?").as_deref(),
Some("16")
);
assert_eq!(solve("arithmetic", "What is 5?").as_deref(), Some("5"));
assert_eq!(
solve("arithmetic", "What is 100 - 40 - 35 + 2?").as_deref(),
Some("27")
);
}
#[test]
fn algebra_linear() {
assert_eq!(
solve("algebra", "Solve for x: 3x + 4 = 19").as_deref(),
Some("5")
);
assert_eq!(
solve("algebra", "Solve for x: 5x - 8 = 12").as_deref(),
Some("4")
);
// both sides: 2x + 16 = 5x - 8 -> 3x = 24 -> x=8
assert_eq!(
solve("algebra", "Solve for x: 2x + 16 = 5x - 8").as_deref(),
Some("8")
);
// parenthesized: 3(x - 4) = 9 -> x-4=3 -> x=7
assert_eq!(
solve("algebra", "Solve for x: 3(x - 4) = 9").as_deref(),
Some("7")
);
// negative solution: 2x + 10 = 4 -> x=-3
assert_eq!(
solve("algebra", "Solve for x: 2x + 10 = 4").as_deref(),
Some("-3")
);
}
#[test]
fn sequence_patterns() {
assert_eq!(
solve(
"sequence",
"What is the next number in this sequence? 2, 4, 6, 8, 10, ?"
)
.as_deref(),
Some("12")
);
assert_eq!(
solve(
"sequence",
"What is the next number in this sequence? 3, 6, 12, 24, 48, ?"
)
.as_deref(),
Some("96")
);
assert_eq!(
solve(
"sequence",
"What is the next number in this sequence? 1, 4, 9, 16, 25, ?"
)
.as_deref(),
Some("36")
);
assert_eq!(
solve(
"sequence",
"What is the next number in this sequence? 1, 1, 2, 3, 5, ?"
)
.as_deref(),
Some("8")
);
}
#[test]
fn unsupported_types_return_none() {
assert_eq!(solve("anagram", "Unscramble: tca"), None);
assert_eq!(solve("riddle", "What has keys but no locks?"), None);
}
}
+1 -1
View File
@@ -120,7 +120,7 @@ Positive:
Risks:
- Many commands default to the public node; `git-remote-gitlawb` defaults to localhost. This split should be called out in README examples.
- `gl sync trigger` signs the request when a local identity is available and falls back to the legacy unsigned request for live compatibility.
- `gl sync trigger` requires a local identity and always sends a signed request; the `/api/v1/sync/trigger` route rejects unsigned calls, so the command fails locally when no identity is configured.
- Several CLI commands parse dynamic JSON responses permissively; good for compatibility, but error messages can hide response-shape regressions.
## CI and release readiness
+10
View File
@@ -58,6 +58,16 @@ services:
GITLAWB_P2P_PORT: "${p2p_port}"
GITLAWB_REPOS_DIR: /data/repos
GITLAWB_KEY: /data/keys/identity.pem
# Only trust X-Forwarded-For when the Caddy sidecar above is actually
# deployed (domain_name != ""). Caddy appends the real client as the
# rightmost hop, letting the push rate limiter key on the true client IP
# rather than Caddy's container address. Without Caddy, this header is
# fully client-controlled and must not be trusted.
GITLAWB_TRUSTED_PROXY: "${domain_name != "" ? "x-forwarded-for" : ""}"
# iCaptcha proof-of-intelligence gate (spam protection on create/fork/register)
ICAPTCHA_MODE: ${icaptcha_mode}
ICAPTCHA_PUBKEY: ${icaptcha_pubkey}
ICAPTCHA_REQUIRED_LEVEL: "${icaptcha_required_level}"
GITLAWB_PUBLIC_URL: $${GITLAWB_PUBLIC_URL}
GITLAWB_BOOTSTRAP_PEERS: $${GITLAWB_BOOTSTRAP_PEERS}
GITLAWB_AUTO_SYNC: $${GITLAWB_AUTO_SYNC}
+13 -10
View File
@@ -290,16 +290,19 @@ locals {
)
compose_yaml = templatefile("${path.module}/compose.yaml.tftpl", {
image_repo = local.image_repo
image_tag = var.image_tag
domain_name = var.domain_name
db_host = var.use_rds ? aws_db_instance.node[0].address : ""
gitlawb_port = var.gitlawb_port
p2p_port = var.gitlawb_p2p_port
metrics_port = var.metrics_port
expose_metrics = local.expose_metrics
pg_user = var.postgres_user
pg_db = var.postgres_db
image_repo = local.image_repo
image_tag = var.image_tag
domain_name = var.domain_name
db_host = var.use_rds ? aws_db_instance.node[0].address : ""
gitlawb_port = var.gitlawb_port
p2p_port = var.gitlawb_p2p_port
metrics_port = var.metrics_port
expose_metrics = local.expose_metrics
pg_user = var.postgres_user
pg_db = var.postgres_db
icaptcha_mode = var.icaptcha_mode
icaptcha_pubkey = var.icaptcha_pubkey
icaptcha_required_level = var.icaptcha_required_level
})
user_data = templatefile("${path.module}/user-data.sh.tftpl", {
+22
View File
@@ -81,6 +81,28 @@ variable "image_tag" {
default = "latest"
}
# ---------------------------------------------------------------------------
# iCaptcha proof-of-intelligence gate (spam protection on writes)
# ---------------------------------------------------------------------------
variable "icaptcha_mode" {
description = "iCaptcha gate mode: off | shadow | enforce. The live gitlawb network enforces."
type = string
default = "enforce"
}
variable "icaptcha_pubkey" {
description = "base64url Ed25519 public key of the iCaptcha service, pinned so startup doesn't depend on fetching it. This is the live icaptcha.gitlawb.com key."
type = string
default = "xjyPNqIbvc9U-kwXW6u9mDqRJ7E2UUMOaJdUWhpEXq8"
}
variable "icaptcha_required_level" {
description = "Minimum iCaptcha proof level required to pass the gate."
type = number
default = 3
}
# ---------------------------------------------------------------------------
# Networking / ingress
# ---------------------------------------------------------------------------
+14
View File
@@ -16,6 +16,10 @@ primary_region = "iad"
GITLAWB_BOOTSTRAP_PEERS = "https://node.gitlawb.com,https://node2.gitlawb.com,https://node3.gitlawb.com"
GITLAWB_AUTO_SYNC = "true"
GITLAWB_MAX_PACK_BYTES = "524288000"
# On Fly: the edge sets Fly-Client-IP (overwriting any client-supplied value),
# so the push rate limiter keys on the real client. Without this the limiter
# would fall back to the socket peer (Fly's proxy) and over-throttle.
GITLAWB_TRUSTED_PROXY = "fly"
[http_service]
internal_port = 7545
@@ -34,6 +38,16 @@ primary_region = "iad"
# slots (hard_limit) — root factor in the 2026-06-12 prod outage
idle_timeout = 120
[[http_service.checks]]
# Gate routing (and deploys) on real readiness: /ready probes the DB, so a
# node stuck retrying a dead or misconfigured database is never routed to,
# and a deploy with a bad DATABASE_URL fails visibly instead of going green.
interval = "15s"
timeout = "5s"
grace_period = "75s"
method = "GET"
path = "/ready"
[[services]]
protocol = "udp"
internal_port = 7546