Compare commits

...
Author SHA1 Message Date
JATMNandClaude Opus 5.5 309fc8360d ci: run PR checks and CodeQL only for the surfaces a PR touches (#519)
* ci: run PR checks and CodeQL only for the surfaces a PR touches

Every PR ran every PR Checks job on the whole workspace, and CodeQL default
setup analyzed every language, including a ~13 min macOS Swift autobuild
(55 min when it hung on main) for an 8-file macos-app/ that 4 of the last
~280 commits touched.

scripts/ci-surfaces.sh maps a PR's changed paths to the jobs that validate
them, and scopes cargo test / build --release to the touched crates plus
their transitive reverse dependencies (derived from crates/*/Cargo.toml).
Root Cargo.toml/Cargo.lock/.cargo, the CI wiring itself, and unknown crate
dirs widen to the full suite. Push to main, merge_group, and the release
flow's workflow_dispatch always run everything on the whole workspace.

codeql.yml replaces default setup with the same scoping per language and a
full analysis on push to main and weekly. It needs default setup disabled.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs: say which changes widen PR CI to the whole workspace

ci-surfaces.sh has two widening tiers. A root Cargo.toml/Cargo.lock/.cargo
change (or toolchain config, or an unknown crate dir) runs the Rust jobs on
--workspace, while the other jobs still follow their own inputs. Only the CI
wiring itself (pr-checks.yml, codeql.yml, ci-surfaces.sh) and non-PR events
run every surface. AGENTS.md and the script header described both tiers as
"everything" / "the full suite" and left "CI wiring" undefined, so an edit
to, say, release.yml read as if it ran the full PR Checks suite; it runs
none of those jobs and only CodeQL's Actions analysis.

Comment and docs only; the mapper's output is unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* ci: test gitlawb-node when bootstrap-peers.json changes

ci-surfaces.sh attributed Rust crates only by crates/<dir>/ paths, but
gitlawb-node also compiles in the repo-root bootstrap-peers.json through
include_str!. A PR that changed only that file therefore ran the Docker
smoke (docker build + --version, which embeds the JSON without parsing
it) and skipped cargo test, so embedded_seed_list_parses_successfully
("embedded bootstrap-peers.json must always parse") no longer guarded it.

Add a path -> crate table for files outside crates/ that a crate
compiles in, routed through the same check as crates/<dir>/ paths (an
unknown crate still widens to --workspace). bootstrap-peers.json now
yields rust=true with -p gitlawb-node and keeps its Docker trigger; every
other mapper output is unchanged.

The header also said anything the script cannot attribute precisely
widens the run. That holds only inside the Rust surface; a path no
surface lists (docs, README, infra/, other workflows) runs no PR Checks
job, as AGENTS.md says. Reword it to the two tiers the script implements.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* ci: feed ci-surfaces NUL-delimited paths and pin its mapping in a test

The mapper read newline-delimited `git diff --name-only` output. In that
mode git C-quotes any path with non-ASCII, tab, double-quote or newline
characters, so crates/gl/src/é.rs arrived as "crates/gl/src/\303\251.rs",
matched no surface, and a PR whose only Rust change was such a file ran no
Rust job (Café.swift likewise skipped Swift CodeQL). Newline-delimited
input also cannot carry a path that contains a newline.

Both workflows now pass `git diff -z` output and the mapper reads it with
`read -r -d ''`; input that is not NUL-terminated is rejected (exit 2)
instead of being misread. Mapping output for every ASCII path is
unchanged (checked against the last 43 commit ranges).

scripts/test-ci-surfaces.sh pins one must-not-skip case per surface row,
the crate fan-out and widening rules, --all, the wiring rows, quoted
names, and rejection of newline-delimited input. PR Checks runs it in the
`changed surfaces` job before mapping, on every event, so a broken mapper
turns the run red rather than silently narrowing it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-11-10 00:41:03 +00:00
f68d0dd3ff fix(node): durable post-receive outbox for receive-pack (#26 split 1/4) (#558)
* fix(node): durable post-receive outbox at the DB layer (#26 split 1/4)

Reviewer 2 closed PR #224 on 2026-08-28 with a directive: split the
work into four narrow PRs. This is Split PR 1 (durable post-receive
lifecycle) at the DB layer; the handler refactor in
crates/gitlawb-node/src/api/repos.rs:2007 (git_receive_pack) lands in
the next slice so the test can drive the failure injection end-to-end.

The pre-outbox crash window the reviewer flagged: receive_pack can
apply a ref to disk and return Ok, and a process exit, a dropped
future, or a DB failure before the bookkeeping at
crates/gitlawb-node/src/api/repos.rs:2361 (push event + cert + webhook)
loses the recovery record. Startup drain enumerates only sources
written from that bookkeeping, so it cannot reconstruct the missing
work. The partial fallback that re-derives from a row present in the
bookkeeping substitutes did:key:recovered and an empty attestation,
which is not equivalent to the original authenticated push.

This commit adds the durable boundary the handler will lean on.

NEW TABLE pending_ref_transitions (migration v27):
  - Written by the handler BEFORE smart_http::receive_pack, in state
    'prepared', carrying the verified pusher DID, the raw RFC 9421
    signature header, signature-input, and content-digest that
    authorized the push, the request id, and the parsed ref update.
  - The handler transitions the row to 'applied' on receive_pack Ok
    or 'cancelled' on Err. The drain reads only 'applied'.
  - A failed or cancelled receive-pack therefore leaves the row in
    'prepared' or 'cancelled', which the drain never promotes. This
    is what closes the reviewer's second proof ("a failed or cancelled
    receive-pack does not turn a prepared intent into completed
    accounting or anchoring").

NEW TABLE anchor_jobs (migration v27, owned by PR 1, consumed by PR 2):
  - One row per (repo_id, ref_name, old_sha, new_sha) transition.
    PR 1 inserts it on 'applied'; PR 2 reads it and updates claimed_at.
  - ON CONFLICT (id) DO NOTHING makes the insert idempotent on the
    deterministic id, so a recovery re-pass cannot create a second
    upload request. This is the handoff boundary; the bundler call
    itself is PR 2.

NEW DB METHODS on Db:
  - insert_pending_ref_transitions: writes one 'prepared' row per
    ref update, returns the persisted rows.
  - mark_pending_ref_transitions_applied / _cancelled: state flip,
    gated on the FROM state, idempotent.
  - list_pending_ref_transitions_applied: drain query, oldest first.
  - delete_pending_ref_transition: called by recovery after the
    artifacts land; a third pass is a no-op.
  - record_push_with_id: ON CONFLICT (id) DO NOTHING on the
    deterministic id.
  - insert_ref_certificate_idempotent: ON CONFLICT (repo_id, ref_name)
    DO NOTHING, returns None if a live-path cert already exists.
  - insert_anchor_job_idempotent: ON CONFLICT (id) DO NOTHING on the
    deterministic per-transition id.

NEW HELPERS in db/mod.rs:
  - deterministic_id: SHA-256 hex with an ASCII Unit Separator
    between fields so two distinct tuples never collide on prefix
    overlap.
  - push_event_id_for, ref_cert_id_for, anchor_job_id_for: the
    derived ids above, one helper per artifact so a caller cannot
    derive a wrong id by mistake.

NEW STRUCTS:
  - PendingRefTransition: the row shape.
  - AnchorJob: the handoff row shape.
  - pending_state: const strings ('prepared' / 'applied' /
    'cancelled') shared by tests, the producer, and the drain so a
    typo on one side cannot silently mismatch the other.

NEW TESTS in db::pending_ref_transition_tests (8 tests, all green):
  - insert_then_mark_applied_flips_state_for_every_ref: producer
    contract.
  - mark_applied_is_idempotent_on_repeat: re-fire is a no-op.
  - cancelled_rows_are_not_returned_by_the_drain: reviewer's second
    proof at the DB layer.
  - prepared_rows_are_not_returned_by_the_drain: same proof for the
    pre-flip state (handler crashed before reaching post-Ok).
  - mark_cancelled_is_idempotent_on_repeat: counterpart.
  - drain_then_re_derive_is_idempotent: reviewer's first proof at
    the DB layer. Inserts a row in 'applied' state directly via
    insert_pending_ref_transition_for_test, drains it, derives the
    artifact ids twice, exercises record_push_with_id and
    insert_anchor_job_idempotent directly, asserts exactly one push
    event row and exactly one anchor job row regardless of how many
    times the drain runs.
  - deterministic_id_avoids_prefix_overlap_collisions: the
    separator regression test.
  - push_event_id_for_is_stable: derived ids match across calls and
    differ on each varied input.

OTHER:
  - Make RefUpdate and its fields pub(crate) so the DB methods can
    iterate the parsed ref updates. No public API change.

NOT IN THIS SLICE (the handler refactor, next commit):
  - The receive-pack handler does not yet call insert_pending_ref_
    transitions before the receive_pack call, nor mark_applied /
    mark_cancelled after. The DB layer is in place for it; the
    handler will call these methods and the startup drain will be
    wired in main.rs.
  - The startup drain in main.rs is not yet called; it will iterate
    list_pending_ref_transitions_applied, re-derive the artifacts,
    and delete the row.
  - The cert/push event issuance in cert.rs and the bookkeeping in
    api/repos.rs:2361 are not yet changed to use the deterministic
    ids. The helper functions exist and are tested; the callers
    follow.

Compiles clean, clippy clean under -D warnings, fmt clean.

* fix(node): wire durable outbox into the receive-pack handler (#26 split 1/4)

This is the handler-level half of Split PR 1. The previous commit
added the migration and the DB methods; this one threads them
through crates/gitlawb-node/src/api/repos.rs:2007 (git_receive_pack),
the cert issuer, and the startup drain.

CHANGES IN THE HANDLER
======================

In git_receive_pack, AT THE LAST POSSIBLE MOMENT before the
smart_http::receive_pack call, the handler now:

  1. Generates a per-handler request_id (UUID).
  2. Captures the raw Signature, Signature-Input, and Content-Digest
     headers from the request.
  3. Calls db.insert_pending_ref_transitions(request_id, ...) which
     writes one row per ref update in state 'prepared'.

The receive_pack call runs as before. After it returns:

  4. On Ok: db.mark_pending_ref_transitions_applied(request_id) —
     the row is the ONLY thing that promotes a 'prepared' row to
     'applied', and the drain reads only 'applied' rows. A process
     crash before this call leaves the row in 'prepared', which the
     drain never promotes.
  5. On Err: db.mark_pending_ref_transitions_cancelled(request_id) —
     a failed receive_pack leaves the row in 'cancelled', which the
     drain never promotes.

This is what closes the reviewer's two proofs:

  Proof 1 (crash window): if the process dies after
  mark_pending_ref_transitions_applied but before the bookkeeping
  writes, the row is in 'applied' and the next startup drain
  re-derives the push event, the per-ref certificate (carrying the
  ORIGINAL pusher DID, not a placeholder), and the anchor handoff.
  The drain uses the persisted authentic pusher DID and signature
  header, not a recovered placeholder.

  Proof 2 (failed receive-pack): the row is only ever flipped to
  'applied' in the explicit Ok branch above. A 'prepared' or
  'cancelled' row is invisible to the drain, so a failed or dropped
  receive_pack cannot turn a prepared intent into completed
  accounting or anchoring.

BOOKKEEPING IS NOW DETERMINISTIC-ID
===================================

The post-Ok bookkeeping at api/repos.rs:2448 now uses:

  - record_push_with_id with push_event_id_for(request_id, first_ref)
    — ON CONFLICT (id) DO NOTHING, so a recovery re-pass is a no-op.
  - issue_ref_certificate_idempotent with
    ref_cert_id_for(request_id, ref_name) — ON CONFLICT (repo_id,
    ref_name) DO NOTHING, returns None if a live-path cert already
    exists.
  - insert_anchor_job_idempotent with
    anchor_job_id_for(repo_id, ref_name, old_sha, new_sha) — the
    per-transition tuple key, so two pushes to the same ref produce
    one anchor upload per landed state.

The legacy entry points (record_push, issue_ref_certificate,
insert_ref_certificate) remain for callers that prefer a fresh UUID
per cert; they are #[allow(dead_code)] for the PR 3 cert/CLI compat
pass to decide whether to keep or remove.

STARTUP DRAIN
=============

crates/gitlawb-node/src/main.rs calls
durable_outbox::drain_pending_ref_transitions(state, 1000) ONCE
before serving, after migrations and after the existing peer /
quarantine prunes. Non-fatal: a transient drain failure logs and
leaves the rows for the next startup.

durable_outbox::drain_pending_ref_transitions reads every 'applied'
row, calls derive_one (which re-derives the three artifacts using the
persisted authentic pusher DID and signature header), then deletes
the row. A second drain pass is a no-op for both the artifacts
(idempotent inserts) and the row (gone after the first pass).

NEW END-TO-END TESTS
====================

crates/gitlawb-node/src/durable_outbox.rs adds three end-to-end
tests in drain_tests, complementing the eight DB-layer tests in
db::pending_ref_transition_tests:

  - drain_re_derives_all_three_artifacts_for_an_applied_row: the
    reviewer's first proof. Inserts a row in 'applied' state (the
    crash window), drains, asserts exactly one push event row,
    exactly one cert row carrying the original pusher DID (not a
    placeholder), and exactly one anchor job row. Asserts the
    deterministic cert id matches. Asserts a second drain pass is a
    no-op.
  - cancelled_row_produces_no_artifacts: the reviewer's second proof
    for the cancelled state. A row in 'cancelled' (receive_pack
    returned Err) is invisible to the drain.
  - prepared_row_produces_no_artifacts: the reviewer's second proof
    for the prepared state. A row in 'prepared' (handler crashed
    between insert_prepared and the post-Ok branch) is invisible to
    the drain.

Each test names the invariant it pins and the production line it
covers. Reverting that line turns the named assertion red.

Compiles clean, 1099 tests pass with 0 regressions, clippy clean
under -D warnings, fmt clean.

Cross-PR overlap (declared in the PR description):

  - #134 (anchors auth): composes. The /arweave/anchors route
    already requires auth; this PR does not change the route.
  - #285 (advisory-lock session affinity): composes. The durable
    intent is written inside the same handler that holds the lock
    from #285; no changes to the lock layer.
  - #306 (Content-Digest on signed requests): composes. PR 1
    persists the Content-Digest header that #306 makes mandatory.
  - #314 (small-order Ed25519): independent. PR 1's tests use strong
    keys.
  - #324 (libp2p keypair persistence): independent. PR 1 does not
    touch p2p identity.
  - #325 (gossip ref-update auth): independent. PR 1's signed
    envelope is the HTTP-side equivalent, not the gossip-side.
  - #382 (replication withheld-subtree trees): independent. PR 1
    does not touch replication or pin selection.

* fix(node): address four reviewer findings on #26 split 1/4

- P1-A: add startup reconcile step that promotes `prepared` rows to
  `applied` when the on-disk ref matches the row's `new_sha`. The
  recovery drain (which only reads `applied` rows) can now pick up
  a ref that landed when the live handler's
  `mark_pending_ref_transitions_applied` call errored or was
  interrupted. Strict SHA equality is the load-bearing check — a
  `prepared` row whose target did NOT actually land stays
  `prepared`.
- P1-B: route the live handler's cert issuance through
  `cert::issue_ref_certificate` (the upsert) instead of
  `issue_ref_certificate_idempotent` (DO NOTHING). A re-push to the
  same ref now updates the cert's `old_sha` / `new_sha` /
  `pusher_did` / `issued_at` / `signature` to the new transition
  while preserving the deterministic `cert_id`. The recovery drain
  keeps the idempotent variant; both paths collapse to one row.
- P2-A: refactor the drain into a `drain_pending_ref_transitions_with`
  testable seam that does per-row log-and-continue, and add
  `drain_pending_ref_transitions_all` that loops
  `DRAIN_PER_PASS_LIMIT=1000` rows for `DRAIN_MAX_PASSES=10` passes.
  A failing row no longer stalls the batch; a backlog above 1000
  rows is fully processed across passes.
- P2-B: add a `first_ref_name` column to `pending_ref_transitions`
  via migration v28. The live handler hoists a `first_ref_name`
  local and persists it on every row of the same `request_id`. The
  drain's `derive_one` keys the push event id on
  `row.first_ref_name` instead of `row.ref_name`, so live and
  recovery produce the same id and `ON CONFLICT (id) DO NOTHING`
  collapses a multi-ref push to one push event row (and one trust-
  score bump). Cert and anchor ids stay per-ref / per-transition.

* fix(node): address four reviewer findings on #26 split 1/4 (round 2)

* fix(node): address reviewer round-3 findings on #26 split 1/4

- P1: Delete outbox rows after live durable effects complete so they
  don't replay on every restart
- P1: Parse git report-status for per-ref ok/ng results; mark only
  proven rejections as cancelled, uncertain outcomes as recoverable
- P1: Introduce 'uncertain' state for receive-pack errors where some
  refs may have landed; reconcile checks these against disk at startup
- P2: Promote deletion transitions during reconcile (new_sha == ZERO_SHA
  with absent ref = successful deletion)
- P2: Loop reconcile across multiple passes so backlogs beyond the first
  page are processed in the same startup

Closes review round 3 findings from reviewer-1 and reviewer-2.

* fix(node): fix CI failures from round-3 changes

- Add COMMENT ON TABLE to v29 migration so migration_bodies_are_non_empty passes
- Return error on non-zero receive-pack exit (preserving backward compat
  with tests that expect Err(AppError::Git(_))) while still parsing
  report-status for outbox row handling

* fix(node): update inv22 gate tests for receive_pack_raw refactor

- F3 gate: scan for smart_http::receive_pack_raw( instead of receive_pack(
- U5 gate: scan for all_refs_ok instead of receive_result.is_ok()

* fix(node): require reflog proof before the reconcile promotes a row (#26 split 1/4)

Closes the second half of the reviewer's round-3 [P1]: the startup
reconcile still promoted on `disk_refs.get(ref) == row.new_sha`, which
also matches a coincidental current tip (old=B, new=A while the ref is
already A). That is not an exotic case — it is the ordinary shape of a
REJECTED push, since git refuses an update whose expected old value does
not match the ref. Both the SHA check and the MAX_RECONCILE_AGE window
pass, the push never happened, and the drain then writes a push event, a
signed certificate and an anchor for it.

The live path proves landings from git's report-status body; that body is
gone by the time the reconcile runs, so the proof is re-derived from the
repository. `git::store::ref_reflog_entries` reads `logs/<ref>` and a row
is promoted only when an entry carries its exact `old -> new` pair,
stamped at or after the row was written (REFLOG_CLOCK_SKEW absorbs git's
whole-second truncation against the row's sub-second `created_at`). The
timestamp floor is what separates a landing from a later push that
re-introduced the same pair: proof must postdate the intent it proves.

Deletions stay exempt from the reflog half and keep the absence-plus-age
rule already in place — git removes a ref's reflog along with the ref, so
no other evidence can exist for one.

`init_bare` now sets `core.logAllRefUpdates`, which bare repos default
off; a repo with no reflog yields no proof and its rows stay put for
human-attended recovery rather than becoming accounting the node cannot
substantiate.

Tests: reconcile_refuses_a_coincidental_tip_with_no_reflog_proof,
reconcile_promotes_a_row_the_reflog_proves,
reconcile_leaves_a_row_prepared_when_the_repo_keeps_no_reflog,
reconcile_refuses_a_reflog_entry_older_than_the_row,
reconcile_still_promotes_a_landed_deletion_which_can_have_no_reflog,
init_bare_keeps_reflogs_so_a_landing_can_be_proven,
ref_reflog_entries_is_none_when_the_repo_kept_no_log.

No migrations added or renumbered.

* fix(node): walk the reconcile backlog on a keyset cursor (#26 split 1/4)

The multi-pass reconcile looped but never advanced. Each pass re-issued
the same cursor-less `LIMIT n` query, and unlike the drain — which
deletes every row it finishes, so its next page is always new work — the
reconcile leaves every row it cannot promote exactly where it was. One
unprovable row at the head of the ordering therefore pinned page one on
every pass, and the backlog behind it was never examined at all, which
is the condition the multi-pass loop was added to remove. Rows keep
ageing toward MAX_RECONCILE_AGE while they wait, so "picked up next
restart" can mean "never recovered".

Pages now resume strictly after the `(created_at, id)` of the last row
EXAMINED, via `list_pending_ref_transitions_prepared_or_uncertain_after`.
A keyset cursor rather than an OFFSET because rows leave the set
underneath the walk when they are promoted: an absent row simply does
not appear on a later page, and it can never shift an unvisited row into
a page that was already read. Pass budget, per-pass limit, short-page
exit and the residual-backlog warning are unchanged.

This matters more with the reflog gate in place: a landing in a repo
that keeps no reflog is PERMANENTLY unprovable, so before this it would
jam page one on every startup rather than once.

Tests: reconcile_all_walks_the_backlog_past_the_first_page,
reconcile_all_advances_past_an_unprovable_row (whose blocker is a
no-reflog row, the class the gate introduces).

* fix(node): bound the reflog read to a recent tail (#26 split 1/4)

`ref_reflog_entries` slurped the whole reflog. A reflog grows one line
(~150 bytes) per ref update and how often a ref is updated is
PUSHER-controlled, so that was an attacker-sized allocation, taken once
per stranded row, at startup — the moment the node can least absorb it.
Every other attacker-influenced read in this crate is bounded
(read_body_capped, MAX_SIGNATURE_ENTRIES, the pin/walk budgets); this one
should be too.

The bound costs nothing in proving power, which is why it is a ceiling
rather than a tradeoff: the landing gate only ever accepts an entry
stamped at or after `created_at - REFLOG_CLOCK_SKEW`, and reflogs are
append-ordered oldest-first, so every entry that could possibly qualify
is at the tail. The read now seeks to the last REFLOG_TAIL_BYTES (256KB,
~2000 recent updates to one ref) and `take`s that many bytes, so the
allocation stays bounded even if the file grows between the stat and the
read.

A window into the middle of a file starts mid-record, so the bytes before
its first newline are discarded: without that, a boundary landing inside
a record's old SHA leaves a suffix that still tokenizes and yields a
truncated SHA as a bogus `old -> new` pair — and landing proof is an
exact pair match, so a bogus pair is a fabricated proof. An entry older
than the window reads as absent, which is the safe direction: the caller
treats "no matching entry" as unproven and leaves the row where it is.
`Ok(None)` for a missing file, and the absence-of-evidence contract, are
unchanged.

Tests: ref_reflog_entries_reads_whole_records_from_the_recent_tail (lays
the file out so the boundary lands 20 bytes into a record's old SHA — the
alignment that would otherwise parse), an_entry_older_than_the_reflog_tail_reads_as_unproven.

* fix(node): read the report through both side-band frames

The report-status is framed twice whenever the client negotiates
side-band-64k, which `git push` over smart HTTP does: the outer
side-band envelope carries a stream that is itself pkt-line encoded.
One strip pass left the first line as `000eunpack ok`, so the unpack
check failed and parse_report_status returned None — read by the caller
as "no report", which keeps every declared ref. The per-ref landing
gate was therefore inert for exactly the pushes it exists to filter.

The regression test is a byte-for-byte capture from git 2.50.1
rejecting one ref of a two-ref push, because a hand-written
single-framed fixture parses under the old code and hides the case.

* fix(node): address round-4 reviewer findings on #26 split 1/4

* fix(node): rustfmt round-4 reviewer fixes

* fix(node): remove empty line after doc comment

* fix(node): drop redundant reflog gate; implicit-ok on exit-zero no-report

* fix(node): address round-5 reviewer findings on #26 split 1/4

* fix(db): v30 migration — add receive_pack_requests table and ordinal column

The reviewer's round-5 finding: the push event identity was encoded
into a mutable per-ref column (first_ref_name) whose correct value is
knowable only after git, and the correction was not committed
atomically with the per-ref outcomes. A crash between git updating
a later ref and the rewrite left the durable rows naming the
rejected ref, and derive_one's row.ref_name == row.first_ref_name
guard then meant no push event ever landed for the accepted child.

Per the state-transition model at
.gravirei/plans/state-model-durable-post-receive.md, this migration
introduces the request-level record that owns the push event and the
trust score, and extends the per-ref child with an ordinal column
the drain and the effect executor walk together. first_ref_name is
dropped; the push event id is keyed on
(request_id, accepted_ordinal) and lives on the request row.

The next commit (handler rewrite) wires the live path and the
recovery drain against the new tables.

BREAKING CHANGE: pending_ref_transitions.first_ref_name is dropped.
Callers that read or write that column must move to
receive_pack_requests.accepted_ordinal.

* fix(node): rewrite receive-pack handler against the request-level model (#26 split 1/4 step 2)

The v30 migration added receive_pack_requests and dropped first_ref_name,
but no Rust code used either. The handler still ran a four-branch per-ref
state flip plus a first_ref_name rewrite after git returned, which is the
mixed-outcome bug a push with the first ref rejected exposes: the live
path and the drain computed different push-event ids.

- Insert a receive_pack_requests row in state `received` BEFORE git runs
  (insert_receive_pack_request), carrying the request bytes and the
  SHA-256 of the body. Crash between intent and git return is now
  recoverable via the reconcile step.
- Drop the first_ref_name rewrite at api/repos.rs:2454. The push event
  identity is now (request_id, accepted_ordinal); accepted_ordinal is
  computed once at the per-ref state flip.
- Re-key push_event_id_for and ref_cert_id_for on (request_id, ordinal).
  Anchor job id stays per-transition.
- Transition the request row to outcomes_committed (with parsed_report
  and accepted_ordinal) or rejected_at_git (no report + non-zero exit)
  in the same handler tail as the per-ref state flip. The drain (step 3)
  picks up outcomes_committed rows; today the live path also runs the
  per-ref effects inline.
- Stub the step-3 surface (mark_request_effects_pending, complete,
  list_receive_pack_requests_due, update_request_attempt) with
  #[allow(dead_code)] + contract-pin comments so the rewrite is
  bisectable: reverting this commit restores the pre-rewrite live path.

The U5 gate (replication tail spawn inside push_succeeded, before
guard.release) is preserved; inv22_replication_tail_spawns_at_the_
durability_boundary stays green.

* fix(node): use raw bytes for request_bytes_hash (#26 split 1/4 step 2)

The v30 migration defined request_bytes_hash as BYTEA but the handler
was binding a hex-encoded String. Postgres rejected the insert with
"column request_bytes_hash is of type bytea but expression is of type
text" and the handler shed the push with a 503.

Switch the handler to bind the raw 32-byte digest and update the
ReceivePackRequest struct field type to Vec<u8> to match. The
drain-side test fixture already used Vec<u8> and needs no change.

Pinned by the bin test suite going from 24 failures (all the
receive-pack-cap tests that send a 4-byte body) to 0 regressions
on the new model.

* fix(node): factor out apply_request_effects; drain walks receive_pack_requests (#26 split 1/4 step 3)

The step-2 commit moved the request row to be the unit of work but
left the per-ref effects fan-out inline in the handler and the
drain walking pending_ref_transitions per-ref. Step 3 factors the
effects fan-out into apply_request_effects(state, request_id) and
rewrites the drain to walk receive_pack_requests.

- apply_request_effects lives in durable_outbox.rs; the live handler
  in api/repos.rs and the drain both call it. Idempotent on
  (request_id, accepted_ordinal). Returns EffectsOutcome::{Done,
  Nothing, Retry}.
- Drain switches from list_pending_ref_transitions_applied +
  derive_one to list_receive_pack_requests_due +
  apply_request_effects. derive_one and the per-ref drain entry
  points are deleted.
- The step-3 stubs in db/mod.rs (mark_request_effects_pending,
  mark_request_complete, list_receive_pack_requests_due) lose
  their #[allow(dead_code)] annotations; every stub has a caller
  in this PR.
- Reconcile (reconcile_prepared_from_disk_all) is unchanged.
  The crash window between intent durable and outcomes commit
  still routes through per-ref reflog proof.
- Drain tests rewritten to stage receive_pack_requests rows; the
  per-ref fixtures (make_row, lookup_accepted_ordinal) are gone.
- New inv26_step3_live_and_drain_share_apply_request_effects
  assertion pins the live/drain sharing and the per-request
  drain walk.

* fix(node): bounded retirement purge for terminal request rows (#26 split 1/4 step 4)

The v30 partial index idx_receive_pack_requests_completed_at
exists but no code reads it. Step 4 wires a periodic purge task
that deletes terminal `complete` and `rejected_at_git` rows
older than the retention window, along with their per-ref
children. `quarantined` (a step-5 state) is never purged by the
timer.

- `purge_completed_receive_pack_requests(older_than, limit)` —
  deletes parent requests using the v30 partial index.
- `purge_completed_pending_ref_transitions(older_than, limit)` —
  deletes children of purged parents, gated on the child's own
  applied_at / cancelled_at.
- `purge_request_queue(db, retention_days, limit)` — orchestrator
  in durable_outbox.rs that calls both and returns the totals.
  The deletion order (parents first, then children) is the
  contract; a crash mid-purge leaves orphaned children that the
  next pass will pick up.
- `spawn_queue_lifecycle_sweep` in main.rs — periodic task on
  the same detached pattern as `spawn_legacy_cid_sweep`, 24-hour
  interval, shutdown-aware. The interval matches the spec's "one
  per cluster per day" target.
- Config knobs `queue_retention_days` (default 7, range 1..=365)
  and `queue_purge_batch` (default 1000, matches
  DRAIN_PER_PASS_LIMIT).
- 3 new `durable_outbox::drain_tests::purge_*` tests pin the
  contract: only old terminal rows are deleted, the second pass
  is a no-op, and a row inside the retention window survives.
- New `inv26_step4_queue_lifecycle_purge_is_wired` gate asserts
  the wiring (main.rs calls the purge, the DB helpers exist, the
  Config knobs exist).

* fix(node): reference-transaction marker + quarantined state + failure matrix (#26 split 1/4 step 5)

Steps 2-4 gave the request row the unit of work, the shared
executor, and the bounded retirement policy. Step 5 closes the
evidence gap: the reconcile now requires a per-request marker
ref (refs/gitlawb/requests/<id>) whose value matches
request_bytes_hash. A missing or mismatched marker quarantines
the request; an operator reclassifies it.

- v31 migration: adds `quarantined` to the state vocabulary and a
  partial index for operator queries.
- Handler writes the marker ref before git-receive-pack; the
  marker is causally bound by being in the same async task as
  the receive-pack call. The marker's value is content-addressed
  (git hash-object of the request bytes), so the gate compares
  consistent SHAs on both sides.
- git::store::read_ref reads a single ref's value, returning
  Ok(None) for absent refs. Used by the marker gate.
- git::store::marker_value_for computes the content-addressed
  marker value; both the live handler and the reconcile use it
  so the write and the read agree.
- Reconcile gains a marker gate between the age check and the
  reflog proof. Mismatch or absent ⇒ mark_request_quarantined +
  mark_children_rejected_for_quarantined_parent.
- effects_max_attempts bound (config knob, default 8) flips
  retry-stuck requests to `quarantined` after N attempts,
  closing the infinite-retry DoS window.
- New failure_matrix_tests submodule covers the spec's outcome ×
  ref-kind × exit-point × recovery-scenario matrix (6 cells).
- New inv26_step5_marker_quarantine_and_bound_are_wired gate
  asserts the marker gate, the bound check, the handler's
  pre-receive-pack ordering, and every load-bearing helper.
- Existing 7 reconcile tests updated to stage a marker ref via
  the new `stage_marker` test helper.

* fix(node): converge durable post-receive to singular request-level lifecycle

Unify producer, evidence, aggregate outcome, executor, retry, and
retirement on the request row: atomic intent and outcome commits,
synthetic normalized reports for implicit-ok/reconciled paths,
reconcile parent promotion with fail-closed deletions and
competing-claimant guard, retry progression from both executable
states with backoff/quarantine, terminal-parent-gated purge with
marker cleanup and hidden refs, minimal digest-only intent plus
request-level RFC9421 proof (v32).

* fix(node): harden request occurrence lifecycle, proof, retry, and retirement

Reject incomplete report framing and omitted refs as indeterminate,
preserve uncertain children on completion, terminalize all-rejected,
bind reconcile with landing history, durable proof with ack gate,
occurrence-keyed anchors, background due loop, deletion quarantine
with operator resolve, children-before-parent purge with marker
tombstones, recovery prereq upgrade check, and bounded marker runner.

* fix(node): keep recovery prereq upgrade best-effort on push path

Refusing pushes when git config upgrade fails broke fake-git and
non-repo disk-path tests with 503. Reconcile already fails closed
on missing reflogs, so downgrade to warn-and-proceed.

* fix(node): close reviewer round on outcome authority and lifecycle ownership

Gate effects on unpack_ok with cleared ordinal and terminal no-effects,
intersect executor with applied children, treat absent report-status as
indeterminate regardless of exit, require landing history before child
delete, claim due requests with recoverable leases, dedupe webhooks by
occurrence ledger, ack-gate proof retention with terminal disposition,
fair marker tombstones with dead-letter, kill_on_drop marker bounds,
and loud migration version collision guard.

* fix(node): load-bearing coverage for intent, guards, and delivery ledger

Handler intent test plus inv22 gate so disabling the durable insert
turns red; unpack-false, mixed ok/ng, divergent cancelled, partial
sibling, and proof-ack tests pin each executor guard; webhook
deliveries record sent_at with stale-pending reclaim; stale
parsed_report comment corrected.

* fix(node): centralize live retry, bound outcome commit, close retention orphans

Route live Retry/Err through schedule_request_retry_or_quarantine so
attempt progression matches the drain; retry post-git outcome commits
3x before restart repair; block terminal purge while non-terminal
children exist; add v27 upgrade-path test.

* fix(node): document attended-restart commit policy and pin executor retry arms

Track outcome-commit success, skip metrics/touch on failure, correct
the lease-expiry comment to startup reconcile; add restart-repair test
for stuck received parents plus RED tests for landing-history and
proof-ack failure paths.

* fix(node): correct marker prereq comment to warn-and-proceed

* fix(node): single partial-completion model with disposition gate

Split apply_request_effects into phase-1 bundle (push, cert, anchor,
history, webhook, proof ack before child deletion) and phase-2 gate
that retries while non-cancelled siblings remain; Nothing only when
nothing owed. Gate tail, Tigris, and effects on PostGitDisposition
with documented replication carve-out; add partial-sibling webhook
and disposition tests; fix stale comments.

* fix(node): close review round on outbox ordering, marker visibility, and repair reachability

Address the P1/P2 review findings: await the webhook delivery claim
before accepted-child deletion; exempt the marker namespace from the
fail-closed ref peeling guard so visibility walks tolerate blob-pointing
request markers; keep request marker deletion bounded in the purge path;
promote cross-page siblings via idempotent report superset; run
stuck-parent repair when the prepared queue is empty; bound the durable
intent write while admission permits are held; remove dead proof
verification scaffolding.

* fix(node): close review round on pre-git refusal lifecycle, namespace gate, and promotion merge

* fix(node): close review round on namespace gate, intent lifecycle, and promotion atomicity

Namespace gate hardening:
- Refuse non-UTF-8 ref-update lines (git accepts bytes ≥0x80; gate never sees them)
- Catch bare 'refs/gitlawb' name (starts_with missed it)
- Add deny test for both holes plus refusal wiring verification

Intent lifecycle guards:
- Add IntentDropGuard: terminalize stranded intents on dropped futures
- Bound refuse_receive_pack_before_git to 10s (no statement_timeout on pool)
- Guard mirror fetch: hideRefs blocks non-exempt refs/gitlawb/* from +refs/*:refs/*

Promotion merge atomicity:
- Take row under SELECT FOR UPDATE so merge is atomic with state read
- Prevents racing due worker from stranding newly applied children outside report

* fix(test): correct inv22 namespace gate assertions to match actual implementation

The test was looking for the wrong patterns:
- UTF-8 check: error message string, not pre-loop condition
- Namespace check: `*r == "refs/gitlawb"`, not `ref_name ==`
- Visibility exemption: `&&` logic (both false to exclude), not `||`

* chore: cargo fmt

* fix(node): phase-aware intent guard, mirror import prune, terminal-state merge

* fix(node): guard cleanup retry with fates, fail-closed mirror prune, refusal coverage

* fix(node): bound guard attempts, sibling-wait outcome, fail-closed prune enumeration

* fix(node): sibling-wait outcome, no-report path, bounded prune and guard retry

* fix(node): no-report durability, cert landing order, bounded reconcile git, flush checks

* fix(node): gate trust-score bump on a new push event row

* chore: update rustls 0.23.37 → 0.23.45 to resolve RUSTSEC-2026-0285

* fix(node): surface git's report-status when a stdin EPIPE follows child exit

* fix(node): cap receive-pack ref-update count before per-ref work and intent

* fix(node): cap ref-update parsing in-loop and dedupe identical commands

* docs(node): correct round-4 comment claims on client outcome and gate cost

* fix(node): single replication tail, proven marker taxonomy, claimed webhook delivery (#558)

- One post_receive_replication_tail per push: the carve-out spawn and the
  durability-bound spawn are mutually exclusive positions on the same
  proven-landed ref set; inv22 now fails on any unfiltered or ungated spawn.
- Marker reconcile: bounded-read errors (git Err/deadline) stay prepared
  and retry instead of quarantining as 'missing marker ref'; only a
  successfully read marker proves missing or mismatch. Err-injection test
  pins no-quarantine on unproven reads.
- Claimed webhook delivery: the synchronous claim phase returns the
  claimed Webhook rows so the send task never re-lists; a list failure
  after claim can no longer drop claimed deliveries. Source gate pins it.

* docs(node): name claimed-hooks variant in fire_event_occurrence doc

* fix(node): order pending_ref_transitions create before outbox alters, fill v29 (#558)

* fix(node): proven-missing marker taxonomy, v38 upgrade test, report-ok tail fixtures (#558)

* fix(node): read marker refs via rev-parse so present refs return their oid (#558)

---------

Co-authored-by: Gravirei <[email protected]>
Co-authored-by: Kevin Codex <[email protected]>
2026-11-09 23:40:38 +00:00
beardthelionandcarlitos973 3c1ee27b60 fix(node): make creation-route rate limit configurable; fix(gl): surface real errors on pr create failure (#560)
- gitlawb-node: rate limiter was hardcoded to 10 req/hour per DID across all
  creation routes (repo/issue/PR create, register, fork). Adds
  GITLAWB_RATE_LIMIT_MAX and GITLAWB_RATE_LIMIT_WINDOW_SECS config fields
  (clap env-var pattern matching the rest of Config), defaults unchanged.
  Lets node operators tune this for real agent/bot usage without recompiling.

- gl pr create: previously called resp.json() before checking response
  status, so any non-2xx response with an empty or non-JSON body crashed
  with a generic "invalid JSON" error instead of surfacing the real cause.
  Now reads the raw response body first and includes it in the error message
  on failure.

Co-authored-by: carlitos973 <[email protected]>
2026-11-09 23:02:05 +00:00
beardthelionandKRATOS 209618a2fe feat(node): add scheduled inbound mirror worker (#555)
* feat(node): persist upstream mirror transition state

Origin-Session: local-d6a143 | Codex | 12 prompts

* fix(node): reject null mirror transition state

Origin-Session: local-d6a143 | Codex | 14 prompts

* fix(node): address mirror state review findings

Origin-Session: local-d6a143 | Codex | 16 prompts

* fix(node): close mirror review blockers

Origin-Session: local-d6a143 | Codex | 20 prompts

* fix(node): align peer URL validation message

Origin-Session: local-d6a143 | Codex | 20 prompts

* fix(node): close remaining mirror review findings

Origin-Session: local-d6a143 | Codex | 21 prompts

* feat(node): add scheduled inbound mirror worker

Origin-Session: local-d6a143 | Codex | 18 prompts

* fix(node): harden scheduled mirror fetches

Origin-Session: local-d6a143 | Codex | 24 prompts

* fix(node): close inbound mirror review blockers

Origin-Session: local-d6a143 | Codex | 24 prompts

* fix(node): reject special-use upstream literals at configure time (#555)

validate_mirror_upstream_url used only the peer public-host classifier, so
an IP literal the fetch worker's stricter permit denies (192.0.2/24,
198.18/15, 2001:db8::/32, ...) could be stored and then hard-fail on every
scheduled scan. The IANA special-use permit moves to peers::
mirror_upstream_ip_permitted, shared by configure validation and the
resolve-path check; the negative test set now covers the denied literals.

* test(node): scrub inherited git env in mirror fixture spawns (#555)

The repository_local_* fixture tests ran plain Command::new("git") and
inherited any GIT_DIR/GIT_CONFIG_* state from an outer checkout, so a
fixture spawn could act on the wrong repository. A shared fixture_git
helper now sets the cwd and removes git environment vars; all five spawn
sites use it.

* fix(node): bound mirror preflight probes separately, align operator docs (#555)

* chore(node): retrigger checks after code-scanning alert triage (#555)

* chore(node): retrigger checks after code-scanning alert triage (#555)

---------

Co-authored-by: KRATOS <[email protected]>
2026-11-09 22:48:47 +00:00
beardthelion a00d7d5c48 fix(gl): pin secret-file modes at creation instead of chmod after (#354) (#557)
* fix(gl): pin secret-file modes at creation instead of chmod after (#354)

identity.pem and ucan.json were written with fs::write and then chmod'd,
leaving the private key world-readable between the two syscalls, and
ucan.json was never chmod'd at all. A shared helper now opens with
mode(0o600) so no permissive window exists, adds O_NOFOLLOW so a
pre-planted symlink is refused rather than written through, and re-pins
the mode after write so a pre-existing loose file is tightened. The
containing directories are created 0700 via DirBuilder and re-pinned the
same way. gitlawb-node's own key write gets the same treatment.

* fix(gl): tighten secret handling before, not after, replacing contents (#459)

Address PR review feedback:

- secret_file::write and the node key write now open without truncate,
  pin mode 0600 on the descriptor, then set_len(0) and write, so a
  pre-existing permissive file never holds new secret contents while
  still group/world-readable.
- Directory re-pins skip symlinked paths instead of chmodding through
  them to the link target.
- The node's existing-key read opens with O_NOFOLLOW and reads from the
  descriptor instead of read_to_string on the path.
- An existing permissive key parent dir is re-pinned to 0700.

* fix(gl): adopt writable --dir leaves the user cannot chmod (#557)

create_dir re-pinned every existing final directory to 0700. On a leaf the
caller can write but does not own (group 0775, ACL grant, a system dir like
/tmp), fchmod fails EPERM and the whole command aborted, a regression over
create_dir_all's no-op. The re-pin now applies only when the euid owns the
opened directory; residual EPERM on an owned leaf adopts rather than aborts.
Regression test proves /tmp is adopted with its mode unchanged.
2026-11-09 20:13:51 +00:00
4cd0a6a306 fix(node)!: gate agent-task reads behind visibility rules (#553)
* fix(node): gate agent-task reads behind the same visibility rules as repo data

list_tasks and get_task had no authorization at all: any anonymous caller
could enumerate every task on the node, including another party's
repo-less task, its ucan_token, and its payload (#268). Add task_visible,
mirroring the repo read-visibility gate already used by the ref-updates
feed: the delegator and assignee can always read their own task, a
repo-scoped task follows that repo's normal visibility rules, and a task
naming no repo (or a repo this node doesn't host) is visible only to its
delegator/assignee. Both REST and GraphQL now route through the same
collect_visible_tasks/get_visible_task collectors so the two surfaces
cannot drift, and neither read path echoes ucan_token back, since the
holder already received it via the create/claim response.

Fixes #268

* fix(node): query the task limit-clamp test as the delegator

tasks_limit_ceiling_clamped_to_200 seeded 201 repo-less tasks and read
them back anonymously, expecting all 200. That read is exactly the
enumeration #268 closes, so the new visibility gate correctly returns
none of them and the test went red. The clamp ceiling is what this test
pins, not the gate, so query as the tasks' delegator, who can legitimately
see all 201 rows.

Refs #268

* fix(node): scope task visibility lookups to the page's repos

collect_visible_tasks loaded every repo on the node and every visibility
rule in order to gate at most 200 tasks, so an anonymous request paid for
the whole node's repo and rule set. Narrow both lookups to the repo ids the
fetched page actually names, and skip them when no task names a repo.

The deduped repo snapshot stays the source of truth for resolving a
repo_id: it collapses mirror and canonical pairs and omits quarantined
repos, and an id missing from it has to keep failing closed. Resolving ids
straight from the repos table would surface exactly those withheld rows.

Add GraphQL denial tests as well. Nothing pinned that the task resolvers
delegate to the shared collectors, so a resolver that queried the database
directly would not have gone red.

Refs #268

* Preserve authorized task reads across shipped clients.

Refs #268

* Bound denied task history scans.

Refs #268

* Signal incomplete task scans with recoverable cursors and map read errors to AppError.

Refs #268

* Stop disclosing denied task rows in the scan-wall cursor and add GraphQL pagination state.

Refs #268

* Normalize task cursors and gate mutation endpoints behind visibility.

Canonicalize RFC 3339 timestamps in parse_after_cursor to handle URL-decoded spaces, reject mixed cursor alias families, gate complete_task and fail_task behind get_visible_task so unreadable tasks 404 instead of leaking existence with 403, and only flag incomplete when hitting candidate ceilings on full SQL batches.

Refs #268

* Update complete_task test in test_support to exercise both 404 on unreadable and 403 on non-assignee tasks.

Refs #268

* Bind task to repo id in complete_task test in test_support.

Refs #268

* Preserve timestamp fractional width and anchor continuation tests on legitimate cursors.

Refs #268

* Match on Value::Object to extract task ID in GraphQL continuation test.

Refs #268

* Remove unused mut binding on visible_newer in GraphQL query test.

Refs #268

* Hide unauthorized claim and keep assigned tasks from being stolen

Gate REST and GraphQL claim behind the same visibility check as complete and fail, refuse claim when another assignee already holds the task, and only broadcast publicly visible task events. Treat a full list page as incomplete when more candidates remain. Surface HTTP errors from CLI and MCP claim and complete helpers.

Refs #327

Co-Authored-By: cairn-code <[email protected]>

* Keep task-list incomplete for the scan ceiling only.

A full visible page was flagged incomplete whenever the SQL batch was full, so the first page of any list with more than 200 candidates looked stalled. Align the GraphQL claim test with the visibility gate's not-found message.

Refs #268

* fix(node): pin claim guards and map mutation errors through AppError

Review required tests that go red if the pre-assigned claim predicate or
the anonymous announce gate is deleted, and incomplete must not stay
true when the candidate stream is exhausted at the scan ceiling. Route
claim, complete, and fail through AppError so closed-pool outages stay
503 and 404s match the read envelope.

Refs #268

* fix(node): match bare and did:key assignee forms in claim and list

create_task stores the supplied assignee unchanged, so a raw SQL
equality check drops a designated assignee who presents the other
did:key form. Compare the normalized key so claim and filtered list
agree with did_matches.

Refs #268

* fix(node): check task creation status and add assignee expression index

- Add error_for_status() to cmd_create and task_create MCP tool
- Update test_create_task_server_error to assert failure on 500
- Add migration v18 creating expression index idx_agent_tasks_assignee_key matching ASSIGNEE_DID_CASE_SQL
- Add did:web:z6Mkfoo single-residual shape to parity boundary matrix

Refs #327

# Conflicts:
#	crates/gitlawb-node/src/db/mod.rs

* fix(node)!: Replace task cursors with opaque continuation tokens

The task read path treated visibility, pagination, and error vocabulary as
separate edits, so each one broke where they met. Rework them as one contract.

A raw (created_at, id) cursor forced a choice between two broken options: it
could name the last visible row, and then a denied window longer than the
1,000-candidate scan budget was unpageable forever; or it could name the last
examined row, and then a denied read leaked the id and timestamp of a task
GET /tasks/{id} otherwise 404s. Continuation tokens remove the choice. They
carry the last examined candidate, so paging always advances a full scan budget
per request, and they are encrypted and authenticated under a node-derived key,
so the caller learns nothing from one and cannot forge one naming a row of
their choosing. Encryption is a synthetic-IV construction over the hmac/sha2
pair already used for webhook signatures, so it adds no dependency and needs no
randomness source.

Making the token the only accepted cursor also gives the ordering key one
domain. agent_tasks.created_at is TEXT and compared as TEXT, so a caller-typed
'...Z' and '...+00:00' denote one instant but sort differently, and a client
could silently skip or repeat same-time rows. The token carries the stored
string verbatim, so the value compared is always one the server wrote. The raw
after_*/cursor_* pairs are removed rather than kept alongside it, since a second
domain is the bug.

Separate the two facts the old single incomplete flag conflated: has_more says
candidates remain, incomplete says this page is short only because the
authorization scan hit its ceiling. Both REST and GraphQL now return has_more,
incomplete, and next_cursor from the shared collector, and REST echoes the limit
it actually applied so a clamped request is visible as clamped.

Have gl task list and MCP task_list follow next_cursor instead of issuing one
request: --limit 500 returned a successful but silently truncated 200 rows.
Following is bounded by a page cap and a no-progress guard, and a run stopped by
either reports an explicit incomplete result with a resume cursor.

Route claimTask, completeTask, and failTask through the same task_write_conflict
classifier the REST handlers use, via curated helpers in the graphql module so
the map_err source guard still holds. A claim race or stale finish reached
GraphQL clients as a generic database error while REST clients got an actionable
conflict; genuine sqlx faults stay opaque on both.

Refs #327

* fix(node): End the candidate stream only after the batch is examined

A short SQL batch means no rows exist past it, not that every row in it was
examined. When the page filled mid-batch the collector treated the two as the
same, marked the stream ended, and suppressed the continuation, so every row
after the one that filled the page was unreachable. The equal-timestamp paging
tests caught it: three rows with a limit of one returned only the first.

Track how much of each batch was consumed and end the stream only when the
whole of a short batch has been examined. Otherwise leave `has_more` to the
probe row, which resumes from the last examined candidate.

Refs #327

* fix(gl): Reject a non-positive task-list limit and fault the write under test

A `--limit 0` reached the node, which clamped it to zero and answered with
an empty page marked complete, so an invalid request read as proof that no
tasks exist. Reject a non-positive limit in `fetch_tasks()`, the helper the
CLI and MCP share, so the guard cannot drift between the two surfaces.

`task_write_sql_faults_stay_opaque` did not exercise what it named. Dropping
`updated_at` also broke the SELECT in `get_task()`, so the fault surfaced
from the `get_visible_task()` pre-check through `graphql_app_err` and never
reached `graphql_claim_conflict`. A `BEFORE UPDATE` trigger keeps every read
valid and faults only inside `Db::claim_task`, and the test now also asserts
that a write-time fault is not reclassified as a claim race.

Refs #268

* fix(node): Bind task cursors to the caller and brake the task read routes

A continuation token names the last candidate a scan examined, not the last
row it returned, so it encodes how far that scan got under one caller's
visibility. The MAC bound the page filter but not the presenting identity,
so resuming a token as a different caller started the scan past rows that
caller was entitled to read and dropped them from the answer with nothing
to signal the loss. Bind the caller's normalized DID into the MAC, with
anonymous flagged absent rather than encoded as empty. Normalization goes
through normalize_owner_key so the two spellings of one did:key identity
bind identically, matching did_matches on the read path: a caller who
presents the other form of their own DID keeps their own page. A mismatched
token renders the existing single rejection message, so this adds no oracle.

GET /api/v1/tasks and GET /api/v1/tasks/{id} are anonymously reachable, and
the visibility gate costs a task lookup plus deduped-repo and
visibility-rule queries before it can return the opaque 404. An
unauthenticated prober therefore pays nothing while the node pays per
request, whether or not the id exists. Attach the per-IP limiter already
used on /ipfs/{cid}, configurable through GITLAWB_TASK_READ_RATE_LIMIT and
swept by the periodic task like every other per-key limiter.

Refs #268

* fix(node): Brake anonymous GraphQL task reads on the same per-IP bucket

The per-IP brake added for the task read routes covered only
/api/v1/tasks*, so an anonymous caller reached the same
collect_visible_tasks and get_visible_task gate over /graphql with no
bucket at all. The fence had an open lane beside it.

Carry the brake as GraphQL request data and debit it in the tasks and
task resolvers rather than layering rate_limit_by_ip onto the GraphQL
router: /graphql is one endpoint for every operation, so a router layer
would charge unrelated queries and every mutation against the task-read
bucket. Debiting per resolved field also prices an aliased query
honestly, since ten aliased tasks fields run the gate ten times.

Extract RATE_LIMIT_MESSAGE so the GraphQL surface, which cannot return a
429 status inside a 200 envelope, refuses with the same text the REST
routes use.

/graphql/ws serves the query root as well and stays unbraked; closing it
needs a WebSocketUpgrade handler and is left for a follow-up.

Refs #268
Refs #327

* fix(gl,node): parse typed task pages, track cursor cycles, and normalize assignee filter MAC

* fix(node): Cap aliased GraphQL task reads per request and derive has_more from visible rows

- Cap aliased GraphQL task read fields per request using an atomic counter
  on TaskReadBrake (MAX_GRAPHQL_TASK_READS_PER_REQUEST = 5).
- Derive has_more in collect_visible_tasks by scanning for bounded_limit + 1
  visible rows, eliminating the un-gated keyset probe that could leak the
  presence of trailing denied tasks.
- Add regression tests covering aliased GraphQL capping and trailing denied
  task has_more privacy.

Refs #327

* fix(node): Check database continuation when scan budget ends on exact batch boundary

When candidate scanning reaches MAX_TASK_SCAN_CANDIDATES without finding
a target_visible row and the final batch was full, probe the database for
rows beyond the scan position so an exhausted candidate stream is not
erroneously marked incomplete.

Refs #327

* fix(node): Document and pin the bound on the task scan-ceiling probe

The scan-ceiling branch of collect_visible_tasks settles has_more with an
un-gated LIMIT 1 probe, so a caller can learn whether any row - readable
or not - trails the position the scan stopped at. Withholding the probe
does not remove that bit: enumeration past a denied window longer than
one scan budget requires handing back a continuation, and following that
continuation returns the same terminal page one round trip later.

State what the probe discloses (one bit, only at server-chosen positions
a full scan budget apart, reachable only through a MAC'd cursor, never a
denied row's id, payload or ucan_token) and pin it end to end. Also
correct the comment above the branch, which claimed has_more never comes
from an un-gated probe while the code below it did exactly that.

Refs #327

* Index task keyset pages so the scan ceiling bounds database work.

Optional IS NULL predicates kept the planner from using a created_at/id order, so every list_tasks_keyset batch could sort a growing match set before LIMIT. Dedicated per-domain SQL plus v28 indexes make the candidate ceiling a database bound.

Refs #327

* Reject oversized task pages so list never returns more than the requested limit.

fetch_tasks asked for the remaining total, then appended every row on a valid-shaped page. A remote that sent more tasks than want could make gl and MCP expose more than --limit. Treat that page as protocol-invalid before any extra row is kept.

Refs #327

* Enforce task page limits on legacy responses and drop redundant single-column indexes.

Refs #327

* Address review feedback on task-read authorization gating and client safety.

- Wire TaskReadBrake into /graphql/ws subscriptions and verify per-request field caps and per-IP rate limiting over WebSocket connections.
- Define open-claim eligibility separately from read visibility so unassigned tasks on readable/unscoped domains can be claimed without making task bodies enumerable.
- Propagate identity errors on explicit key directories in CLI and MCP instead of silently falling back to anonymous mode.
- Enforce response byte limits before deserializing task pages, validate row schema and page-local uniqueness before row commit, and sanitize continuation cursors in terminal diagnostics.
- Add GraphQL denial assertions for repo-less tasks and token isolation.

Refs #268
Refs #327

* Address review feedback on task-read authorization gating, WebSocket budgets, and test framing.

Refs #395

* Address review feedback on quarantined task claims, WebSocket auth, and MCP limit parsing.

Refs #395

* Reject forged signatures during WebSocket authentication upgrade.

Refs #395

* Accumulate forged WebSocket handshake response until headers terminate.

Refs #395

* Withhold quarantined-repo tasks on all read surfaces before party checks.

Refs #395

* docs: Align task-read documentation with enforced visibility and rate limits.

Refs #395

* docs: Distinguish task list and get behavior in rate-limit guidance.

Refs #395

* fix(node): Batch task quarantine checks to bound read costs.

Refs #395

* docs(node): clarify the shared task-read budget

Document that GraphQL task queries over HTTP and WebSocket debit the same per-IP hourly budget as REST task reads.

Refs #395

* fix(node,gl): bound scoped repo lookup and validate MCP cursor

Bound repository access in list_repos_deduped_by_ids to requested groups via idx_repos_owner_key_name, leaving dedup_cte unscoped. Strictly validate MCP task_list cursor argument before network dispatch.

Refs #396

* test(db): verify scoped lookup work in prepared plans

Measure discarded rows and verify actual custom and generic prepared-plan counters, including the previous full-index-scan regression.

Fixup for a0ea4c78e4.
Refs #396

* docs(db): remove stray dedup comment escapes

Restore readable root-path notation in the deduplicated listing documentation.

Fixup for a0ea4c78e4.
Refs #396

* fix(node): address #464 review findings

- Hold migration 27/28 numbering: reservation comment now names the
  competing Gitlawb/node#384 range (27-35) and states the numbering is
  on hold pending #384. No renumber.
- Restore crates/gl/src/identity.rs to LF endings so the diff shows
  only the added load_optional_keypair function.
- Drop the dead single-column idx_agent_tasks_assignee_key build from
  v27 (v28 builds the keyset-ordered variants directly); adjust the
  v27 test to assert the raw index is dropped and the expression
  index is never built.
- Align task_claimable with claim_task's conditional write: drop the
  delegator short-circuit and correct the docstring.
- Pin error_for_status on the gl task list path with a 403 mock that
  serves a well-formed empty page, so the test goes red if the call
  is removed.

* test(node): pin the delegator deny on a pre-assigned claim

- Add claim_task_delegator_cannot_claim_preassigned_task: pins opaque 404
  for delegator claiming a pre-assigned task (regression guard for P2)
- Fix test_list_tasks_server_error_surfaces regex: /api/v1/tasks\? ->
  /api/v1/tasks\? (escaped ? for literal question mark)
- Assert actual reqwest StatusCode::FORBIDDEN instead of matching "403"
  in error string (avoids spurious pass from random mock port)

* test(node): exercise streamed byte-budget guard with chunked body

Use with_chunked_body instead of with_body so mockito sends no
Content-Length, pinning the no-length streamed accumulation guard.
Assert the "exceeded" message to distinguish from the declared-length
check.

* fix(node): pin declared-length byte-budget arm in oversized content-length test

Addresses beardthelion review finding (2026-09-26): test_read_task_page_json_oversized_content_length now asserts on the "(declared" arm the way the chunked twin asserts "(exceeded", so the Content-Length pre-check at crates/gl/src/task.rs:211 is genuinely pinned. Verified the pin by neutering the pre-check (test goes red), then restored. fmt clean.

* fix(node): fail closed on claim for unresolvable repo

task_claimable returned true for an unassigned task whose repo_id is
slash-form (mirror row) or names a repo this node does not host, while
task_visible fails closed on the same rows. A signed stranger could claim
such a task even though reads 404. Return false for a Some(repo_id) the
node cannot resolve to a locally-hosted repo, matching task_visible's
fail-closed check; keep the None arm open (repo-less tasks claimable by
design). Adds a regression test asserting a stranger's claim 404s and
leaves the assignee unchanged for both row classes.

* fix(node): pad sealed task cursor to fixed width

The scan-ceiling token seals the last examined row's (created_at, id) as
variable-length JSON under a length-preserving keystream, so token length
leaked the withheld row's field widths: a 32-char to_rfc3339() rendering
mints a 148-char token where a 35-char one mints 161 chars. Pad the
serialized payload with trailing JSON whitespace to a fixed 128-byte
bucket before the SIV tag is computed; decode tolerates the padding
(serde_json ignores trailing whitespace) and tokens minted before this
change still verify. REST and GraphQL share task_cursor::encode, so one
fix covers both.

Adds token_length_does_not_vary_with_position_width: asserts identical
token length across narrow/wide timestamp positions and that the padded
token still decodes to the exact stored position.

* fix(node): address beardthelion round-six P3 findings on task cursor

- Pin the encode docstring compat claim: add unpadded_legacy_token_still_verifies
  test minting a token via the pre-padding recipe (siv + apply_keystream with
  no padding) and asserting decode returns the position.
- Pin the PAYLOAD_BUCKET round-up claim: add oversized_payload_rounds_up_to_next_bucket
  test with a 140-char id crossing the 128-byte bucket, asserting round-trip
  at two buckets.
- Correct timestamp-width figures: to_rfc3339() renders 25 chars with no
  fractional seconds (not 32); 32 is the microsecond width.

* fix(node): address beardthelion round-seven P3 findings on task cursor

- Correct wire-form docstring to v1.<iv>.<body>
- Fix payload width figures: 101 bytes total, to_rfc3339() precision ladder
- Tighten oversized-payload test to assert exactly 2*PAYLOAD_BUCKET
- Pin legacy back-compat test with a frozen v1 token fixture

* test(node): address beardthelion round-eight P3 findings on task cursor

- Pin the one-bucket width: round_trips_position_verbatim now asserts the
  decoded body of a normal token is exactly PAYLOAD_BUCKET, catching a pad
  that skips next-multiple or a drifted bucket constant. Every other width
  assertion here is bucket-relative, so only an absolute check goes red.

- Assert the frozen fixture is the unpadded shape:
  unpadded_legacy_token_still_verifies decodes the body segment and asserts
  raw.len() < PAYLOAD_BUCKET, so a literal re-minted under the padded
  recipe would retire the back-compat pin loudly instead of passing
  silently.

- Narrow the recipe-pin claim: the docstring now names what the frozen
  literal actually binds (domains, MAC input, keystream, segment layout,
  field names) instead of claiming padding and serialization changes break
  the test. The padding-width claim stays with
  oversized_payload_rounds_up_to_next_bucket.

* fix(node): trim stored assignee on every task gate; align SECURITY.md task-read wording

* test(node): pin mirror-row list-vs-get asymmetry; fix plan-test bind (#553)

- EXPLAIN helper binds normalize_owner_key(assignee) to match the
  production list_tasks_keyset bind.
- Pin the deliberate asymmetry jatmn asked about: a delegator sees a
  slash-form (mirror-hosted) task in a list, while get fails closed for
  every caller because the row has no local repo backing. Notes in
  get_visible_task and a regression test keep it from drifting.
- Repair test-mod drift from the main merge: build_schema gains its
  TaskCursorKey arg at three sites and list_tasks calls move to
  list_tasks_keyset.

* test(node): reconcile tasks complexity contract with TaskPageType (#553)

The four GraphQL task/brake tests failed on this branch: contract tests
still queried the flat tasks { id } shape that TaskPageType replaced, and
the alias-brake tests ran default-size lists whose six aliases exceed
GRAPHQL_MAX_COMPLEXITY before the brake can debit.

- Contract tests now select through items { ... } with refreshed cost
  arithmetic.
- Brake tests use tasks(limit: 1), keeping six aliases under the budget
  so the 6th field reaches TaskReadBrake and fails RATE_LIMIT_MESSAGE.
  Complexity stays the outer gate (#552); the brake stays the per-request
  field-count gate.

* docs: note mirror-hosted task list-vs-get asymmetry (#553)

Option 3 for slash-form mirror repo_id rows was pinned in code and test at
6c0926de; this adds the operator-facing half jatmn asked for. README and
SECURITY task-read bullets now state that mirror rows list for party readers
while get-by-id and GraphQL task fail closed for all callers.

* ci: retrigger codeql scan after alert 86 dismissal

---------

Co-authored-by: euxarist <[email protected]>
Co-authored-by: cairn-code <[email protected]>
Co-authored-by: cairn-intern <[email protected]>
Co-authored-by: Marlowe <[email protected]>
2026-11-09 19:02:54 +00:00
0919cf7c62 fix(core): ignore invalid signatures in satisfies_threshold (#349) (#554)
* fix(core): ignore invalid signatures in satisfies_threshold (#349)

RefUpdateCert::satisfies_threshold called verify_all, which fails closed
on the first malformed or invalid signature entry. Signature entries live
outside the signed body, so any third party can append a junk entry with
no key material — appending one denied an otherwise valid certificate, a
DoS on the threshold check.

Add a lenient valid_signers() helper that skips entries failing to parse,
decode, or verify, and have satisfies_threshold count distinct valid
signers from it. verify_all stays strict (fail-closed) so its API and the
tampered_signature_fails_verify_all test are unchanged. A forged entry
naming a real maintainer DID with a bad signature is skipped, so it cannot
inflate the count either.

Adds tests: appended junk entries do not deny a valid 2-of-2 cert, and a
forged maintainer signature is not counted toward the threshold.

Closes #349

* fix(core): bound the tolerant signature scan before cryptographic work

satisfies_threshold verified every entry, so a peer could append
arbitrarily many syntactically valid entries with bogus signatures and
buy attacker-controlled CPU and allocation on every threshold check.
Bound each stage ahead of the loop: reject a list over
MAX_SIGNATURE_ENTRIES loudly (a prefix scan would let junk starve real
signatures out of the window), refuse to base64-decode any encoded
signature that is not exactly the 86 chars of a 64-byte Ed25519
signature, skip non-maintainer and already-counted signers on a string
compare before any curve work, and return as soon as the threshold is
met.

* docs(cert): correct ENCODED_SIG_LEN length-gate comment (#554)

The comment claimed anything longer cannot decode to a valid signature.
The same 64 bytes appear in padded or non-url-safe encodings; the gate is a
canonical-form DoS bound for the tolerant scan, not a crypto impossibility.

---------

Co-authored-by: andreolf <[email protected]>
Co-authored-by: Kevin Codex <[email protected]>
2026-11-09 17:42:13 +00:00
beardthelionandRapyBus S.A.C. 05afd68e41 bootstrap-peers: add moonwake-ledger with a dialable p2p multiaddr (#559)
Co-authored-by: RapyBus S.A.C. <[email protected]>
2026-11-09 17:21:54 +00:00
beardthelion 6c971e7ea0 ci(audit): ignore RUSTSEC-2026-0459, unlinked libsqlite3-sys advisory (#564)
The advisory covers bundled SQLite in libsqlite3-sys 0.30.1, which sits in
Cargo.lock only because sqlx resolves sqlx-sqlite unconditionally. sqlx builds
postgres-only, so the bundled C source is never compiled or linked. The fix
needs libsqlite3-sys >= 0.39 but sqlx-sqlite 0.8.6 pins ^0.30.1; the waiver is
documented to retire if the sqlite feature ever enters the build.
2026-11-09 16:54:05 +00:00
beardthelion 10d5b98b2b fix(gl): report quickstart registration failure when the node returns no UCAN (#355) (#556)
A 2xx with a non-JSON or ucan-less body still printed "Registered
successfully / UCAN saved to ..." while nothing was written, and left a
stale ucan.json from another node in place. Print the success lines only
when a UCAN was actually persisted.
2026-11-09 16:11:08 +00:00
a2262d92b2 fix(node): implement reconciliation sweep as durability backstop (#218) (#551)
* feat(node): implement reconciliation sweep as durability backstop (#218)

Reconciliation sweep: periodic background worker that re-derives pin/seal
sets and fills gaps so a dropped replication job never means data loss.

- visibility_pack.rs: all_object_paths (two-phase ls-tree + cat-file
  --batch-all-objects), allowed_blob_tree_sets_bounded, ObjectPath/BlobTreeSets
  type aliases
- db/mod.rs: atomic policy transactions, get/set_node_state, has_ipfs_cid,
  filter_ipfs_pinned_oids, filter_pinata_pinned_oids, repo_policy_epoch,
  migration v27 (node_state table + policy_epoch column)
- ipfs_pin.rs: PolicyFence struct, dispatch fence re-check before POST
- pinata.rs: dispatch fence re-check before POST
- encrypted_pin.rs: dispatch fence re-check
- reconciliation.rs: full sweep worker with keyset cursor, PolicyFence
- api/ipfs.rs: effective_cid fallback in list_pins
- main.rs: spawn reconciliation sweep worker
- .cargo/audit.toml: RUSTSEC-2026-0258 h2 ignore (pending #368)

* fix: add missing parameters to pinning functions and adjust timeout settings

* fix(tests): update CID assertions to handle Option type and adjust parameters in pinning functions

* fix(node): close review findings on reconciliation sweep (#218)

Reviewer-1 P1 / Reviewer-2 P1: v27/v28/v29 were referenced by tests but
never added to the MIGRATIONS array, so every existing deployed node
would fail to boot (node_state, repos.policy_epoch, pinned_cids.cid
NOT NULL → NULL all unreadable). Forward-only migrations:
  v27 pinata_only_clear_legacy_equal_cid: ALTER cid DROP NOT NULL,
       UPDATE cid = NULL WHERE cid = pinata_cid
  v28 node_state_key_value: new key/value table for the sweep cursor
  v29 repos_policy_epoch: BIGINT NOT NULL DEFAULT 0

Reviewer-2 P1: visibility_pack.rs:542 phase 1 used ls-tree -rz, which
under -r recurses into blobs and never emits tree entries. Trees then
arrived only from the phase-2 catch-all with empty path and the
fail-closed filter denied every one — the sweep could repair a
non-flat repo's git graph. Switched phase 1 to ls-tree -r -t -z and
add a per-commit rev-parse <commit>^{tree} that registers the root
tree at "/". 48 visibility_pack unit tests pass.

Reviewer-1 P2 / Reviewer-2 P2: list_pinned_cids called is_raw_cidv1 on
the cid column before decoding it as Option<String>, so SQL NULL
failed the decode and the new Pinata-only rows never reached the
handler. Decode cid as Option first, drop only the both-NULL case.
The is_raw_cidv1 filter was removed entirely: the new contract lists
every row that has something to advertise, and the handler at
api/ipfs.rs:list_pins is the seam that decides what to do with a
legacy-shape row (the resolver 404s on a mismatched key, per #173 U4).
Updated list_pinned_cids_omits_unrepaired_legacy_row to match.

Reviewer-1 P2: two new sqlx::test cases pin the gate order:
sweep_skips_quarantined_repos_before_scan (SQL filter, plus a
defense-in-depth per-row re-check) and
sweep_skips_private_repos_before_scan (per-repo listable_at_root).
Both use expect(0) on the mock IPFS so any future reorder that
moves the scan ahead of the gate fails the build.

Also: record_pinned_cid ON CONFLICT now rewrites cid (R1-P2: a
stale wrong CID from a previous push is repaired by a subsequent
one). record_pinata_cid ON CONFLICT clears cid to NULL when the
existing row has the legacy cid = pinata_cid shape; on INSERT, stores
cid = NULL when raw_cid == pinata_cid (the Pinata-only signal).

* feat: implement per-backend continuation offset for missing OIDs

- Enhance `missing_oids` function to accept a `start_after` parameter, allowing for rotation of the missing set based on the last attempted OID.
- Update `run_pass` to load and save the continuation offset for both IPFS and Pinata backends, ensuring that previously attempted OIDs are retried last in subsequent passes.
- Introduce tests to validate the behavior of the new offset mechanism, ensuring deterministic ordering and proper handling of gaps in the missing set.
- Ensure that the offset is cleared when no missing OIDs are found, allowing for fresh starts in subsequent passes.

* fix(node): trace provenance/structural invariants through every consumer

The v30 local_ipfs_provenance column was set by the writers but
only consulted by has_ipfs_cid and filter_ipfs_pinned_oids. The pin
loop's "is this already done" check still used row existence
(is_pinned), so a Pinata-only row hit the early-return branch
without ever reaching the local writer, and the flag stayed FALSE
forever. Three other seams carried the same shape-vs-provenance
confusion.

P1 (ipfs_pin.rs:1843): pin loop's skip keys on has_ipfs_cid
(writer-owned local_ipfs_provenance = TRUE), not on row existence.
A Pinata-only row now falls through to the local writer path,
which calls record_pinned_cid_with_source and flips the flag on
the conflict branch. The Pinata pin loop's skip remains keyed on
has_pinata_cid (a separate seam).

P1 (db/mod.rs v30 backfill): tighten the WHERE clause to
cid IS NOT NULL AND pinata_cid IS NULL. The previous permissive
(cid <> pinata_cid) clause marked pre-v30 Pinata-first uploads
(where record_pinata_cid deliberately stored cid = raw_cid
without a local Kubo push) as local IPFS pins, so the sweep
filtered them as already-durable and an operator enabling IPFS
later never got a local recovery copy. Ambiguous dual-shape rows
stay at FALSE under the strict rule and the sweep re-derives on
re-pin (cheap, idempotent).

P1 (db/mod.rs record_pinned_cid_with_source ON CONFLICT): restore
cid = COALESCE(pinned_cids.cid, EXCLUDED.cid). A Pinata-only row
with cid = NULL (the raw_cid == pinata_cid shape) would otherwise
be marked complete with local_ipfs_provenance = TRUE and a NULL
resolver key.

P1b (visibility_pack.rs): remove the synthetic "/" path on root
trees and add a structural entry-level check
(structurally_safe_root_tree / _for_caller) in both the sweep
gate (allowed_blob_tree_sets_bounded) and the per-request tree
gate (allowed_tree_set_for_caller_bounded). A tree is admitted
only if every direct entry in its serialized bytes is safe at
/{filename} and (for tree entries) the child tree is itself in
the allow set. The previous path-based "/" check let the root
tree of a public repo slip through even when its bytes name a
denied subtree entry.

P2 (api/ipfs.rs list_pins): add local_ipfs_provenance to
PinnedCidRecord, include in list_pinned_cids SELECT, and surface
as local_pinned / pinata_pinned in the JSON response. A
Pinata-only row previously returned the raw resolver key as
local_cid, making it indistinguishable from a real dual-backed
row. The new fields let a gl consumer distinguish local-only,
remote-only, and dual rows without re-inferring semantics from
nullability. local_cid is preserved for backward compatibility.

Tests:
- migration_v30_backfills_local_ipfs_provenance_heuristically:
  flipped both-distinct expectation to FALSE under strict rule.
- migration_v12_makes_cid_nullable_and_preserves_classification
  and migration_v27_clears_legacy_equal_cid: both-distinct /
  distinct row expectations flipped to FALSE.
- sweep_never_pins_root_tree_naming_withheld_subtree (new):
  assert root tree + secret subtree tree absent from pinned_cids.
- allowed_tree_set_gates_withheld_subtree_tree: root tree
  assertion inverted to EXCLUDED for anon.
- list_pins_reports_writer_owned_provenance_for_all_shapes (new):
  assert 4 row shapes produce correct local_pinned /
  pinata_pinned JSON.

Scope discipline preserved: PolicyFence capture order,
_pin_permit reuse for the seal phase, fresh per-arm
rederive_budget, keyset repo cursor with lookahead, mirror-row
skip, Kubo 2xx-without-Hash refusal, v27 provenance-clearing
migration all untouched. v31 schema unchanged.

* fix(node): structural tree gate at every depth + non-commit ref tolerance

The prior round installed a `local_ipfs_provenance` column and
applied a structural tree gate only to root trees. The fresh
review pass found two structural gaps the writer contract
didn't propagate to:

P1 (visibility_pack.rs:1519): the path-based tree admission
admitted a tree whenever its path was allowed, regardless of
whether its serialized bytes named a denied subtree. Concrete
failure: a public repo with `/public/secret/file` and a
`/public/secret/**` deny admitted the `/public` tree (its path
is `/public`, allowed) but `/public` still contained a
`secret/` entry pointing at the denied subtree — anyone who
obtained `/public`'s CID could `cat-file -p` it and read the
subtree's name and child OID. The prior round's post-pass
structural check ran on root trees only.

The fix: replace the prior round's
`structurally_safe_root_tree` / `_for_caller` / `root_tree_oids`
helpers with one recursive `tree_structurally_safe` that walks
every tree's entries at every depth, recursing on tree entries.
Bundled the rule/path args into a `TreeCheckCtx` struct to fix
the clippy `too_many_arguments` warnings the prior round
introduced. A tree is admitted iff every direct entry is safe
at its path AND (for tree entries) the child tree is itself
structurally safe there. A denied-subtree tree fails at its
level; the parent fails because the denied-path child entry
fails; the propagation continues up to the root. The
`admitted` set memoizes trees proven safe at some path so the
recursion short-circuits on cycles and on trees reachable at
multiple allowed paths.

P1 (visibility_pack.rs:533): `all_object_paths` called
`assert_all_refs_are_commits`, which bailed on any ref that
didn't peel to a commit (tag-of-tree, tag-of-blob). The
sweep (`run_pass`) converted that error into a full-scan
failure that skipped the whole repo. A repo with a pushable
tag-of-tree never got its missing public pins repaired by the
new hourly worker, even though its commit-reachable graph
was otherwise safe to classify (`ipfs_cid_tree_served_despite_non_commit_ref`
is the in-repo example of that supported shape).

The fix: remove the `assert_all_refs_are_commits` call from
`all_object_paths` (and the analogous one in `blob_paths`,
used by the encrypted recovery path — the same reasoning
applies). `git rev-list --all` already silently skips
non-commit refs, so the commit-reachable object set is what
the sweep needs. The unused `assert_all_refs_are_commits`
helper is removed. Unclassifiable ref targets still fail
closed at a different layer: the cat-file catch-all
enumerates them with no path, and the path-based allow
filter drops empty-path entries.

Test inversions and new regressions:

- `migration_v12_makes_cid_nullable_and_preserves_classification`
  and `migration_v27_clears_legacy_equal_cid` — both-distinct /
  distinct row expectations flipped to FALSE under the v30
  strict backfill (prior round's tightening is re-affirmed
  here; the v30 backfill is `cid IS NOT NULL AND
  pinata_cid IS NULL`, the only unambiguous local-IPFS
  shape).
- `ipfs_cid_gate_withholds_blob_from_unauthorized` — root
  tree assertion inverted to NOT_FOUND (the recursive
  structural gate denies root trees whose entries name a
  denied subtree).
- `ipfs_cid_legacy_provider_cid_repaired_on_repush` —
  legacy-shape row insert now sets `local_ipfs_provenance
  = TRUE` to reflect that the bytes ARE on local IPFS (the
  legacy CID is just the wrong key).
- `ipfs_cid_tree_served_despite_non_commit_ref` — root
  tree assertion inverted (same reason as the gate test).
- `fails_closed_on_annotated_tag_of_a_blob` and
  `fails_closed_when_a_ref_cannot_be_traversed` — renamed
  to `skips_*`, assertions inverted from `Err` to `Ok` (a
  ref pointing at a non-commit object is no longer a walk
  error; the blob falls out of the gap set because
  `git rev-list --all` skips it).
- `ipfs_cid_walk_error_fails_closed` — removed (relied on
  the now-removed `assert_all_refs_are_commits` to force an
  error; the fail-closed arm is still covered by
  `ipfs_cid_commit_tag_walk_error_fails_closed`, which uses
  a different trigger — a nonexistent ref object makes
  `rev-list --all` fail).
- `sweep_does_not_publish_public_ancestor_tree_naming_withheld_subtree`
  (new) — recursive depth test: a `/public/secret/**` repo
  asserts root tree, `/public` tree, and `/public/secret`
  subtree are all absent from `pinned_cids` while
  `/public/visible.txt` is IPFS-pinned.
- `sweep_repairs_commit_reachable_object_in_repo_with_tag_of_tree`
  (new) — tag-of-tree repo with a separate `mktree`'d tree
  (not commit-reachable); asserts the commit-reachable blob
  is IPFS-pinned and the unclassifiable tag-of-tree is not.

Scope discipline preserved: PolicyFence capture order,
`_pin_permit` reuse for the seal phase, fresh per-arm
`rederive_budget`, keyset repo cursor with lookahead,
mirror-row skip, Kubo 2xx-without-Hash refusal, v27
provenance-clearing migration, v30 strict backfill, v31
schema, per-(repo, backend) reconciliation offset, and the
`local_ipfs_provenance` writer/reader contract are all
untouched.

* test(node): ignore pre-existing RED replication-tail test on CI

`api::repos::tests::receive_pack_tail_survives_a_disconnect_during_release`
documents an open bug in `api/repos.rs` (the replication tail must
be spawned above `release` and gated on
`receive_result.is_ok()`; today it's spawned below, so a
disconnect during `release` drops the pins, recovery copy, and
announce). The test's `assert!` message starts with "RED:" —
this is a TDD-style "keep the design intent visible in the code"
test, not a working-behavior assertion.

The test was added 2026-07-29 by `beardthelion` (PR #215
lineage), well before the #218 sweep work, and is unrelated to
this PR. CI is failing on it under load (passes in
isolation) because the open bug hasn't been fixed yet.

`#[ignore]` until the open bug is fixed. The test's design
intent and the fix description stay in the doc comment and the
assert message; only the CI failure is suppressed. Run the
test explicitly with `cargo test -- --ignored` to verify
the bug is fixed when the tail-spawn moves above `release`.

* fix(node): align encrypted manifest anchor log with one-shot contract

The "will retry next pass" log at reconciliation.rs:1034 was false:
plan_seal returns SkipUnchanged on every subsequent pass once the
recipients tag matches, so sealed stays empty and the anchor block
never runs again. A transient Irys outage at the moment of a fresh
seal therefore LOSES that anchor permanently — the next pass has no
delta to anchor and no retry fires. This is intentionally best-effort:
re-anchoring an unchanged manifest would burn Irys writes for no
recovery benefit.

Factored the warn message into ENCRYPTED_MANIFEST_ANCHOR_FAILED_MSG
and added a unit test that pins the constant; the test goes RED if
"will retry" reappears in the log.

* fix(node): address eight reviewer findings on #218 reconciliation sweep (round 3)

* fix(node): close audit.toml array and drop inline comment

* fix(node): repair the stale for-each-ref fake-git fixture (round 8 P1)

`upload_pack_shares_one_deadline_across_walk_and_filtered_serve` — the
#174 shared-deadline guard for the FILTERED serve arm — was red on both
`test (stable)` and `test (beta)`.

The fake git answered `for-each-ref` with the single token
`refs/heads/main`, a ref NAME, where `blob_paths` phase 2 asks for
`--format=%(objectname) %(objecttype)`. The phase-2 parse fails closed on
a line it cannot read, so the walk returned an error and the request
surfaced as a generic 500 instead of being reaped into a 504. That
silently repurposed the test from "the filtered serve shares one
deadline" into "the walk errors" — the guard proved nothing while it was
red.

The fixture is what was stale, not the production path: the arm now
emits the column shape the walk parses. A commit tip has no peeled
referent, so two columns is the whole line. The 1.2s walk cost stays on
`ls-tree` and the serve's on `pack-objects`, so the two phases remain
independently attributable.

* fix(node): peel annotated tags in the blob_paths ref walk (round 8 P1)

`%(objecttype)` reports the type of the ref's OWN object, which for an
annotated tag is `tag`. The phase-2 `if kind == "blob" || kind == "tree"`
arms therefore never saw the referent, and a blob reachable ONLY through
an annotated tag escaped the withheld set — while `rev_list_keep`'s
`git rev-list --objects --all` does peel tags and served it. Phase 2
closed the leak for a direct ref-to-blob and for that shape only.

The walk now asks for the peeled atoms `%(*objectname) %(*objecttype)`
and inserts the referent with an empty path, alongside the direct
blob/tree tips it already handled. Line parsing moves to a field-count
match: two columns for a non-tag tip (the peeled atoms expand to empty),
four for a tag tip, anything else still fails closed.

Measured against git 2.50, the `*` atoms peel the WHOLE chain — a
tag-of-a-tag-of-a-tag-of-a-blob reports the blob — so a `tag` peeled
type does not occur on stock git. That arm is kept anyway, because
`push_delta.rs`'s ref-type guard documents `%(*objecttype)` as a
one-level peel: under such a git it finishes the chain with
`rev-parse <tip>^{}` plus a `cat-file -t` probe rather than bailing and
500ing every clone of a repo that merely carries a nested tag. Both
children are bounded by the walk's shared deadline and are unreachable
on the common one-line-per-ref path.

* fix(node): one empty-path decision for both blob_paths consumers (round 8 P1)

`blob_paths` has two consumers and only one of them carried the
empty-path policy. `withheld_from_pairs` (the deny side, what the
smart-http serve filter excludes) withheld a phase-2 empty-path entry
from everyone but the owner; `allowed_blob_set_for_caller_bounded` (the
allow side, what `GET /ipfs/{cid}` hands over) called
`visibility_check(..., "")` directly. On a public repo no glob matches
the empty path, so that returns `Allow`: the serve filter withheld the
OID while the IPFS gate served the bytes — the leak phase 2 exists to
close, reopened one layer over.

The policy moves into `pair_decision`, which both consumers now call, so
the two gates cannot disagree and a future change to the empty-path rule
moves both at once. `withheld_blob_recipients_bounded` routes through it
too: an unclassifiable blob grants a recovery copy to the owner only,
never to a rule's named reader whose grant was written against a path
this object does not have.

No behavior change for non-empty paths — `pair_decision` delegates to
`visibility_check` unchanged.

* test(node): bind the phase-2 ref tests to a commit-unreachable blob (round 8 P2)

`skips_a_ref_pointing_at_a_blob` and `skips_an_annotated_tag_of_a_blob`
were vacuous. Both hung their ref on `fixture()`'s `secret` blob, which
is COMMITTED at `secret/b.txt`, so phase 1's per-commit `ls-tree`
already produced `(secret, "/secret/b.txt")`, the `/secret/**` rule
already denied it, and `withheld.contains(&secret)` passed with phase 2
deleted outright.

Both now use `orphan_blob`, a blob written straight to the object store
that no commit's tree names, so the ONLY route into the withheld set is
the `for-each-ref` phase. That is also the real leak shape:
`rev_list_keep`'s `git rev-list --objects --all` does follow a ref to
such a blob, so an under-withheld one ships in the clone pack.

Verified by mutation: with `if false` injected at the phase-2 filter,
every test below goes RED (they were green under that mutation before).

New coverage:
- `ref_only_blob_is_denied_on_the_allow_side_too` — the allow-side
  denial for the shared `pair_decision`; it also asserts deny and allow
  agree on one OID, and that the owner IS still admitted, so it cannot
  pass by denying everything. Reverting the allow side to
  `visibility_check` turns it red on exactly that assertion.
- `skips_a_nested_annotated_tag_of_a_blob` — real-git tag-of-a-tag.
- `peels_a_tag_whose_peeled_target_is_still_a_tag` — drives the
  one-level-peel fallback through the fake-git seam, so that arm is
  tested rather than dead.

* fix(node): advance the reconciliation continuation only for dispatched work (round 8 P2)

`ipfs_last` / `pinata_last` were captured from the missing set above the
pin permit, both `PolicyFence` captures and both pin loops, then written
under nothing but `if ipfs_enabled` / `if pinata_enabled`. Every stage in
between can legitimately produce nothing — a fence capture that fails, a
quarantine/visibility recheck that says skip, a pin-boundary
re-derivation that errors — and each is transient.

Advancing past OIDs that were never attempted is not a harmless retry
delay: `missing_oids` rotates strictly past the stored offset, so the
whole unattempted prefix lands BEHIND the entire backlog on the next
pass. For an at-cap repo — the only kind the continuation exists for —
the backlog never drains inside one cap window, so those objects are
starved indefinitely. That is a durability hole in the durability
backstop.

The offset now moves only for work actually handed to a backend, tracked
at the dispatch boundary as `to_pin.last()` (not the missing set's last:
an OID the pin-boundary re-derivation dropped was never offered). It is
recorded before the call, so a pin phase that times out mid-batch still
counts as dispatched.

The write becomes three outcomes rather than two, via one
`next_offset_write` helper so the backends cannot drift apart:
  * work dispatched            -> advance to the last dispatched OID
  * nothing missing, query OK  -> None, marking the row done
  * nothing dispatched, or the missing-set query failed
                               -> write NOTHING, leaving the resume point
                                  a capped pass already paid for

`*_scan_ok` is what separates the last two: an empty missing set means
"everything is pinned" or "the filter query failed and we know nothing",
and only the first should mark the row done.

Test: `sweep_leaves_offset_untouched_when_nothing_was_dispatched`, using
a `#[cfg(test)]` seam on the pin-boundary re-derivation — that stage
cannot be starved from the outside, because the mid-scan re-filter runs
first on the same rules and budget and makes the sweep `continue` well
before the offset write. It asserts the stored offset comes back
byte-identical after a declining pass, then that the same repo does
advance it once dispatch happens, so the assertion cannot pass on a dead
write site. Restoring the old semantics turns it red.

* fix(gl): render the Pinata provider CID in `gl ipfs list` (round 8 P2)

The server-side wire change landed without its client half. `cmd_list`
read `pin["cid"].as_str().unwrap_or("?")` as the entry heading and
nothing in the CLI read `pinata_cid` at all, so a Pinata-only row
printed as a bare `?` — the listing showed nothing usable for an object
that is in fact durably stored.

`cid` is legitimately null for such a row: it is the key
`GET /ipfs/{cid}` resolves against, and the node stopped aliasing a
Pinata provider CID into it (round 3) precisely because that CID 404s
there. So the fix is on this side.

Rendering moves into `render_pin`, which leads with the node-resolvable
CID when there is one and otherwise the provider CID, labelled so nobody
feeds it back to this node's resolver. A dual row keeps both visible. A
`backends:` line names where the bytes are, read from the writer-owned
`local_pinned` / `pinata_pinned` booleans rather than re-inferred from
CID shape — which is what the node's own comment warns against.

Older nodes send neither flag; those `cid`-only rows fall back to CID
presence and render as they always did
(`render_pin_handles_a_legacy_cid_only_row`).

* fix(node): implement third fence check for IPFS pinning and enhance visibility rules handling

* fix(node): enhance documentation and add tests for ProgressState and helper functions

* fix(node): address round-9 reviewer findings on #218 reconciliation sweep

* fix(node): correct consumer-4 recipient test for call-invariant set

* fix(git): fail closed on non-UTF-8 tree listing in walk_tree_oids_inner (#218 round 10 P1)

walk_tree_oids_inner previously returned Ok(()) on a non-UTF-8
ls-tree -z listing, inserting only the tree OID and skipping
every child blob/tree OID. A direct tree ref (or an annotated
tag peeling to a tree) is valid Git input; git rev-list --objects
--all still enumerates the tree and every descendant, so the
keep-side removed the tree from the served set but passed the
child blob OIDs to pack-objects, exposing their bytes to an
anonymous clone.

Phase 1 already bails on the same input at the blob_paths walk;
this fix puts the tree-walk on the same fail-closed outcome.
The new fails_closed_on_non_utf8_tree_tip test pins the
invariant for both a direct refs/tags/direct-tree ref and an
annotated tag-of-tree (the two non-commit shapes round 9 added
tolerance for), confirming the walker bails rather than
returning Ok with a partial withheld set.

* fix(recon): run encrypted recovery even when public list is empty (#218 round 10 P1)

A path-scoped repo whose only reachable object is a direct blob
or direct tree ref yields an empty public list (the anonymous
public classifier removes the only object from the served set)
while withheld_blob_recipients_bounded still assigns that
object to the owner recovery set. The early `continue` on
object_list.is_empty() (and the second one after the
mid-pass refilter) suppressed encrypted recovery too, and a
lost or failed encrypted copy was never repaired.

Track empty-public-work as a flag and gate the public phase
on it. Encrypted phase 2 runs regardless. The backend enable
flags and `_pin_permit` move out of the gated block so phase
2 can read them when the public phase did not run.

* fix(db): rename migration v12 test to v32 (#218 round 10 P3)

The test name and its docstring/comments still say "v12", but
the cid-nullable migration is at v32 (line 1163). The test
itself is correct — it drives the real path — so this is
naming drift rather than dead coverage. A future reader
looking for the migration would otherwise chase a v12 that
no longer exists.

* fix(git): index OID membership before quadratic classifier scan (#218 round 10 P2)

all_object_paths was doing O(O×P) work in two places: the
catch-all cat-file branch checked `blob_set.iter().any(|o|
o == oid)` and `tree_set.iter().any(...)` for every reachable
object, and the phase 1 loop also did a per-commit unused
`git rev-parse <commit>^{tree}` probe whose result was
discarded (the root-tree OID is recomputed by
`root_tree_oids`).

On the 50k-object repos the sweep exists for, the quadratic
scan and the redundant per-commit child process can consume
each authorization deadline before any object reaches a
backend, producing a permanent hourly skip.

Maintain a separate `blob_oids` / `tree_oids` HashSet alongside
the path-pair sets so the membership check is O(1). Drop the
unused per-commit rev-parse probe.

* fix(git): bound tree walk by ls-tree invocation count, not just depth (#218 round 10 P2)

walk_tree_oids_inner stopped at MAX_TREE_WALK_DEPTH and the
shared deadline. A wide shallow tree reachable from N ref tips
or from multiple parents still spawned one ls-tree child
process per subtree well inside the depth cap, and the
wall-clock bound could not stop that — expiry was the only
stop. The ceil was wall-clock only.

Add a structural invocation cap (MAX_TREE_WALK_INVOCATIONS)
the walker fails closed at, and a memo of walked tree OIDs so
the same tree reachable from multiple ref tips is walked once.
Round 10 P2.

* fix(pinata): suppress push when DB record is not durable (#218 round 10 P2)

pin_new_objects pushed (sha, cid) on provider POST success
regardless of whether record_pinata_cid persisted the row.
A closed/failed DB write or an explicit-transaction source
record timeout meant the next sweep queried the DB, found no
row, and re-offered the same gap — with reconciliation having
counted a fill that never happened.

Track a `db_record_durable` flag. On hard failure of either
record_pinata_cid (non-timeout) or record_pin_source
(any failure, since it's a multi-statement transaction), set
the flag false and skip the `pinned.push`. The
`BoundedDbError::Elapsed` autocommit case stays as the
previous comment's "may have committed" reasoning: the push
fires so the reconcile still sees the fill. Round 10 P2.

* fix(node): serialize policy narrow with irreversible upload via in-process mutex (#218 round 10 P1)

The PolicyFence's epoch read was a one-way read: a visibility
narrow (rule insert, quarantine set) could commit between the
fence's epoch read and the upload's HTTP POST, and the
upload's DB-record INSERT's epoch check would see the new
epoch and reject — but the provider already accepted the
object. Encrypted envelopes had no fence at all.

Add a per-repo in-process mutex that PolicyFence acquires
at capture and releases at drop. The narrow paths
(set_repo_quarantine, set_visibility_rule,
remove_visibility_rule) acquire the same lock for the
duration of their DB transaction. A narrow therefore blocks
until the in-flight batch finishes, and an in-flight batch
waits for the narrow to commit before it even reads the
epoch.

Multi-process / multi-node is a known gap: a Postgres
advisory lock per repo would extend the same guarantee
across processes. Deferred to a follow-up.

* fix(pinata): update post-upload stalled record test for round 10 P2 contract

The previous test asserted that a successful upload with a
timed-out post-upload source record still returned the (sha,
cid) pair. That assertion pinned the buggy behavior the
reviewer called out: the reconcile would count a fill that
was not durable, then re-offer the same gap.

The test name and the `record_pin_source` arm of `pin_new_objects`
are unchanged; the assertion now reflects the new contract:
when the post-upload source record times out (its `mark_pin_sources_incomplete`
already sets the source set to incomplete, so the next
pass re-offers), the pair is suppressed so the reconcile
sees the gap as not-filled. Round 10 P2 follow-up.

* fix: clippy allow + rustfmt on round 10 commits

clippy::too_many_arguments on walk_tree_oids_inner (8 args after
the round 10 memo + invocation counter threading). rustfmt
re-ran on the if/else blocks added in the has_public_work
refactor (round 10 P1 #42). No behavior change.

* fix(node): address round-11 review findings on #218 reconciliation sweep

P1: remove the whole-batch policy mutex. PolicyFence no longer holds a
per-repo lock from capture through drop and the narrow paths
(set_visibility_rule, remove_visibility_rule, set_repo_quarantine) no
longer acquire it, so a visibility narrow commits immediately and the
batch aborts on its next is_current check instead of the narrow
blocking behind the sweep. Accepted residual is a single in-flight
object; the fenced DB record still refuses to land a raced row as
durable. encrypt_and_pin_stops_sealing_when_reader_removed_mid_batch
is green again. PolicyMutexes is deleted, resolving the key-leak P2
by removal.

P2: cover the round-10 has_public_work branch with
sweep_seals_withheld_blob_when_public_list_is_empty: a direct-blob-ref
repo yields an empty public list with a non-empty owner recovery set;
asserts no public gaps/fills, no cleartext pin, exactly one POST (the
seal envelope), and a recorded encrypted copy.

P3: peel-case isolation in the non-UTF-8 tree-tip test (delete the
direct-tree ref before the second call so the peel arm must run);
v12->v32 rename drift in the migration test (boundary, comments, and
the stale (2)->true doc line); accurate comments for the
pre-refilter flag and the post-refilter fall-through.

* fix(node): address merge-readiness findings on #218 reconciliation sweep

Reworks the sweep around explicit state-machine contracts instead of
adding conditionals: backend results carry last_attempted plus
confirmed durability separately (PinBatchOutcome), provider permits
scope to provider effects only, discovery pages by commit window with
a persisted cursor, legacy provenance migrates as unknown, and the
pin list advertises only validated raw local keys.

- Empty-path owner recovery without path rules: phase 2 drops the
  has_path_scoped_rule gate and runs on the actual recipient result,
  with a dedicated recovery recheck (no anonymous-listability
  requirement); matrix test over blob/tree/tag refs x zero/root
  rules, including recreate-after-delete.
- Pinata Elapsed is non-durable unless verify_pinata_record proves
  the exact row and fence epoch; loop test with row-locked repos.
- list_pins gates cid/local_cid on is_raw_cidv1 plus confirmed
  provenance; response-shape matrix uses real CIDs with a legacy
  row and GET /ipfs follow-through; docs match the nullable wire
  contract.
- Permits acquired per provider effect and dropped after; no nesting
  at pool size 1 (test renamed to the new shape).
- Scan discovery pages SCAN_COMMIT_WINDOW commits oldest-first with
  a node_state cursor (advance on drained/empty/attempted, delete on
  cover); refilters and phase-2 recipients run on the same window;
  walk-layer paging/invocation/union tests plus a 1050-commit
  cursor-to-completion sweep test.
- v35 migrates legacy rows as unknown (no TRUE backfill); v35 test
  asserts unknown + preserved Pinata history; sweep re-derivation
  test re-pins instead of filtering.
- Drained completions DELETE continuation rows (no tombstones);
  FK cascade on repo delete; row-count regression test.
- Audit: lru waiver gets an explicit retirement owner plus a weekly
  drift guard; revocation model recorded as a product decision on
  PolicyFence (prompt revocation, single-in-flight residual).

* fix(tests): configure git identity in bare clone for tag fixture

The windowed-union test creates its annotated tag in the bare clone,
which carries no config. It passed locally only via the developer's
global git identity; CI has none, so 'git tag -a' failed there.
Verified with an empty GIT_CONFIG_GLOBAL/SYSTEM.

* fix(tests): ignore pre-existing hung-git pid race on CI

read_object_bounded_returns_by_deadline_with_a_hung_git reads the
fake's pid file with no readiness handshake; a slow scheduler yields
'No such file or directory' (stable-only red, identical beta green).
Unrelated to #218 (store.rs untouched); re-enable with a readiness
signal from the fake.

* fix(node): address round-12 review findings on #218 reconciliation sweep

- P1 roots: admit structurally safe root trees at "/" into the allow
  set and candidates, so safe snapshots fully replicate; roots naming
  a withheld subtree stay excluded via the recursive structural check.
- P1 per-pass WalkBudget on sweep enumeration (entries / output bytes /
  git invocations) with fail-closed budget-exhaustion; serve paths keep
  the uncapped budget.
- P2 recovery lane: independent recovery cursor pages unlistable
  history across restarts instead of sharing the scan window.
- P2 gap-hold: failed gap queries hold discovery advance but not the
  healthy backend, via test-only failure seams.
- P2 nested tags: collect full tag chains so nested tag objects pin
  without an inner ref.
- P2 RepoStore: thread the Tigris-backed store into the sweep with an
  acquire timeout and shutdown select; restore cache-miss and skip
  missing.
- P2 Pinata source: keep confirmed pairs when the primary already
  covers the repo; suppress push for stalled post-upload records
  without coverage.
- Update large-history sweep expectation to blob+root+commit per round.

* fix(node): address consolidated review findings on #218 reconciliation sweep

P1: wire WalkBudgetExceeded through the sweep cursor logic. Both the
public scan and the recovery walk now dispatch the typed
budget-exhaustion signal at the sweep boundary: a budget-hit window
advances past content the sweep refuses to classify partially
(scan cursor save/delete via the existing persist site; recovery
cursor likewise), while git/policy/panic/timeout failures keep the
cursor for retry. visibility_pack documents the obligation; the
match arms are the handler, with a one-line pattern note for future
typed walk errors.

P2: run encrypted recovery when the public scan fails. Scan
walk/panic/timeout arms now fall through with an empty public list
instead of continuing the whole repo iteration, so phase 2 runs on
the recovery lane's own cursor like the unlistable-repo path. The
scan cursor stays untouched for retry. Refilter/recheck-public
failures remain fail-closed whole-iteration continues by design.
Includes a repo-scoped test seam (process-global set, safe across
spawn_blocking and parallel tests via fresh UUIDs).

P2: bump policy_epoch only when remove_visibility_rule deletes a
row, mirroring set_repo_quarantine's rows-affected guard, so a
typo'd or double-deleted glob cannot abort fenced batches.

Also adds the repo-loop outcome matrix (every early exit × scan
cursor / recovery cursor / phase 1 / phase 2) and deny tests:
budget-hit advances both lane cursors with zero pins/seals
(real 250k-entry trip, early-trip fixture runs in ~30s), scan
failure still seals the recovery copy, delete-missing-rule leaves
the epoch unchanged. Each deny was verified by temporarily
restoring the old branch.

* fix(node): address jatmn review findings on reconciliation sweep re-home

* fix(node): reload persisted sweep cursor between passes

* fix(node,gl): pinata offset warn wording and boundary-safe pin timestamp

---------

Co-authored-by: Gravirei <[email protected]>
Co-authored-by: Kevin Codex <[email protected]>
2026-11-09 15:35:18 +00:00
64 changed files with 40741 additions and 1312 deletions

No files matched your search

+31
View File
@@ -21,6 +21,25 @@ ignore = [
# the build, at which point it becomes a real reachable advisory.
"RUSTSEC-2023-0071", # rsa Marvin attack (no fix; not linked in our build)
# lru 0.16.3 (use-after-free in pop()). Reachable via alloy. The
# lockfile holds only 0.16.3 — the aws-sdk-s3 path that once pulled
# 0.12.5 no longer does. The RUSTSEC-2026-0253 advisory covers this
# range, so the ignore stays load-bearing until a fixed 0.16.x lands;
# REMOVE when the locked version is fixed or lru leaves the tree.
# Retirement owner: node maintainers, enforced by the weekly
# audit-schedule drift guard ("Fail if lru moved while its ignore
# remains"), which fails if either locked version moves or lru
# leaves the tree. No separate tracking issue exists; file one
# against the lru upgrade if this waiver needs discussion outside
# the schedule job.
"RUSTSEC-2026-0253",
#
# Round-3 P2 (reviewer 2): the previous version of this file
# ignored RUSTSEC-2026-0258 for h2 0.4.13, but the lockfile
# already has h2 0.4.18 (verified 2026-08-31: cargo tree -p h2
# shows 0.4.18) — the fix is in. The ignore was stale and would
# have silently re-accepted the advisory if h2 dropped back to
# 0.4.13. Removed.
# crypto-bigint 0.5.5 (secret-dependent branches in operations meant
# to be constant-time). REACHABLE accepted risk, on two paths:
# - alloy-signer -> elliptic-curve 0.13 -> crypto-bigint 0.5: the k256
@@ -39,4 +58,16 @@ ignore = [
# (verified 2026-11-03). REMOVE only when Cargo.lock has no crypto-bigint
# below 0.7.6: `grep -A1 'name = "crypto-bigint"' Cargo.lock`.
"RUSTSEC-2026-0387", # crypto-bigint timing variability (no compatible fix)
# libsqlite3-sys 0.30.1 (bundled SQLite WAL uninitialized-memory disclosure,
# CVE-2026-119375). Present in Cargo.lock because sqlx resolves sqlx-sqlite
# unconditionally; sqlx is built postgres-only
# (crates/gitlawb-node/Cargo.toml), so sqlx-sqlite and its bundled SQLite C
# source are never compiled or linked (`cargo tree -i sqlx-sqlite` prints
# nothing) and there is no reachable WAL open to attack. The fix requires
# libsqlite3-sys >= 0.39, but sqlx-sqlite 0.8.6 pins ^0.30.1; a compatible
# bump needs a sqlx upgrade, not a lockfile refresh. REMOVE this ignore if
# sqlx is ever built with the `sqlite` feature or any other libsqlite3-sys
# consumer enters the build.
"RUSTSEC-2026-0459", # libsqlite3-sys bundled SQLite CVE (not linked in our build)
]
+50
View File
@@ -326,6 +326,19 @@ GITLAWB_IPFS_RESOLVE_BUDGET_SECS=10
# from the concurrency caps above). 0 disables. Default 600.
GITLAWB_IPFS_RATE_LIMIT=600
# Per-client-IP rate limit for the anonymous task read routes
# (GET /api/v1/tasks, GET /api/v1/tasks/{id}), in requests per hour.
# Both are publicly reachable (optional_signature). GET /api/v1/tasks runs
# collect_visible_tasks and returns a visibility-filtered page;
# GET /api/v1/tasks/{id} runs get_visible_task and returns an opaque 404 when
# the task is hidden or missing. These reads can require task, repository,
# and visibility-rule queries even when no task is returned, so the brake
# bounds the cost of anonymous probes. Keyed on the resolved client IP via
# GITLAWB_TRUSTED_PROXY. 0 disables. Default: 1200 (a list page
# followed by per-task reads is a normal client pattern, so this sits above
# the /ipfs budget). The GraphQL/WS task-read brake shares this budget.
GITLAWB_TASK_READ_RATE_LIMIT=1200
# ── Creation rate limiting (repo/agent/issue/PR flood brake) ──────────────
# Max creation requests (POST /api/v1/repos, /api/register, fork, issues,
# pulls) per client IP per hour, in addition to the per-DID limit. The per-DID
@@ -334,6 +347,14 @@ GITLAWB_IPFS_RATE_LIMIT=600
# brake caps a single-source flood. Uses GITLAWB_TRUSTED_PROXY below to resolve
# the client IP. 0 disables. Default 120.
GITLAWB_CREATE_RATE_LIMIT=120
# Per-DID creation requests allowed inside the window below (repo create,
# issue create, PR create, register, fork). 0 is rejected at startup: it would
# silently disable the limiter. Default 10.
GITLAWB_RATE_LIMIT_MAX=10
# Window the per-DID creation count above applies over, in seconds. 0 is
# rejected at startup: every timestamp would prune immediately, disabling the
# limiter. Default 3600.
GITLAWB_RATE_LIMIT_WINDOW_SECS=3600
# ── Peer-sync rate limiting (per client IP, uses GITLAWB_TRUSTED_PROXY below) ─
# /api/v1/peers/announce and /api/v1/sync/notify accept unsigned requests from
@@ -359,6 +380,35 @@ GITLAWB_TRUSTED_PROXY=
# Enable automatic background sync from known peers
GITLAWB_AUTO_SYNC=false
# ── External upstream mirroring ───────────────────────────────────────────
# Default-off scheduled fetch worker for repositories configured as INBOUND.
# Enabling requires GITLAWB_ENFORCE_OWNER_PUSH=true; the node refuses to start
# otherwise. The worker updates branches and tags atomically while preserving
# Gitlawb-owned review/internal refs. Enabling also requires Git >= 2.37 on
# the host for http.curloptResolve DNS pinning; the node refuses to start
# when the installed Git is older.
GITLAWB_UPSTREAM_MIRROR_ENABLED=false
# Pause after each complete scan. Must be 5..=86400 seconds.
GITLAWB_UPSTREAM_MIRROR_INTERVAL_SECS=300
# Deadline for the external fetch child only; the preflight probes run on
# their own short bound. Must be 1..=3153600000 seconds.
GITLAWB_UPSTREAM_MIRROR_FETCH_TIMEOUT_SECS=600
# Keyset page size; every page is processed before the next scan delay.
GITLAWB_UPSTREAM_MIRROR_PAGE_SIZE=100
# ── Reconciliation sweep ─────────────────────────────────────────────────
# Periodic durability sweep: re-derives the public pin set and the withheld-blob
# recovery set each hour and fills gaps so a dropped replication job never means
# data loss. Defaults to true; set to false to disable the sweep even when a pin
# backend (IPFS/Pinata) is configured.
#
# Phase-capability matrix:
# - Public pin repair: IPFS-only, Pinata-only, or both (requires the
# respective backend to be configured).
# - Encrypted recovery repair: requires local IPFS (GITLAWB_IPFS_API).
# Pinata-only nodes reconcile public pins only; encrypted recovery
# reconciliation is not performed.
GITLAWB_RECONCILIATION_SWEEP=true
# ── iCaptcha proof-of-intelligence gate ───────────────────────────────────
# Optional gate on create_repo + register: require callers to present an
# iCaptcha proof (X-ICaptcha-Proof header) earned at icaptcha.gitlawb.com.
+24
View File
@@ -86,3 +86,27 @@ jobs:
exit 1
fi
echo "No drift: ignore list is consistent with the pinned hickory-proto version."
# Drift guard: the RUSTSEC-2026-0253 (lru) ignore is only valid while
# the lockfile holds exactly the version it was written for (0.16.3
# via alloy). Any move — an upgrade that may carry the upstream fix,
# or removal of the dependency — must revisit the ignore. Retirement
# owner: node maintainers, via this weekly job (no separate tracking
# issue; see the waiver comment in .cargo/audit.toml).
- name: Fail if lru moved while its ignore remains
run: |
set -euo pipefail
versions="$(grep -A1 'name = "lru"' Cargo.lock | grep '^version' | cut -d'"' -f2 | sort -u | tr '\n' ' ')"
echo "lru versions in Cargo.lock: ${versions:-not present}"
lru_ignore_present=false
if grep -q 'RUSTSEC-2026-0253' .cargo/audit.toml; then
lru_ignore_present=true
fi
# Drift = the lockfile no longer matches what the ignore assumes:
# the version moved (possible fix) or lru left the tree
# (dead ignore). Both must fail for explicit re-triage.
if [ "$lru_ignore_present" = true ] && [ "${versions}" != "0.16.3 " ]; then
echo "::error::lru versions are [${versions:-absent from Cargo.lock}] but the RUSTSEC-2026-0253 ignore is still in .cargo/audit.toml. Re-triage the waiver against the advisory's fixed range and either drop the ignore or update the pinned set."
exit 1
fi
echo "No drift: lru ignore is consistent with the pinned versions."
+100
View File
@@ -0,0 +1,100 @@
name: CodeQL
# Advanced-setup replacement for CodeQL default setup. Default setup analyzes
# every detected language on every PR, including Swift (macos-app/), which
# autobuilds on a macOS runner for ~13 minutes per PR and has hung for 55 on
# main; that single job was the longest check on most PRs. Here a PR analyzes
# only the languages it touches (scripts/ci-surfaces.sh), while push to main
# and the weekly schedule analyze every language, so main's code-scanning
# baseline stays complete.
#
# Requires CodeQL default setup to be DISABLED (Settings -> Code security ->
# Code scanning): GitHub rejects advanced-setup uploads while it is enabled.
#
# A PR that skips a language may show "configuration not found" on the CodeQL
# check for that language; that is informational and expected.
on:
pull_request:
branches: [main]
push:
branches: [main]
schedule:
- cron: "17 4 * * 1"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: codeql-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
changes:
name: changed languages
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
codeql: ${{ steps.surfaces.outputs.codeql }}
matrix: ${{ steps.surfaces.outputs.codeql_matrix }}
steps:
- name: Check out repository
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
fetch-depth: 2
- name: Map changed paths to languages
id: surfaces
env:
EVENT_NAME: ${{ github.event_name }}
run: |
set -euo pipefail
if [ "$EVENT_NAME" = "pull_request" ]; then
# -z: literal paths; plain --name-only C-quotes unusual names.
git diff -z --name-only --no-renames HEAD^1 HEAD > changed-paths
tr '\0' '\n' < changed-paths >&2
scripts/ci-surfaces.sh < changed-paths > surfaces.out
else
scripts/ci-surfaces.sh --all > surfaces.out
fi
cat surfaces.out
grep -E '^(codeql|codeql_matrix)=' surfaces.out >> "$GITHUB_OUTPUT"
analyze:
name: Analyze (${{ matrix.language }})
needs: changes
if: needs.changes.outputs.codeql == 'true'
runs-on: ${{ matrix.runner }}
# Normal runs take <=14 min (Swift autobuild is the longest); cap a hung
# autobuild instead of letting it hold a runner for an hour.
timeout-minutes: 30
permissions:
contents: read
security-events: write
actions: read
packages: read
strategy:
fail-fast: false
matrix: ${{ fromJSON(needs.changes.outputs.matrix) }}
steps:
- name: Check out repository
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- name: Initialize CodeQL
uses: github/codeql-action/init@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
with:
languages: ${{ matrix.language }}
build-mode: ${{ matrix.build-mode }}
- name: Autobuild
if: matrix.build-mode == 'autobuild'
uses: github/codeql-action/autobuild@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
with:
category: "/language:${{ matrix.language }}"
+98 -3
View File
@@ -30,8 +30,62 @@ env:
RUST_BACKTRACE: 1
jobs:
# Decide which checks this event needs. On pull_request, each job below runs
# only when the PR touches the surface it validates, and the cargo test /
# release-build jobs are scoped to the touched crates plus their reverse
# dependencies (scripts/ci-surfaces.sh). Every other event (push to main,
# merge_group, the release flow's workflow_dispatch) runs the full suite, so
# main and the merge queue keep whole-workspace coverage and main-red-alert
# still sees a complete run. A skipped job reports as "skipped", not green.
changes:
name: changed surfaces
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
rust: ${{ steps.surfaces.outputs.rust }}
rust_packages: ${{ steps.surfaces.outputs.rust_packages }}
windows: ${{ steps.surfaces.outputs.windows }}
windows_packages: ${{ steps.surfaces.outputs.windows_packages }}
docker: ${{ steps.surfaces.outputs.docker }}
audit: ${{ steps.surfaces.outputs.audit }}
core_deps: ${{ steps.surfaces.outputs.core_deps }}
release_versioning: ${{ steps.surfaces.outputs.release_versioning }}
release_tag: ${{ steps.surfaces.outputs.release_tag }}
steps:
- name: Check out repository
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
# The pull_request checkout is the test-merge commit; depth 2 brings
# its first parent (the base tip) so the diff is exactly this PR.
fetch-depth: 2
# Every job's gate comes from the mapper, so pin its mapping first; a
# failure here fails this job and the run is red, not silently narrowed.
- name: Test the surface mapper
run: scripts/test-ci-surfaces.sh
- name: Map changed paths to surfaces
id: surfaces
env:
EVENT_NAME: ${{ github.event_name }}
run: |
set -euo pipefail
if [ "$EVENT_NAME" = "pull_request" ]; then
# -z: literal paths; plain --name-only C-quotes unusual names.
git diff -z --name-only --no-renames HEAD^1 HEAD > changed-paths
tr '\0' '\n' < changed-paths >&2
scripts/ci-surfaces.sh < changed-paths > surfaces.out
else
scripts/ci-surfaces.sh --all > surfaces.out
fi
cat surfaces.out
cat surfaces.out >> "$GITHUB_OUTPUT"
fmt-clippy:
name: fmt + clippy
needs: changes
if: needs.changes.outputs.rust == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
@@ -57,6 +111,8 @@ jobs:
release-tag-validation:
name: release tag validation
needs: changes
if: needs.changes.outputs.release_tag == 'true'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
@@ -86,6 +142,8 @@ jobs:
test:
name: test (${{ matrix.toolchain }})
needs: changes
if: needs.changes.outputs.rust == 'true'
runs-on: ubuntu-latest
timeout-minutes: 45
strategy:
@@ -130,10 +188,17 @@ jobs:
- name: cargo test
env:
DATABASE_URL: postgres://postgres:postgres@localhost:5432/gitlawb_test
run: cargo test --locked --workspace
# `--workspace`, or `-p <crate>...` for the crates a PR touches plus
# every crate that depends on them.
PACKAGES: ${{ needs.changes.outputs.rust_packages }}
run: |
# shellcheck disable=SC2086 # PACKAGES is a word list by design
cargo test --locked $PACKAGES
build-release:
name: build --release
needs: changes
if: needs.changes.outputs.rust == 'true'
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
@@ -153,10 +218,20 @@ jobs:
key: release
- name: cargo build --release
run: cargo build --locked --release --workspace
env:
PACKAGES: ${{ needs.changes.outputs.rust_packages }}
run: |
# shellcheck disable=SC2086 # PACKAGES is a word list by design
cargo build --locked --release $PACKAGES
audit:
name: cargo audit
# cargo audit reads only Cargo.lock (+ .cargo/audit.toml), so a PR that
# leaves both unchanged cannot change its verdict; a newly published
# advisory against main is caught by the push-to-main run and the weekly
# Scheduled Audit instead of turning every unrelated PR red.
needs: changes
if: needs.changes.outputs.audit == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
@@ -216,6 +291,8 @@ jobs:
msrv:
name: MSRV (Rust 1.91)
needs: changes
if: needs.changes.outputs.rust == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
@@ -239,6 +316,12 @@ jobs:
docker-build:
name: Docker build smoke test
# Runs when the image's own inputs change: the Dockerfile, the manifests
# its dependency layer copies, the lockfile, bootstrap-peers.json. A
# source-only Rust change is already compiled by `build --release` (stable)
# and `MSRV` (1.91, the image's toolchain); push to main still runs this.
needs: changes
if: needs.changes.outputs.docker == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
@@ -255,6 +338,8 @@ jobs:
test-windows:
name: test (windows, non-blocking)
needs: changes
if: needs.changes.outputs.windows == 'true'
runs-on: windows-latest
timeout-minutes: 45
# Non-blocking, mirroring the informational `beta` lane above. We ship `gl`
@@ -282,7 +367,13 @@ jobs:
key: windows
- name: cargo test (shipped Windows crates)
run: cargo test --locked -p gl -p git-remote-gitlawb
shell: bash
env:
# `-p gl -p git-remote-gitlawb`, or whichever of the two a PR affects.
PACKAGES: ${{ needs.changes.outputs.windows_packages }}
run: |
# shellcheck disable=SC2086 # PACKAGES is a word list by design
cargo test --locked $PACKAGES
# gitlawb-core is embedded by every consumer (gl, git-remote-gitlawb, the node
# daemon), so it must stay lean. This gate fails if gitlawb-core's normal
@@ -292,6 +383,8 @@ jobs:
# cannot do. Regen instructions live in the allowlist header.
core-deps-purity:
name: gitlawb-core dependency purity
needs: changes
if: needs.changes.outputs.core_deps == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
@@ -324,6 +417,8 @@ jobs:
# not its author knew this gate existed.
release-versioning:
name: release versioning coverage
needs: changes
if: needs.changes.outputs.release_versioning == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
+1 -1
View File
@@ -40,5 +40,5 @@ When a node denies a request, surface the denial to the user. Never render a den
## Toolchain and CI reality
- MSRV is Rust 1.91 (`rust-version` in the workspace `Cargo.toml`). The MSRV CI job runs `cargo check --workspace --all-targets` on exactly 1.91, so check that an API you rely on is stable there; the full test suite runs on stable.
- The PR workflow runs these jobs on every PR: `cargo fmt --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo test --workspace` (against Postgres), a release build, `cargo audit`, the MSRV check, and a Docker build. The `main` branch ruleset currently requires one approving review but does not require these checks to pass before merge; a failing job should block review approval in practice, but nothing enforces that mechanically today.
- The PR workflow's jobs are `cargo fmt --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo test` (against Postgres), a release build, `cargo audit`, the MSRV check, and a Docker build. On a PR, `scripts/ci-surfaces.sh` runs each job only when the PR touches its inputs, and scopes `cargo test` and the release build to the touched crates plus every crate that depends on them. Root `Cargo.toml`/`Cargo.lock`/`.cargo/` changes run the Rust jobs (fmt, clippy, tests, release build, MSRV) on the whole workspace; every other job still runs only when its own inputs change. Edits to the CI wiring itself (`pr-checks.yml`, `codeql.yml`, `scripts/ci-surfaces.sh`), push to `main`, and the merge queue run every job on the whole workspace. Edits to any other workflow run none of these jobs on a PR; CodeQL only analyzes them as Actions code. Run `cargo test --workspace` locally regardless. The same script limits CodeQL (`codeql.yml`) to the languages a PR touches. The `main` branch ruleset currently requires one approving review but does not require these checks to pass before merge; a failing job should block review approval in practice, but nothing enforces that mechanically today.
- Conventional commit titles (`feat:`, `fix:`, `docs:`, ...) are required by convention and drive releases, but no CI job checks them; a bad title fails review, not the pipeline.
Generated
+5 -3
View File
@@ -3085,7 +3085,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
dependencies = [
"libc",
"windows-sys 0.52.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -3565,6 +3565,7 @@ dependencies = [
"multiaddr",
"percent-encoding",
"prometheus",
"rand 0.8.6",
"reqwest",
"serde",
"serde_json",
@@ -3600,6 +3601,7 @@ dependencies = [
"ed25519-dalek 2.2.0",
"gitlawb-core",
"icaptcha-client",
"libc",
"mockito",
"percent-encoding",
"reqwest",
@@ -6227,7 +6229,7 @@ dependencies = [
"errno",
"libc",
"linux-raw-sys",
"windows-sys 0.52.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -7161,7 +7163,7 @@ dependencies = [
"getrandom 0.4.2",
"once_cell",
"rustix",
"windows-sys 0.52.0",
"windows-sys 0.61.2",
]
[[package]]
+3 -1
View File
@@ -70,7 +70,7 @@ Known limitations:
- Repository write authorization is secure by default: `GITLAWB_ENFORCE_OWNER_PUSH` defaults to `true`, so a valid HTTP Signature identifies a pusher and the node additionally requires that pusher to be the repository owner. Set `GITLAWB_ENFORCE_OWNER_PUSH=false` only for the duration of a rolling upgrade whose pushers are not yet the repo owner. Authentication is not authorization on its own (anyone can mint a `did:key` and sign), so owner enforcement is the only thing that prevents every signed caller from pushing to every repository, private ones included.
- UCAN proof chains are validated when supplied, but UCAN capabilities are not consulted by write authorization and the root issuer is not independently trust-anchored. UCANs therefore do not yet grant scoped collaborator access.
- Agent lifecycle revocation is not enforced by HTTP Signature authorization; do not rely on removing or revoking an agent record to block a compromised signer.
- Read visibility is not a blanket data-classification boundary: task, IPFS-pin, and Arweave-anchor listings are not repository-gated; withheld path names can be visible to a root reader; and later visibility changes cannot retract content already announced or externally anchored.
- Task list/get reads (REST and GraphQL) are repository/task-gated and omit `ucan_token`. Slash-form `repo_id` rows (mirror-hosted tasks) can appear on list for the delegator or trimmed assignee, but `GET /api/v1/tasks/{id}` and GraphQL `task` fail closed for every caller because there is no local repository backing to serve. Read visibility is not a blanket data-classification boundary: IPFS-pin and Arweave-anchor listings are not repository-gated; withheld path names can be visible to a root reader; and later visibility changes cannot retract content already announced or externally anchored.
- Peer writes are signed by upgraded nodes, but strict signed-peer enforcement is opt-in during rolling upgrades.
- GraphQL mutations require a signed identity (a verified signer bound to the acting DID), are limited to the agent-task queue, and are covered by a CI completeness fence that requires signer acquisition and actor binding on every registered mutation; repo writes are not exposed over GraphQL and remain REST-only.
- Pull-request review comments do not yet have threaded line-level anchors, and merges do not enforce approval requirements.
@@ -404,6 +404,7 @@ Important node settings:
| `GITLAWB_REQUIRE_SIGNED_PEER_WRITES` | Require peer writes to prove their DID on every transport: RFC 9421 signatures on the announce/sync routes, and an Ed25519 payload signature on gossip ref-update events. Defaults to `false` during the staged rollout below. Upgrade every gossip publisher before enabling, or their updates are dropped with no error on their side. |
| `GITLAWB_ENFORCE_OWNER_PUSH` | Require the authenticated pusher to be the repo owner on `git-receive-pack`. **Defaults to `true`.** A `did:key` signature is authentication, not authorization (anyone can mint a key and sign), so with this off every signed caller may push to every repository, private ones included. Delegated and CI keys count as non-owners: a UCAN `git/push` capability is verified but not yet honored for authorization, so they cannot push while this is on. Set `false` only for a rolling upgrade; see [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md). |
| `GITLAWB_AUTO_SYNC` | Enable automatic sync from known peers. |
| `GITLAWB_RECONCILIATION_SWEEP` | Enable the hourly durability sweep that re-pins/backstops missing objects (default `true`; disabled when no IPFS/Pinata backend is configured). Public pin repair runs against any configured backend (IPFS, Pinata, or both). Encrypted recovery repair requires local IPFS (`GITLAWB_IPFS_API`); Pinata-only nodes reconcile public pins only. |
| `GITLAWB_MAX_PACK_BYTES` | Max git pack body size for smart-HTTP routes. |
| `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack, receive-pack, or `info/refs` advertisement may run before it is aborted (504). Default 600. Also bounds the withheld-blob classification walk (on both the upload-pack serve and receive-pack replication paths) and the push-side pin-candidate discovery (`rev-list` / `cat-file`), each reaped via process-group teardown at the deadline. On the path-scoped upload-pack path the classification walk and the pack serve share ONE deadline, so this value bounds their combined duration rather than granting each stage a full budget: a walk that consumes it leaves the serve nothing and the clone gets a 504. Serving large path-scoped repos may therefore need a higher value than they did when each stage was budgeted separately. Accepted range is 1 to 3153600000 (100 years), since the node derives deadlines from this value and a larger one cannot be represented. |
| `GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS` | Max seconds the storage-acquisition phase (Tigris HEAD/GET, push advisory-lock) of a served git op may run before the request is shed with a 503, separate from the git-run timeout. The concurrency permit is released on expiry so a stalled backend cannot pin the pool. Default 30. |
@@ -421,6 +422,7 @@ Important node settings:
| `GITLAWB_IPFS_REQUEST_BUDGET_SECS` | Absolute wall-clock budget for one admitted `/ipfs/{cid}` request's acquire+walk lifetime. Per-stage clamps bound the acquire and walk stages to the remaining budget, and no stage starts once it is exhausted; the scan then stops with a retryable 503. The object-type probe and content-read `cat-file` subprocesses are budget-checked before starting and each also run under their own deadline (the lesser of `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` and the remaining budget), reaped via process-group teardown, so a hung `cat-file` cannot hold the request's walk slot past it. One hang path is still unbounded: the probe's object-store readability check is a plain filesystem sweep with nothing to reap, so a wedged filesystem can hold the slot past the deadline. Default 600. Accepted range is 1 to 3153600000 (100 years), since the node derives a deadline from this value and a larger one cannot be represented. |
| `GITLAWB_IPFS_RESOLVE_BUDGET_SECS` | Shorter budget for the pre-walk CID resolve inside an admitted `/ipfs/{cid}` request: the lookup that maps the requested CID to its git oid(s), which runs while the scarce walk admission is already held. A well-formed CID with no pin row does no probe and no walk work, so without this it could hold a walk slot for the whole request budget while nothing walked, and enough such requests shed every real retrieval at admission. The effective deadline is the lesser of this and the remaining request budget, so a value above `GITLAWB_IPFS_REQUEST_BUDGET_SECS` degrades to the request budget. Only the resolve is on this clock; walk and probe work stay on the request budget, so a slow but progressing scan is never shed by it. Default 10. Accepted range is 1 to 3153600000 (100 years). |
| `GITLAWB_IPFS_RATE_LIMIT` | Max `/ipfs/{cid}` requests per client IP per hour (route flood brake). 0 disables. Default 600. |
| `GITLAWB_TASK_READ_RATE_LIMIT` | Max anonymous task-read requests (`GET /api/v1/tasks`, `GET /api/v1/tasks/{id}`) per client IP per hour, sharing the budget with the GraphQL/WS task-read brake. Uses `GITLAWB_TRUSTED_PROXY` to resolve the client IP. 0 disables. Default 1200 (above the `/ipfs` budget to allow list pages followed by per-task reads). |
| `GITLAWB_TIGRIS_BUCKET` | Optional S3/Tigris shared repo storage bucket. |
| `GITLAWB_PINATA_JWT` | Optional Pinata/IPFS warm-storage pinning. |
| `GITLAWB_IRYS_URL` | Optional Irys/Arweave permanent anchoring. |
+9 -9
View File
@@ -26,7 +26,7 @@ We will acknowledge receipt within 48 hours and aim to release a fix within 14 d
**Content addressing**
- Every git object is content-addressed via CIDv1 (SHA-256)
- Tamper-evident by construction — a modified object changes its CID
- Tamper-evident by construction: a modified object changes its CID
**UCAN token validation**
- Bootstrap UCAN tokens are issued at registration.
@@ -35,8 +35,8 @@ We will acknowledge receipt within 48 hours and aim to release a fix within 14 d
- Capability grants are not yet consulted by repository write authorization; see the limitations below.
**Smart contracts (Base Sepolia testnet)**
- `GitlawbDIDRegistry` — on-chain DID → document registry
- `GitlawbNameRegistry` — human name → DID registry
- `GitlawbDIDRegistry`: on-chain DID → document registry
- `GitlawbNameRegistry`: human name → DID registry
- Both auditable on-chain, no admin keys
---
@@ -55,7 +55,7 @@ These are documented limitations of the current live release. They should be pri
### Repository write authorization defaults
- `git-receive-pack` verifies HTTP Signatures and, with the default configuration, also requires the pusher DID to be the repository owner: `GITLAWB_ENFORCE_OWNER_PUSH` defaults to `true`.
- **Impact:** With the default setting, a valid signature authenticates the pusher *and* the pusher must be the repository owner. Authentication is not authorization on its own — anyone can mint a `did:key` and sign — so owner enforcement is the only thing that prevents every signed caller from pushing to every repository, including private ones.
- **Impact:** With the default setting, a valid signature authenticates the pusher, and the pusher must be the repository owner. Authentication is not authorization on its own, since anyone can mint a `did:key` and sign, so owner enforcement is the only thing that prevents every signed caller from pushing to every repository, including private ones.
- **Mitigation:** `GITLAWB_ENFORCE_OWNER_PUSH=false` is the explicit rolling-upgrade compatibility override. Disable it only for the duration of an upgrade whose pushers are not yet the repo owner, then turn it back on. Delegated and CI keys count as non-owners: a UCAN `git/push` capability is verified but not yet honored for authorization, so they cannot push while this is on.
### UCAN delegation and revocation
@@ -73,7 +73,7 @@ These are documented limitations of the current live release. They should be pri
### Private repository reads
- Repository and path-scoped visibility checks are enforced for repository API and Git content reads. A denied whole-repository or root read returns the same 404 shape as a missing repository, so the denial does not reveal private-repository existence.
- Sparse-clone support exposes withheld path globs to callers who may read the repository root. Do not put sensitive information in withheld path names.
- `GET /api/v1/tasks`, `/api/v1/ipfs/pins`, and `/api/v1/arweave/anchors` are not repository-gated. Task records include a UCAN token; pin and anchor listings expose object and ref metadata.
- Task list/get reads (`GET /api/v1/tasks`, `GET /api/v1/tasks/{id}`, and GraphQL `tasks` / `task` over HTTP and WebSocket) are repository/task-gated through shared visibility collectors and omit `ucan_token`. Slash-form `repo_id` rows (mirror-hosted tasks) can appear on list for the delegator or trimmed assignee, but `GET /api/v1/tasks/{id}` and GraphQL `task` fail closed for every caller because there is no local repository backing to serve. All of those transports share the same per-IP `GITLAWB_TASK_READ_RATE_LIMIT` budget. `/api/v1/ipfs/pins` and `/api/v1/arweave/anchors` are not repository-gated; pin and anchor listings expose object and ref metadata.
- Changing a repository's visibility controls future serving, but cannot retract ref metadata or configured external pins and anchors already announced while the repository was public. Do not push secrets to an announceable repository.
- **Impact:** Visibility policies protect the repository and Git content routes they gate, not every metadata endpoint or previously published content.
- **Remaining boundary:** This read control does not address the independent write-authorization and UCAN-delegation limitations described above.
@@ -119,10 +119,10 @@ These are documented limitations of the current live release. They should be pri
## Threat Model
Twigpine is designed to be secure against:
- **Unauthenticated writes** — HTTP Signature auth on protected write endpoints; see Known Limitations for owner and capability authorization gaps
- **Tampered git objects** — CIDv1 content addressing detects modification
- **Identity spoofing** — DIDs derived from public keys, unforgeable without the private key
- **Centralized takedown** — no single point of control; data on IPFS + Arweave
- **Unauthenticated writes:** HTTP Signature auth on protected write endpoints; see Known Limitations for owner and capability authorization gaps
- **Tampered git objects:** CIDv1 content addressing detects modification
- **Identity spoofing:** DIDs derived from public keys, unforgeable without the private key
- **Centralized takedown:** no single point of control; data on IPFS + Arweave
Twigpine is **not yet** designed to defend against:
- A compromised node operator (node operators are trusted for their own node)
+9 -1
View File
@@ -1,7 +1,7 @@
{
"$comment": "Canonical seed list for the Twigpine network. Merged with GITLAWB_BOOTSTRAP_PEERS at startup. PRs to add public nodes welcome.",
"version": 1,
"updated": "2026-06-30",
"updated": "2026-10-02",
"peers": [
{
"name": "gitlawb",
@@ -42,6 +42,14 @@
"http_url": "https://rapybus-gitlawb.com",
"p2p_multiaddr": null,
"added": "2026-05-16"
},
{
"name": "moonwake-ledger",
"operator": "Moonwake Ledger",
"did": "did:key:z6Mkh9AZ46Z3RkGZfFCom5vzX2535f54d6G2o2ZGkCqjwW8K",
"http_url": "https://node.moonwakeledger.com",
"p2p_multiaddr": null,
"added": "2026-10-02"
}
]
}
+37
View File
@@ -539,6 +539,43 @@ mod tests {
assert!(matches!(err, Error::Signature(_)));
}
/// The identity-point forgery must be rejected: the shared attestation
/// verifier is a cert-bound provenance gate, so accepting the weak-key
/// signature would let anyone mint a forged attestation that verifies.
/// Strict verification rejects small-order public keys and R (the identity
/// point here), which ordinary verification does not.
#[test]
fn verify_rejects_identity_point_forgery() {
let cert_hash = sample_cert_hash();
let mut att = dummy_attestation(&fresh(), cert_hash);
// Public key A = identity point (0,1); signature R = identity, S = 0.
// The equation `[S]B = R + [k]A` then holds for any k and any message.
let identity = [
1u8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0,
];
let mut buf = Vec::with_capacity(34);
buf.extend_from_slice(&ED25519_MULTICODEC);
buf.extend_from_slice(&identity);
att.signer = format!(
"did:key:{}",
multibase::encode(multibase::Base::Base58Btc, &buf)
);
let mut sig = [0u8; 64];
sig[..32].copy_from_slice(&identity);
att.sig = B64U.encode(sig);
let err = att.verify_signature(cert_hash).unwrap_err();
match err {
Error::Did(msg) => assert!(
msg.contains("small-order"),
"identity-point signer must be refused as a small-order key, got: {msg}"
),
other => panic!("expected Error::Did for an identity-point signer, got {other:?}"),
}
}
/// A payload that happens to contain a `cert_hash` field of its own does
/// not interfere with the outer binding: the attestation envelope's
/// `cert_hash` is the only field consulted by `verify_signature`, and the
+228 -11
View File
@@ -41,6 +41,30 @@ use crate::{Error, Result};
/// The certificate type discriminant. Always `"gitlawb/ref-update/v1"`.
pub const CERT_TYPE: &str = "gitlawb/ref-update/v1";
/// Exact url-safe unpadded base64 length of a 64-byte Ed25519 signature.
/// The tolerant threshold scan requires this canonical unpadded length before
/// attempting decode: the length gate is a DoS bound that keeps an
/// attacker-sized sig string from costing allocation before the signature is
/// rejected. A longer string is not meaningless (the same 64 bytes can appear
/// in padded or non-url-safe encodings) but no in-tree signer produces one,
/// so the scan skips non-canonical entries rather than decoding them.
const ENCODED_SIG_LEN: usize = 86;
/// Hard ceiling on signature entries a certificate may carry into the
/// tolerant threshold scan. A legitimate certificate holds one entry per
/// maintainer (a re-sign after key rotation may briefly leave a second), so
/// this is far above any real signer set, while turning the scan's worst case
/// from "one curve verification per attacker-appended entry" into a small
/// constant. Rejection is loud (`Err`), not a silent prefix scan: scanning
/// only the first N entries would let junk placed ahead of the real
/// signatures starve them out of the window, which is the #349 denial again
/// in positional form. An attacker positioned to inflate the list past this
/// ceiling can already suppress the certificate outright. The loud
/// rejection does deny a legitimately over-long certificate; that is an
/// explicit, detectable denial above the cap rather than a silent window
/// junk placement could exploit.
const MAX_SIGNATURE_ENTRIES: usize = 64;
/// A signature on a ref-update certificate.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CertSignature {
@@ -138,9 +162,15 @@ impl RefUpdateCert {
Ok(())
}
/// Verify all signatures on this certificate.
/// Verify all signatures on this certificate, failing closed.
///
/// Returns the list of DIDs whose signatures are valid.
/// On success returns the list of DIDs whose signatures are valid. This is
/// strict: it returns an error if *any* signature entry is malformed or
/// invalid. Because signature entries live outside the signed body (anyone
/// can append one without key material), do not use this to make a
/// threshold decision on an untrusted certificate — a single appended junk
/// entry would reject the whole cert. Use [`Self::satisfies_threshold`],
/// which ignores invalid entries and counts only the valid signers.
pub fn verify_all(&self) -> Result<Vec<Did>> {
use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine};
let signing_bytes = self.body.to_signing_bytes()?;
@@ -169,21 +199,65 @@ impl RefUpdateCert {
/// from the provided set of authorized maintainer DIDs.
///
/// Counts distinct signer DIDs, not signature entries: a repeated
/// signature from the same maintainer counts once.
/// signature from the same maintainer counts once. Malformed and invalid
/// signature entries are skipped, not fatal — entries live outside the
/// signed body, so any third party can append junk without key material,
/// and short-circuiting on the first bad one (as [`Self::verify_all`]
/// does) would let that junk deny an otherwise valid certificate (#349).
///
/// `threshold` must be at least 1. Zero is rejected instead of being
/// treated as "no signatures required": a caller that derives the
/// threshold from configuration and gets 0 would otherwise accept any
/// certificate, whether its signatures verify or not.
/// Tolerance must not become an unbounded verification path (#349 review
/// round 2): entries are attacker-extensible, so the work done for them is
/// bounded BEFORE the loop, not discovered inside it:
///
/// - a list over [`MAX_SIGNATURE_ENTRIES`] is rejected outright (see the
/// constant for why loud rejection beats a silent prefix scan);
/// - an encoded signature is length-gated before base64 work — a 64-byte
/// Ed25519 signature is exactly [`ENCODED_SIG_LEN`] characters in
/// url-safe unpadded base64, so an oversized blob never allocates;
/// - entries whose signer is not in `maintainers`, or is already counted,
/// are skipped on a string compare, before any decode or curve work —
/// so even within the cap, junk naming strangers costs no crypto;
/// - the scan returns as soon as the threshold is met.
pub fn satisfies_threshold(&self, maintainers: &[Did], threshold: usize) -> Result<bool> {
use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine};
if threshold == 0 {
return Err(Error::RefCert("threshold must be at least 1".to_string()));
}
if self.signatures.len() > MAX_SIGNATURE_ENTRIES {
return Err(Error::RefCert(format!(
"certificate carries {} signature entries (max {})",
self.signatures.len(),
MAX_SIGNATURE_ENTRIES
)));
}
let signing_bytes = self.body.to_signing_bytes()?;
let mut counted: HashSet<&Did> = HashSet::new();
let valid = self.verify_all()?;
let distinct_signers: HashSet<&Did> =
valid.iter().filter(|d| maintainers.contains(d)).collect();
Ok(distinct_signers.len() >= threshold)
for cert_sig in &self.signatures {
if counted.contains(&cert_sig.signer) || !maintainers.contains(&cert_sig.signer) {
continue;
}
if cert_sig.sig.len() != ENCODED_SIG_LEN {
continue;
}
let Ok(vk) = cert_sig.signer.to_verifying_key() else {
continue;
};
let Ok(sig_bytes_vec) = URL_SAFE_NO_PAD.decode(&cert_sig.sig) else {
continue;
};
let Ok(sig_bytes) = <[u8; 64]>::try_from(sig_bytes_vec) else {
continue;
};
if verify(&vk, &signing_bytes, &sig_bytes).is_ok() {
counted.insert(&cert_sig.signer);
if counted.len() >= threshold {
return Ok(true);
}
}
}
Ok(counted.len() >= threshold)
}
/// Validate the certificate structure (not signatures).
@@ -416,6 +490,149 @@ mod tests {
assert!(!cert.satisfies_threshold(&maintainers, 2).unwrap());
}
/// An attacker can append junk signature entries to a cert (they live
/// outside the signed body, so no key material is needed). `satisfies_threshold`
/// must ignore them and still count the real signers — before the fix,
/// `verify_all`'s `?` on the junk made the whole check error out, denying a
/// valid 2-of-2 cert.
#[test]
fn satisfies_threshold_ignores_appended_junk_signatures() {
let kp1 = Keypair::generate();
let kp2 = Keypair::generate();
let mut cert = RefUpdateCert::new(
kp1.did(),
"refs/heads/main".to_string(),
dummy_hash('0'),
dummy_hash('a'),
1,
&kp1,
)
.unwrap();
cert.countersign(&kp2).unwrap();
// Junk sits AHEAD of the real entries so the scan must actually skip
// it: appended last, the threshold-2 early return at the second real
// signature would fire before any junk is examined and the test would
// stay green even if the skip logic never ran.
cert.signatures.insert(
0,
CertSignature {
signer: kp1.did(),
sig: "!!!not-base64!!!".to_string(),
},
);
// Valid base64 but only 3 bytes — fails the 64-byte length check.
cert.signatures.insert(
0,
CertSignature {
signer: kp2.did(),
sig: "AAAA".to_string(),
},
);
// Exactly ENCODED_SIG_LEN of valid base64 decoding to garbage —
// exercises the decode+verify path before the skip, not just the
// length gate.
cert.signatures.insert(
0,
CertSignature {
signer: kp1.did(),
sig: "A".repeat(ENCODED_SIG_LEN),
},
);
let maintainers = vec![kp1.did(), kp2.did()];
assert!(cert.satisfies_threshold(&maintainers, 2).unwrap());
}
/// A junk entry that names a real maintainer DID but carries a signature
/// that does not verify must not be counted — otherwise appending garbage
/// under a maintainer's DID could forge a threshold.
#[test]
fn satisfies_threshold_does_not_count_a_forged_maintainer_signature() {
let kp1 = Keypair::generate();
let kp2 = Keypair::generate();
let mut cert = RefUpdateCert::new(
kp1.did(),
"refs/heads/main".to_string(),
dummy_hash('0'),
dummy_hash('a'),
1,
&kp1,
)
.unwrap();
// Well-formed 64-byte signature from kp2, but over unrelated bytes, so
// it fails verification against the cert body.
cert.signatures.push(CertSignature {
signer: kp2.did(),
sig: kp2.sign_b64(b"unrelated bytes"),
});
let maintainers = vec![kp1.did(), kp2.did()];
// Only kp1 actually signed the body: 2-of-2 must fail, 1-of-2 holds.
assert!(!cert.satisfies_threshold(&maintainers, 2).unwrap());
assert!(cert.satisfies_threshold(&maintainers, 1).unwrap());
}
/// The tolerant scan must not be an unbounded verification path: a list
/// past MAX_SIGNATURE_ENTRIES is rejected loudly before any curve work,
/// while a list at the cap still verifies normally.
#[test]
fn satisfies_threshold_rejects_an_oversized_signature_list() {
let kp = Keypair::generate();
let mut cert = RefUpdateCert::new(
kp.did(),
"refs/heads/main".to_string(),
dummy_hash('0'),
dummy_hash('a'),
1,
&kp,
)
.unwrap();
let junk = || CertSignature {
signer: Keypair::generate().did(),
sig: "AAAA".to_string(),
};
while cert.signatures.len() < MAX_SIGNATURE_ENTRIES {
cert.signatures.push(junk());
}
let maintainers = vec![kp.did()];
// At the cap: still fine, the real signature counts.
assert!(cert.satisfies_threshold(&maintainers, 1).unwrap());
// One past the cap: loud rejection, no tolerant scan.
cert.signatures.push(junk());
let err = cert.satisfies_threshold(&maintainers, 1).unwrap_err();
assert!(err.to_string().contains("signature entries"));
}
/// An entry whose encoded signature is not the exact base64 length of a
/// 64-byte Ed25519 signature is skipped before any base64 decode — an
/// attacker-sized sig string must cost neither allocation nor curve work,
/// and must not affect the verdict.
#[test]
fn satisfies_threshold_skips_an_oversized_encoded_signature() {
let kp1 = Keypair::generate();
let kp2 = Keypair::generate();
let mut cert = RefUpdateCert::new(
kp1.did(),
"refs/heads/main".to_string(),
dummy_hash('0'),
dummy_hash('a'),
1,
&kp1,
)
.unwrap();
// Names a real maintainer, valid base64 alphabet, but megabytes long:
// the length gate must refuse it without decoding.
cert.signatures.push(CertSignature {
signer: kp2.did(),
sig: "A".repeat(2 * 1024 * 1024),
});
let maintainers = vec![kp1.did(), kp2.did()];
assert!(cert.satisfies_threshold(&maintainers, 1).unwrap());
assert!(!cert.satisfies_threshold(&maintainers, 2).unwrap());
}
#[test]
fn threshold_zero_is_rejected() {
let kp = Keypair::generate();
+40
View File
@@ -77,6 +77,14 @@ impl Keypair {
}
/// Verify an Ed25519 signature.
///
/// Strict verification: rejects small-order `R` and small-order public keys
/// (the identity point, and any point of low order). Ordinary `verify` accepts
/// a signature forged with the identity point as the public key plus
/// `R = identity, S = 0`, which verifies for *any* message. `identity::verify`
/// is the shared primitive behind HTTP request authentication, UCANs, and
/// certificates, so weak-key acceptance is an authentication bypass, not a
/// malleability nuance.
pub fn verify(verifying_key: &VerifyingKey, msg: &[u8], sig_bytes: &[u8; 64]) -> Result<()> {
let sig = Signature::from_bytes(sig_bytes);
verifying_key
@@ -208,6 +216,38 @@ mod tests {
);
}
/// The identity-point forgery: with public key A = identity, R = identity,
/// and S = 0, the equation `[S]B = R + [k]A` holds for every message,
/// because `[k]·identity = identity`. Ordinary (non-strict) Ed25519
/// verification accepts it, so the shared `verify` primitive must use
/// strict verification, which rejects small-order R and public keys.
#[test]
fn verify_rejects_identity_point_forgery() {
use ed25519_dalek::Verifier;
// The identity point (0,1) compresses to y = 1 with sign bit 0.
let identity = [
1u8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0,
];
let vk = VerifyingKey::from_bytes(&identity).expect("identity point is on the curve");
let mut sig_bytes = [0u8; 64];
sig_bytes[..32].copy_from_slice(&identity);
let msg = b"arbitrary message the key owner never signed";
// Prove the forged signature satisfies the ordinary verification
// equation, so the strict check below is what actually defends the
// boundary (not a signature that was already invalid everywhere).
assert!(
vk.verify(msg, &Signature::from_bytes(&sig_bytes)).is_ok(),
"identity-point forgery must satisfy ordinary verification (this is why strict is needed)"
);
assert!(
verify(&vk, msg, &sig_bytes).is_err(),
"strict verification must reject the identity-point forgery"
);
}
#[test]
fn verify_rejects_weak_key_signature() {
// Regression guard for strict verification: a signature forged under a
+1
View File
@@ -75,6 +75,7 @@ alloy = { version = "1", default-features = false, features = [
"network",
"rpc-types-eth",
] }
rand = { workspace = true }
libp2p-dns = { version = "0.45.0", features = ["tokio"] }
[dev-dependencies]
+421 -3
View File
@@ -2130,9 +2130,32 @@ async fn gate_and_serve(
/// GET /api/v1/ipfs/pins
///
/// Returns all CIDs that have been pinned to the local IPFS node from git
/// objects received via push. Each entry includes the git SHA-256 hex, the
/// CIDv1 string, and the timestamp when it was pinned.
/// Returns all CIDs that have been pinned from git objects received via push.
/// Each entry includes the git SHA-256 hex and the timestamp when it was
/// pinned. The wire contract is nullable by design:
///
/// - `cid` / `local_cid` — node-local resolver keys: the raw CIDv1 the
/// `GET /ipfs/{cid}` endpoint serves. Non-null ONLY when the stored value
/// parses as a raw CIDv1 AND writer-owned local provenance confirms the
/// bytes went through this node's IPFS daemon. Otherwise null, even when
/// the database holds some other CID-shaped string for the row.
/// - `pinata_cid` — the provider identifier Pinata returned (dag-pb/UnixFS,
/// not a node resolver key). The only usable identifier for remote-only
/// rows; `gl` consumers must read this field, never `cid`, for those.
/// - `local_pinned` — `true` iff the local-IPFS writer path
/// (`record_pinned_cid_with_source`) pushed the bytes into the local
/// daemon. Writer-owned; never inferred from CID shape. A row can carry
/// `local_pinned = true` with a null `cid` when its stored key predates
/// the raw-key contract (legacy provider key awaiting repair).
/// - `pinata_pinned` — `true` iff the row has a non-null `pinata_cid`.
///
/// Rows with neither a local nor a Pinata CID are omitted so the response
/// only contains rows with at least one backend. Remote-only rows stay
/// visible through `pinata_cid`/`pinata_pinned`; nothing is hidden.
///
/// #218 review P2: the response surfaces writer-owned provenance so a
/// `gl` consumer can distinguish local-only, remote-only, and dual rows
/// without re-inferring semantics from nullability.
pub async fn list_pins(
State(state): State<AppState>,
auth: Option<Extension<crate::auth::AuthenticatedDid>>,
@@ -2146,6 +2169,38 @@ pub async fn list_pins(
// map to 503 `db_unavailable` (not 500 via `.map_err(AppError::Internal)`) (#251).
let pins = state.db.list_pinned_cids().await?;
let pins: Vec<serde_json::Value> = pins
.into_iter()
.filter(|p| p.cid.is_some() || p.pinata_cid.is_some())
.map(|p| {
// A stored `cid` is advertised as a node-local resolver key
// only when it passes the raw-CIDv1 contract AND local
// provenance is confirmed. The historical `cid` column has
// held more than one namespace (Kubo dag-pb, Pinata CIDv0),
// and the `/ipfs/{cid}` resolver recomputes the raw CID
// from object bytes and rejects anything else — so a
// legacy provider key in `cid`/`local_cid` would promise a
// local object the node immediately refuses to serve.
// Such rows stay listed (with their provenance flags) but
// carry no local key until repair rewrites them.
let local_key = match &p.cid {
Some(c) if gitlawb_core::cid::is_raw_cidv1(c) && p.local_ipfs_provenance => {
Some(c.clone())
}
_ => None,
};
serde_json::json!({
"sha256_hex": p.sha256_hex,
"cid": local_key.clone(),
"local_cid": local_key,
"pinata_cid": p.pinata_cid,
"local_pinned": p.local_ipfs_provenance,
"pinata_pinned": p.pinata_cid.is_some(),
"pinned_at": p.pinned_at,
})
})
.collect();
Ok(Json(serde_json::json!({
"pins": pins,
"count": pins.len(),
@@ -2459,6 +2514,369 @@ mod closed_pool_tests {
);
}
/// #218 review P2: the `list_pins` API response surfaces writer-owned
/// provenance (`local_pinned`, `pinata_pinned`) so a `gl` consumer
/// can distinguish local-only, remote-only, and dual rows without
/// re-inferring semantics from nullability. `cid`/`local_cid` carry a
/// value ONLY for rows whose stored key parses as a raw CIDv1 with
/// confirmed local provenance; anything else (Pinata-only rows,
/// legacy provider keys) exposes its usable identifier through
/// `pinata_cid` alone, and every non-null local key must resolve
/// through `GET /ipfs/{cid}`.
#[sqlx::test]
async fn list_pins_reports_writer_owned_provenance_for_all_shapes(pool: sqlx::PgPool) {
use sqlx::Row as _;
let state = crate::test_support::test_state(pool.clone()).await;
let db = &state.db;
// Raw resolver keys must be REAL raw CIDv1 strings: the handler
// gates `cid`/`local_cid` on `is_raw_cidv1`, so placeholder text
// would fail the gate for the wrong reason. Provider CIDs stay
// opaque Qm… strings (never syntax-checked, only echoed).
fn raw_cid_for(content: &[u8]) -> String {
gitlawb_core::cid::Cid::from_git_object_bytes(content).to_string()
}
// Two real blobs on disk in one public repo, so the local and
// dual rows below can be followed through `GET /ipfs/{cid}`.
// The resolver locates repos through `repo_store.acquire`, not
// the row's `disk_path`, so the recipe overwrites the acquired
// bare path with a real clone: workdir commit of both files,
// cloned --bare into the store path, oids via rev-parse, raw
// keys recomputed from the file bytes exactly as the serve
// path does.
fn run_git(args: &[&str], cwd: &std::path::Path) {
let out = std::process::Command::new("git")
.args(args)
.current_dir(cwd)
.output()
.expect("git runs");
assert!(
out.status.success(),
"git {args:?} failed: {}",
String::from_utf8_lossy(&out.stderr)
);
}
let content1 = b"pins follow-through one\n";
let content2 = b"pins follow-through two\n";
let tmp = tempfile::TempDir::new().unwrap();
let work = tmp.path().join("work-pins-list");
std::fs::create_dir_all(&work).unwrap();
std::fs::write(work.join("one.txt"), content1).unwrap();
std::fs::write(work.join("two.txt"), content2).unwrap();
run_git(
&["init", "-q", "--object-format=sha256", "-b", "main"],
&work,
);
run_git(&["config", "user.email", "t@t"], &work);
run_git(&["config", "user.name", "t"], &work);
run_git(&["add", "."], &work);
run_git(&["commit", "-qm", "seed"], &work);
let oid_of = |spec: &str| {
let out = std::process::Command::new("git")
.args(["rev-parse", spec])
.current_dir(&work)
.output()
.expect("git rev-parse runs");
assert!(out.status.success(), "rev-parse {spec} failed");
String::from_utf8_lossy(&out.stdout).trim().to_string()
};
let sha_local = oid_of("HEAD:one.txt");
let sha_dual = oid_of("HEAD:two.txt");
let raw1 = raw_cid_for(content1);
let raw4 = raw_cid_for(content2);
db.upsert_mirror_repo(
"zPinsListOwner",
"pins-list",
"/unused-pins-list",
None,
false,
)
.await
.unwrap();
let rec = db
.get_repo("zPinsListOwner", "pins-list")
.await
.unwrap()
.expect("mirror repo row");
let bare = state
.repo_store
.acquire(&rec.owner_did, &rec.name)
.await
.unwrap();
let _ = std::fs::remove_dir_all(&bare);
std::fs::create_dir_all(&bare).unwrap();
run_git(
&[
"clone",
"--bare",
"-q",
work.to_str().unwrap(),
bare.to_str().unwrap(),
],
tmp.path(),
);
// The combination table for what the response must carry:
//
// shape | local_pinned | pinata_pinned | local_cid | pinata_cid
// -------------------+--------------+---------------+-----------+-----------
// local-only | true | false | raw | null
// pinata-only (raw) | false | true | null | provider
// pinata-only (null) | false | true | null | raw(=provider)
// dual | true | true | raw | provider
// legacy provider key| true | false | null | null
//
// The pinata-only (raw) row keeps a genuine raw key in the
// column but must NOT surface it: provenance is unconfirmed,
// so no local field may carry it. The legacy row predates the
// raw-key contract (dag-pb key, provenance claimed) and must
// surface no local key until repair rewrites it — while the
// row itself (and its `local_pinned` flag) stays visible.
//
// Distinct shas (the table is keyed on sha256_hex) keep the
// rows independent.
// (1) local-only: real local pin via the production writer.
db.record_pinned_cid_with_source(&sha_local, &raw1, &rec.id)
.await
.unwrap();
// (2) pinata-only (raw != provider): the post-v27/v30 row shape
// produced by `record_pinata_cid`. The stored raw key is genuine
// but local provenance is unconfirmed, so neither local field
// may carry it; the provider field is the usable identifier.
let sha_pinata_raw = "sha_p2_pinata_only_distinct_cids";
let raw2 = raw_cid_for(b"pins pinata-only bytes\n");
let pinata2 = "QmPinataProviderCidForRawOnlyRow";
assert_ne!(raw2, pinata2);
db.record_pinata_cid(
sha_pinata_raw,
&raw2,
pinata2,
Some("repo-p2-pinata-raw"),
i64::MAX,
)
.await
.unwrap();
// (3) pinata-only (raw == provider): `record_pinata_cid` stores
// `cid = NULL` so the resolver key isn't aliased to a dag-pb
// provider CID that doesn't hash to the raw bytes (#173).
let sha_pinata_null = "sha_p2_pinata_only_null_cid";
let same = "QmPinataOnlyCidRawEqualsProvider";
db.record_pinata_cid(
sha_pinata_null,
same,
same,
Some("repo-p2-pinata-null"),
i64::MAX,
)
.await
.unwrap();
// (4) dual: local first (sets the flag and `cid`), then Pinata
// (sets `pinata_cid` and preserves `cid` and `local_ipfs_provenance`).
let pinata4 = "QmPinataProviderCidForDualRow";
db.record_pinned_cid_with_source(&sha_dual, &raw4, &rec.id)
.await
.unwrap();
db.record_pinata_cid(&sha_dual, &raw4, pinata4, Some(&rec.id), i64::MAX)
.await
.unwrap();
// (5) legacy provider key: a pre-fix row whose `cid` is a dag-pb
// provider identifier with claimed local provenance. Production
// writers can no longer produce this shape, so it is seeded with
// raw SQL — the point is that the projection must not trust column
// shape as proof of a raw local key.
let sha_legacy = "sha_p2_legacy_provider_key";
sqlx::query(
"INSERT INTO pinned_cids (sha256_hex, cid, pinned_at, pinata_cid, repo_id, local_ipfs_provenance)
VALUES ($1, $2, $3, NULL, $4, TRUE)",
)
.bind(sha_legacy)
.bind("QmLegacyProviderKeyPredatesRawContract")
.bind("2026-07-01T00:00:00Z")
.bind(&rec.id)
.execute(&pool)
.await
.unwrap();
// Hit the production handler end-to-end through a one-shot router
// request, so the test exercises the actual response shape
// (not a unit test on PinnedCidRecord fields).
let resp = Router::new()
.route("/api/v1/ipfs/pins", axum::routing::get(list_pins))
.layer(axum::Extension(crate::auth::AuthenticatedDid(
"did:key:zPinListTest".to_string(),
)))
.with_state(state.clone())
.oneshot(
axum::http::Request::builder()
.uri("/api/v1/ipfs/pins")
.body(axum::body::Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(resp.status(), axum::http::StatusCode::OK);
let bytes = axum::body::to_bytes(resp.into_body(), usize::MAX)
.await
.expect("read body");
let v: Value = serde_json::from_slice(&bytes).expect("json body");
let pins = v
.get("pins")
.and_then(|p| p.as_array())
.expect("pins array");
assert_eq!(
pins.len(),
5,
"all five shapes must appear in the response (every row has at least one of cid/pinata_cid set)"
);
// Build a map sha -> pin object for ergonomic assertions.
let by_sha: std::collections::HashMap<String, &Value> = pins
.iter()
.map(|p| {
(
p.get("sha256_hex")
.and_then(|s| s.as_str())
.unwrap()
.to_string(),
p,
)
})
.collect();
// Helper: assert a single pin's fields.
let assert_pin = |sha: &str,
local_pinned: bool,
pinata_pinned: bool,
local_cid: Option<&str>,
pinata_cid: Option<&str>| {
let pin = by_sha.get(sha).unwrap_or_else(|| {
panic!(
"pin row for {sha} missing; got shas {:?}",
by_sha.keys().collect::<Vec<_>>()
)
});
assert_eq!(
pin.get("local_pinned").and_then(|v| v.as_bool()),
Some(local_pinned),
"{sha}: local_pinned mismatch",
);
assert_eq!(
pin.get("pinata_pinned").and_then(|v| v.as_bool()),
Some(pinata_pinned),
"{sha}: pinata_pinned mismatch",
);
assert_eq!(
pin.get("local_cid").and_then(|v| v.as_str()),
local_cid,
"{sha}: local_cid mismatch",
);
assert_eq!(
pin.get("pinata_cid").and_then(|v| v.as_str()),
pinata_cid,
"{sha}: pinata_cid mismatch",
);
// Round-3 P2: `cid` is the LOCAL resolver key, full
// stop. It is the raw CID when the row has a local pin
// (local-only or dual shape) and `None` for Pinata-only
// rows. The previous contract aliased the Pinata provider
// CID into `cid` for Pinata-only rows, which made
// `gl ipfs list` advertise a CID the node's own
// `/ipfs/{cid}` resolver cannot serve (404). The
// provenance split is exposed via `local_cid`,
// `pinata_cid`, and the boolean `local_pinned` /
// `pinata_pinned` flags.
let cid_value = pin.get("cid").cloned().unwrap_or(serde_json::Value::Null);
let expected_cid = local_cid
.map(|s| serde_json::Value::String(s.to_string()))
.unwrap_or(serde_json::Value::Null);
assert_eq!(cid_value, expected_cid, "{sha}: cid mismatch");
};
// (1) local-only
assert_pin(&sha_local, true, false, Some(raw1.as_str()), None);
// (2) pinata-only (raw != provider): the stored raw key is
// genuine but provenance is unconfirmed, so NO local field may
// carry it — remote-only state is visible through `pinata_cid`
// alone.
assert_pin(sha_pinata_raw, false, true, None, Some(pinata2));
// (3) pinata-only (raw == provider) — cid NULL, pinata_cid set
assert_pin(sha_pinata_null, false, true, None, Some(same));
// (4) dual
assert_pin(&sha_dual, true, true, Some(raw4.as_str()), Some(pinata4));
// (5) legacy provider key: no local field carries the dag-pb
// key, but the row and its claimed provenance stay visible so
// the repair gap is observable rather than hidden.
assert_pin(sha_legacy, true, false, None, None);
// Sanity: the `local_ipfs_provenance` column itself is what
// powers `local_pinned`, so the response field must agree
// with the database column. Reading directly avoids any
// confusion if the writer path changes.
let rows = sqlx::query("SELECT sha256_hex, local_ipfs_provenance FROM pinned_cids")
.fetch_all(&pool)
.await
.unwrap();
let mut db_provenance: std::collections::HashMap<String, bool> = Default::default();
for r in rows {
let sha: String = r.get("sha256_hex");
let p: bool = r.get("local_ipfs_provenance");
db_provenance.insert(sha, p);
}
for (sha, expected) in [
(sha_local.as_str(), true),
(sha_pinata_raw, false),
(sha_pinata_null, false),
(sha_dual.as_str(), true),
(sha_legacy, true),
] {
assert_eq!(
db_provenance.get(sha).copied(),
Some(expected),
"db column for {sha} does not match expected writer-owned provenance"
);
}
// Follow-through: every non-null local key in the response must
// resolve through `GET /ipfs/{cid}`. The local-only and dual rows
// above name real blobs on disk in a public repo, so both must
// serve their exact bytes; the Pinata-only and legacy rows carry
// no local key and have nothing to follow.
let follow_router = Router::new()
.route("/ipfs/{cid}", axum::routing::get(get_by_cid))
.with_state(state.clone());
for (raw, content) in [(&raw1, &content1[..]), (&raw4, &content2[..])] {
let resp = follow_router
.clone()
.oneshot(
axum::http::Request::builder()
.uri(format!("/ipfs/{raw}"))
.body(axum::body::Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(
resp.status(),
axum::http::StatusCode::OK,
"advertised local key {raw} must resolve"
);
let body = axum::body::to_bytes(resp.into_body(), usize::MAX)
.await
.expect("read body");
assert_eq!(
body.as_ref(),
content,
"resolved bytes for {raw} must match the pinned blob"
);
}
}
/// #251 / CodeRabbit nit: cover `get_by_cid`'s DB-error conversion path — a
/// valid CID must still yield 503 on a closed pool.
#[sqlx::test]
+1
View File
@@ -22,6 +22,7 @@ pub mod replicas;
pub mod repos;
pub mod resolve;
pub mod stars;
pub mod task_cursor;
pub mod tasks;
pub mod visibility;
pub mod webhooks;
+116 -55
View File
@@ -84,7 +84,7 @@ pub(crate) const PUBLIC_HTTP_URL_REQUIREMENT: &str = "must be a public http(s) U
/// sits at a prefix-length-dependent offset (RFC 6052 §2.2) — so they return
/// `None`. Any caller that needs them blocked must reject the wider
/// `64:ff9b::/32` itself; `is_public_http_url` does this in its native-v6 arm.
fn embedded_ipv4(v6: std::net::Ipv6Addr) -> Option<std::net::Ipv4Addr> {
pub(crate) fn embedded_ipv4(v6: std::net::Ipv6Addr) -> Option<std::net::Ipv4Addr> {
if let Some(v4) = v6.to_ipv4_mapped().or_else(|| v6.to_ipv4()) {
return Some(v4);
}
@@ -104,6 +104,116 @@ fn embedded_ipv4(v6: std::net::Ipv6Addr) -> Option<std::net::Ipv4Addr> {
None
}
/// Whether an IP literal is outside the local/private ranges that an
/// attacker-controlled outbound URL must never reach. Kept separate from URL
/// parsing so callers that resolve a DNS name themselves can apply the same
/// classification to every answer and then pin the approved addresses.
pub(crate) fn is_public_ip(ip: std::net::IpAddr) -> bool {
// Reject loopback/unspecified on the literal as given — catches `::1` and
// `::` before the IPv4-folding below (`::1`.to_ipv4() would otherwise map
// to a non-loopback `0.0.0.1`).
if ip.is_loopback() || ip.is_unspecified() {
return false;
}
// Fold any IPv6 literal that embeds an IPv4 address (mapped, compatible,
// 6to4, NAT64) down to that IPv4 so the v4 range checks catch
// loopback/private addresses smuggled in via an IPv6 encoding, then
// re-check loopback/unspecified.
let ip = match ip {
std::net::IpAddr::V6(v6) => embedded_ipv4(v6).map(std::net::IpAddr::V4).unwrap_or(ip),
v4 => v4,
};
if ip.is_loopback() || ip.is_unspecified() {
return false;
}
match ip {
std::net::IpAddr::V4(v4) => {
let o = v4.octets();
// RFC1918 private, link-local, CGNAT (100.64.0.0/10), or the
// RFC1122 "this host" block 0.0.0.0/8 (never a valid destination;
// 0.0.0.0 itself is already caught by the is_unspecified check).
if v4.is_private()
|| v4.is_link_local()
|| (o[0] == 100 && (o[1] & 0xc0) == 64)
|| o[0] == 0
{
return false;
}
}
std::net::IpAddr::V6(v6) => {
let s = v6.segments();
// fc00::/7 (unique-local), fe80::/10 (link-local), or fec0::/10
// (site-local, deprecated by RFC 3879; the link-local mask does
// not cover it).
if (s[0] & 0xfe00) == 0xfc00 || (s[0] & 0xffc0) == 0xfe80 || (s[0] & 0xffc0) == 0xfec0 {
return false;
}
// Any NAT64 address (64:ff9b::/32) that is not the cleanly
// decodable well-known /96 — e.g. the RFC 8215 local-use
// 64:ff9b:1::/48 — carries a translated target whose embedded v4
// sits at a prefix-length-dependent offset (RFC 6052 §2.2).
// Rather than risk a wrong decode across every prefix length we
// reject the whole NAT64 space here. The well-known /96 was already
// folded to its v4 above and never reaches this arm.
if s[0] == 0x0064 && s[1] == 0xff9b {
return false;
}
}
}
true
}
/// Mirror-upstream variant of [`is_public_ip`]: in addition to the shared
/// peer classification, rejects IANA special-purpose ranges that are
/// non-global, locally meaningful, or unsafe as owner-controlled outbound
/// fetch endpoints. For IP-literal upstreams, configure-time URL validation
/// and the fetch-time egress policy share this classifier, so a stored
/// literal can never fail the worker it is scheduled for. Hostname upstreams
/// are only shape-checked at configure time: each fetch resolves DNS and
/// applies this classifier to the answers, so a hostname whose answers turn
/// non-public still fails at fetch time.
pub(crate) fn mirror_upstream_ip_permitted(ip: std::net::IpAddr) -> bool {
if !is_public_ip(ip) {
return false;
}
let classified = match ip {
std::net::IpAddr::V6(v6) => embedded_ipv4(v6).map(std::net::IpAddr::V4).unwrap_or(ip),
_ => ip,
};
match classified {
std::net::IpAddr::V4(v4) => {
let [a, b, c, _] = v4.octets();
// IANA special-purpose ranges that are not globally reachable but
// may be routed inside labs, clouds, or operator networks. They are
// unsuitable for an owner-controlled outbound fetch even though
// `Ipv4Addr::is_private` does not classify them as RFC1918.
!(a >= 224
|| (a == 192 && b == 0 && c == 0)
|| (a == 192 && b == 0 && c == 2)
|| (a == 192 && b == 88 && c == 99)
|| (a == 198 && matches!(b, 18 | 19))
|| (a == 198 && b == 51 && c == 100)
|| (a == 203 && b == 0 && c == 113))
}
std::net::IpAddr::V6(v6) => {
let s = v6.segments();
// IANA special-purpose ranges that are non-global, locally
// meaningful, or unsafe to treat as ordinary forge endpoints.
// Reject the complete IETF protocol-assignment /23 rather than
// trying to maintain its narrow globally reachable exceptions;
// a public forge has no reason to depend on those anycast ranges.
!(v6.is_multicast()
|| (s[0] & 0xffc0) == 0xfec0
|| (s[0] == 0x0100 && s[1] == 0 && s[2] == 0 && s[3] == 0)
|| (s[0] == 0x0100 && s[1] == 0 && s[2] == 0 && s[3] == 1)
|| (s[0] == 0x2001 && (s[1] & 0xfe00) == 0)
|| (s[0] == 0x2001 && s[1] == 0x0db8)
|| (s[0] == 0x3fff && (s[1] & 0xf000) == 0)
|| s[0] == 0x5f00)
}
}
}
/// Whether a peer `http_url` is a public http(s) endpoint safe to register.
/// Rejects non-http(s) schemes, loopback/unspecified/private/link-local IPs,
/// and `localhost` / `.localhost` / `.local` / `.internal` hostnames. Used at
@@ -137,60 +247,9 @@ pub fn is_public_http_url(raw: &str) -> bool {
// before parsing as an IP.
let ip_candidate = host.trim_start_matches('[').trim_end_matches(']');
if let Ok(ip) = ip_candidate.parse::<std::net::IpAddr>() {
// Reject loopback/unspecified on the literal as given — catches `::1`
// and `::` before the IPv4-folding below (`::1`.to_ipv4() would
// otherwise map to a non-loopback `0.0.0.1`).
if ip.is_loopback() || ip.is_unspecified() {
return false;
}
// Fold any IPv6 literal that embeds an IPv4 address (mapped, compatible,
// 6to4, NAT64) down to that IPv4 so the v4 range checks catch
// loopback/private addresses smuggled in via an IPv6 encoding, then
// re-check loopback/unspecified.
let ip = match ip {
std::net::IpAddr::V6(v6) => embedded_ipv4(v6).map(std::net::IpAddr::V4).unwrap_or(ip),
v4 => v4,
};
if ip.is_loopback() || ip.is_unspecified() {
return false;
}
match ip {
std::net::IpAddr::V4(v4) => {
let o = v4.octets();
// RFC1918 private, link-local, CGNAT (100.64.0.0/10), or the
// RFC1122 "this host" block 0.0.0.0/8 (never a valid destination;
// 0.0.0.0 itself is already caught by the is_unspecified check).
if v4.is_private()
|| v4.is_link_local()
|| (o[0] == 100 && (o[1] & 0xc0) == 64)
|| o[0] == 0
{
return false;
}
}
std::net::IpAddr::V6(v6) => {
let s = v6.segments();
// fc00::/7 (unique-local), fe80::/10 (link-local), or fec0::/10
// (site-local, deprecated by RFC 3879 and so never a legitimate
// peer address, but still routable on networks that kept it).
if (s[0] & 0xfe00) == 0xfc00
|| (s[0] & 0xffc0) == 0xfe80
|| (s[0] & 0xffc0) == 0xfec0
{
return false;
}
// Any NAT64 address (64:ff9b::/32) that is not the cleanly
// decodable well-known /96 — e.g. the RFC 8215 local-use
// 64:ff9b:1::/48 — carries a translated target whose embedded v4
// sits at a prefix-length-dependent offset (RFC 6052 §2.2).
// Rather than risk a wrong decode across every prefix length we
// reject the whole NAT64 space here. The well-known /96 was
// already folded to its v4 above and never reaches this arm.
if s[0] == 0x0064 && s[1] == 0xff9b {
return false;
}
}
}
// IP literals share the single DNS/literal classification in
// `is_public_ip`; hostname-only rules stay local to this function.
return is_public_ip(ip);
} else if !host.contains('.') {
// A name with no dot at all. The `.local` / `.internal` rules above are
// suffix checks, so a single label slips both, and the resolver decides
@@ -616,6 +675,8 @@ mod tests {
fn rejects_deprecated_site_local_v6() {
// fec0::/10, deprecated by RFC 3879. The link-local mask (0xffc0 ==
// 0xfe80) does not cover it: 0xfec0 & 0xffc0 is 0xfec0.
assert!(!super::is_public_ip("fec0::1".parse().unwrap()));
assert!(!super::is_public_ip("feff:ffff::1".parse().unwrap()));
for bad in ["http://[fec0::1]/", "http://[feff:ffff::1]:7545"] {
assert!(!is_public_http_url(bad), "{bad:?} must be rejected");
}
File diff suppressed because it is too large. Load diff
+782
View File
@@ -0,0 +1,782 @@
//! Opaque, integrity-protected continuation tokens for the task read surfaces.
//!
//! The task list pages by keyset over `(created_at, id)`, but the rows a
//! caller may *see* are a filtered subset of the rows the query has to
//! *examine*. Handing the caller a raw `(created_at, id)` cursor therefore
//! forced a choice between two broken options (#327 review): anchor the cursor
//! on the last visible row, and a window of denied tasks longer than the scan
//! budget stalls paging forever; or anchor it on the last examined row, and a
//! denied read leaks the id and timestamp of a task `GET /tasks/{id}` other-
//! wise 404s.
//!
//! A server-issued token removes the choice. The position it carries is the
//! last *examined* candidate, so paging always advances, and the payload is
//! opaque and MAC'd, so the caller learns nothing from it and cannot forge one
//! naming a row of their choosing.
//!
//! The position must be *confidential*, not merely unforgeable: a token that
//! merely signed a base64 payload would still let its holder read the id and
//! timestamp of the denied row it names, which is the disclosure the token
//! exists to prevent. The payload is therefore encrypted under a
//! synthetic-IV construction (SIV): the tag is an HMAC over the filter and the
//! plaintext, and it doubles as the IV seeding the keystream the plaintext is
//! XORed with. That needs no randomness source and no dependency beyond the
//! `hmac`/`sha2` pair already used for webhook signatures and blob recipient
//! tags, and it is decrypt-last: nothing is parsed until the tag verifies.
//!
//! Making the token the *only* accepted cursor also fixes the ordering-domain
//! bug the same review found. `agent_tasks.created_at` is TEXT and compared as
//! TEXT, so `...Z` and `...+00:00` denote one instant but sort differently. A
//! caller-typed timestamp could silently skip or repeat same-time rows. The
//! token instead carries the stored string verbatim, so the value compared is
//! always a value the server wrote.
//!
//! A token is bound to the *caller* as well as the filter. The position it
//! names is the last examined candidate, not the last visible one, so it
//! encodes how far a scan got under one caller's visibility. Resuming it as a
//! different caller would start the scan past rows that caller is entitled to
//! read, silently dropping them from the answer (#327 review). The presenting
//! caller's normalized DID (empty and flagged absent when anonymous) is
//! therefore part of the MAC input, so a token presented by anyone else fails
//! to verify rather than under-reporting.
//!
//! Wire form: `v1.<iv>.<body>`, both parts base64url unpadded: the 16-byte
//! synthetic IV (the truncated MAC, which also seeds the keystream) and the
//! padded, encrypted JSON payload.
use base64::engine::general_purpose::URL_SAFE_NO_PAD;
use base64::Engine;
use hmac::{Hmac, Mac};
use serde::{Deserialize, Serialize};
use sha2::Sha256;
use crate::error::AppError;
type HmacSha256 = Hmac<Sha256>;
const CURSOR_PREFIX: &str = "v1";
const KEY_DERIVATION_LABEL: &[u8] = b"gitlawb/tasks-cursor-key/v1";
const TAG_DOMAIN: &[u8] = b"gitlawb/tasks-cursor-tag/v1";
const STREAM_DOMAIN: &[u8] = b"gitlawb/tasks-cursor-stream/v1";
/// Truncated MAC length, and the synthetic IV width. 128 bits is far beyond
/// forgery reach for a token that carries no authority of its own (every page
/// re-runs the visibility gate against the presenting caller), and keeps the
/// token short enough to sit in a query string.
const TAG_LEN: usize = 16;
/// Padding bucket for the serialized cursor payload, in bytes. The keystream
/// XORs the plaintext byte for byte, so token length would otherwise track
/// payload length and leak the named row's field widths to a holder who
/// cannot decrypt anything. 128 comfortably covers the widest realistic
/// payload: a 35-char RFC3339 timestamp (`to_rfc3339()` renders 29 chars at
/// millisecond precision and 32 at microsecond, 35 only with all nine
/// fractional digits), a 36-char server-minted UUID, and the fixed JSON
/// envelope plus a 10-digit expiry, which together total 101 bytes. Should a future payload ever exceed the bucket, the length
/// rounds up to the next multiple instead of failing, so `encode` never
/// breaks — it just mints a longer token.
const PAYLOAD_BUCKET: usize = 128;
/// How long an issued cursor stays acceptable. Keyset positions never go stale
/// on their own — `created_at`/`id` are immutable — so this is not a
/// correctness bound. It bounds how long a token stays valid across a node
/// restart-and-rotate and keeps an abandoned page from being resumed
/// indefinitely.
const CURSOR_TTL_SECS: i64 = 24 * 60 * 60;
/// Node-keyed MAC key for continuation tokens, derived from the node keypair
/// seed so it needs no configuration and survives restarts of the same node.
/// Derived rather than used directly so a token forgery oracle could not
/// bear on the signing key itself.
#[derive(Clone)]
pub struct TaskCursorKey([u8; 32]);
impl std::fmt::Debug for TaskCursorKey {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("TaskCursorKey(<redacted>)")
}
}
impl TaskCursorKey {
pub fn derive(node_seed: &[u8; 32]) -> Self {
let mut mac = HmacSha256::new_from_slice(node_seed).expect("HMAC accepts any key length");
mac.update(KEY_DERIVATION_LABEL);
let mut key = [0u8; 32];
key.copy_from_slice(&mac.finalize().into_bytes());
Self(key)
}
}
/// The keyset position a token carries: the last candidate row the previous
/// request examined, whether or not the caller was allowed to see it.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct TaskPosition {
pub created_at: String,
pub id: String,
}
impl TaskPosition {
pub fn new(created_at: impl Into<String>, id: impl Into<String>) -> Self {
Self {
created_at: created_at.into(),
id: id.into(),
}
}
pub fn as_pair(&self) -> (&str, &str) {
(self.created_at.as_str(), self.id.as_str())
}
}
#[derive(Serialize, Deserialize)]
struct CursorPayload<'a> {
/// `created_at` of the last examined candidate, verbatim as stored.
t: &'a str,
/// `id` of the last examined candidate.
i: &'a str,
/// Unix expiry.
e: i64,
}
/// The filter a page was issued under. A cursor is only meaningful against the
/// same `status`/`assignee_did` filter that produced it: resuming a filtered
/// scan from an unfiltered position (or the reverse) silently skips rows.
/// Bound into the MAC rather than stored in the payload, so it costs no token
/// length and cannot be edited without invalidating the tag.
#[derive(Debug, Clone, Copy)]
pub struct TaskFilter<'a> {
pub status: Option<&'a str>,
pub assignee_did: Option<&'a str>,
}
/// The visibility identity a token was issued to. Normalized through
/// `normalize_owner_key` so the two spellings of one `did:key` identity
/// (`did:key:X` and bare `X`) bind identically, matching `did_matches` on the
/// read path: a caller who authenticates with the other spelling of their own
/// DID must be able to resume their own page.
fn caller_binding(caller: Option<&str>) -> &str {
caller.map(crate::db::normalize_owner_key).unwrap_or("")
}
/// The assignee filter a token was issued under. Normalized through
/// `normalize_owner_key` so the two spellings of one `did:key` identity
/// (`did:key:X` and bare `X`) bind identically, matching `list_tasks_keyset`
/// in SQL: minting under `did:key:X` and resuming with bare `X` (or the reverse)
/// hits the same rows and must verify under the MAC.
fn assignee_binding(assignee: Option<&str>) -> &str {
assignee.map(crate::db::normalize_owner_key).unwrap_or("")
}
fn cursor_mac(
key: &TaskCursorKey,
filter: TaskFilter<'_>,
caller: Option<&str>,
plaintext: &[u8],
) -> HmacSha256 {
let mut mac = HmacSha256::new_from_slice(&key.0).expect("HMAC accepts any key length");
mac.update(TAG_DOMAIN);
// Length-prefix every field so no two distinct filter/caller/plaintext
// tuples can produce the same MAC input.
for field in [
filter.status.unwrap_or("").as_bytes(),
assignee_binding(filter.assignee_did).as_bytes(),
caller_binding(caller).as_bytes(),
plaintext,
] {
mac.update(&(field.len() as u64).to_be_bytes());
mac.update(field);
}
// Presence is distinct from emptiness for all three optional fields, so an
// anonymous token cannot collide with one issued to a caller whose
// normalized DID is the empty string.
mac.update(&[
u8::from(filter.status.is_some()),
u8::from(filter.assignee_did.is_some()),
u8::from(caller.is_some()),
]);
mac
}
/// Synthetic IV: the authentication tag over the filter and plaintext, which
/// also seeds the keystream. Deterministic by construction, so no randomness
/// source is needed and two tokens for the same page are byte-identical.
fn siv(
key: &TaskCursorKey,
filter: TaskFilter<'_>,
caller: Option<&str>,
plaintext: &[u8],
) -> [u8; TAG_LEN] {
let mut out = [0u8; TAG_LEN];
out.copy_from_slice(
&cursor_mac(key, filter, caller, plaintext)
.finalize()
.into_bytes()[..TAG_LEN],
);
out
}
/// XOR `buf` with the keystream for `iv`. Its own inverse, so encrypt and
/// decrypt are the same call.
fn apply_keystream(key: &TaskCursorKey, iv: &[u8; TAG_LEN], buf: &mut [u8]) {
for (block_index, chunk) in buf.chunks_mut(32).enumerate() {
let mut mac = HmacSha256::new_from_slice(&key.0).expect("HMAC accepts any key length");
mac.update(STREAM_DOMAIN);
mac.update(iv);
mac.update(&(block_index as u64).to_be_bytes());
let block = mac.finalize().into_bytes();
for (byte, k) in chunk.iter_mut().zip(block.iter()) {
*byte ^= k;
}
}
}
/// Mint a token resuming at `position` for `filter`, usable only by `caller`.
///
/// The serialized payload is padded with trailing whitespace to a fixed
/// bucket before the SIV tag is computed: the keystream preserves plaintext
/// length, so an unpadded payload would let the token's length leak the
/// named row's field widths (a 25-char `to_rfc3339()` rendering mints a
/// shorter token than a 35-char one). `decode` tolerates the padding because
/// `serde_json` ignores trailing whitespace, and the tag covers the padded
/// bytes, so a token minted before padding landed still verifies.
pub fn encode(
key: &TaskCursorKey,
filter: TaskFilter<'_>,
caller: Option<&str>,
position: &TaskPosition,
) -> String {
let mut payload = serde_json::to_vec(&CursorPayload {
t: &position.created_at,
i: &position.id,
e: chrono::Utc::now().timestamp() + CURSOR_TTL_SECS,
})
.expect("cursor payload is plain strings and an integer");
let padded_len = payload.len().next_multiple_of(PAYLOAD_BUCKET);
payload.resize(padded_len, b' ');
let iv = siv(key, filter, caller, &payload);
apply_keystream(key, &iv, &mut payload);
format!(
"{CURSOR_PREFIX}.{}.{}",
URL_SAFE_NO_PAD.encode(iv),
URL_SAFE_NO_PAD.encode(&payload)
)
}
/// One rejection message for every way a token can fail to verify. A caller
/// who mangled a token, replayed an expired one, or tried to move one to a
/// different filter or a different presenting identity learns only that the
/// cursor is not usable — never which of those it was, and never anything
/// about the row it named.
const INVALID_CURSOR: &str = "invalid or expired cursor";
fn reject() -> AppError {
AppError::BadRequest(INVALID_CURSOR.into())
}
/// Verify a token against `filter` and the presenting `caller`, and return the
/// position it carries.
pub fn decode(
key: &TaskCursorKey,
filter: TaskFilter<'_>,
caller: Option<&str>,
token: &str,
) -> crate::error::Result<TaskPosition> {
let mut parts = token.split('.');
let (Some(version), Some(iv_b64), Some(body_b64), None) =
(parts.next(), parts.next(), parts.next(), parts.next())
else {
return Err(reject());
};
if version != CURSOR_PREFIX {
return Err(reject());
}
let iv_bytes = URL_SAFE_NO_PAD.decode(iv_b64).map_err(|_| reject())?;
let mut plaintext = URL_SAFE_NO_PAD.decode(body_b64).map_err(|_| reject())?;
let iv: [u8; TAG_LEN] = iv_bytes.try_into().map_err(|_| reject())?;
apply_keystream(key, &iv, &mut plaintext);
// Authenticate before parsing: until the tag matches, `plaintext` is just
// attacker-chosen bytes run through a keystream. `verify_truncated_left`
// is the constant-time compare, so a forgery attempt cannot be steered by
// timing the first differing byte.
cursor_mac(key, filter, caller, &plaintext)
.verify_truncated_left(&iv)
.map_err(|_| reject())?;
let decoded: CursorPayload<'_> = serde_json::from_slice(&plaintext).map_err(|_| reject())?;
if decoded.e < chrono::Utc::now().timestamp() {
return Err(reject());
}
Ok(TaskPosition::new(decoded.t, decoded.i))
}
#[cfg(test)]
mod tests {
use super::*;
fn key() -> TaskCursorKey {
TaskCursorKey::derive(&[7u8; 32])
}
fn unfiltered() -> TaskFilter<'static> {
TaskFilter {
status: None,
assignee_did: None,
}
}
#[test]
fn round_trips_position_verbatim() {
let k = key();
// A stored timestamp the server wrote, fractional digits and all.
let pos = TaskPosition::new("2026-01-03T00:00:00.123456789+00:00", "task-a");
let token = encode(&k, unfiltered(), None, &pos);
assert_eq!(decode(&k, unfiltered(), None, &token).unwrap(), pos);
// A sub-bucket payload pads to exactly one bucket: the padding
// claim is an exact width, and every other width assertion here is
// bucket-relative, so only an absolute check catches a pad that
// skips "next multiple" or a drifted bucket constant.
let body = token.split('.').nth(2).expect("token has a body part");
let raw = URL_SAFE_NO_PAD.decode(body).unwrap();
assert_eq!(
raw.len(),
PAYLOAD_BUCKET,
"normal token must pad to exactly one bucket"
);
}
/// The whole point of the token is that the caller may hold it without
/// learning the denied row it names. A signed-but-plaintext payload would
/// pass every other test here and still fail this one.
#[test]
fn token_does_not_expose_the_row_it_names() {
let k = key();
let token = encode(
&k,
unfiltered(),
None,
&TaskPosition::new("2026-01-03T00:00:00+00:00", "denied-task-id"),
);
let body = token.split('.').nth(2).expect("token has a body part");
let raw = URL_SAFE_NO_PAD.decode(body).unwrap();
let as_text = String::from_utf8_lossy(&raw);
for secret in ["denied-task-id", "2026-01-03", "\"t\"", "\"i\""] {
assert!(
!as_text.contains(secret),
"token body must not carry {secret:?} in the clear: {as_text:?}"
);
}
// Also assert it is not merely reordered or whitespace-mangled JSON.
assert!(
serde_json::from_slice::<serde_json::Value>(&raw).is_err(),
"token body must not be parseable as JSON"
);
}
/// The keystream preserves plaintext length, so without the fixed-width
/// padding a token naming a 35-char timestamp would be measurably longer
/// than one naming a 32-char one: the holder would learn the withheld
/// row's field widths without decrypting anything. Mint the same shape
/// both ways and require identical token length.
#[test]
fn token_length_does_not_vary_with_position_width() {
let k = key();
// Ids are server-minted fixed-length UUIDs; the timestamp is the
// width that varies in production (`to_rfc3339()` renders 35 chars
// with nanoseconds, 25 without fractional seconds).
let id = "b3f7a1c2-4d5e-4f6a-8b9c-0d1e2f3a4b5c";
let narrow = encode(
&k,
unfiltered(),
None,
&TaskPosition::new("2026-01-03T00:00:00+00:00", id),
);
let wide = encode(
&k,
unfiltered(),
None,
&TaskPosition::new("2026-01-03T00:00:00.123456789+00:00", id),
);
assert_eq!(
narrow.len(),
wide.len(),
"tokens must not leak the named row's field widths: {narrow} vs {wide}"
);
// The padded token still decodes to the exact stored string.
assert_eq!(
decode(&k, unfiltered(), None, &wide).unwrap(),
TaskPosition::new("2026-01-03T00:00:00.123456789+00:00", id)
);
}
/// A token minted before the fixed-width padding landed (unpadded payload,
/// tag over the raw bytes) must still verify: `decode` MACs whatever
/// plaintext the keystream recovers and `serde_json` tolerates the
/// missing trailing whitespace. This pins the back-compat claim in the
/// `encode` docstring with a frozen v1 artifact, so a change to the v1
/// recipe pins (domains, MAC input, keystream, segment layout, field
/// names) breaks this test instead of silently re-deriving under the
/// new internals. It does not pin the padding: `decode` never consults
/// `PAYLOAD_BUCKET` and the fixture is unpadded, so the padding-width
/// claim belongs to `oversized_payload_rounds_up_to_next_bucket`.
///
/// Fixture expiry is fixed at 2100-01-01 (`4102444800`): it must stay in
/// the future whenever the test runs.
#[test]
fn unpadded_legacy_token_still_verifies() {
let k = key();
let filter = unfiltered();
let position = TaskPosition::new("2026-01-03T00:00:00+00:00", "legacy-task");
// Pre-padding recipe minted once and frozen: serialize with no
// padding, tag, and encrypt. Deterministic by construction (SIV needs
// no randomness), so this literal is stable across runs.
let token = "v1.ILJyy7JZScm_h-HM1-CggA.l4kJUoHTbqSGTVKL0DP4hORnbOS1_gHYV2fwJYXd2QtM-6TBMCKh-wvluIiaEYC9rUSUPVSteAKkp7jBCmK_bR2D";
assert_eq!(
decode(&k, filter, None, token).unwrap(),
position,
"pre-padding token must verify under the current decode"
);
// The fixture is a genuine pre-padding artifact, so its decoded
// body must stay below one bucket: a literal re-minted under the
// padded recipe would retire this back-compat pin silently.
let body = token.split('.').nth(2).expect("token has a body part");
let raw = URL_SAFE_NO_PAD.decode(body).unwrap();
assert!(
raw.len() < PAYLOAD_BUCKET,
"legacy fixture must be the unpadded shape: {} bytes",
raw.len()
);
}
/// The `PAYLOAD_BUCKET` docstring promises an oversized payload rounds up
/// to the next multiple instead of failing. Mint a position whose id
/// pushes the serialized payload past one bucket and require the token
/// to round-trip (at two buckets).
#[test]
fn oversized_payload_rounds_up_to_next_bucket() {
let k = key();
// ~140-char id: payload crosses the 128-byte bucket.
let long_id = "a".repeat(140);
let position = TaskPosition::new("2026-01-03T00:00:00+00:00", long_id.clone());
let probe = serde_json::to_vec(&CursorPayload {
t: &position.created_at,
i: &position.id,
e: 1_700_000_000,
})
.expect("cursor payload is plain strings and an integer");
assert!(
probe.len() > PAYLOAD_BUCKET,
"fixture payload must exceed one bucket: {} bytes",
probe.len()
);
assert!(
probe.len() <= 2 * PAYLOAD_BUCKET,
"fixture payload must fit in two buckets: {} bytes",
probe.len()
);
let token = encode(&k, unfiltered(), None, &position);
assert_eq!(
decode(&k, unfiltered(), None, &token).unwrap(),
position,
"oversized payload must round-trip at two buckets"
);
// The padded body must be exactly two buckets, not merely longer
// than one: the padding claim is an exact width, not a direction.
let body = token.split('.').nth(2).expect("token has a body part");
let raw = URL_SAFE_NO_PAD.decode(body).unwrap();
assert_eq!(
raw.len(),
2 * PAYLOAD_BUCKET,
"oversized payload must pad to exactly two buckets"
);
}
/// Callers paste the token straight into a query string, so it must carry
/// no character that needs percent-encoding.
#[test]
fn token_is_url_safe_verbatim() {
let token = encode(
&key(),
TaskFilter {
status: Some("pending"),
assignee_did: Some("did:key:z6MkAssignee"),
},
None,
&TaskPosition::new("2026-01-03T00:00:00.123456789+00:00", "task-a"),
);
assert!(
token
.chars()
.all(|c| c.is_ascii_alphanumeric() || matches!(c, '-' | '_' | '.')),
"token must be query-safe without encoding: {token}"
);
}
/// Flipping any byte of a token must fail the tag, not decode to a
/// different position: the keystream is malleable on its own, the tag is
/// what stops a chosen-position forgery.
#[test]
fn rejects_any_tampered_byte() {
let k = key();
let token = encode(
&k,
unfiltered(),
None,
&TaskPosition::new("2026-01-03T00:00:00+00:00", "task-a"),
);
let parts: Vec<&str> = token.split('.').collect();
for part in [1usize, 2] {
for position in [0usize, 3] {
let mut bytes = parts[part].as_bytes().to_vec();
bytes[position] = if bytes[position] == b'A' { b'B' } else { b'A' };
let mut mangled: Vec<String> = parts.iter().map(|p| p.to_string()).collect();
mangled[part] = String::from_utf8(bytes).unwrap();
assert!(
decode(&k, unfiltered(), None, &mangled.join(".")).is_err(),
"byte {position} of part {part} must not be malleable"
);
}
}
// Sanity: the untouched token still decodes, so the loop above is not
// passing because every token is rejected.
assert!(decode(&k, unfiltered(), None, &token).is_ok());
}
#[test]
fn rejects_token_from_another_node() {
let pos = TaskPosition::new("2026-01-03T00:00:00+00:00", "task-a");
let token = encode(&TaskCursorKey::derive(&[1u8; 32]), unfiltered(), None, &pos);
assert!(decode(
&TaskCursorKey::derive(&[2u8; 32]),
unfiltered(),
None,
&token
)
.is_err());
}
#[test]
fn rejects_cursor_moved_to_a_different_filter() {
let k = key();
let pos = TaskPosition::new("2026-01-03T00:00:00+00:00", "task-a");
let token = encode(
&k,
TaskFilter {
status: Some("pending"),
assignee_did: None,
},
None,
&pos,
);
assert!(decode(&k, unfiltered(), None, &token).is_err());
assert!(decode(
&k,
TaskFilter {
status: Some("claimed"),
assignee_did: None
},
None,
&token
)
.is_err());
assert!(decode(
&k,
TaskFilter {
status: Some("pending"),
assignee_did: None
},
None,
&token
)
.is_ok());
}
/// A filter of `Some("")` must not verify a token minted with `None`.
#[test]
fn distinguishes_absent_filter_from_empty_filter() {
let k = key();
let pos = TaskPosition::new("2026-01-03T00:00:00+00:00", "task-a");
let token = encode(&k, unfiltered(), None, &pos);
assert!(decode(
&k,
TaskFilter {
status: Some(""),
assignee_did: None
},
None,
&token
)
.is_err());
}
/// A cursor names how far a scan got under one caller's visibility. Moved
/// to a different presenting identity it would start that caller's scan
/// past rows they are entitled to read, so it must fail to verify rather
/// than silently under-report (#327 review).
#[test]
fn rejects_cursor_moved_to_a_different_caller() {
let k = key();
let pos = TaskPosition::new("2026-01-03T00:00:00+00:00", "task-a");
let alice = Some("did:key:z6MkAlice");
let bob = Some("did:key:z6MkBob");
let anon_token = encode(&k, unfiltered(), None, &pos);
assert!(
decode(&k, unfiltered(), alice, &anon_token).is_err(),
"an anonymously minted cursor must not resume an authenticated scan"
);
assert!(decode(&k, unfiltered(), None, &anon_token).is_ok());
let alice_token = encode(&k, unfiltered(), alice, &pos);
assert!(
decode(&k, unfiltered(), bob, &alice_token).is_err(),
"one caller's cursor must not resume another caller's scan"
);
assert!(
decode(&k, unfiltered(), None, &alice_token).is_err(),
"an authenticated cursor must not resume an anonymous scan"
);
assert!(decode(&k, unfiltered(), alice, &alice_token).is_ok());
}
/// The binding is on identity, not spelling: `did:key:X` and bare `X` are
/// one caller everywhere else on the read path (`did_matches`), so a caller
/// who presents the other form of their own DID must keep their own page.
#[test]
fn caller_binding_is_normalized_across_did_key_spellings() {
let k = key();
let pos = TaskPosition::new("2026-01-03T00:00:00+00:00", "task-a");
let token = encode(&k, unfiltered(), Some("did:key:z6MkAlice"), &pos);
assert_eq!(
decode(&k, unfiltered(), Some("z6MkAlice"), &token).unwrap(),
pos
);
// A different method sharing the base58 tail is a different principal.
assert!(decode(&k, unfiltered(), Some("did:web:z6MkAlice"), &token).is_err());
}
/// The filter binding is on identity, not spelling: `assignee_did` matches
/// normalized keys in SQL (`normalize_owner_key`), so minting under
/// `did:key:X` and resuming under bare `X` (or vice versa) must succeed,
/// while `did:web:X` remains distinct.
#[test]
fn assignee_filter_binding_is_normalized_across_did_key_spellings() {
let k = key();
let pos = TaskPosition::new("2026-01-03T00:00:00+00:00", "task-a");
let filter_did_key = TaskFilter {
status: Some("pending"),
assignee_did: Some("did:key:z6MkAssignee"),
};
let filter_bare = TaskFilter {
status: Some("pending"),
assignee_did: Some("z6MkAssignee"),
};
let filter_web = TaskFilter {
status: Some("pending"),
assignee_did: Some("did:web:z6MkAssignee"),
};
let token_from_did_key = encode(&k, filter_did_key, None, &pos);
assert_eq!(
decode(&k, filter_bare, None, &token_from_did_key).unwrap(),
pos
);
assert!(decode(&k, filter_web, None, &token_from_did_key).is_err());
let token_from_bare = encode(&k, filter_bare, None, &pos);
assert_eq!(
decode(&k, filter_did_key, None, &token_from_bare).unwrap(),
pos
);
assert!(decode(&k, filter_web, None, &token_from_bare).is_err());
}
/// Anonymous is a distinct binding from a caller whose normalized DID is
/// the empty string, the same way `Some("")` is distinct from `None` for
/// the filter fields.
#[test]
fn distinguishes_anonymous_from_empty_caller() {
let k = key();
let pos = TaskPosition::new("2026-01-03T00:00:00+00:00", "task-a");
let token = encode(&k, unfiltered(), None, &pos);
assert!(decode(&k, unfiltered(), Some(""), &token).is_err());
}
/// Length-prefixing must stop a status/assignee pair from being re-split.
#[test]
fn rejects_field_boundary_shift() {
let k = key();
let pos = TaskPosition::new("2026-01-03T00:00:00+00:00", "task-a");
let token = encode(
&k,
TaskFilter {
status: Some("pend"),
assignee_did: Some("ing"),
},
None,
&pos,
);
assert!(decode(
&k,
TaskFilter {
status: Some("pending"),
assignee_did: Some("")
},
None,
&token
)
.is_err());
}
#[test]
fn rejects_expired_token() {
let k = key();
// Minted the same way `encode` does, but already past its expiry.
let mut payload = serde_json::to_vec(&CursorPayload {
t: "2026-01-03T00:00:00+00:00",
i: "task-a",
e: chrono::Utc::now().timestamp() - 1,
})
.unwrap();
let iv = siv(&k, unfiltered(), None, &payload);
apply_keystream(&k, &iv, &mut payload);
let expired = format!(
"v1.{}.{}",
URL_SAFE_NO_PAD.encode(iv),
URL_SAFE_NO_PAD.encode(&payload)
);
let err = decode(&k, unfiltered(), None, &expired).unwrap_err();
assert!(err.to_string().contains(INVALID_CURSOR));
}
#[test]
fn rejects_malformed_shapes() {
let k = key();
for bad in [
"",
"v1",
"v1.",
"v1.abc",
"v2.abc.def",
"v1.abc.def.ghi",
"v1.!!!.def",
] {
assert!(
decode(&k, unfiltered(), None, bad).is_err(),
"must reject {bad:?}"
);
}
}
/// Every rejection path renders the same text, so a caller cannot tell a
/// forged token from an expired one from a filter mismatch.
#[test]
fn every_rejection_is_indistinguishable() {
let k = key();
for bad in ["v1.abc.def", "not-a-cursor", "v1..", "v9.a.b"] {
assert_eq!(
decode(&k, unfiltered(), None, bad).unwrap_err().to_string(),
format!("invalid request: {INVALID_CURSOR}")
);
}
}
}
File diff suppressed because it is too large. Load diff
+4
View File
@@ -634,10 +634,12 @@ mod tests {
.connect_lazy("postgres://localhost/gitlawb_test_placeholder")
.expect("lazy pool creation should not fail");
let db = Arc::new(crate::db::Db::for_testing(pool.clone()));
let task_cursor_key = crate::api::task_cursor::TaskCursorKey::derive(&keypair.to_seed());
let schema = Arc::new(graphql::build_schema(
db.clone(),
ref_tx.clone(),
task_tx.clone(),
task_cursor_key.clone(),
));
crate::state::AppState {
config: Arc::new(Config::parse_from(["gitlawb-node"])),
@@ -649,6 +651,7 @@ mod tests {
ref_update_tx: ref_tx,
task_event_tx: task_tx,
graphql_schema: schema,
task_cursor_key,
machine_id: None,
repo_store: crate::git::repo_store::RepoStore::for_testing(PathBuf::from("/tmp"), pool),
rate_limiter: RateLimiter::new(100, Duration::from_secs(60)),
@@ -684,6 +687,7 @@ mod tests {
git_ipfs_walk_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_ipfs_walk_per_caller:
crate::rate_limit::PerCallerConcurrency::with_default_max_keys(16),
task_read_rate_limiter: RateLimiter::new(1200, Duration::from_secs(3600)),
git_bin: "git".to_string(),
}
}
+124 -8
View File
@@ -11,6 +11,8 @@ use uuid::Uuid;
use crate::db::RefCertificate;
use crate::state::AppState;
/// Issue a signed ref-update certificate for a successful push. The
/// live receive-pack handler calls this on every successful push.
/// Build the v1 signing payload — the EXACT bytes the node signs and
/// gl's `verify_signature` reconstructs. This is the single source
/// of truth shared by `issue_ref_certificate` and the frozen
@@ -81,8 +83,36 @@ pub(crate) fn v2_signing_payload(
/// Issue a signed ref-update certificate for a successful push.
///
/// Builds a canonical JSON payload, signs it with the node's Ed25519 key,
/// persists the certificate, and returns it.
/// `cert_id` is the deterministic id derived from `(request_id,
/// ref_name)` (see [`crate::db::ref_cert_id_for`]). It is required so
/// the recovery drain and the live handler produce the same primary
/// key: a live push followed by a recovery pass collapses to a
/// single cert row, and a re-push to the same `(repo, ref)` updates
/// the existing row's `old_sha` / `new_sha` / `pusher_did` /
/// `issued_at` / `signature` to the new transition while preserving
/// the original `id` (the `insert_ref_certificate` upsert is
/// keyed on `(repo_id, ref_name)` and only updates fields when the
/// new `issued_at` is strictly greater).
///
/// #26 Split PR 1 P1-B: the live handler routes through this
/// function (the upsert). After the reviewer-1 round-2 fix, the
/// recovery drain also routes through this function (P1: refresh a
/// stale cert), so both paths use the same deterministic `cert_id`
/// and the same upsert. A re-pass is always safe:
///
/// - Live handler → live upsert: re-push updates the row, preserves
/// the original `id`. The contract pinned by
/// `insert_ref_certificate_upserts_on_repo_ref` is restored.
/// - Live handler → recovery: live's `ON CONFLICT (id) DO UPDATE`
/// preserves the original `id`; the recovery's same upsert is
/// a no-op for an equal-`issued_at` re-run and a refresh for a
/// strictly-newer one.
/// - Recovery → live handler: the recovery wrote a row with the
/// deterministic `id`; the live upsert (which preserves `id` and
/// only updates other fields when `issued_at` is strictly newer)
/// is a no-op for an equal-`issued_at` re-run and a refresh for
/// a strictly-newer one.
#[allow(dead_code)] // round-trip test in db/mod.rs pins the upsert contract; the live path and the drain use issue_ref_certificate_with_issued_at
pub async fn issue_ref_certificate(
state: &AppState,
repo_id: &str,
@@ -90,9 +120,92 @@ pub async fn issue_ref_certificate(
old_sha: &str,
new_sha: &str,
pusher_did: &str,
cert_id: &str,
) -> Result<RefCertificate> {
issue_ref_certificate_with_issued_at(
state, repo_id, ref_name, old_sha, new_sha, pusher_did, cert_id, None,
)
.await
}
/// #26 Split PR 1 round 4 — variant that lets the caller stamp the
/// cert's `issued_at` with a transition-time timestamp instead of
/// `Utc::now()`. The recovery drain passes the persisted
/// `row.created_at` so a replay after a later live cert does not
/// outrank the live cert in the `EXCLUDED.issued_at >
/// ref_certificates.issued_at` upsert guard.
///
/// The live handler uses the default `issue_ref_certificate` (no
/// override), which keeps `Utc::now()` — the reviewer's invariant
/// is that `issued_at` reflects the transition time, and for a
/// live push the transition time and the wall-clock are the same.
///
/// `issued_at_override` is honored verbatim; passing a value not in
/// RFC 3339 form is a logic bug (the upsert will mis-order), so
/// callers must use the row's persisted `created_at`.
///
/// # Clippy allow — too many arguments
/// This is the explicit "stamp a transition-time `issued_at`"
/// variant of `issue_ref_certificate`. The drain
/// (`durable_outbox::derive_one`) is the in-crate caller; the
/// test `replay_of_stale_row_does_not_overwrite_live_cert_b` pins
/// the contract that a recovery replay's `issued_at` does NOT
/// outrank a later live cert. Adding a struct-arg would be a
/// larger refactor for two callers (live + drain) and obscure the
/// parallel to `issue_ref_certificate` (which is `#[allow]`'d for
/// the same reason historically).
#[allow(clippy::too_many_arguments)]
pub async fn issue_ref_certificate_with_issued_at(
state: &AppState,
repo_id: &str,
ref_name: &str,
old_sha: &str,
new_sha: &str,
pusher_did: &str,
cert_id: &str,
issued_at_override: Option<String>,
) -> Result<RefCertificate> {
let cert = build_ref_certificate(
state,
repo_id,
ref_name,
old_sha,
new_sha,
pusher_did,
Some(cert_id.to_string()),
issued_at_override,
)
.await?;
state.db.insert_ref_certificate(&cert).await
}
/// Shared cert construction: build the JSON payload, sign it with the
/// node key, and assemble the `RefCertificate` row. `cert_id_override`
/// lets the recovery path plug in a deterministic id; the live path
/// passes `None` and gets a fresh UUID. `issued_at_override` lets
/// the recovery path stamp the cert with the original transition
/// time so the upsert's `issued_at > issued_at` guard correctly
/// orders transitions regardless of write order.
#[allow(clippy::too_many_arguments)]
async fn build_ref_certificate(
state: &AppState,
repo_id: &str,
ref_name: &str,
old_sha: &str,
new_sha: &str,
pusher_did: &str,
cert_id_override: Option<String>,
issued_at_override: Option<String>,
) -> Result<RefCertificate> {
let node_did = state.node_did.to_string();
let issued_at = Utc::now().to_rfc3339();
// P1 (reviewer-1 round 4): when the caller passes a transition-
// time `issued_at` (the recovery drain passes `row.created_at`),
// use it verbatim so the upsert's per-column guard
// `EXCLUDED.issued_at > ref_certificates.issued_at` correctly
// orders transitions regardless of write order. The live handler
// passes `None` and gets `Utc::now()` — for a live push the
// transition time and the wall-clock are the same.
let issued_at = issued_at_override.unwrap_or_else(|| Utc::now().to_rfc3339());
// Build the canonical signing payload via the shared builder so
// the frozen-vector test below and the live signer cannot drift
@@ -104,8 +217,9 @@ pub async fn issue_ref_certificate(
let signature = state.node_keypair.sign_b64(&payload_bytes);
let id = cert_id_override.unwrap_or_else(|| Uuid::new_v4().to_string());
let cert = RefCertificate {
id: Uuid::new_v4().to_string(),
id,
repo_id: repo_id.to_string(),
ref_name: ref_name.to_string(),
old_sha: old_sha.to_string(),
@@ -376,14 +490,16 @@ mod v1_payload_tests {
let old = "0".repeat(40);
let new = "a".repeat(40);
let pusher = "did:key:z6MkPusher";
let cert_id = "cert-issuer-observe";
// Drive the live issuer and read the persisted row back by
// id, so the test covers the stored `version` column and
// not just the in-process return value.
let cert =
crate::cert::issue_ref_certificate(&state, repo_id, ref_name, &old, &new, pusher)
.await
.expect("issue_ref_certificate must succeed");
let cert = crate::cert::issue_ref_certificate(
&state, repo_id, ref_name, &old, &new, pusher, cert_id,
)
.await
.expect("issue_ref_certificate must succeed");
// The cert returned by the issuer is the source of truth
// for both the version claim and the signed bytes.
+204
View File
@@ -152,6 +152,57 @@ pub struct Config {
#[arg(long, env = "GITLAWB_AUTO_SYNC", default_value_t = false)]
pub auto_sync: bool,
/// Periodically fetch configured INBOUND repositories from their external
/// HTTPS upstream. Disabled by default: enabling this also requires
/// `GITLAWB_ENFORCE_OWNER_PUSH=true`, so an upstream-authoritative mirror
/// cannot accept writes from arbitrary signed identities between fetches.
#[arg(long, env = "GITLAWB_UPSTREAM_MIRROR_ENABLED", default_value_t = false)]
pub upstream_mirror_enabled: bool,
/// Pause between complete upstream-mirror scans. A scan is sequential and
/// paginated; the pause begins only after the prior scan has finished, so a
/// slow forge cannot create overlapping worker cycles.
#[arg(
long,
env = "GITLAWB_UPSTREAM_MIRROR_INTERVAL_SECS",
default_value_t = 300,
value_parser = clap::value_parser!(u64).range(5..=86_400)
)]
pub upstream_mirror_interval_secs: u64,
/// Maximum wall-clock time for one external `git fetch`. The child runs in
/// its own process group and is fully reaped on timeout. The preflight
/// URL-rewrite and local-config probes run on their own short bounds and
/// do not consume this budget.
#[arg(
long,
env = "GITLAWB_UPSTREAM_MIRROR_FETCH_TIMEOUT_SECS",
default_value_t = 600,
value_parser = clap::value_parser!(u64).range(1..=GIT_SERVICE_TIMEOUT_SECS_MAX)
)]
pub upstream_mirror_fetch_timeout_secs: u64,
/// Number of durable INBOUND rows read per database page. Every page is
/// processed before the scan sleeps; this bounds query memory without
/// permanently starving repositories beyond the first page.
#[arg(
long,
env = "GITLAWB_UPSTREAM_MIRROR_PAGE_SIZE",
default_value_t = 100,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_000)
)]
pub upstream_mirror_page_size: usize,
/// Enable the periodic reconciliation sweep that re-derives pin/seal sets
/// and fills durability gaps. Defaults to true; set to false to disable
/// the sweep even when a pin backend (IPFS/Pinata) is configured.
#[arg(
long,
env = "GITLAWB_RECONCILIATION_SWEEP",
default_value_t = true,
action = clap::ArgAction::Set
)]
pub reconciliation_sweep: bool,
/// Irys URL for Arweave permanent anchoring.
/// Leave empty to disable. Use https://devnet.irys.xyz for free devnet.
#[arg(long, env = "GITLAWB_IRYS_URL", default_value = "")]
@@ -233,6 +284,29 @@ pub struct Config {
#[arg(long, env = "GITLAWB_SHUTDOWN_GRACE_SECS", default_value_t = 30)]
pub shutdown_grace_secs: u64,
/// Maximum number of creation requests (repo create, issue create, PR
/// create, register, fork) allowed per DID within the rate-limit window.
/// 0 is rejected at parse time: it would silently disable the limiter
/// (RateLimiter treats max_requests == 0 as allow-all), not block every
/// creation request.
#[arg(
long,
env = "GITLAWB_RATE_LIMIT_MAX",
default_value_t = 10,
value_parser = clap::value_parser!(u32).range(1..)
)]
pub rate_limit_max_requests: u32,
/// Rate-limit window size, in seconds, over which
/// `rate_limit_max_requests` applies. 0 is rejected at parse time: every
/// timestamp would prune immediately, silently disabling the limiter.
#[arg(
long,
env = "GITLAWB_RATE_LIMIT_WINDOW_SECS",
default_value_t = 3600,
value_parser = clap::value_parser!(u64).range(1..)
)]
pub rate_limit_window_secs: u64,
/// Maximum wall-clock time a single served git operation (upload-pack /
/// receive-pack through `run_git_service`) may run before it is aborted and
/// its process group torn down, in seconds. Bounds a git that neither
@@ -762,6 +836,60 @@ pub struct Config {
value_parser = clap::builder::RangedU64ValueParser::<u64>::new().range(0..=86_400)
)]
pub pin_repair_sweep_delay_secs: u64,
/// #26 Split PR 1 step 4 — receive-pack queue retention window.
/// Terminal `complete` and `rejected_at_git` rows older than
/// this are eligible for the periodic purge. `quarantined`
/// rows are never purged on a timer. The v30 partial index
/// `idx_receive_pack_requests_completed_at` keeps the scan
/// cheap regardless of the value.
#[arg(
long,
env = "GITLAWB_QUEUE_RETENTION_DAYS",
default_value_t = 7,
value_parser = clap::builder::RangedU64ValueParser::<i64>::new().range(1..=365)
)]
pub queue_retention_days: i64,
/// #26 Split PR 1 step 4 — receive-pack queue purge batch size.
/// Each periodic purge pass deletes at most this many terminal
/// rows per batch. The drain and the purge share the same
/// `DRAIN_PER_PASS_LIMIT` budget; see the spawn function in
/// `main.rs` for the wiring.
#[arg(
long,
env = "GITLAWB_QUEUE_PURGE_BATCH",
default_value_t = 1000,
value_parser = clap::builder::RangedU64ValueParser::<i64>::new().range(1..=100_000)
)]
pub queue_purge_batch: i64,
/// #26 Split PR 1 step 5 — effects-executor retry bound. The
/// drain flips a request to `quarantined` after this many
/// `EffectsOutcome::Retry` returns, closing the
/// infinite-retry DoS window. Default 8 matches the spec; set
/// lower in tests.
#[arg(
long,
env = "GITLAWB_EFFECTS_MAX_ATTEMPTS",
default_value_t = 8,
value_parser = clap::builder::RangedU64ValueParser::<i32>::new().range(1..=1000)
)]
pub effects_max_attempts: i32,
/// Per-client-IP rate limit for the anonymous task read routes
/// (`GET /api/v1/tasks`, `GET /api/v1/tasks/{id}`), in requests per hour.
/// Both are publicly reachable (`optional_signature`). `GET /api/v1/tasks`
/// runs `collect_visible_tasks` and returns a visibility-filtered page;
/// `GET /api/v1/tasks/{id}` runs `get_visible_task` and returns an opaque 404
/// when the task is hidden or missing. These reads can require task,
/// repository, and visibility-rule queries even when no task is returned,
/// so the brake bounds the cost of anonymous probes. Keyed on the resolved
/// client IP via `GITLAWB_TRUSTED_PROXY`. `0` disables. Default: 1200 (a list page
/// followed by per-task reads is a normal client pattern, so this sits above
/// the `/ipfs` budget). GraphQL `tasks` / `task` queries, including WebSocket
/// operations, share this per-IP budget with REST task reads.
#[arg(long, env = "GITLAWB_TASK_READ_RATE_LIMIT", default_value_t = 1200)]
pub task_read_rate_limit: usize,
}
impl Config {
@@ -832,6 +960,13 @@ impl Config {
Self::DB_LOCK_POOL_NON_PUSH_HEADROOM
));
}
if self.upstream_mirror_enabled && !self.enforce_owner_push {
return Err(
"GITLAWB_UPSTREAM_MIRROR_ENABLED requires GITLAWB_ENFORCE_OWNER_PUSH=true: \
an upstream-authoritative mirror must not accept writes from arbitrary signed identities"
.to_string(),
);
}
let main_floor = Self::DB_POOL_APP_HEADROOM as u64;
if (self.db_max_connections as u64) < main_floor {
return Err(format!(
@@ -1031,6 +1166,51 @@ pub(crate) fn validate_web_url(raw: &str) -> Result<(), String> {
mod tests {
use super::*;
#[test]
fn upstream_mirror_worker_is_default_off_and_bounded() {
let mut config = Config::parse_from(["gitlawb-node"]);
assert!(!config.upstream_mirror_enabled);
assert_eq!(config.upstream_mirror_interval_secs, 300);
assert_eq!(config.upstream_mirror_fetch_timeout_secs, 600);
assert_eq!(config.upstream_mirror_page_size, 100);
config.validate().unwrap();
assert!(
Config::try_parse_from(["gitlawb-node", "--upstream-mirror-interval-secs", "4",])
.is_err()
);
assert!(
Config::try_parse_from(["gitlawb-node", "--upstream-mirror-page-size", "0",]).is_err()
);
}
#[test]
fn upstream_mirror_worker_requires_owner_push_enforcement() {
// enforce_owner_push defaults true on main, so the only unsafe shape
// is the explicit opt-out: mirror enabled plus enforcement refused.
let mut unsafe_config = Config::parse_from([
"gitlawb-node",
"--upstream-mirror-enabled",
"--enforce-owner-push",
"false",
]);
let error = unsafe_config.validate().unwrap_err();
assert!(error.contains("GITLAWB_ENFORCE_OWNER_PUSH=true"));
Config::parse_from([
"gitlawb-node",
"--upstream-mirror-enabled",
"--enforce-owner-push",
"true",
])
.validate()
.unwrap();
// The default posture alone also validates: enforcement is on.
Config::parse_from(["gitlawb-node", "--upstream-mirror-enabled"])
.validate()
.unwrap();
}
#[test]
fn lock_pool_size_defaults_to_40_and_rejects_zero() {
assert_eq!(
@@ -1259,6 +1439,30 @@ mod tests {
);
}
#[test]
fn queue_lifecycle_knobs_default_conservatively() {
let c = Config::parse_from(["gitlawb-node"]);
// 7-day retention matches the v30 partial index comment
// and the spec at .gravirei/plans/state-model-durable-post-receive.md.
assert_eq!(c.queue_retention_days, 7);
// 1000 rows per pass matches DRAIN_PER_PASS_LIMIT in durable_outbox.
assert_eq!(c.queue_purge_batch, 1000);
assert_eq!(
Config::parse_from(["gitlawb-node", "--queue-retention-days", "30"])
.queue_retention_days,
30
);
assert!(Config::try_parse_from(["gitlawb-node", "--queue-retention-days", "0"]).is_err());
assert!(Config::try_parse_from(["gitlawb-node", "--queue-retention-days", "366"]).is_err());
assert_eq!(
Config::parse_from(["gitlawb-node", "--queue-purge-batch", "500"]).queue_purge_batch,
500
);
assert!(Config::try_parse_from(["gitlawb-node", "--queue-purge-batch", "0"]).is_err());
}
#[test]
fn ipfs_walk_per_source_defaults_and_rejects_out_of_range() {
assert_eq!(Config::parse_from(["gitlawb-node"]).ipfs_walk_per_source, 4);
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+387 -2
View File
@@ -6,6 +6,7 @@
use std::collections::{BTreeSet, HashMap};
use std::path::Path;
use std::str::FromStr;
use std::time::Duration;
use ed25519_dalek::VerifyingKey;
use gitlawb_core::did::Did;
@@ -127,17 +128,60 @@ fn plan_seal(node_seed: &[u8; 32], dids: &BTreeSet<String>, stored_tag: Option<&
/// `node_seed` keys the opaque recipients tag. Returns `(oid, cid)` for each blob
/// actually sealed and recorded this call (the per-push delta), used by Option B3
/// to anchor a manifest. Recipient identities are never stored or returned.
///
/// Nine args (the fence joins the seal's eight) but grouping them would churn
/// both callers and the race/hung-git tests for no behavioral gain.
#[allow(clippy::too_many_arguments)]
pub async fn encrypt_and_pin(
ipfs_api: &str,
repo_path: &Path,
db: &Db,
repo_id: &str,
node_seed: &[u8; 32],
git_bin: &str,
batch_budget: Duration,
recipients: &HashMap<String, BTreeSet<String>>,
fence: Option<&crate::ipfs_pin::PolicyFence>,
) -> Vec<(String, String)> {
let mut sealed = Vec::new();
let mut skipped_unresolvable = 0usize;
for (oid, dids) in recipients {
// One shared read deadline for the whole batch, like `pin_new_objects`: a
// hung git child is watchdog-reaped at this bound, so the outer
// `PIN_PHASE_DEADLINE` timeout cannot be held open by a blocking read
// (R1-P2). Each read runs under `spawn_blocking` — it is synchronous child
// spawn + pipe drain + watchdog join.
let read_deadline = std::time::Instant::now() + batch_budget;
let total = recipients.len();
for (attempted, (oid, dids)) in recipients.iter().enumerate() {
// Batch budget gate (R2-P3), mirroring the public pin loops: an object
// is never started with a remainder too small to cover a bounded read's
// teardown. This is consistency (the seal is bounded by the outer
// `PIN_PHASE_DEADLINE` either way), but it keeps the three loops from
// drifting apart in how they report a truncated batch.
if crate::ipfs_pin::batch_budget_gate(
"encrypted-seal",
read_deadline,
sealed.len(),
total - attempted,
)
.is_none()
{
break;
}
// Policy fence (R1-P1): the recipients snapshot was derived before the
// long withheld-blob walk; if a visibility rule moved while that walk
// ran (a reader added or removed), stop sealing instead of pinning to a
// stale recipient set. Checked FIRST so a changed policy costs nothing.
if let Some(f) = fence {
if !f.is_current().await {
tracing::warn!(
repo = %f.repo_id(),
oid = %oid,
"visibility policy changed after the recipients snapshot; stopping the seal loop"
);
break;
}
}
// A DB read failure is not a cache miss: re-sealing here would do an
// avoidable IPFS write during a partial outage. Skip and retry next push.
let stored_tag = match db.encrypted_blob_recipients_tag(repo_id, oid).await {
@@ -188,7 +232,9 @@ pub async fn encrypt_and_pin(
}
SealPlan::Seal { keys, tag } => (keys, tag),
};
let data = match crate::git::store::read_object(repo_path, oid) {
let data = match read_object_bounded_spawn_blocking(git_bin, repo_path, oid, read_deadline)
.await
{
Ok(Some((_t, bytes))) => bytes,
Ok(None) => {
tracing::warn!(oid = %oid, "git object not found; skipping encrypted pin");
@@ -206,6 +252,22 @@ pub async fn encrypt_and_pin(
continue;
}
};
// Dispatch fence (R1-P1): re-read the policy epoch immediately before
// the irreversible HTTP POST. The iteration-top check catches a narrow
// that landed before work began; THIS check catches a narrow that landed
// during the tag lookup, recipient resolution, git read, or seal — all
// of which can take seconds. Without this, a reader removed during
// preparation can still receive a newly published envelope.
if let Some(f) = fence {
if !f.is_current().await {
tracing::warn!(
repo = %f.repo_id(),
oid = %oid,
"visibility policy changed during encrypted seal preparation; aborting upload"
);
break;
}
}
let cid = match crate::ipfs_pin::pin_git_object(ipfs_api, oid, &envelope, None).await {
Ok(c) if !c.is_empty() => c,
Ok(_) => {
@@ -237,10 +299,33 @@ pub async fn encrypt_and_pin(
sealed
}
/// Bounded, reaped git object read for the seal loop, run off the async thread:
/// `read_object_bounded` is synchronous child spawn + pipe drain + watchdog
/// join, so blocking the runtime task on it would let a hung git hold a worker
/// thread (R1-P2). The `deadline` is the batch's shared read deadline; a child
/// still alive at it is SIGTERM/SIGKILL group-reaped by the watchdog.
async fn read_object_bounded_spawn_blocking(
git_bin: &str,
repo_path: &Path,
sha256_hex: &str,
deadline: std::time::Instant,
) -> anyhow::Result<Option<(String, Vec<u8>)>> {
let git_bin = git_bin.to_string();
let repo_path = repo_path.to_path_buf();
let sha256_hex = sha256_hex.to_string();
tokio::task::spawn_blocking(move || {
crate::git::store::read_object_bounded(&git_bin, &repo_path, &sha256_hex, deadline)
.map_err(anyhow::Error::from)
})
.await
.map_err(|e| anyhow::anyhow!("read_object spawn_blocking join failed: {e}"))?
}
#[cfg(test)]
mod tests {
use super::*;
use ed25519_dalek::SigningKey;
use std::time::Duration;
/// A did:key encoding a small-order point. Well-formed, indistinguishable
/// from a real one by eye, and refused at resolution since the choke-point
@@ -475,4 +560,304 @@ mod tests {
other => panic!("changed recipient set must re-seal; got {other:?}"),
}
}
/// A reader removed mid-seal must stop the seal loop (R1-P1 "race test for
/// reader removal"): `encrypt_and_pin` re-checks the policy fence before
/// each blob, so a `remove_visibility_rule` landing while the first seal is
/// in flight aborts before a later blob is pinned to a stale recipient set.
#[sqlx::test]
async fn encrypt_and_pin_stops_sealing_when_reader_removed_mid_batch(pool: sqlx::PgPool) {
let db = crate::db::Db::for_testing(pool);
db.run_migrations().await.expect("migrations");
let tmp = tempfile::TempDir::new().unwrap();
let repo_path = tmp.path().join("seal-race.git");
// Three loose blobs, each withheld (path-scoped deny exists so the sweep
// would have derived recipients for them).
let oids: Vec<String> = {
crate::git::store::init_bare(&repo_path).expect("init bare repo");
(0..3)
.map(|i| {
let mut cmd = std::process::Command::new("git");
cmd.args(["hash-object", "-w", "--stdin"])
.current_dir(&repo_path)
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::piped());
let mut child = cmd.spawn().expect("spawn git hash-object");
{
use std::io::Write;
child
.stdin
.as_mut()
.expect("stdin")
.write_all(format!("secret blob {i}\n").as_bytes())
.expect("write stdin");
}
let out = child.wait_with_output().expect("hash-object output");
assert!(out.status.success());
String::from_utf8_lossy(&out.stdout).trim().to_string()
})
.collect()
};
// A real repos row so the fence has an epoch and a reader can be removed.
let now = chrono::Utc::now();
let repo_id = uuid::Uuid::new_v4().to_string();
db.create_repo(&crate::db::RepoRecord {
id: repo_id.clone(),
name: "seal-race-repo".into(),
owner_did: "did:key:zSealRaceOwner".into(),
description: None,
is_public: true,
default_branch: "main".into(),
created_at: now,
updated_at: now,
disk_path: repo_path.display().to_string(),
forked_from: None,
machine_id: None,
})
.await
.expect("create repo");
// A rule whose removal is the "reader removed" mutation: one reader per
// blob, all under the same path glob.
let reader = did_key(1);
db.set_visibility_rule(
&repo_id,
"**/secret/*",
crate::db::VisibilityMode::B,
std::slice::from_ref(&reader),
"did:key:zSealRaceOwner",
)
.await
.expect("set rule");
// IPFS endpoint that delays the FIRST add 2s so the removal lands while
// that seal is in flight, then answers immediately.
let endpoint = delaying_cid_endpoint(vec![Duration::from_secs(2)]).await;
let recipients: HashMap<String, BTreeSet<String>> = oids
.iter()
.cloned()
.map(|oid| {
let mut s = BTreeSet::new();
s.insert(reader.clone());
(oid, s)
})
.collect();
let fence = crate::ipfs_pin::PolicyFence::capture(&db, &repo_id)
.await
.expect("fence captures");
let sealed = tokio::time::timeout(Duration::from_secs(30), async {
let seal_db = db.clone();
let seal_repo = repo_path.clone();
let seal_endpoint = endpoint.clone();
let seal_repo_id = repo_id.clone();
let handle = tokio::spawn(async move {
encrypt_and_pin(
&seal_endpoint,
&seal_repo,
&seal_db,
&seal_repo_id,
&SEED,
"git",
Duration::from_secs(60),
&recipients,
Some(&fence),
)
.await
});
// Let the first add start (endpoint sleeps 2s), then remove the
// reader so the fence is stale before the loop checks again.
// The window has to cover blob 1's encrypt phase: if the removal
// lands before the loop reaches dispatch, the fence aborts with
// nothing sealed, so give the in-flight add room to start under
// a loaded box while staying inside the 2s endpoint delay.
tokio::time::sleep(Duration::from_millis(1200)).await;
db.remove_visibility_rule(&repo_id, "**/secret/*")
.await
.expect("remove rule");
handle.await.expect("seal task")
})
.await
.expect("wedge guard: the fence abort must not take 30s");
assert!(
sealed.len() < oids.len(),
"a reader removal landing mid-batch must abort before every blob is sealed: {}",
sealed.len()
);
assert!(
!sealed.is_empty(),
"at least the blob already in flight before the removal completes"
);
}
/// A hung git must not hold the seal loop past its read budget (R1-P2): the
/// git read runs under `spawn_blocking` against `read_object_bounded`, so
/// the watchdog reaps a wedged child at the batch deadline and the loop
/// keeps its shape instead of blocking a runtime worker indefinitely.
#[cfg(unix)]
#[sqlx::test]
async fn encrypt_and_pin_returns_by_budget_with_a_hung_git(pool: sqlx::PgPool) {
use std::os::unix::fs::PermissionsExt;
let db = crate::db::Db::for_testing(pool);
db.run_migrations().await.expect("migrations");
let tmp = tempfile::TempDir::new().unwrap();
let repo_path = tmp.path().join("seal-hung.git");
let oids: Vec<String> = {
crate::git::store::init_bare(&repo_path).expect("init bare repo");
(0..2)
.map(|i| {
let mut cmd = std::process::Command::new("git");
cmd.args(["hash-object", "-w", "--stdin"])
.current_dir(&repo_path)
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::piped());
let mut child = cmd.spawn().expect("spawn git hash-object");
{
use std::io::Write;
child
.stdin
.as_mut()
.expect("stdin")
.write_all(format!("secret blob {i}\n").as_bytes())
.expect("write stdin");
}
let out = child.wait_with_output().expect("hash-object output");
assert!(out.status.success());
String::from_utf8_lossy(&out.stdout).trim().to_string()
})
.collect()
};
// A git that wedges forever, ignoring SIGTERM, so only the watchdog's
// SIGKILL can reap it.
let fake = tmp.path().join("hanging-git");
std::fs::write(&fake, "#!/bin/sh\ntrap '' TERM\necho $$ > pid\nsleep 30\n").unwrap();
let mut perm = std::fs::metadata(&fake).unwrap().permissions();
perm.set_mode(0o755);
std::fs::set_permissions(&fake, perm).unwrap();
let now = chrono::Utc::now();
let repo_id = uuid::Uuid::new_v4().to_string();
db.create_repo(&crate::db::RepoRecord {
id: repo_id.clone(),
name: "seal-hung-repo".into(),
owner_did: "did:key:zSealHungOwner".into(),
description: None,
is_public: true,
default_branch: "main".into(),
created_at: now,
updated_at: now,
disk_path: repo_path.display().to_string(),
forked_from: None,
machine_id: None,
})
.await
.expect("create repo");
db.set_visibility_rule(
&repo_id,
"**/secret/*",
crate::db::VisibilityMode::B,
&[did_key(1)],
"did:key:zSealHungOwner",
)
.await
.expect("set rule");
let recipients: HashMap<String, BTreeSet<String>> = oids
.iter()
.cloned()
.map(|oid| {
let mut s = BTreeSet::new();
s.insert(did_key(1));
(oid, s)
})
.collect();
// Unreachable endpoint: even if a read somehow succeeded, the pin would
// fail; the read itself is the thing under test.
let started = std::time::Instant::now();
let sealed = tokio::time::timeout(
Duration::from_secs(60),
encrypt_and_pin(
"http://127.0.0.1:9",
&repo_path,
&db,
&repo_id,
&SEED,
fake.to_str().unwrap(),
Duration::from_secs(2),
&recipients,
None,
),
)
.await
.expect("a hung git must not hold the seal past the outer wedge guard");
let elapsed = started.elapsed();
assert!(
elapsed < Duration::from_secs(10),
"a hung git must be watchdog-reaped inside the read budget, not block the loop for ~10s+ (took {elapsed:?})"
);
assert!(
sealed.is_empty(),
"with a hung git no blob can be read, so nothing may be reported sealed"
);
}
/// Local TCP endpoint that answers `{ "Hash": "QmMock" }` after an optional
/// per-request delay, so a seal can be made to straddle a policy mutation.
async fn delaying_cid_endpoint(delays: Vec<Duration>) -> String {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let endpoint = format!("http://{}", listener.local_addr().unwrap());
tokio::spawn(async move {
let mut seen = 0usize;
while let Ok((mut sock, _)) = listener.accept().await {
let delay = *delays
.get(seen)
.or_else(|| delays.last())
.unwrap_or(&Duration::ZERO);
seen += 1;
tokio::spawn(async move {
let mut acc = Vec::new();
let mut buf = [0u8; 4096];
loop {
let n = match sock.read(&mut buf).await {
Ok(0) | Err(_) => break,
Ok(n) => n,
};
acc.extend_from_slice(&buf[..n]);
if let Some(hdr_end) =
acc.windows(4).position(|w| w == b"\r\n\r\n").map(|p| p + 4)
{
let headers = String::from_utf8_lossy(&acc[..hdr_end]).to_lowercase();
let len: usize = headers
.lines()
.find_map(|l| l.strip_prefix("content-length:"))
.and_then(|v| v.trim().parse().ok())
.unwrap_or(0);
if acc.len() >= hdr_end + len {
break;
}
}
}
tokio::time::sleep(delay).await;
let body = br#"{"Hash":"bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4"}"#;
let _ = sock
.write_all(
format!("HTTP/1.1 200 OK\r\nContent-Length: {}\r\n\r\n", body.len())
.as_bytes(),
)
.await;
let _ = sock.write_all(body).await;
let _ = sock.flush().await;
});
}
});
endpoint
}
}
+14
View File
@@ -12,6 +12,9 @@ pub enum AppError {
#[error("repo already exists: {0}")]
RepoExists(String),
#[error("conflict: {0}")]
Conflict(String),
#[error("not found: {0}")]
NotFound(String),
@@ -189,6 +192,7 @@ impl IntoResponse for AppError {
"repo_exists",
format!("repository '{r}' already exists"),
),
AppError::Conflict(msg) => (StatusCode::CONFLICT, "conflict", msg.clone()),
AppError::NotFound(msg) => (StatusCode::NOT_FOUND, "not_found", msg.clone()),
AppError::Unauthorized(msg) => (StatusCode::UNAUTHORIZED, "not_an_agent", msg.clone()),
AppError::Forbidden(msg) => (StatusCode::FORBIDDEN, "forbidden", msg.clone()),
@@ -321,6 +325,16 @@ pub type Result<T> = std::result::Result<T, AppError>;
mod tests {
use super::*;
#[test]
fn conflict_maps_to_409() {
assert_eq!(
AppError::Conflict("task not claimable".into())
.into_response()
.status(),
StatusCode::CONFLICT
);
}
#[test]
fn timeout_maps_to_504_distinct_from_git_500() {
assert_eq!(
+15 -8
View File
@@ -185,10 +185,13 @@ fn rev_list_delta(
/// List every object in the repository via
/// `git cat-file --batch-all-objects --batch-check='%(objectname)'`.
///
/// This is the whole-repo enumeration the push path falls back to and the
/// reconciliation sweep relies on. It returns *all* objects (including
/// unreachable/dangling ones), which is what the sweep needs to catch
/// stragglers — do not swap it for a reachability walk.
/// This is the whole-repo enumeration the push path falls back to. It
/// returns *all* objects (including unreachable/dangling ones), which is
/// what the push path needs to catch stragglers — do not swap it for a
/// reachability walk. The reconciliation sweep no longer uses it: sweep
/// candidates come from the windowed walk (window commits plus walked
/// pairs), which are reachable by construction, so dangling objects are
/// absent rather than filtered.
pub fn list_all_objects(repo_path: &Path, git_bin: &str, deadline: Instant) -> Result<Vec<String>> {
let out = crate::git::visibility_pack::run_bounded_git(
git_bin,
@@ -214,6 +217,7 @@ pub fn list_all_objects(repo_path: &Path, git_bin: &str, deadline: Instant) -> R
/// filter needs to tell blobs (content, withholdable) and trees (content,
/// withholdable since #135/#173) from commits/tags (structural, never withheld)
/// without typing the candidate list itself.
#[allow(dead_code)] // used by tests and all_blob_oids
pub fn list_all_objects_with_type(
repo_path: &Path,
git_bin: &str,
@@ -261,6 +265,7 @@ pub fn all_blob_oids(
/// candidate blob or tree absent from the matching reachable visibility-allowed
/// set; a dangling or withheld object is in this universe but not the allowed
/// set, so it never replicates (#99, #172).
#[allow(dead_code)] // production went to the typed sweep path; tests still use it
pub fn all_blob_and_tree_oids(
repo_path: &Path,
git_bin: &str,
@@ -303,10 +308,12 @@ pub struct PinCandidateSet {
/// Every degraded path is **logged**, not silent: a full-scan fallback, a
/// failed full scan, and a panicked blocking task each emit a warning. On a
/// failed full scan or a task panic the candidate set is empty (pin nothing
/// this push); that is a durability gap the reconciliation sweep backstops, and
/// it can never leak because the withheld/fail-closed filter still runs on
/// whatever set is returned. `full_scan` rides on the returned set so the caller
/// knows when the dangling-inclusive filter is required.
/// this push); that is a durability gap the reconciliation sweep backstops
/// when it is enabled and a pin backend is configured (a node running with the
/// sweep disabled or with no IPFS/Pinata backend has no backstop), and it can
/// never leak because the withheld/fail-closed filter still runs on whatever
/// set is returned. `full_scan` rides on the returned set so the caller knows
/// when the dangling-inclusive filter is required.
///
/// `scan_sem` is the post-receive scan admission pool (`git_encrypt_semaphore`,
/// #174 F4): both git-spawning stages — the per-tip `cat-file` probe + delta
+465
View File
@@ -220,6 +220,7 @@ pub fn response_served_pack(output: &[u8]) -> bool {
///
/// Accepts a push. The caller MUST verify HTTP Signature auth before
/// calling this function.
#[allow(dead_code)] // used by tests; production uses receive_pack_raw
pub async fn receive_pack(
git_bin: &str,
repo_path: &Path,
@@ -244,6 +245,219 @@ pub async fn receive_pack(
.body(Body::from(output))?)
}
/// Run `git-receive-pack` and return the raw stdout bytes together with
/// the process exit status. Unlike [`receive_pack`], this does NOT bail
/// on a non-zero exit: the caller needs the stdout (which contains the
/// report-status with per-ref ok/ng results) even when the process
/// exits non-zero. A timeout still returns `Err`.
///
/// `reflog_action` (typically `gitlawb-request:<request_id>`) is
/// exported as `GIT_REFLOG_ACTION` so the ref transaction writes
/// request-bound landing evidence into the reflog. Recovery requires
/// that evidence before auto-promoting a stranded row; without it a
/// later request recreating the same old/new tuple is
/// indistinguishable from the stranded one.
#[allow(dead_code)]
pub async fn receive_pack_raw(
git_bin: &str,
repo_path: &Path,
request_body: Bytes,
timeout: Duration,
admission: Option<AdmissionGuard>,
) -> Result<(Vec<u8>, bool)> {
receive_pack_raw_with_reflog(git_bin, repo_path, request_body, timeout, admission, None).await
}
/// [`receive_pack_raw`] with explicit reflog binding.
pub async fn receive_pack_raw_with_reflog(
git_bin: &str,
repo_path: &Path,
request_body: Bytes,
timeout: Duration,
admission: Option<AdmissionGuard>,
reflog_action: Option<&str>,
) -> Result<(Vec<u8>, bool)> {
let mut command = Command::new(git_bin);
command
.arg("receive-pack")
.arg("--stateless-rpc")
.arg(repo_path);
if let Some(action) = reflog_action {
command.env("GIT_REFLOG_ACTION", action);
}
let (out, err, status, _admission) = drive_git_child_raw(
command,
request_body,
timeout,
"git-receive-pack",
admission,
)
.await?;
// On timeout, drive_git_child_raw returns Err — we never reach here.
// On success/non-zero exit, we have the stdout and exit status.
if !status.success() {
let stderr = String::from_utf8_lossy(&err);
tracing::warn!(stderr = %stderr, "git-receive-pack exited non-zero");
}
Ok((out, status.success()))
}
/// Parse the git-receive-pack report-status output to determine per-ref
/// success/failure. Returns `(unpack_ok, per_ref_results)` where
/// `per_ref_results` is a list of `(ref_name, is_ok)`.
///
/// The report-status format (after the sideband framing) is:
/// ```text
/// unpack ok\n (or: unpack fail\n)
/// ok <refname>\n (per successful ref)
/// ng <refname> <reason>\n (per rejected ref)
/// \n (empty line terminates)
/// ```
///
/// Returns `None` if the output cannot be parsed (e.g. the client did
/// not request report-status, or the output is truncated). In that
/// case the caller should treat all refs as uncertain.
pub fn parse_report_status(output: &[u8]) -> Option<(bool, Vec<(String, bool)>)> {
let text = std::str::from_utf8(output).ok()?;
// Strip sideband framing: each line starts with a pkt-line length
// prefix and a channel byte (1 = stdout, 2 = stderr). The actual
// data starts after the first `0000` flush packet or after we
// strip sideband bytes.
//
// Framing completeness is part of authority, at whichever level the
// framing lives. A double-framed capture carries the report stream
// (itself pkt-line framed, flush-terminated) inside the sideband
// envelope — real git output has been observed with NO outer flush,
// so the outer level cannot require one; the inner frame's flush is
// authoritative there. A single-framed (or unframed) payload has no
// inner frame, so its only completion marker is the outer flush:
// output ending after a complete `unpack ok` / `ok <ref>` packet but
// before the outer `0000` is a truncated report and yields `None`
// (uncertain), not a successful report.
let (stripped, outer_flush) = strip_sideband(text, false)?;
// The report-status vocabulary (`unpack ...`, `ok ...`, `ng ...`)
// never opens with four hex digits, so a hex quartet here is inner
// pkt-line framing, not report text.
let stripped = if stripped.len() >= 4
&& stripped.as_bytes()[..4]
.iter()
.all(|b| b.is_ascii_hexdigit())
{
strip_sideband(&stripped, true)?.0
} else if !outer_flush {
return None;
} else {
stripped
};
// The report is framed TWICE when the client negotiated side-band-64k,
// which `git push` over smart HTTP does — so this is the common case, not
// an exotic one. The outer frame is the side-band envelope; band 1 carries
// the report-status stream, which is ITSELF pkt-line encoded. Real bytes
// from git 2.50.1 rejecting one ref of a two-ref push:
//
// 0057\x01000eunpack ok\n0028ng refs/heads/main non-fast-forward\n...
//
// After one pass the first line reads `000eunpack ok`, which fails the
// `unpack ok` check below and makes this return None — i.e. "no report",
// which the caller treats as inconclusive and keeps every declared ref.
// The per-ref gate would then be inert for exactly the pushes it exists to
// filter. The second pass above already removed the inner pkt-line
// framing where present (and required its flush); it is a no-op
// on the single-framed shape, because plain report text does not begin
// with four hex digits.
let lines: Vec<&str> = stripped.lines().collect();
if lines.is_empty() {
return None;
}
// First non-empty line is "unpack ok" or "unpack fail".
let unpack_line = lines.iter().find(|l| !l.is_empty())?;
let unpack_ok = if unpack_line.starts_with("unpack ok") {
true
} else if unpack_line.starts_with("unpack fail") {
false
} else {
return None;
};
let mut results = Vec::new();
for line in &lines[1..] {
let line = line.trim();
if line.is_empty() {
break;
}
if let Some(rest) = line.strip_prefix("ok ") {
results.push((rest.to_string(), true));
} else if let Some(rest) = line.strip_prefix("ng ") {
// "ng <refname> <reason>" — skip the reason
let ref_name = rest.split_whitespace().next()?.to_string();
results.push((ref_name, false));
}
}
Some((unpack_ok, results))
}
/// Strip git sideband framing from a pkt-line encoded output.
/// Sideband-encoded lines start with a 4-hex-digit length, then a
/// channel byte (0x01=stdout, 0x02=stderr), then payload. Returns the
/// decoded payload plus whether the stream saw its terminating flush
/// packet, or `None` if the framing is malformed. With `require_flush`,
/// a stream that never sees its flush is also `None` — the caller uses
/// this for the level whose completion marker is authoritative (see
/// [`parse_report_status`]).
fn strip_sideband(text: &str, require_flush: bool) -> Option<(String, bool)> {
let mut output = String::new();
let mut pos = 0;
let bytes = text.as_bytes();
let mut saw_flush = false;
loop {
if pos == bytes.len() {
break;
}
if pos + 4 > bytes.len() {
// Truncated framing mid-header: fail closed.
return None;
}
let len_str = std::str::from_utf8(&bytes[pos..pos + 4]).ok()?;
let len = usize::from_str_radix(len_str, 16).ok()?;
if len == 0 {
// Flush packet — end of sideband stream
saw_flush = true;
break;
}
if len < 4 || pos + len > bytes.len() {
// Truncated pkt-line extends past EOF: incomplete framing.
return None;
}
let pkt_data = std::str::from_utf8(&bytes[pos + 4..pos + len]).ok()?;
pos += len;
// Sideband: first byte is channel (1=stdout, 2=stderr)
if let Some(payload) = pkt_data.strip_prefix('\x01') {
output.push_str(payload);
} else if pkt_data.starts_with('\x02') {
// stderr — skip (git error messages)
} else {
// Not sideband encoded — pass through
output.push_str(pkt_data);
}
}
if require_flush && !saw_flush {
// Truncated at a packet boundary: every packet parsed, but the
// stream never completed. Indistinguishable from a report git was
// killed mid-write, so no authority.
return None;
}
if output.is_empty() {
None
} else {
Some((output, saw_flush))
}
}
/// Sends SIGTERM to a child's whole process group on drop, unless disarmed first.
///
/// A served `git upload-pack`/`receive-pack` forks helpers such as `pack-objects`.
@@ -578,6 +792,121 @@ async fn drive_git_child(
Ok((out, admission))
}
/// Like [`drive_git_child`], but returns stdout and stderr even on a
/// non-zero exit status. Used by [`receive_pack_raw`] so the caller
/// can parse the report-status output from a failed `git-receive-pack`.
async fn drive_git_child_raw(
mut command: Command,
input: Bytes,
timeout: Duration,
what: &str,
admission: Option<AdmissionGuard>,
) -> Result<(
Vec<u8>,
Vec<u8>,
std::process::ExitStatus,
Option<AdmissionGuard>,
)> {
command
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped());
#[cfg(unix)]
command.process_group(0);
let mut child = command.spawn()?;
let mut stdin = child.stdin.take();
let mut stdout = child.stdout.take().context("git stdout was not piped")?;
let mut stderr = child.stderr.take().context("git stderr was not piped")?;
#[cfg(unix)]
let pgid = child.id().map(|id| id as i32);
#[cfg(unix)]
let mut group_guard = KillGroupOnDrop {
child: Some(child),
pgid,
admission,
};
let mut out = Vec::new();
let mut err = Vec::new();
let interact = async {
let write = async {
match stdin.take() {
Some(mut s) => s.write_all(&input).await,
None => Ok(()),
}
};
#[cfg(unix)]
let child_ref = group_guard.child_mut();
#[cfg(not(unix))]
let child_ref = &mut child;
let (write_result, r_out, r_err, status) = tokio::join!(
write,
stdout.read_to_end(&mut out),
stderr.read_to_end(&mut err),
child_ref.wait(),
);
r_out?;
r_err?;
Ok::<_, anyhow::Error>((write_result, status?))
};
let timed = tokio::time::timeout(timeout, interact).await;
let (write_result, status, admission) = match timed {
Ok(result) => {
#[cfg(unix)]
let admission = group_guard.disarm();
let (write_result, status) = result?;
(write_result, status, admission)
}
Err(_elapsed) => {
#[cfg(unix)]
{
reap_group_on_timeout(group_guard.child_mut()).await;
drop(group_guard.disarm());
}
#[cfg(not(unix))]
{
let _ = child.start_kill();
let _ = tokio::time::timeout(Duration::from_secs(2), child.wait()).await;
drop(admission);
}
return Err(GitServiceTimeout.into());
}
};
// Prefer the child's collected answer over a stdin-write error: an early
// rejection (unpack failure, command-level deny) makes git print a complete
// report-status and exit WITHOUT draining stdin, so a body larger than the
// pipe buffer EPIPEs the write while the answer is already on stdout. The
// child is reaped (the join above waited on it), so `out`/`err`/`status`
// are complete and authoritative — returning Err here would discard them:
// `receive_pack_raw_with_reflog` would propagate the write error and the
// handler would mark every prepared row `uncertain` instead of driving
// `cancelled` per the report. That durable fate is the effect: on a
// non-zero exit the client still gets `Err(AppError::Git)` (the handler
// maps the failed run itself, not this write), while on exit 0 an error
// here would answer a push git actually landed with a server error. The
// non-raw `drive_git_child` orders the same way (exit status and git's own
// failure first, stdin-write error last — pinned by
// `run_git_service_surfaces_git_stderr_over_a_stdin_epipe`), but it must
// fail on non-zero because its callers do not parse output from a failed
// child; the raw driver exists precisely to parse that output.
if let Err(e) = &write_result {
tracing::warn!(
err = %e,
what = %what,
"git stdin write failed after child exit; surfacing collected output"
);
}
Ok((out, err, status, admission))
}
/// Map a validated service name to its git subcommand. Only the two fixed
/// subcommands can come out, so an unrecognized service can never reach git's argv.
fn service_to_command(service: &str) -> Result<&'static str> {
@@ -868,6 +1197,86 @@ mod tests {
use std::process::Command;
use tempfile::TempDir;
/// Byte-for-byte capture from `git receive-pack` 2.50.1 rejecting one ref of
/// a two-ref push, with side-band-64k negotiated — what `git push` over
/// smart HTTP actually sends. The report is framed twice: the outer
/// side-band envelope, then the report-status stream's own pkt-lines.
///
/// This fixture is the point of the test. A hand-written single-framed
/// string parses fine with only one strip pass, so a fixture invented to
/// match the parser hides the exact case the parser is for.
const REAL_SIDEBAND_REPORT: &[u8] =
b"0057\x01000eunpack ok\n0028ng refs/heads/main non-fast-forward\n0018ok refs/heads/third\n0000";
#[test]
fn a_real_double_framed_sideband_report_is_read_not_treated_as_absent() {
let (unpack_ok, refs) =
parse_report_status(REAL_SIDEBAND_REPORT).expect("a real git report must parse");
assert!(unpack_ok, "unpack line must be read through both frames");
assert_eq!(
refs,
vec![
("refs/heads/main".to_string(), false),
("refs/heads/third".to_string(), true),
],
"the rejected ref must be distinguished from the accepted one"
);
}
/// The single-framed shape must keep working: the second pass has to be a
/// no-op there, not a corruption. Plain report text does not begin with
/// four hex digits, which is what makes that safe.
#[test]
fn a_single_framed_report_still_parses_after_the_second_pass() {
// Framed programmatically rather than by a hand-counted hex prefix: a
// wrong length makes the parser return None, which would look exactly
// like the regression this pair of tests exists to catch.
let payload = "\x01unpack ok\nok refs/heads/x\n";
let single = format!("{:04x}{payload}0000", payload.len() + 4);
let (unpack_ok, refs) =
parse_report_status(single.as_bytes()).expect("single-framed must parse");
assert!(unpack_ok);
assert_eq!(refs, vec![("refs/heads/x".to_string(), true)]);
}
#[test]
fn truncated_mid_packet_report_is_indeterminate_not_partial() {
// Truncate a valid single-framed report mid-packet (overrun):
// must return None so the handler persists the whole request as
// indeterminate instead of committing a partial ok set.
let payload = "\x01unpack ok\nok refs/heads/x\n";
let single = format!("{:04x}{payload}0000", payload.len() + 4);
let truncated = &single.as_bytes()[..single.len() - 6];
assert!(
parse_report_status(truncated).is_none(),
"mid-packet truncation must be indeterminate"
);
}
/// Truncation exactly at a packet boundary (every packet complete, but
/// the outer flush never arrived — e.g. git killed after the last `ok`
/// line) must also be indeterminate: the payload is complete but the
/// frame is not, and framing completeness is part of authority. The
/// mid-packet sibling above cannot catch this; only the flush gate can.
/// Removing the `require_flush` enforcement returns `Some` and turns
/// this red.
#[test]
fn truncated_at_packet_boundary_report_is_indeterminate_not_partial() {
let payload = "\x01unpack ok\nok refs/heads/x\n";
let single = format!("{:04x}{payload}0000", payload.len() + 4);
let no_flush = &single.as_bytes()[..single.len() - 4];
assert!(
parse_report_status(no_flush).is_none(),
"a report missing its outer flush must be indeterminate, \
even with every packet intact"
);
// Sanity: the untruncated fixture still parses (flush present).
assert!(
parse_report_status(single.as_bytes()).is_some(),
"the same bytes with the flush must parse"
);
}
#[test]
fn service_to_command_maps_allowlist_and_rejects_passthrough() {
assert_eq!(
@@ -2641,6 +3050,62 @@ mod tests {
);
}
// The raw driver's counterpart: a rejection-shaped early exit where git
// prints a COMPLETE report-status and exits without draining stdin, so a
// body larger than the pipe buffer (~64 KiB) EPIPEs the write while the
// answer is already on stdout. `receive_pack_raw` must surface that report
// (Ok + stdout + exit status), not discard it as a stdin-write error — an
// Err here loses parseable output: `receive_pack_raw_with_reflog`
// propagates it and the handler marks every prepared row `uncertain`
// instead of driving `cancelled` from the report (the durable fate is the
// effect; the client-visible status on a failed run is `AppError::Git`
// either way). Goes RED if the stdin-write check returns Err after the
// join. Both exit codes are pinned: receive-pack rejections and successes
// both carry reports the handler must parse.
#[cfg(unix)]
#[tokio::test]
async fn receive_pack_raw_surfaces_report_status_over_a_stdin_epipe() {
for exit_code in [0, 1] {
let tmp = tempfile::TempDir::new().unwrap();
std::fs::write(tmp.path().join("report.bin"), REAL_SIDEBAND_REPORT).unwrap();
// Never reads stdin: the pending body blocks the pipe, then EPIPEs
// once the fake exits. `cat` delivers the report first.
let git_bin = write_fake_git(
tmp.path(),
&format!("#!/bin/sh\ncat \"$(dirname \"$0\")/report.bin\"\nexit {exit_code}\n"),
);
let body = Bytes::from(vec![0u8; 256 * 1024]);
let (out, ok) = receive_pack_raw(
git_bin.to_str().unwrap(),
tmp.path(),
body,
Duration::from_secs(60),
None,
)
.await
.expect(
"a report-status printed before exit must not be discarded as a \
stdin-write error",
);
assert_eq!(
ok,
exit_code == 0,
"the exit status rides the Ok return for both report shapes"
);
let (unpack_ok, results) = parse_report_status(&out)
.expect("git's report survives the EPIPE and stays parseable");
assert!(unpack_ok, "unpack ok parses");
assert_eq!(
results,
vec![
("refs/heads/main".to_string(), false),
("refs/heads/third".to_string(), true),
],
"the per-ref report reaches the handler instead of an HTTP error"
);
}
}
// ── #192 F1: response_served_pack detector (real git output) ────────────
// A completing request (want+flush+done) yields NAK + a real packfile;
File diff suppressed because it is too large. Load diff
+3 -1
View File
@@ -627,7 +627,9 @@ impl TigrisClient {
}
/// Compress a bare repo directory into a tar.zst byte vector.
fn compress_repo(repo_path: &Path) -> Result<Vec<u8>> {
/// `pub(crate)` for sweep tests that seed a fake object store with a
/// byte-identical archive (rather than duplicating the format).
pub(crate) fn compress_repo(repo_path: &Path) -> Result<Vec<u8>> {
let buf = Vec::new();
let encoder = zstd::stream::Encoder::new(buf, 3)?; // level 3 = fast + decent ratio
let mut tar = tar::Builder::new(encoder);
File diff suppressed because it is too large. Load diff
+93 -12
View File
@@ -76,6 +76,7 @@ pub(crate) fn graphql_app_err(e: crate::error::AppError) -> async_graphql::Error
// Curated client-safe variants — `Display` is intentional API text.
safe @ (crate::error::AppError::RepoNotFound(_)
| crate::error::AppError::RepoExists(_)
| crate::error::AppError::Conflict(_)
| crate::error::AppError::NotFound(_)
| crate::error::AppError::Unauthorized(_)
| crate::error::AppError::Forbidden(_)
@@ -92,15 +93,81 @@ pub(crate) fn graphql_app_err(e: crate::error::AppError) -> async_graphql::Error
}
}
/// Classify a failed `claim_task` for GraphQL exactly as REST's claim handler
/// classifies it: a lost race is a client-safe conflict, a real sqlx fault
/// stays opaque. Lives here rather than at the three call sites so the
/// classification cannot drift between transports (#327 review), and so
/// `every_graphql_map_err_uses_opaque_helpers` can keep whitelisting by name.
pub(crate) fn graphql_claim_conflict(e: anyhow::Error) -> async_graphql::Error {
graphql_app_err(crate::api::tasks::task_write_conflict(
e,
"task not claimable: not found or already claimed",
))
}
/// The `finish_task` half of [`graphql_claim_conflict`], covering both
/// `completeTask` and `failTask`.
pub(crate) fn graphql_finish_conflict(e: anyhow::Error) -> async_graphql::Error {
graphql_app_err(crate::api::tasks::task_write_conflict(
e,
"task not found or not in claimed state",
))
}
pub struct TaskReadBrakeExtension;
impl async_graphql::extensions::ExtensionFactory for TaskReadBrakeExtension {
fn create(&self) -> Arc<dyn async_graphql::extensions::Extension> {
Arc::new(TaskReadBrakeExtensionImpl)
}
}
use async_graphql::async_trait::async_trait;
struct TaskReadBrakeExtensionImpl;
#[async_trait]
impl async_graphql::extensions::Extension for TaskReadBrakeExtensionImpl {
async fn prepare_request(
&self,
ctx: &async_graphql::extensions::ExtensionContext<'_>,
mut request: async_graphql::Request,
next: async_graphql::extensions::NextPrepareRequest<'_>,
) -> async_graphql::ServerResult<async_graphql::Request> {
if let Some(session_brake) = ctx
.session_data
.get(&std::any::TypeId::of::<crate::rate_limit::TaskReadBrake>())
.and_then(|d| d.downcast_ref::<crate::rate_limit::TaskReadBrake>())
{
request = request.data(crate::rate_limit::TaskReadBrake {
limiter: session_brake.limiter.clone(),
key: session_brake.key.clone(),
request_count: Arc::new(std::sync::atomic::AtomicUsize::new(0)),
});
}
if let Some(did) = ctx
.session_data
.get(&std::any::TypeId::of::<crate::auth::AuthenticatedDid>())
.and_then(|d| d.downcast_ref::<crate::auth::AuthenticatedDid>())
{
request = request.data(did.clone());
}
next.run(ctx, request).await
}
}
pub fn build_schema(
db: Arc<Db>,
ref_update_tx: tokio::sync::broadcast::Sender<RefUpdateBroadcast>,
task_event_tx: tokio::sync::broadcast::Sender<TaskEventBroadcast>,
task_cursor_key: crate::api::task_cursor::TaskCursorKey,
) -> GitlawbSchema {
apply_query_limits(Schema::build(QueryRoot, MutationRoot, SubscriptionRoot))
.data(db)
.data(ref_update_tx)
.data(task_event_tx)
.data(task_cursor_key)
.extension(TaskReadBrakeExtension)
.finish()
}
@@ -192,6 +259,7 @@ mod tests {
db,
tokio::sync::broadcast::channel(1).0,
tokio::sync::broadcast::channel(1).0,
crate::api::task_cursor::TaskCursorKey::derive(&[42u8; 32]),
)
}
@@ -217,6 +285,7 @@ mod tests {
Arc::new(Db::for_testing(pool)),
ref_tx.clone(),
task_tx.clone(),
crate::api::task_cursor::TaskCursorKey::derive(&[42u8; 32]),
);
for count in [7, 8] {
let fields = (0..count)
@@ -313,7 +382,10 @@ mod tests {
assert_eq!(response.errors.len(), 1);
assert_eq!(response.errors[0].message, "Query is too complex.");
assert_eq!(
db.list_tasks(Some("pending"), None, 8).await.unwrap().len(),
db.list_tasks_keyset(Some("pending"), None, 8, None)
.await
.unwrap()
.len(),
8
);
continue;
@@ -330,7 +402,7 @@ mod tests {
7
);
assert_eq!(
db.list_tasks(Some("claimed"), Some(caller), 8)
db.list_tasks_keyset(Some("claimed"), Some(caller), 8, None)
.await
.unwrap()
.len(),
@@ -352,7 +424,7 @@ mod tests {
// documented maximums instead of going stale as literals.
for (field, selection, max) in [
("refUpdates", "repo", MAX_VISIBLE_REF_UPDATES),
("tasks", "id", MAX_VISIBLE_TASK_PAGE_SIZE),
("tasks", "items { id }", MAX_VISIBLE_TASK_PAGE_SIZE),
(
"reposPage",
"nodes { name }",
@@ -595,7 +667,12 @@ mod tests {
pool.close().await;
let (ref_tx, _) = tokio::sync::broadcast::channel(16);
let (task_tx, _) = tokio::sync::broadcast::channel(16);
let schema = build_schema(Arc::new(Db::for_testing(pool)), ref_tx, task_tx);
let schema = build_schema(
Arc::new(Db::for_testing(pool)),
ref_tx,
task_tx,
crate::api::task_cursor::TaskCursorKey::derive(&[42u8; 32]),
);
// refUpdates(limit: MAX_VISIBLE_REF_UPDATES):
// Single-field selection has child_complexity 1 -> cost 100 + 200 + 1 = 301 <= 400.
@@ -649,12 +726,12 @@ mod tests {
);
// tasks(limit: MAX_VISIBLE_TASK_PAGE_SIZE):
// Single-field selection has child_complexity 1, so the cost is
// 50 + MAX_VISIBLE_TASK_PAGE_SIZE + 1 <= 400.
// Selection items { id } has child_complexity 2, so the cost is
// 50 + MAX_VISIBLE_TASK_PAGE_SIZE + 2 <= 400.
// It passes validation and reaches the resolver.
let single_task = schema
.execute(format!(
"{{ tasks(limit: {MAX_VISIBLE_TASK_PAGE_SIZE}) {{ id }} }}"
"{{ tasks(limit: {MAX_VISIBLE_TASK_PAGE_SIZE}) {{ items {{ id }} }} }}"
))
.await;
assert!(
@@ -674,14 +751,14 @@ mod tests {
single_task.errors
);
// Two-field selection has child_complexity 2, so the cost is
// 50 + MAX_VISIBLE_TASK_PAGE_SIZE + 2 <= 400.
// Selection items { id status } has child_complexity 3, so the cost is
// 50 + MAX_VISIBLE_TASK_PAGE_SIZE + 3 <= 400.
// A request at the documented maximum stays reachable with multiple
// fields (the old multiplicative price rejected it at 450); it passes
// validation and reaches the resolver like the single-field case.
let multi_task = schema
.execute(format!(
"{{ tasks(limit: {MAX_VISIBLE_TASK_PAGE_SIZE}) {{ id status }} }}"
"{{ tasks(limit: {MAX_VISIBLE_TASK_PAGE_SIZE}) {{ items {{ id status }} }} }}"
))
.await;
assert!(
@@ -843,8 +920,10 @@ mod tests {
}
/// Every `.map_err(` in the GraphQL query/mutation resolvers must route
/// through the opaque helpers, or discard the error (`|_|`). Same source-
/// scrape pattern as `api::authz_guard` (#255 review).
/// through one of the curated helpers in this module, or discard the error
/// (`|_|`). Same source-scrape pattern as `api::authz_guard` (#255
/// review). The list is deliberately a whitelist of names rather than a
/// prefix match, so adding a new mapper is a decision a reviewer sees.
#[test]
fn every_graphql_map_err_uses_opaque_helpers() {
for (file, src) in [
@@ -859,6 +938,8 @@ mod tests {
let after = code[idx + ".map_err(".len()..].trim_start();
let ok = after.starts_with("crate::graphql::graphql_db_err")
|| after.starts_with("crate::graphql::graphql_app_err")
|| after.starts_with("crate::graphql::graphql_claim_conflict")
|| after.starts_with("crate::graphql::graphql_finish_conflict")
|| after.starts_with("|_|")
|| after.starts_with("|_ ");
assert!(
+327 -43
View File
@@ -106,25 +106,37 @@ impl MutationRoot {
let assignee_did = caller.to_string();
let db = ctx.data_unchecked::<Arc<Db>>();
let tx = ctx.data_unchecked::<tokio::sync::broadcast::Sender<TaskEventBroadcast>>();
crate::api::tasks::get_claimable_task(db, &id, caller)
.await
.map_err(crate::graphql::graphql_app_err)?
.ok_or_else(|| async_graphql::Error::new("task not found"))?;
// Same classification REST's claim handler applies: a claim race is a
// client-safe conflict, a genuine sqlx failure stays opaque. Routing
// both transports through `task_write_conflict` is what stops a normal
// race from reaching a GraphQL caller as a generic database error
// (#327 review).
let task = db
.claim_task(&id, &assignee_did)
.await
.map_err(crate::graphql::graphql_db_err)?;
// Report the stored assignee form like REST, so subscribers see the
// same identity whichever API surface ran the transition.
.map_err(crate::graphql::graphql_claim_conflict)?;
let by_did = task
.assignee_did
.as_deref()
.map(crate::db::trim_assignee_did)
.unwrap_or(assignee_did.as_str())
.to_string();
let _ = tx.send(TaskEventBroadcast {
task_id: id,
old_status: "pending".to_string(),
new_status: "claimed".to_string(),
by_did,
at: Utc::now().to_rfc3339(),
});
crate::api::tasks::announce_task_event(
db,
tx,
TaskEventBroadcast {
task_id: id,
old_status: "pending".to_string(),
new_status: "claimed".to_string(),
by_did,
at: Utc::now().to_rfc3339(),
},
)
.await;
Ok(AgentTaskType::from(task))
}
@@ -145,12 +157,12 @@ impl MutationRoot {
let by_did = caller.to_string();
let db = ctx.data_unchecked::<Arc<Db>>();
let tx = ctx.data_unchecked::<tokio::sync::broadcast::Sender<TaskEventBroadcast>>();
// Authorize the actor: binding by_did to the signer is necessary but not
// sufficient — only the task's assignee may finish it.
let existing = db
.get_task(&id)
// Authorize the actor: the task must be visible to the caller (returning
// not found for invisible tasks so existence is not leaked), and only
// the task's assignee may finish it.
let existing = crate::api::tasks::get_visible_task(db, &id, Some(caller))
.await
.map_err(crate::graphql::graphql_db_err)?
.map_err(crate::graphql::graphql_app_err)?
.ok_or_else(|| async_graphql::Error::new("task not found"))?;
if !crate::api::did_matches(
caller,
@@ -163,20 +175,25 @@ impl MutationRoot {
let task = db
.finish_task(&id, "completed", input.result.as_deref(), caller)
.await
.map_err(crate::graphql::graphql_db_err)?;
.map_err(crate::graphql::graphql_finish_conflict)?;
let by_did = task
.assignee_did
.as_deref()
.map(crate::db::trim_assignee_did)
.unwrap_or(by_did.as_str())
.to_string();
let _ = tx.send(TaskEventBroadcast {
task_id: id,
old_status: "claimed".to_string(),
new_status: "completed".to_string(),
by_did,
at: Utc::now().to_rfc3339(),
});
crate::api::tasks::announce_task_event(
db,
tx,
TaskEventBroadcast {
task_id: id,
old_status: "claimed".to_string(),
new_status: "completed".to_string(),
by_did,
at: Utc::now().to_rfc3339(),
},
)
.await;
Ok(AgentTaskType::from(task))
}
@@ -197,11 +214,12 @@ impl MutationRoot {
let by_did = caller.to_string();
let db = ctx.data_unchecked::<Arc<Db>>();
let tx = ctx.data_unchecked::<tokio::sync::broadcast::Sender<TaskEventBroadcast>>();
// Authorize the actor: only the task's assignee may fail it.
let existing = db
.get_task(&id)
// Authorize the actor: the task must be visible to the caller (returning
// not found for invisible tasks so existence is not leaked), and only
// the task's assignee may fail it.
let existing = crate::api::tasks::get_visible_task(db, &id, Some(caller))
.await
.map_err(crate::graphql::graphql_db_err)?
.map_err(crate::graphql::graphql_app_err)?
.ok_or_else(|| async_graphql::Error::new("task not found"))?;
if !crate::api::did_matches(
caller,
@@ -215,20 +233,25 @@ impl MutationRoot {
let task = db
.finish_task(&id, "failed", Some(&reason), caller)
.await
.map_err(crate::graphql::graphql_db_err)?;
.map_err(crate::graphql::graphql_finish_conflict)?;
let by_did = task
.assignee_did
.as_deref()
.map(crate::db::trim_assignee_did)
.unwrap_or(by_did.as_str())
.to_string();
let _ = tx.send(TaskEventBroadcast {
task_id: id,
old_status: "claimed".to_string(),
new_status: "failed".to_string(),
by_did,
at: Utc::now().to_rfc3339(),
});
crate::api::tasks::announce_task_event(
db,
tx,
TaskEventBroadcast {
task_id: id,
old_status: "claimed".to_string(),
new_status: "failed".to_string(),
by_did,
at: Utc::now().to_rfc3339(),
},
)
.await;
Ok(AgentTaskType::from(task))
}
}
@@ -326,7 +349,7 @@ mod tests {
);
if *name == "claimTask" {
assert!(
errs.contains("task not claimable"),
errs.contains("task not found"),
"claim race / missing task must keep its business message: {errs}"
);
assert!(
@@ -370,6 +393,210 @@ mod tests {
);
}
/// #327 review: `claimTask`/`completeTask`/`failTask` called
/// `graphql_db_err` directly on db-layer business failures, so an ordinary
/// claim race or stale finish reached GraphQL clients as a generic
/// database error while REST clients got an actionable conflict. All three
/// now route through the shared `task_write_conflict` classifier.
///
/// Deleting either `map_err` from a mutation turns this red: the message
/// becomes the raw anyhow text instead of the fixed conflict form.
#[sqlx::test]
async fn task_write_races_surface_as_conflicts_not_db_errors(pool: PgPool) {
let state = crate::test_support::test_state(pool).await;
let schema = state.graphql_schema.as_ref();
let assignee = "did:key:zGQLRACEASSIGNEEAAAAAAAAAAAAAAAAAAAAAAA";
let rival = "did:key:zGQLRACERIVALBBBBBBBBBBBBBBBBBBBBBBBBBB";
let delegator = "did:key:zGQLRACEDELEGATORCCCCCCCCCCCCCCCCCCCCC";
let now = chrono::Utc::now().to_rfc3339();
state
.db
.create_repo(&crate::db::RepoRecord {
id: "race-repo".into(),
name: "race".into(),
owner_did: delegator.into(),
description: None,
is_public: true,
default_branch: "main".into(),
created_at: chrono::Utc::now(),
updated_at: chrono::Utc::now(),
disk_path: "/tmp/race-repo".into(),
forked_from: None,
machine_id: None,
})
.await
.unwrap();
state
.db
.create_task(&crate::db::AgentTask {
id: "race-task".into(),
repo_id: Some("race-repo".into()),
kind: "build".into(),
status: "pending".into(),
delegator_did: delegator.into(),
assignee_did: None,
capability: "repo:write".into(),
ucan_token: None,
payload: None,
result: None,
created_at: now.clone(),
updated_at: now,
deadline: None,
})
.await
.unwrap();
// The assignee wins the claim.
let claim = |did: &str| {
format!(r#"mutation {{ claimTask(id: "race-task", assigneeDid: "{did}") {{ id }} }}"#)
};
let resp = schema
.execute(Request::new(claim(assignee)).data(AuthenticatedDid(assignee.into())))
.await;
assert!(resp.errors.is_empty(), "first claim: {}", errors(&resp));
// The rival loses the race: ineligible caller receives opaque not-found.
let resp = schema
.execute(Request::new(claim(rival)).data(AuthenticatedDid(rival.into())))
.await;
let errs = errors(&resp);
assert!(
errs.contains("task not found"),
"ineligible caller after task is claimed must receive opaque not-found: {errs}"
);
assert!(
!errs.contains(crate::graphql::GRAPHQL_DB_ERROR_MESSAGE),
"ineligible claim is not a database failure: {errs}"
);
// A second claim that reaches the conditional SQL write (e.g. assignee claiming
// again on an already-claimed task) exercises the conflict mapper:
let resp = schema
.execute(Request::new(claim(assignee)).data(AuthenticatedDid(assignee.into())))
.await;
let errs = errors(&resp);
assert!(
errs.contains("task not claimable: not found or already claimed"),
"a lost claim race must render the fixed conflict message: {errs}"
);
assert!(
!errs.contains(crate::graphql::GRAPHQL_DB_ERROR_MESSAGE),
"a claim race is not a database failure: {errs}"
);
// A stale finish on an already-completed task is the same class.
let complete = format!(
r#"mutation {{ completeTask(id: "race-task", byDid: "{assignee}", input: {{ result: "ok" }}) {{ id }} }}"#
);
let resp = schema
.execute(Request::new(&complete).data(AuthenticatedDid(assignee.into())))
.await;
assert!(resp.errors.is_empty(), "first complete: {}", errors(&resp));
let resp = schema
.execute(Request::new(&complete).data(AuthenticatedDid(assignee.into())))
.await;
let errs = errors(&resp);
assert!(
errs.contains("task not found or not in claimed state"),
"a stale complete must render the fixed conflict message: {errs}"
);
assert!(
!errs.contains(crate::graphql::GRAPHQL_DB_ERROR_MESSAGE),
"a stale complete is not a database failure: {errs}"
);
let fail = format!(
r#"mutation {{ failTask(id: "race-task", byDid: "{assignee}", input: {{ reason: "nope" }}) {{ id }} }}"#
);
let resp = schema
.execute(Request::new(&fail).data(AuthenticatedDid(assignee.into())))
.await;
let errs = errors(&resp);
assert!(
errs.contains("task not found or not in claimed state"),
"a stale fail must render the fixed conflict message: {errs}"
);
assert!(
!errs.contains(crate::graphql::GRAPHQL_DB_ERROR_MESSAGE),
"a stale fail is not a database failure: {errs}"
);
}
/// The other half of the same classification: a genuine SQL fault on the
/// write itself must stay opaque. Without this, routing conflicts through
/// `task_write_conflict` could be "fixed" by making every db failure a
/// client-visible conflict message.
#[sqlx::test]
async fn task_write_sql_faults_stay_opaque(pool: PgPool) {
let state = crate::test_support::test_state(pool.clone()).await;
let assignee = "did:key:zGQLOPAQUEASSIGNEEAAAAAAAAAAAAAAAAAAAAA";
let delegator = "did:key:zGQLOPAQUEDELEGATORBBBBBBBBBBBBBBBBBBB";
let now = chrono::Utc::now().to_rfc3339();
state
.db
.create_task(&crate::db::AgentTask {
id: "opaque-task".into(),
repo_id: None,
kind: "build".into(),
status: "pending".into(),
delegator_did: delegator.into(),
assignee_did: Some(assignee.into()),
capability: "repo:write".into(),
ucan_token: None,
payload: None,
result: None,
created_at: now.clone(),
updated_at: now,
deadline: None,
})
.await
.unwrap();
// Fault the write itself, not the schema. Dropping a column the
// visibility pre-check also SELECTs would fail in `get_visible_task`
// and never reach `graphql_claim_conflict`, so this test would pass
// against a claim mapper that leaks (#327 review). A BEFORE UPDATE
// trigger keeps every read valid and faults only inside
// `Db::claim_task`.
sqlx::raw_sql(
"CREATE FUNCTION gl_test_fault_task_update() RETURNS trigger
LANGUAGE plpgsql AS $fn$
BEGIN RAISE EXCEPTION 'gl_test_forced_update_fault'; END;
$fn$;
CREATE TRIGGER gl_test_fault_task_update
BEFORE UPDATE ON agent_tasks
FOR EACH ROW EXECUTE FUNCTION gl_test_fault_task_update();",
)
.execute(&pool)
.await
.unwrap();
let resp = state
.graphql_schema
.execute(
Request::new(format!(
r#"mutation {{ claimTask(id: "opaque-task", assigneeDid: "{assignee}") {{ id }} }}"#
))
.data(AuthenticatedDid(assignee.into())),
)
.await;
let errs = errors(&resp);
assert!(
errs.contains(crate::graphql::GRAPHQL_DB_ERROR_MESSAGE),
"a real sqlx fault on the write must stay opaque: {errs}"
);
assert!(
!errs.contains("task not claimable"),
"a write-time sqlx fault must not be reclassified as a claim race: {errs}"
);
assert!(
!errs.contains("gl_test_forced_update_fault") && !errs.contains("trigger"),
"database text leaked through the conflict mapper: {errs}"
);
}
/// Adversarial-review GATE-1 (GraphQL): completing a task requires being its
/// assignee, not merely signing as the by_did you pass. A signer who is not
/// the assignee is rejected even though the by_did binding passes; the
@@ -392,7 +619,7 @@ mod tests {
payload: None,
result: None,
created_at: now.clone(),
updated_at: now,
updated_at: now.clone(),
deadline: None,
};
state.db.create_task(&task).await.expect("seed task");
@@ -409,14 +636,69 @@ mod tests {
)
};
// Stranger signs as themselves and passes byDid=self (so the signer
// binding passes), but is not the assignee → rejected by authorization.
// Stranger signs as themselves on a repo-less task they cannot see:
// invisible task returns "task not found" so existence is not leaked.
let resp = schema
.execute(Request::new(q(stranger)).data(AuthenticatedDid(stranger.into())))
.await;
assert!(
errors(&resp).contains("task not found"),
"an invisible task must return not found, got: {}",
errors(&resp)
);
// Seed a task on a public repo that stranger CAN see, but is not assignee of:
let pub_repo = crate::db::RepoRecord {
id: "pub-r".into(),
name: "pub-r".into(),
owner_did: "did:key:zGQLDELEGATORCCCCCCCCCCCCCCCCCCCCCCCCCC".into(),
description: None,
is_public: true,
default_branch: "main".into(),
created_at: chrono::Utc::now(),
updated_at: chrono::Utc::now(),
disk_path: "/tmp/pub-r".into(),
forked_from: None,
machine_id: None,
};
state.db.create_repo(&pub_repo).await.expect("create repo");
let pub_task = crate::db::AgentTask {
id: "task-pub".into(),
repo_id: Some("pub-r".into()),
kind: "build".into(),
status: "pending".into(),
delegator_did: "did:key:zGQLDELEGATORCCCCCCCCCCCCCCCCCCCCCCCCCC".into(),
assignee_did: None,
capability: "repo:write".into(),
ucan_token: None,
payload: None,
result: None,
created_at: now.clone(),
updated_at: now,
deadline: None,
};
state
.db
.create_task(&pub_task)
.await
.expect("seed pub task");
state
.db
.claim_task("task-pub", assignee)
.await
.expect("claim pub task");
let q_pub = |actor: &str| {
format!(
r#"mutation {{ completeTask(id: "task-pub", byDid: "{actor}", input: {{}}) {{ id status }} }}"#
)
};
let resp = schema
.execute(Request::new(q_pub(stranger)).data(AuthenticatedDid(stranger.into())))
.await;
assert!(
errors(&resp).contains("assignee"),
"a non-assignee signer must be rejected: {}",
"a non-assignee signer on a visible task must be rejected: {}",
errors(&resp)
);
@@ -482,7 +764,7 @@ mod tests {
// repo id afterwards.
let stored = state
.db
.list_tasks(None, None, 50)
.list_tasks_keyset(None, None, 50, None)
.await
.expect("list tasks");
assert!(
@@ -644,9 +926,11 @@ mod tests {
.execute(Request::new(q(thief)).data(AuthenticatedDid(thief.into())))
.await;
let errs = errors(&resp);
// The claim gate hides existence: an ineligible claimer gets the same
// not-found as a missing task, not a reserved deny that would leak it.
assert!(
errs.contains("reserved"),
"stranger claim must surface reserved deny distinctly: {errs}"
errs.contains("task not found"),
"stranger claim must read as not-found, got: {errs}"
);
assert!(
!errs.contains("gql-ucan-secret"),
+481 -29
View File
@@ -5,7 +5,7 @@ use std::sync::Arc;
use crate::api::events::MAX_VISIBLE_REF_UPDATES;
use crate::db::{Db, RepoRecord, MAX_VISIBLE_REPO_PAGE_SIZE, MAX_VISIBLE_TASK_PAGE_SIZE};
use super::types::{AgentTaskType, RefUpdateType, RepoPageType, RepoType};
use super::types::{AgentTaskReadType, RefUpdateType, RepoPageType, RepoType, TaskPageType};
fn repo_type(repo: RepoRecord) -> RepoType {
RepoType {
@@ -45,6 +45,23 @@ fn parse_repo_cursor(cursor: &str) -> Result<(String, String)> {
pub struct QueryRoot;
/// Debit the per-IP task-read brake for one task field, mirroring the
/// `rate_limit_by_ip` layer on `task_read_routes`. Both surfaces run the same
/// #268 visibility gate and pay the same queries before answering, so an
/// anonymous prober must not get an unbraked lane by asking over GraphQL
/// (#327 review).
///
/// A schema built without the brake (unit tests, subscriptions) is not braked,
/// exactly as `rate_limit_by_ip` is a no-op without its extension.
async fn task_read_brake(ctx: &Context<'_>, field: &str) -> Result<()> {
match ctx.data::<crate::rate_limit::TaskReadBrake>() {
Ok(brake) if !brake.check(field).await => Err(async_graphql::Error::new(
crate::rate_limit::RATE_LIMIT_MESSAGE,
)),
_ => Ok(()),
}
}
#[Object]
impl QueryRoot {
// DB-backed roots carry a base cost so aliases consume the request budget
@@ -217,30 +234,74 @@ impl QueryRoot {
assignee_did: Option<String>,
#[graphql(
default = 50,
desc = "Requests above the documented maximum are clamped to it (no error). Negative values clamp to 0."
desc = "Max 200; larger requests are clamped to 200 (no error). Negative values clamp to 0. Use `nextCursor` to read past the clamp."
)]
limit: i64,
) -> Result<Vec<AgentTaskType>> {
#[graphql(
desc = "Opaque continuation token from a previous page's `nextCursor`. Must be presented with the same `status`/`assigneeDid` filter that issued it."
)]
cursor: Option<String>,
) -> Result<TaskPageType> {
use crate::api::task_cursor::{self, TaskCursorKey, TaskFilter};
task_read_brake(ctx, "tasks").await?;
let db = ctx.data_unchecked::<Arc<Db>>();
// Clamp before SQL: a negative LIMIT is a client fault that Postgres
// rejects with 2201W, which would otherwise trip the opaque DB path
// and write an error-level log on every probe (#250 review).
let limit = limit.clamp(0, MAX_VISIBLE_TASK_PAGE_SIZE);
let tasks = db
.list_tasks(status.as_deref(), assignee_did.as_deref(), limit)
.await
.map_err(crate::graphql::graphql_db_err)?;
Ok(tasks.into_iter().map(AgentTaskType::from).collect())
// #268: gate rows via the same collector the REST list route uses (like
// `ref_updates` shares `collect_visible_ref_updates` with its REST feed),
// so the two surfaces cannot drift. The collector clamps `limit` itself,
// including the negative-LIMIT case #250 called out for this resolver.
let caller = ctx
.data::<crate::auth::AuthenticatedDid>()
.ok()
.map(|d| d.0.as_str());
let key = ctx.data_unchecked::<TaskCursorKey>();
let filter = TaskFilter {
status: status.as_deref(),
assignee_did: assignee_did.as_deref(),
};
let resume = cursor
.as_deref()
.map(|token| task_cursor::decode(key, filter, caller, token))
.transpose()
.map_err(crate::graphql::graphql_app_err)?;
let result = crate::api::tasks::collect_visible_tasks(
db,
status.as_deref(),
assignee_did.as_deref(),
limit,
resume.as_ref(),
caller,
)
.await
.map_err(crate::graphql::graphql_app_err)?;
Ok(TaskPageType {
next_cursor: result
.next_position
.as_ref()
.map(|pos| task_cursor::encode(key, filter, caller, pos)),
items: result
.tasks
.into_iter()
.map(AgentTaskReadType::from)
.collect(),
has_more: result.has_more,
incomplete: result.incomplete,
})
}
#[graphql(complexity = "50 + child_complexity")]
async fn task(&self, ctx: &Context<'_>, id: String) -> Result<Option<AgentTaskType>> {
async fn task(&self, ctx: &Context<'_>, id: String) -> Result<Option<AgentTaskReadType>> {
task_read_brake(ctx, "task").await?;
let db = ctx.data_unchecked::<Arc<Db>>();
let t = db
.get_task(&id)
// #268: same gate as the REST get route, via the shared helper.
let caller = ctx
.data::<crate::auth::AuthenticatedDid>()
.ok()
.map(|d| d.0.as_str());
let t = crate::api::tasks::get_visible_task(db, &id, caller)
.await
.map_err(crate::graphql::graphql_db_err)?;
Ok(t.map(AgentTaskType::from))
.map_err(crate::graphql::graphql_app_err)?;
Ok(t.map(AgentTaskReadType::from))
}
}
@@ -889,10 +950,14 @@ mod tests {
Arc::new(db)
}
fn cursor_key() -> crate::api::task_cursor::TaskCursorKey {
crate::api::task_cursor::TaskCursorKey::derive(&[42u8; 32])
}
fn schema(db: Arc<Db>) -> super::super::GitlawbSchema {
let (ref_tx, _) = tokio::sync::broadcast::channel(16);
let (task_tx, _) = tokio::sync::broadcast::channel(16);
super::super::build_schema(db, ref_tx, task_tx)
super::super::build_schema(db, ref_tx, task_tx, cursor_key())
}
fn repo(id: &str, owner_did: &str, name: &str, is_public: bool) -> RepoRecord {
@@ -1202,7 +1267,7 @@ mod tests {
async fn tasks_negative_limit_clamped(pool: PgPool) {
let db = db(pool).await;
let schema = schema(db);
let resp = anon(&schema, "{ tasks(limit: -1) { id } }").await;
let resp = anon(&schema, "{ tasks(limit: -1) { items { id } } }").await;
assert!(
resp.errors.is_empty(),
"negative limit must clamp, not fail: {:?}",
@@ -1237,12 +1302,12 @@ mod tests {
.unwrap();
}
let schema = schema(db);
let resp = anon(&schema, "{ tasks(limit: 5000) { id } }").await;
assert_eq!(
count_tasks(&resp),
crate::db::MAX_VISIBLE_TASK_PAGE_SIZE as usize,
"limit above the maximum must clamp to MAX_VISIBLE_TASK_PAGE_SIZE"
);
// Queried as the delegator, not anonymously: since #268 the task read
// surface is visibility-gated, and an anonymous caller sees none of
// these repo-less tasks at all. The clamp is what this test pins, so it
// needs a caller who can legitimately see all 201 rows.
let resp = authed(&schema, "{ tasks(limit: 5000) { items { id } } }", OWNER).await;
assert_eq!(count_tasks(&resp), 200, "limit above 200 must clamp to 200");
}
/// A limit above MAX_VISIBLE_REF_UPDATES must reach the resolver clamped,
@@ -1283,14 +1348,401 @@ mod tests {
);
}
fn count_tasks(resp: &async_graphql::Response) -> usize {
/// Seed one repo-less task carrying a `ucan_token`, so a leak on any read
/// surface is visible in the response body.
async fn seed_task(db: &Db, id: &str, delegator: &str) {
let now = Utc::now().to_rfc3339();
db.create_task(&crate::db::AgentTask {
id: id.into(),
repo_id: None,
kind: "build".into(),
status: "pending".into(),
delegator_did: delegator.into(),
assignee_did: None,
capability: "repo:write".into(),
ucan_token: Some("SECRET-UCAN-TOKEN".into()),
payload: None,
result: None,
created_at: now.clone(),
updated_at: now,
deadline: None,
})
.await
.unwrap();
}
/// #268: the `tasks` resolver must delegate to the gated collector, not
/// query the DB directly. A repo-less task belonging to someone else is
/// invisible to an anonymous caller. `tasks_negative_limit_clamped` cannot
/// catch a resolver that stops calling `collect_visible_tasks` because it
/// seeds no rows — this seeds one, so the gate is load-bearing here.
#[sqlx::test]
async fn tasks_repo_less_task_hidden_from_anon(pool: PgPool) {
let db = db(pool).await;
seed_task(&db, "t1", OWNER).await;
let schema = schema(db);
let resp = anon(&schema, "{ tasks { items { id } } }").await;
assert_eq!(
count_tasks(&resp),
0,
"anon must not enumerate another party's repo-less task"
);
assert!(
!format!("{:?}", resp.data).contains("SECRET-UCAN-TOKEN"),
"no ucan token may reach an anonymous caller"
);
}
/// #268 sibling for the single-task resolver: an invisible task reads as
/// `null`, indistinguishable from one that does not exist.
#[sqlx::test]
async fn task_by_id_is_null_for_anon(pool: PgPool) {
let db = db(pool).await;
seed_task(&db, "t1", OWNER).await;
let schema = schema(db);
let resp = anon(&schema, r#"{ task(id: "t1") { id } }"#).await;
assert!(resp.errors.is_empty(), "graphql errors: {:?}", resp.errors);
let async_graphql::Value::Object(obj) = &resp.data else {
panic!("data not an object: {:?}", resp.data);
};
let async_graphql::Value::List(rows) = obj.get("tasks").expect("tasks key") else {
panic!("tasks not a list");
assert_eq!(
obj.get("task"),
Some(&async_graphql::Value::Null),
"an invisible task must read as null, got {:?}",
obj.get("task")
);
}
/// #268: `ucanToken` is absent from the read type's schema entirely, so the
/// delegator cannot request it either. Asking for it is a validation error,
/// which pins the redaction at the schema level rather than per-resolver.
#[sqlx::test]
async fn task_read_schema_has_no_ucan_token_field(pool: PgPool) {
let db = db(pool).await;
seed_task(&db, "t1", OWNER).await;
let schema = schema(db);
let resp = authed(&schema, r#"{ task(id: "t1") { id ucanToken } }"#, OWNER).await;
assert!(
!resp.errors.is_empty(),
"ucanToken must not exist on the task read type"
);
}
#[sqlx::test]
async fn tasks_find_older_visible_row_behind_denied_window(pool: PgPool) {
let db = db(pool).await;
db.create_repo(&repo("public-repo", OWNER, "public", true))
.await
.unwrap();
let visible = crate::db::AgentTask {
id: "visible".into(),
repo_id: Some("public-repo".into()),
kind: "build".into(),
status: "pending".into(),
delegator_did: OWNER.into(),
assignee_did: None,
capability: "repo:write".into(),
ucan_token: None,
payload: None,
result: None,
created_at: "2026-01-01T00:00:00Z".into(),
updated_at: "2026-01-01T00:00:00Z".into(),
deadline: None,
};
rows.len()
db.create_task(&visible).await.unwrap();
for i in 0..200 {
let mut hidden = visible.clone();
hidden.id = format!("hidden-{i:03}");
hidden.repo_id = None;
hidden.created_at = "2026-01-02T00:00:00Z".into();
hidden.updated_at = hidden.created_at.clone();
db.create_task(&hidden).await.unwrap();
}
let schema = schema(db);
let resp = anon(&schema, "{ tasks(limit: 1) { items { id } incomplete } }").await;
assert_eq!(count_tasks(&resp), 1);
assert!(!task_incomplete(&resp));
assert!(format!("{:?}", resp.data).contains("visible"));
}
/// The GraphQL surface must page past a denied window on server-issued
/// cursors alone, exactly as REST does (#327 review). Before this, the
/// only cursor a caller could hold named the last row they saw, so the
/// query stalled at `incomplete: true` forever and never reached
/// `past-ceiling`.
#[sqlx::test]
async fn tasks_page_past_candidate_ceiling_on_server_cursors(pool: PgPool) {
let db = db(pool).await;
db.create_repo(&repo("public-repo", OWNER, "public", true))
.await
.unwrap();
let visible_newer = crate::db::AgentTask {
id: "newer-visible".into(),
repo_id: Some("public-repo".into()),
kind: "build".into(),
status: "pending".into(),
delegator_did: OWNER.into(),
assignee_did: None,
capability: "repo:write".into(),
ucan_token: None,
payload: None,
result: None,
created_at: "2026-01-03T00:00:00Z".into(),
updated_at: "2026-01-03T00:00:00Z".into(),
deadline: None,
};
db.create_task(&visible_newer).await.unwrap();
for i in 0..1500 {
let mut hidden = visible_newer.clone();
hidden.id = format!("hidden-{i:04}");
hidden.repo_id = None;
hidden.created_at = "2026-01-02T00:00:00Z".into();
hidden.updated_at = hidden.created_at.clone();
db.create_task(&hidden).await.unwrap();
}
let mut visible_older = visible_newer.clone();
visible_older.id = "past-ceiling".into();
visible_older.created_at = "2026-01-01T00:00:00Z".into();
visible_older.updated_at = visible_older.created_at.clone();
db.create_task(&visible_older).await.unwrap();
let schema = schema(db);
let mut seen: Vec<String> = Vec::new();
let mut cursor: Option<String> = None;
for request in 0..10 {
let query = match &cursor {
Some(c) => format!(
r#"{{ tasks(limit: 1, cursor: "{c}") {{ items {{ id }} hasMore incomplete nextCursor }} }}"#
),
None => {
"{ tasks(limit: 1) { items { id } hasMore incomplete nextCursor } }".to_string()
}
};
let resp = anon(&schema, &query).await;
let rendered = format!("{:?}", resp.data);
assert!(
!rendered.contains("hidden-"),
"request {request} disclosed a denied row: {rendered}"
);
for item in task_items(&resp) {
let async_graphql::Value::Object(row) = item else {
panic!("task item not an object");
};
seen.push(row.get("id").unwrap().to_string().replace('"', ""));
}
// Short page with more behind it is the scan wall, and says so.
if task_page_bool(&resp, "hasMore") && task_items(&resp).is_empty() {
assert!(
task_incomplete(&resp),
"request {request}: an empty page with rows behind it is a paused scan"
);
}
match task_page_cursor(&resp) {
Some(c) => cursor = Some(c),
None => {
assert!(!task_page_bool(&resp, "hasMore"));
assert!(!task_incomplete(&resp));
break;
}
}
}
assert_eq!(
seen,
vec!["newer-visible".to_string(), "past-ceiling".to_string()],
"both visible rows must be reachable using only server-issued cursors"
);
}
/// REST and GraphQL mint the same tokens from the same node key, so a
/// cursor must not be usable against a filter it was not issued for, and
/// must render the same single rejection every other bad cursor renders.
#[sqlx::test]
async fn tasks_rejects_unusable_cursor(pool: PgPool) {
use crate::api::task_cursor::{self, TaskFilter, TaskPosition};
let db = db(pool).await;
let schema = schema(db);
let wrong_filter = task_cursor::encode(
&cursor_key(),
TaskFilter {
status: Some("pending"),
assignee_did: None,
},
None,
&TaskPosition::new("2026-01-01T00:00:00Z", "t1"),
);
let foreign = task_cursor::encode(
&crate::api::task_cursor::TaskCursorKey::derive(&[9u8; 32]),
TaskFilter {
status: None,
assignee_did: None,
},
None,
&TaskPosition::new("2026-01-01T00:00:00Z", "t1"),
);
for (label, query) in [
(
"garbage",
r#"{ tasks(cursor: "not-a-cursor") { items { id } } }"#.to_string(),
),
(
"another node's key",
format!(r#"{{ tasks(cursor: "{foreign}") {{ items {{ id }} }} }}"#),
),
(
"different filter",
format!(r#"{{ tasks(cursor: "{wrong_filter}") {{ items {{ id }} }} }}"#),
),
] {
let resp = anon(&schema, &query).await;
assert_eq!(
resp.errors.len(),
1,
"{label}: must be rejected, not treated as page one"
);
assert!(
resp.errors[0].message.contains("invalid or expired cursor"),
"{label}: unexpected message {:?}",
resp.errors[0].message
);
}
let resp = anon(
&schema,
&format!(
r#"{{ tasks(status: "pending", cursor: "{wrong_filter}") {{ items {{ id }} }} }}"#
),
)
.await;
assert!(resp.errors.is_empty(), "graphql errors: {:?}", resp.errors);
}
#[sqlx::test]
async fn tasks_anonymous_denial_hides_repoless_task_and_leaks_no_token(pool: PgPool) {
let db = db(pool).await;
let task = crate::db::AgentTask {
id: "t1".into(),
repo_id: None,
kind: "code-review".into(),
status: "pending".into(),
delegator_did: "did:key:z6MkDelegator".into(),
assignee_did: None,
capability: "agent:task".into(),
ucan_token: Some("secret-ucan-token".into()),
payload: Some("payload".into()),
result: None,
created_at: "2026-01-01T00:00:00Z".into(),
updated_at: "2026-01-01T00:00:00Z".into(),
deadline: None,
};
db.create_task(&task).await.unwrap();
let schema = schema(db);
let query = "{ tasks { items { id } } }";
let resp = anon(&schema, query).await;
assert!(resp.errors.is_empty(), "graphql errors: {:?}", resp.errors);
assert_eq!(count_tasks(&resp), 0);
let rendered = format!("{:?}", resp.data);
assert!(!rendered.contains("secret-ucan-token"));
}
#[sqlx::test]
async fn task_anonymous_denial_returns_null_and_leaks_no_token(pool: PgPool) {
let db = db(pool).await;
let task = crate::db::AgentTask {
id: "t1".into(),
repo_id: None,
kind: "code-review".into(),
status: "pending".into(),
delegator_did: "did:key:z6MkDelegator".into(),
assignee_did: None,
capability: "agent:task".into(),
ucan_token: Some("secret-ucan-token".into()),
payload: Some("payload".into()),
result: None,
created_at: "2026-01-01T00:00:00Z".into(),
updated_at: "2026-01-01T00:00:00Z".into(),
deadline: None,
};
db.create_task(&task).await.unwrap();
let schema = schema(db);
let query = r#"{ task(id: "t1") { id } }"#;
let resp = anon(&schema, query).await;
assert!(resp.errors.is_empty(), "graphql errors: {:?}", resp.errors);
let async_graphql::Value::Object(obj) = &resp.data else {
panic!("data not an object: {:?}", resp.data);
};
assert_eq!(obj.get("task"), Some(&async_graphql::Value::Null));
let rendered = format!("{:?}", resp.data);
assert!(!rendered.contains("secret-ucan-token"));
}
fn task_page_bool(resp: &async_graphql::Response, field: &str) -> bool {
assert!(resp.errors.is_empty(), "graphql errors: {:?}", resp.errors);
let async_graphql::Value::Object(obj) = &resp.data else {
panic!("data not an object: {:?}", resp.data);
};
let async_graphql::Value::Object(page) = obj.get("tasks").expect("tasks key") else {
panic!("tasks not an object");
};
let async_graphql::Value::Boolean(value) = page.get(field).expect("field present") else {
panic!("{field} not a bool");
};
*value
}
fn task_page_cursor(resp: &async_graphql::Response) -> Option<String> {
assert!(resp.errors.is_empty(), "graphql errors: {:?}", resp.errors);
let async_graphql::Value::Object(obj) = &resp.data else {
panic!("data not an object: {:?}", resp.data);
};
let async_graphql::Value::Object(page) = obj.get("tasks").expect("tasks key") else {
panic!("tasks not an object");
};
match page.get("nextCursor").expect("nextCursor key") {
async_graphql::Value::Null => None,
async_graphql::Value::String(c) => Some(c.clone()),
other => panic!("nextCursor not a string: {other:?}"),
}
}
fn task_items(resp: &async_graphql::Response) -> &Vec<async_graphql::Value> {
assert!(resp.errors.is_empty(), "graphql errors: {:?}", resp.errors);
let async_graphql::Value::Object(obj) = &resp.data else {
panic!("data not an object: {:?}", resp.data);
};
let async_graphql::Value::Object(page) = obj.get("tasks").expect("tasks key") else {
panic!("tasks not an object");
};
let async_graphql::Value::List(rows) = page.get("items").expect("items key") else {
panic!("items not a list");
};
rows
}
fn count_tasks(resp: &async_graphql::Response) -> usize {
task_items(resp).len()
}
fn task_incomplete(resp: &async_graphql::Response) -> bool {
assert!(resp.errors.is_empty(), "graphql errors: {:?}", resp.errors);
let async_graphql::Value::Object(obj) = &resp.data else {
panic!("data not an object: {:?}", resp.data);
};
let async_graphql::Value::Object(page) = obj.get("tasks").expect("tasks key") else {
panic!("tasks not an object");
};
let async_graphql::Value::Boolean(incomplete) =
page.get("incomplete").expect("incomplete key")
else {
panic!("incomplete not a bool");
};
*incomplete
}
}
@@ -11,9 +11,9 @@ pub struct SubscriptionRoot;
#[Subscription]
impl SubscriptionRoot {
/// Live ref-update stream. `/graphql/ws` is mounted outside the
/// `optional_signature` layer, so this resolver has NO caller identity and
/// cannot gate per-subscriber — it relays whatever enters the broadcast
/// Live ref-update stream. `/graphql/ws` is mounted under the
/// `optional_signature` layer and carries optional caller identity, but this
/// resolver does not inspect it — it relays whatever enters the broadcast
/// channel to any anonymous client. Its visibility safety therefore rests
/// entirely on the WRITE side: the push handler only sends a
/// `RefUpdateBroadcast` for repos the anonymous public may read (inside its
+59
View File
@@ -56,6 +56,65 @@ impl From<AgentTask> for AgentTaskType {
}
}
/// Read-only projection of `AgentTask` for the `tasks`/`task` queries, as
/// opposed to `AgentTaskType`, which the task mutations (`createTask`,
/// `claimTask`, `completeTask`, `failTask` — all `require_signer`-gated)
/// return. Identical except for the missing `ucan_token` (#268): a read
/// surface never needs to echo it back, since the assignee already received it
/// at delegation/claim time via the mutation response.
#[derive(SimpleObject, Clone)]
pub struct AgentTaskReadType {
pub id: String,
pub repo_id: Option<String>,
pub kind: String,
pub status: String,
pub delegator_did: String,
pub assignee_did: Option<String>,
pub capability: String,
pub payload: Option<String>,
pub result: Option<String>,
pub created_at: String,
pub updated_at: String,
pub deadline: Option<String>,
}
/// Wraps a `tasks` page with the same completion signals the REST list route
/// exposes, so the two surfaces answer pagination identically.
///
/// `hasMore` and `incomplete` are separate because they are separate facts
/// (#327 review): `hasMore` says more candidates remain, `incomplete` says
/// this page is short *only* because the authorization scan hit its safety
/// wall. `nextCursor` is present exactly when `hasMore` is true and is an
/// opaque MAC'd token — it can name the last examined candidate, denied or
/// not, without disclosing it, which is what lets a caller page past a denied
/// window instead of stalling on it.
#[derive(SimpleObject, Clone)]
pub struct TaskPageType {
pub items: Vec<AgentTaskReadType>,
pub has_more: bool,
pub incomplete: bool,
pub next_cursor: Option<String>,
}
impl From<AgentTask> for AgentTaskReadType {
fn from(t: AgentTask) -> Self {
Self {
id: t.id,
repo_id: t.repo_id,
kind: t.kind,
status: t.status,
delegator_did: t.delegator_did,
assignee_did: t.assignee_did,
capability: t.capability,
payload: t.payload,
result: t.result,
created_at: t.created_at,
updated_at: t.updated_at,
deadline: t.deadline,
}
}
}
#[derive(SimpleObject, Clone)]
pub struct RefUpdateType {
pub repo: String,
+406 -49
View File
@@ -922,6 +922,12 @@ async fn sweep_pass(
// Advance FIRST: every path below this line may skip the row, and none of them
// may wedge the walk (scenario 7).
last = sha.clone();
// Round-3 P1: skip Pinata-only rows whose local cid is NULL.
// The legacy repair walk re-keys `cid` from a provider CID to
// the raw resolver CID; a row with no local cid has no string
// to re-key, and skipping is the natural behavior. The cursor
// still advances so the walk does not loop on this row.
let Some(stored) = stored else { continue };
// Same cost gate as the skip-path repair: a canonical raw CIDv1 key is already
// the resolver key, so it reads no bytes and resolves no repo.
if gitlawb_core::cid::is_raw_cidv1(&stored) {
@@ -1457,6 +1463,97 @@ fn note_legacy_repair_read() {
/// have to be documented, validated, and kept meaningful.
pub const PIN_BATCH_BUDGET: Duration = Duration::from_secs(120);
/// A captured per-repo visibility-policy epoch that fences a pin batch.
///
/// The reconciliation sweep reads the epoch immediately before dispatching a
/// pin loop and passes a fence in; the loop re-reads the epoch before every
/// upload and aborts the batch the moment it moves. A visibility narrow that
/// lands mid-batch (a rule made private, a repo quarantined) must not let the
/// remaining pre-authorized objects still go to a public content-addressed
/// backend — the narrow is a policy change, and dispatching against the stale
/// snapshot is the exact irreversible-publication class this fence exists for
/// (R1-P1). `None` (the push path) means "no fence": the push derives its own
/// object list at admission and holds a write lease, so no sweep-style batch
/// snapshot crosses the dispatch boundary.
///
/// The fence deliberately does NOT hold any lock across the batch: a
/// visibility narrow (rule insert/remove, quarantine set) must commit
/// immediately, and the batch aborts on its next `is_current` check. Holding
/// a per-repo mutex from capture through drop would invert this — the narrow
/// would block behind the background sweep and every object in the batch
/// would still be sealed/posted to the reader being removed. The accepted
/// residual is a single in-flight object: a narrow that commits between the
/// pre-POST `is_current` check and the HTTP POST cannot be recalled, but the
/// next iteration aborts and the fenced DB record (row-locked against the
/// narrow's epoch bump) refuses to land the raced row as durable.
/// Multi-process / multi-node narrows are ordered by the same epoch column;
/// no in-process registry is involved.
///
/// Product decision (revocation model): prompt revocation wins over
/// publication atomicity. The suite intentionally allows one provider
/// upload already in flight when a rule narrows and requires every
/// LATER object to stop (`encrypt_and_pin_stops_sealing_when_reader_
/// removed_mid_batch`). A stronger contract — revocation commit
/// linearizing against every concurrent publication, or compensation
/// (unpin/delete) for an envelope that finishes after removal — is
/// explicitly OUT of scope until decided and documented here. Do not
/// reintroduce batch-spanning mutual exclusion to close the
/// single-in-flight window without that decision: it trades one
/// possibly-raced object for sealing the whole batch to a removed
/// reader.
#[derive(Clone)]
pub struct PolicyFence {
db: crate::db::Db,
repo_id: String,
epoch: i64,
}
impl PolicyFence {
/// Capture the current policy epoch for `repo_id`. A read failure is a
/// skip, not a retry-with-zero: the caller must not dispatch a batch it
/// cannot fence (fail closed on a stale allow).
pub async fn capture(db: &crate::db::Db, repo_id: &str) -> Option<Self> {
match db.repo_policy_epoch(repo_id).await {
Ok(epoch) => Some(PolicyFence {
db: db.clone(),
repo_id: repo_id.to_string(),
epoch,
}),
Err(e) => {
tracing::warn!(repo = %repo_id, err = %e, "policy-epoch read failed; not fencing pin batch");
None
}
}
}
/// Whether the repo's policy epoch is unchanged since capture. A read
/// failure is treated as "changed": never dispatch on a policy we cannot
/// prove current.
pub async fn is_current(&self) -> bool {
match self.db.repo_policy_epoch(&self.repo_id).await {
Ok(epoch) => epoch == self.epoch,
Err(_) => false,
}
}
/// The epoch value captured at `capture` time. Exposed so the
/// pinner can pass it to
/// `Db::record_pinned_cid_with_source_fenced` — the third
/// fence in the same transaction as the row insert.
/// Returning the field directly (rather than a `Option`)
/// matches the contract: a `PolicyFence` always has a
/// captured epoch; `is_current()` reports whether it still
/// matches.
pub fn captured_epoch(&self) -> i64 {
self.epoch
}
/// The repo this fence guards, for log correlation.
pub fn repo_id(&self) -> &str {
&self.repo_id
}
}
/// The smallest remainder worth starting a bounded git read (or an add) with.
///
/// A 1ms remainder otherwise buys a child spawned already past its deadline, which
@@ -1508,6 +1605,7 @@ pub async fn pin_git_object(
return Ok(String::new());
}
let expected_cid = Cid::from_git_object_bytes(data).to_string();
let url = format!(
"{}/api/v0/add?cid-version=1&raw-leaves=true&pin=true",
ipfs_api.trim_end_matches('/')
@@ -1550,6 +1648,15 @@ pub async fn pin_git_object(
// Kubo returns newline-delimited JSON; we only care about the last object
// (there's typically just one for a single-file add).
//
// The response MUST carry a real `Hash`: a misconfigured `GITLAWB_IPFS_API`
// (proxy returning HTML, health check on the wrong port, truncated gateway)
// can otherwise answer 2xx with no JSON, and falling back to the locally
// computed `expected_cid` would record a row for bytes the backend never
// stored. The reconciliation sweep trusts `pinned_cids` rows as durability
// evidence, so a silent false positive at pin time becomes a permanent blind
// spot for the backstop. A missing `Hash` fails the pin rather than recording
// a phantom row (mirrors Pinata's `data.cid` check).
let body = resp
.text()
.await
@@ -1565,13 +1672,26 @@ pub async fn pin_git_object(
let v: serde_json::Value = serde_json::from_str(line).ok()?;
v["Hash"].as_str().map(|s| s.to_string())
})
.next_back()
.ok_or_else(|| {
anyhow::anyhow!(
"IPFS /api/v0/add returned 200 without a usable Hash: {:.200}",
body
)
})?;
.next_back();
let cid = match cid {
Some(cid) => {
if cid != expected_cid {
tracing::warn!(
sha256 = %sha256_hex,
returned = %cid,
expected = %expected_cid,
"IPFS returned a different CID than computed locally (Kubo chunking may differ); recording the backend's answer"
);
}
cid
}
None => {
return Err(anyhow::anyhow!(
"IPFS /api/v0/add returned 2xx without a Hash field; refusing to record \
a CID the backend never acknowledged (misconfigured GITLAWB_IPFS_API?)"
));
}
};
// The string still has to be a CID. A garbage Hash would be recorded and
// later cat-ed the same way a missing one was, so reject it here.
Cid::from_str(&cid).map_err(|e| {
@@ -1654,8 +1774,8 @@ pub(crate) fn batch_budget_gate(
/// than [`PIN_READ_FLOOR`] left. It is a gate, not a hard ceiling, since a started
/// iteration still runs to completion;
/// - the git read: `store::read_object_bounded` runs under `spawn_blocking` against the
/// ABSOLUTE batch deadline (not the loop-top remainder, which the `is_pinned` round-trip
/// sitting between the two would push past it), with SIGTERM-then-SIGKILL
/// ABSOLUTE batch deadline (not the loop-top remainder, which the `has_ipfs_cid`
/// round-trip sitting between the two would push past it), with SIGTERM-then-SIGKILL
/// process-group teardown, so a hung `git cat-file` costs this batch its remaining
/// budget plus one watchdog teardown instead of holding the permit for the child's
/// whole lifetime and blocking a runtime worker while it does;
@@ -1699,10 +1819,11 @@ pub(crate) fn batch_budget_gate(
/// # Truncation semantics
///
/// A batch stopped at the deadline leaves its remaining objects unpinned, and
/// nothing sweeps them up afterwards. There is no reconciliation pass over
/// `pinned_cids`; recovery is opportunistic, happening only if some later push
/// on the repo takes the full-scan fallback (`push_delta::list_all_objects`) and
/// re-derives the whole object set, which then re-offers the skipped OIDs.
/// nothing sweeps them up afterwards on the push path; recovery is opportunistic
/// (a later full-scan push re-offers the skipped OIDs). The reconciliation
/// sweep is the systematic backstop: when it is enabled and a pin backend is
/// configured, it re-derives the public object set each pass and fills any
/// remaining gap.
///
/// The twin in `pinata.rs` is back at parity on everything that bounds or repairs an
/// object: it runs the same shared budget gate at the top of every iteration, the same
@@ -1719,6 +1840,29 @@ pub(crate) fn batch_budget_gate(
///
/// Returns a list of `(sha256_hex, cid)` pairs pinned AND durably recorded this
/// call.
/// What one `pin_new_objects` call (IPFS or Pinata twin) did, with the
/// three backend states kept apart instead of inferred from each other:
///
/// - `confirmed`: `(sha, provider_cid)` pairs whose DB record durably
/// landed. ONLY these may advance `gaps_filled`, branch/gossip CID
/// state, or any "repaired" bookkeeping. A provider upload whose
/// record timed out, failed, or was refused by the fence is absent
/// here even though the bytes may sit on the provider.
/// - `last_attempted`: the last OID whose loop body was entered — i.e.
/// the backend did real work for it (reads, an upload attempt, or a
/// skip-branch decision), whether that work succeeded, failed, or hit
/// an unknown outcome. `None` when nothing was entered (empty input,
/// immediate fence abort, or immediate budget gate). Durable
/// continuation cursors advance to this, never to the tail of the
/// planned vector: the tail may never have been visited, and
/// promoting it would rotate an untouched suffix behind the backlog
/// forever.
#[derive(Debug)]
pub struct PinBatchOutcome {
pub confirmed: Vec<(String, String)>,
pub last_attempted: Option<String>,
}
// Eight because #173's git seam (`git_bin`, `git_timeout`) and pin provenance
// (`repo_id`) sit alongside #174's batch budget. All four callers pass every one, and
// grouping them into a context struct would add a type whose only job is to be
@@ -1733,16 +1877,34 @@ pub async fn pin_new_objects(
db: &crate::db::Db,
repo_id: &str,
batch_budget: Duration,
) -> Vec<(String, String)> {
fence: Option<&PolicyFence>,
) -> PinBatchOutcome {
if ipfs_api.is_empty() {
return vec![];
return PinBatchOutcome {
confirmed: Vec::new(),
last_attempted: None,
};
}
let deadline = Instant::now() + batch_budget;
let total = object_list.len();
let mut pinned = Vec::new();
let mut last_attempted: Option<String> = None;
for (attempted, sha) in object_list.into_iter().enumerate() {
// Policy fence (R1-P1): a visibility narrow that lands after the caller
// built this batch must abort it before the next irreversible upload.
// Checked FIRST so a changed policy costs nothing beyond the read.
if let Some(f) = fence {
if !f.is_current().await {
tracing::warn!(
repo = %f.repo_id,
unattempted = total - attempted,
"visibility policy changed mid-batch; stopping the pin loop"
);
break;
}
}
// Top of the iteration, before any of this object's work: an object is
// never started with a remainder too small to cover a bounded read's
// teardown. Consumed as a guard only: the read below runs against the
@@ -1751,19 +1913,40 @@ pub async fn pin_new_objects(
if batch_budget_gate("IPFS", deadline, pinned.len(), total - attempted).is_none() {
break;
}
// Skip if already pinned, but first backfill provenance if the existing
// pin has none. A legacy pin (recorded before repo_id existed, #173, jatmn)
// is skipped here before record_pinned_cid ever runs, so its NULL provenance
// would never resolve to one repo and known CIDs keep hitting the scan. The
// backfill only sets repo_id (AND repo_id IS NULL guard preserves
// first-pinner-owns) and never re-pins the bytes: the object is already on IPFS.
// Every DB call from here to the end of the iteration is bounded by the
// ABSOLUTE batch deadline (F3, #173): the loop runs under a global pin permit
// and a bare await parked it for the whole stall. The elapsed arm is mapped per
// site below, never as a blanket "existing error arm": a timeout cancels the
// client future but not the statement Postgres is running, so it reports an
// UNKNOWN outcome, not a failed write.
match db_bounded(deadline, db.is_pinned(&sha)).await {
// Attempt progress (not a durability claim): from here the loop does
// real work for this OID — skip-branch reads, airgapped git reads,
// an upload attempt — whatever the outcome. The caller persists
// this as the continuation, never the planned vector's tail.
last_attempted = Some(sha.clone());
// Skip if the object is ALREADY a real local IPFS pin, but first
// backfill provenance if the existing pin has none. A legacy pin
// (recorded before repo_id existed, #173, jatmn) is skipped here
// before record_pinned_cid ever runs, so its NULL provenance would
// never resolve to one repo and known CIDs keep hitting the scan.
// The backfill only sets repo_id (AND repo_id IS NULL guard
// preserves first-pinner-owns) and never re-pins the bytes: the
// object is already on IPFS.
//
// #218 review P1a: this check keys on `has_ipfs_cid` (writer-owned
// `local_ipfs_provenance = TRUE`), NOT on row existence
// (`is_pinned`). A Pinata-only row is `is_pinned = true` but
// `has_ipfs_cid = false`: the bytes never reached the local IPFS
// daemon, only Pinata, and we MUST fall through to the local
// writer path so a real local pin lands. Using `is_pinned` here
// was the gap that made the Pinata-only → local-IPFS repair
// path inert: every sweep pass re-entered this arm, recorded
// the source, and continued without ever calling
// `pin_git_object` or `record_pinned_cid_with_source`. The flag
// stayed FALSE forever.
//
// Every DB call from here to the end of the iteration is bounded
// by the ABSOLUTE batch deadline (F3, #173): the loop runs under
// a global pin permit and a bare await parked it for the whole
// stall. The elapsed arm is mapped per site below, never as a
// blanket "existing error arm": a timeout cancels the client
// future but not the statement Postgres is running, so it
// reports an UNKNOWN outcome, not a failed write.
match db_bounded(deadline, db.has_ipfs_cid(&sha)).await {
Ok(true) => {
// Elapsed here is free to skip: these are reads, so a late server-side
// completion costs nothing, and the backfill's own `AND repo_id IS NULL`
@@ -1873,7 +2056,7 @@ pub async fn pin_new_objects(
}
Ok(false) => {}
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "DB error checking pinned status");
tracing::warn!(sha = %sha, err = %e, "DB error checking IPFS pinned status");
continue;
}
}
@@ -1887,7 +2070,7 @@ pub async fn pin_new_objects(
// own deadline regardless.
//
// The read runs against the ABSOLUTE batch deadline, not against the remainder
// measured at the top of the iteration: the `is_pinned` round-trip above sits
// measured at the top of the iteration: the `has_ipfs_cid` round-trip above sits
// between the two, so `Instant::now() + budget_left` would land past `deadline`
// by however long the DB took, and under a saturated pool that is the dominant
// term. A slow DB check must not push the read's own bound out.
@@ -1972,6 +2155,24 @@ pub async fn pin_new_objects(
break;
};
// Dispatch fence (R1-P1): re-read the policy epoch immediately before
// the irreversible HTTP POST. The iteration-top check catches a narrow
// that landed before work began; THIS check catches a narrow that landed
// during the has_ipfs_cid round-trip or the bounded Git read — both of
// which can take seconds and during which a quarantine or rule change may
// have committed. Without this, stale plaintext can start uploading
// under authorization that is no longer current.
if let Some(f) = fence {
if !f.is_current().await {
tracing::warn!(
repo = %f.repo_id,
unattempted = total - attempted,
"visibility policy changed during preparation; aborting IPFS upload"
);
break;
}
}
// Pin to IPFS
match pin_git_object(ipfs_api, &sha, &data, Some(add_timeout)).await {
Ok(cid) if !cid.is_empty() => {
@@ -2021,7 +2222,31 @@ pub async fn pin_new_objects(
// per-object failure.
match db_bounded(
db_record_deadline(deadline),
retry_db_record(|| db.record_pinned_cid_with_source(&sha, &raw_cid, repo_id)),
retry_db_record(|| {
// #218 review round 9 (guidance #3 —
// linearization): always go through the
// fenced form. The fence is either
// captured (sweep / public-pin path: the
// third fence is the linearization point
// that closes the rule-write /
// record-write race) or absent
// (push-side admission where the
// decision is made at request time —
// we pass `i64::MAX` as a sentinel that
// the fenced form treats as "no fence
// check"). The 3-arg
// `record_pinned_cid_with_source` is
// still available for tests that don't
// own a fence, but the production
// pinner routes through here.
let fence_epoch = fence.map(|f| f.captured_epoch()).unwrap_or(i64::MAX);
db.record_pinned_cid_with_source_fenced(
&sha,
&raw_cid,
repo_id,
fence_epoch,
)
}),
)
.await
{
@@ -2038,7 +2263,10 @@ pub async fn pin_new_objects(
}
}
pinned
PinBatchOutcome {
confirmed: pinned,
last_attempted,
}
}
#[cfg(test)]
@@ -2374,6 +2602,74 @@ mod tests {
);
}
/// The misconfigured-`GITLAWB_IPFS_API` false positive (P3): a 2xx response
/// that carries no `Hash` field (proxy returning HTML, health check on the
/// wrong port, truncated gateway) must FAIL the pin, not fall back to the
/// locally computed `expected_cid`. Falling back records a `pinned_cids`
/// row for bytes the backend never stored, and the reconciliation sweep
/// trusts rows as durability evidence — so the false positive becomes a
/// permanent blind spot for the backstop. A missing `Hash` must surface as
/// an explicit error, never a successful pin.
#[tokio::test]
async fn pin_git_object_rejects_a_2xx_without_a_hash_field() {
let endpoint = empty_ok_endpoint().await;
let inner = tokio::time::timeout(
Duration::from_secs(30),
pin_git_object(&endpoint, "deadbeef", b"some object bytes\n", None),
)
.await
.expect("wedge guard: an immediate empty 200 cannot take 30s");
let err = inner.expect_err(
"a 2xx without a Hash field must not surface as a successful pin \
(would record a phantom pinned_cids row the sweep then trusts)",
);
assert!(
err.to_string().contains("without a Hash field"),
"the error must name the missing Hash so operators diagnose the endpoint: {err:#}"
);
}
/// A 200 that answers with an empty body and no `Hash` — the exact shape of
/// a proxy or health-check endpoint mistaken for a Kubo API.
async fn empty_ok_endpoint() -> String {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let endpoint = format!("http://{}", listener.local_addr().unwrap());
tokio::spawn(async move {
while let Ok((mut sock, _)) = listener.accept().await {
tokio::spawn(async move {
let mut acc = Vec::new();
let mut buf = [0u8; 4096];
loop {
let n = match sock.read(&mut buf).await {
Ok(0) | Err(_) => break,
Ok(n) => n,
};
acc.extend_from_slice(&buf[..n]);
if let Some(hdr_end) =
acc.windows(4).position(|w| w == b"\r\n\r\n").map(|p| p + 4)
{
let headers = String::from_utf8_lossy(&acc[..hdr_end]).to_lowercase();
let len: usize = headers
.lines()
.find_map(|l| l.strip_prefix("content-length:"))
.and_then(|v| v.trim().parse().ok())
.unwrap_or(0);
if acc.len() >= hdr_end + len {
break;
}
}
}
let _ = sock
.write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n")
.await;
let _ = sock.flush().await;
});
}
});
endpoint
}
/// The second unhardened sink, reached from `sync.rs`. Same shape as above.
#[tokio::test]
async fn cat_against_silent_endpoint_errors_within_its_own_timeout() {
@@ -2486,15 +2782,16 @@ mod tests {
&db,
"repo-batch-budget",
Duration::from_millis(5500),
None,
),
)
.await
.expect("wedge guard: a 5.5s budget cannot take 30s");
assert!(
(1..=3).contains(&pinned.len()),
(1..=3).contains(&pinned.confirmed.len()),
"the batch must stop partway, not pin all five and not stall on the first: pinned {}",
pinned.len()
pinned.confirmed.len()
);
let text = logs.text();
let warns: Vec<&str> = text
@@ -2516,9 +2813,52 @@ mod tests {
})
.unwrap_or_else(|| panic!("the deadline warn must name the unattempted count: {text}"));
assert!(
unattempted >= 1 && unattempted + pinned.len() <= 5,
unattempted >= 1 && unattempted + pinned.confirmed.len() <= 5,
"unattempted={unattempted} with {} pinned is not a partial batch of five",
pinned.len()
pinned.confirmed.len()
);
}
/// The outcome reports the last object actually ENTERED, not the planned
/// tail: the first upload hangs 6s against a ~2s per-request timeout, so
/// it fails, and the loop-top gate breaks the batch before the second
/// object starts. `last_attempted` must be the first OID — the only one
/// the loop body entered — even though nothing was confirmed and two
/// OIDs were never visited. A cursor persisted from `to_pin.last()`
/// would rotate the untouched suffix behind the backlog forever.
#[sqlx::test]
async fn pin_new_objects_reports_last_attempted_not_planned_tail(pool: sqlx::PgPool) {
let db = crate::db::Db::for_testing(pool);
db.run_migrations().await.expect("migrations");
let tmp = tempfile::TempDir::new().unwrap();
let repo_path = tmp.path().join("attempted.git");
let oids = seed_loose_blobs(&repo_path, 3);
let endpoint = delaying_endpoint(vec![Duration::from_secs(6)]).await;
let outcome = tokio::time::timeout(
Duration::from_secs(30),
pin_new_objects(
&endpoint,
&repo_path,
"git",
Duration::from_secs(30),
oids.clone(),
&db,
"repo-attempted",
Duration::from_secs(2),
None,
),
)
.await
.expect("a 2s budget cannot take 30s");
assert_eq!(
outcome.last_attempted,
Some(oids[0].clone()),
"only the first object was entered; the planned tail was never visited"
);
assert!(
outcome.confirmed.is_empty(),
"the hung upload timed out, so nothing was confirmed"
);
}
@@ -2553,12 +2893,13 @@ mod tests {
&db,
"repo-batch-continues",
Duration::from_secs(90),
None,
),
)
.await
.expect("wedge guard: a 13s add plus an immediate one cannot take 60s");
assert_eq!(
pinned.len(),
pinned.confirmed.len(),
2,
"a slow but progressing endpoint must pin both objects: an upload past the client's \
10s default is not a dead endpoint"
@@ -2591,11 +2932,12 @@ mod tests {
&db,
"repo-batch-rejects",
Duration::from_secs(60),
None,
),
)
.await
.expect("a rejecting endpoint answers immediately, so this cannot take 30s");
assert!(pinned.is_empty(), "every add was rejected");
assert!(pinned.confirmed.is_empty(), "every add was rejected");
assert_eq!(
requests.load(std::sync::atomic::Ordering::SeqCst),
4,
@@ -2689,6 +3031,7 @@ mod tests {
&db,
"repo-merge-test",
Duration::from_secs(2),
None,
),
)
.await
@@ -2699,7 +3042,7 @@ mod tests {
let elapsed = started.elapsed();
assert!(
pinned.is_empty(),
pinned.confirmed.is_empty(),
"a git that never answers cannot produce a pinned object: {pinned:?}"
);
assert!(
@@ -2776,6 +3119,7 @@ mod tests {
&db,
"repo-merge-test",
Duration::from_millis(1500),
None,
),
)
.await
@@ -2790,7 +3134,7 @@ mod tests {
that can only be spawned and reaped"
);
assert_eq!(
pinned.len(),
pinned.confirmed.len(),
1,
"the first object is inside the budget and must pin: {pinned:?}"
);
@@ -2849,6 +3193,7 @@ mod tests {
&db,
"repo-merge-test",
Duration::from_secs(60),
None,
),
)
.await
@@ -2858,7 +3203,7 @@ mod tests {
if genuinely_unreadable {
assert!(
pinned.is_empty(),
pinned.confirmed.is_empty(),
"nothing can be pinned through a store that cannot be read: {pinned:?}"
);
assert_eq!(
@@ -2923,6 +3268,7 @@ mod tests {
&db,
"repo-merge-test",
Duration::from_secs(60),
None,
),
)
.await
@@ -2938,7 +3284,7 @@ mod tests {
must still be attempted, got {attempted} of {}",
oids.len()
);
let pinned_shas: Vec<&String> = pinned.iter().map(|(sha, _)| sha).collect();
let pinned_shas: Vec<&String> = pinned.confirmed.iter().map(|(sha, _)| sha).collect();
let expected: Vec<&String> = oids.iter().filter(|o| !tainted.contains(o)).collect();
assert_eq!(
pinned_shas, expected,
@@ -2991,6 +3337,7 @@ mod tests {
&db,
"repo-merge-test",
Duration::from_secs(60),
None,
),
)
.await
@@ -3002,7 +3349,7 @@ mod tests {
"an object-scoped fault must not stop the batch: every object must be read"
);
assert_eq!(
pinned.len(),
pinned.confirmed.len(),
4,
"one corrupt object must cost only itself: the other four must still pin"
);
@@ -3225,6 +3572,7 @@ mod tests {
&db,
"repo-stalled-db",
Duration::from_millis(1500),
None,
),
)
.await
@@ -3235,7 +3583,7 @@ mod tests {
let elapsed = started.elapsed();
assert!(
pinned.is_empty(),
pinned.confirmed.is_empty(),
"a stalled pinned-status check cannot produce a pinned object: {pinned:?}"
);
assert!(
@@ -3295,6 +3643,7 @@ mod tests {
&db,
"repo-skip-stalled",
Duration::from_millis(1500),
None,
),
)
.await
@@ -3305,7 +3654,7 @@ mod tests {
let elapsed = started.elapsed();
assert!(
pinned.is_empty(),
pinned.confirmed.is_empty(),
"an already-pinned object is skipped, never re-pinned: {pinned:?}"
);
assert!(
@@ -3361,13 +3710,17 @@ mod tests {
&db,
"repo-multi-stalled",
Duration::from_millis(1500),
None,
),
)
.await
.expect("three stalled objects must still cost one budget, not three");
let elapsed = started.elapsed();
assert!(pinned.is_empty(), "nothing can pin against a stalled DB");
assert!(
pinned.confirmed.is_empty(),
"nothing can pin against a stalled DB"
);
assert!(
elapsed < Duration::from_secs(3),
"three stalled objects must charge ONE budget (1.5s), not one each; got {elapsed:?}"
@@ -3390,7 +3743,7 @@ mod tests {
///
/// The lock time is a MARGIN, not a boundary: taking it at 100ms left `is_pinned`
/// racing it on a loaded box, and losing that race makes the read block, time out,
/// and break the batch, which fails on `pinned.len() == 1` for a reason that has
/// and break the batch, which fails on `pinned.confirmed.len() == 1` for a reason that has
/// nothing to do with the floor. Any time between the `is_pinned` round trip and
/// the add's 1.7s return proves the same thing.
#[sqlx::test]
@@ -3428,6 +3781,7 @@ mod tests {
&db,
"repo-spent-budget",
Duration::from_millis(2000),
None,
),
);
let (pinned, ()) = tokio::join!(pin, locker);
@@ -3440,7 +3794,7 @@ mod tests {
nothing can resolve the CID"
);
assert_eq!(
pinned.len(),
pinned.confirmed.len(),
1,
"the durably recorded pin must be returned: {pinned:?}"
);
@@ -3501,6 +3855,7 @@ mod tests {
&db,
"repo-definite-error",
Duration::from_millis(1200),
None,
),
);
let (pinned, ()) = tokio::join!(pin, commit);
@@ -3582,6 +3937,7 @@ mod tests {
&db,
"repo-marker-floor",
Duration::from_millis(1500),
None,
),
);
let (pinned, ()) = tokio::join!(pin, controller);
@@ -3592,7 +3948,7 @@ mod tests {
drop(sources_lock);
assert!(
pinned.is_empty(),
pinned.confirmed.is_empty(),
"an already-pinned object is skipped, never re-pinned: {pinned:?}"
);
assert!(
@@ -3686,6 +4042,7 @@ mod tests {
&db,
"repo-repair-stalled",
Duration::from_millis(2200),
None,
),
);
let (pinned, mut cids_lock) = tokio::join!(pin, controller);
+311 -52
View File
@@ -7,6 +7,7 @@ mod bootstrap;
mod cert;
mod config;
mod db;
mod durable_outbox;
mod encrypted_pin;
mod error;
mod git;
@@ -18,11 +19,13 @@ mod operator;
mod p2p;
mod pinata;
mod rate_limit;
mod reconciliation;
mod server;
mod state;
mod sync;
#[cfg(test)]
mod test_support;
mod upstream_mirror;
mod visibility;
mod webhooks;
@@ -306,10 +309,15 @@ async fn main() -> Result<()> {
let (ref_update_tx, _) = tokio::sync::broadcast::channel::<state::RefUpdateBroadcast>(256);
let (task_event_tx, _) = tokio::sync::broadcast::channel::<state::TaskEventBroadcast>(256);
// Node-keyed, so continuation tokens this node mints are only accepted by
// this node and survive its restarts without any configured secret.
let task_cursor_key = api::task_cursor::TaskCursorKey::derive(&keypair.to_seed());
let graphql_schema = Arc::new(graphql::build_schema(
Arc::clone(&db),
ref_update_tx.clone(),
task_event_tx.clone(),
task_cursor_key.clone(),
));
let machine_id = std::env::var("FLY_MACHINE_ID").ok();
@@ -350,9 +358,14 @@ async fn main() -> Result<()> {
);
// Per-DID limiter for the creation endpoints. Keyed on the authenticated
// DID (attacker-varied), so bound its key set to cap memory.
let rate_limiter =
rate_limit::RateLimiter::new_bounded(10, std::time::Duration::from_secs(3600), 200_000);
// DID (attacker-varied), so bound its key set to cap memory. Max and window
// are operator-configurable via GITLAWB_RATE_LIMIT_MAX /
// GITLAWB_RATE_LIMIT_WINDOW_SECS; both reject 0 at parse time.
let rate_limiter = rate_limit::RateLimiter::new_bounded(
config.rate_limit_max_requests as usize,
std::time::Duration::from_secs(config.rate_limit_window_secs),
200_000,
);
// Per-client-IP flood brake for the creation endpoints. The per-DID limiter
// above is bypassed by a DID farm (one throwaway did:key per repo), which is
@@ -450,6 +463,7 @@ async fn main() -> Result<()> {
db,
node_did: node_did.clone(),
node_keypair: keypair,
task_cursor_key,
p2p: p2p_handle,
http_client,
ref_update_tx,
@@ -548,11 +562,19 @@ async fn main() -> Result<()> {
std::time::Duration::from_secs(3600),
200_000,
),
task_read_rate_limiter: rate_limit::RateLimiter::new_bounded(
config.task_read_rate_limit,
std::time::Duration::from_secs(3600),
200_000,
),
git_bin: "git".to_string(),
};
if config.ipfs_rate_limit == 0 {
tracing::warn!("GITLAWB_IPFS_RATE_LIMIT=0 — per-IP /ipfs rate limiting disabled");
}
if config.task_read_rate_limit == 0 {
tracing::warn!("GITLAWB_TASK_READ_RATE_LIMIT=0 — per-IP task read rate limiting disabled");
}
// Periodic peer-count poll for the metrics gauge. If p2p is disabled
// we still set the gauge to 0 so dashboards don't show "no data".
@@ -609,6 +631,13 @@ async fn main() -> Result<()> {
}
let _legacy_cid_sweep = spawn_legacy_cid_sweep(&state, &config);
let _queue_lifecycle_sweep = spawn_queue_lifecycle_sweep(&state, &config);
// NOTE: the due-request worker is spawned AFTER the one-shot startup
// reconcile/drain below, not here: its 5s interval would otherwise tick
// during operator/PoS setup and interleave with boot recovery on backlog
// rows (state gates prevent data loss, but the interleave costs recovery
// latency and widens the reconcile/drain race window).
let _marker_cleanup_worker = spawn_marker_cleanup_worker(&state);
let router = server::build_router(state.clone());
// Re-register the socket bound at startup — same fd, so there was never a
@@ -666,6 +695,46 @@ async fn main() -> Result<()> {
info!("auto-sync worker started");
}
// External-forge mirroring is a separate, default-off worker. Startup
// validation already requires owner-only pushes when it is enabled; the
// worker additionally verifies the Git runtime can pin DNS answers before
// any URL reaches `git fetch`.
if config.upstream_mirror_enabled {
upstream_mirror::start(
Arc::clone(&state.db),
Arc::clone(&state.config),
state.repo_store.clone(),
state.repo_write_leases.clone(),
Arc::clone(&state.git_write_semaphore),
state.subscribe_shutdown(),
)
.context("starting upstream mirror worker")?;
}
// Periodic reconciliation sweep: re-derives pin/seal sets and fills gaps
// so a dropped replication job never means data loss.
{
let db = state.db.clone();
let config = Arc::clone(&state.config);
let http_client = Arc::clone(&state.http_client);
let node_keypair = Arc::clone(&state.node_keypair);
let node_did = state.node_did.clone();
let pin_sem = Arc::clone(&state.pin_semaphore);
let shutdown_rx = state.subscribe_shutdown();
if reconciliation::spawn(
db,
config,
http_client,
node_keypair,
node_did,
pin_sem,
shutdown_rx,
Some(state.repo_store.clone()),
) {
info!("reconciliation sweep worker started");
}
}
// On-chain operator setup: verify stake + spawn heartbeat loop
if !state.config.contract_node_staking.is_empty()
&& !state.config.operator_private_key.is_empty()
@@ -700,6 +769,63 @@ async fn main() -> Result<()> {
let grace = std::time::Duration::from_secs(config.shutdown_grace_secs);
info!(grace_secs = config.shutdown_grace_secs, "axum server ready");
// #26 Split PR 1: drain any `applied` rows left by a previous
// process that crashed after Git applied a ref but before the
// bookkeeping landed. Runs once, BEFORE the server accepts new
// pushes, so a recovery re-derivation does not race a fresh push.
// Non-fatal: a transient drain failure is logged and the rows
// remain `applied` for the next startup to pick up.
//
// P1-A: the reconcile step runs FIRST and promotes any `prepared`
// or `uncertain` row whose target SHA actually landed on disk.
// This is the path that recovers a ref when the post-receive
// `mark_pending_ref_transitions_applied` call errored or was
// interrupted after `receive_pack` returned Ok, or when
// receive-pack returned Err but some refs may have landed
// (the `uncertain` state). Without this step, the drain (gated
// on `state = 'applied'`) would never see those rows.
//
// P2 (reviewer-1/2 round 3): use the multi-pass reconcile so
// prepared/uncertain rows beyond the first 1000-row page are
// processed in the same startup, rather than waiting for the
// next restart (where they might age out of MAX_RECONCILE_AGE).
match durable_outbox::reconcile_prepared_from_disk_all(
state.clone(),
durable_outbox::DRAIN_PER_PASS_LIMIT,
durable_outbox::DRAIN_MAX_PASSES,
)
.await
{
Ok(0) => {}
Ok(n) => info!(
n,
"reconciled prepared/uncertain -> applied via on-disk ref match"
),
Err(e) => warn!(
err = %e,
"pending ref transition reconcile failed at startup (non-fatal; will retry on next start)"
),
}
match durable_outbox::drain_receive_pack_requests_all(
state.clone(),
durable_outbox::DRAIN_PER_PASS_LIMIT,
durable_outbox::DRAIN_MAX_PASSES,
)
.await
{
Ok(0) => {}
Ok(n) => info!(n, "drained pending ref transitions from prior run"),
Err(e) => warn!(
err = %e,
"pending ref transition drain failed at startup (non-fatal; will retry on next start)"
),
}
// Start the due-request worker only now that boot recovery is done (see
// the note at the other spawns above): its first tick must not interleave
// with the one-shot reconcile/drain on backlog rows.
let _due_request_worker = spawn_due_request_worker(&state);
// `into_make_service_with_connect_info` exposes the socket peer address as
// `ConnectInfo<SocketAddr>` so the push limiter can key on the real client
// when no trusted proxy header applies (see `rate_limit::client_key`).
@@ -732,6 +858,158 @@ async fn main() -> Result<()> {
Ok(())
}
/// U4 (#173): spawn the periodic legacy provider-CID repair sweep. Releases before this
/// version stored the PROVIDER CID (Kubo dag-pb / Pinata CIDv0) in `pinned_cids.cid`,
/// and this version's `/ipfs/{cid}` resolver withholds any row whose stored key is not
/// the raw-content CID. The opportunistic repair on the pin path only fires when a push
/// re-carries the object, which normal git negotiation makes it not do, so those rows
/// need a walk. DETACHED, never on the boot path: the caller keeps serving while this
/// runs, and the sweep's own batch bound plus inter-batch delay keep it off the DB's
/// critical path. Its cursor is durable, so a restart mid-walk resumes instead of
/// rewinding. It re-arms after every run, including a failed one, so it never returns
/// and there is no awaited value to log here; the shutdown watcher below is what ends
/// it.
///
/// A named function rather than an inline block in `main` so the WIRING has a seam a
/// test can call: that the task is spawned at all, that it reads its batch and delay
/// from the config knobs rather than some other field, that the caller is not blocked
/// on it, and that the shutdown watcher actually ends it mid-walk. The sweep's own
/// behavior is covered elsewhere; this is the boot-path half.
fn spawn_legacy_cid_sweep(state: &AppState, config: &Config) -> tokio::task::JoinHandle<()> {
let db = state.db.clone();
let repos_dir = config.repos_dir.clone();
let git_bin = state.git_bin.clone();
let git_timeout = std::time::Duration::from_secs(config.git_service_timeout_secs);
let batch = config.pin_repair_sweep_batch;
let delay = std::time::Duration::from_secs(config.pin_repair_sweep_delay_secs);
let mut shutdown_rx = state.subscribe_shutdown();
tokio::spawn(async move {
tokio::select! {
_ = ipfs_pin::run_sweep_rearmed(
&repos_dir, &git_bin, git_timeout, batch, delay,
ipfs_pin::SWEEP_REARM_DELAY, &db,
) => {}
// Shutdown mid-walk simply drops the run; the persisted cursor means the
// next boot picks up where this one stopped.
_ = shutdown_rx.changed() => {}
}
})
}
/// #26 Split PR 1 step 4 — periodic queue-lifecycle purge. Runs on
/// the same detached task pattern as `spawn_legacy_cid_sweep`:
/// tokio::spawn with a shutdown watcher, never on the boot path. The
/// interval is fixed at 24 hours (the spec calls for "one per
/// cluster per day"); the inter-batch delay is implicit in the
/// batch size plus the wall-clock cost of each pass. The drain
/// (`drain_receive_pack_requests_all`) and the purge
/// (`purge_request_queue`) share the same `DRAIN_PER_PASS_LIMIT`
/// budget so a 1000-row purge pass takes roughly the same time as
/// a 1000-row drain pass.
fn spawn_queue_lifecycle_sweep(state: &AppState, config: &Config) -> tokio::task::JoinHandle<()> {
let app_state = state.clone();
let retention_days = config.queue_retention_days;
let batch = config.queue_purge_batch;
let git_timeout = std::time::Duration::from_secs(config.git_service_timeout_secs.min(30));
let mut shutdown_rx = state.subscribe_shutdown();
tokio::spawn(async move {
let mut interval = tokio::time::interval(std::time::Duration::from_secs(24 * 3600));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
// Consume the immediate first tick: Tokio intervals fire
// immediately on creation, which would run the purge
// concurrently with startup reconcile/drain. The child purge
// is terminal-parent-gated in SQL, but delaying the first
// sweep a full period removes the race entirely.
interval.tick().await;
loop {
tokio::select! {
_ = interval.tick() => {
if let Err(e) = durable_outbox::purge_request_queue(
&app_state.db,
retention_days,
batch,
).await {
tracing::warn!(err = %e, "queue lifecycle purge failed; will retry on next tick");
}
// Drain marker tombstones with the bounded deleter;
// failures retain the tombstone for the next tick.
if let Err(e) = durable_outbox::drain_marker_cleanup_queue(
&app_state,
batch,
git_timeout,
).await {
tracing::warn!(err = %e, "marker cleanup drain failed");
}
}
_ = shutdown_rx.changed() => {
break;
}
}
}
})
}
/// Shutdown-aware background due-request loop. The startup drain runs
/// once; without this, `effects_pending` rows with a future
/// `next_attempt_at` would wait until the next process restart.
/// Polls the indexed due query with bounded batches and failure
/// isolation (one request's error never aborts the batch).
fn spawn_due_request_worker(state: &AppState) -> tokio::task::JoinHandle<()> {
let app_state = state.clone();
let mut shutdown_rx = state.subscribe_shutdown();
tokio::spawn(async move {
let mut interval = tokio::time::interval(std::time::Duration::from_secs(5));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
loop {
tokio::select! {
_ = interval.tick() => {
match durable_outbox::drain_receive_pack_requests(
app_state.clone(),
durable_outbox::DRAIN_PER_PASS_LIMIT,
)
.await
{
Ok((0, _)) => {}
Ok((n, _)) => tracing::info!(n, "due request worker drained"),
Err(e) => tracing::warn!(err = %e, "due request worker drain failed"),
}
}
_ = shutdown_rx.changed() => break,
}
}
})
}
/// Dedicated marker-tombstone worker at a shorter cadence than the
/// daily purge, so external Git refs do not linger a full period
/// after their SQL owner is gone.
fn spawn_marker_cleanup_worker(state: &AppState) -> tokio::task::JoinHandle<()> {
let app_state = state.clone();
let git_timeout =
std::time::Duration::from_secs(app_state.config.git_service_timeout_secs.min(30));
let mut shutdown_rx = state.subscribe_shutdown();
tokio::spawn(async move {
let mut interval = tokio::time::interval(std::time::Duration::from_secs(60));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
loop {
tokio::select! {
_ = interval.tick() => {
if let Err(e) = durable_outbox::drain_marker_cleanup_queue(
&app_state,
durable_outbox::DRAIN_PER_PASS_LIMIT,
git_timeout,
)
.await
{
tracing::warn!(err = %e, "marker cleanup worker failed");
}
}
_ = shutdown_rx.changed() => break,
}
}
})
}
fn spawn_shutdown_signal(tx: watch::Sender<bool>) {
tokio::spawn(async move {
#[cfg(unix)]
@@ -1187,6 +1465,8 @@ mod rate_limiter_sweep_tests {
state.peer_write_rate_limiter = RateLimiter::new(10, window);
state.federated_rate_limiter = RateLimiter::new(10, window);
state.ipfs_rate_limiter = RateLimiter::new(10, window);
state.ipfs_work_rate_limiter = RateLimiter::new(10, window);
state.task_read_rate_limiter = RateLimiter::new(10, window);
let limiters = |s: &crate::state::AppState| {
[
@@ -1198,6 +1478,8 @@ mod rate_limiter_sweep_tests {
s.peer_write_rate_limiter.clone(),
s.federated_rate_limiter.clone(),
s.ipfs_rate_limiter.clone(),
s.ipfs_work_rate_limiter.clone(),
s.task_read_rate_limiter.clone(),
]
};
for l in limiters(&state) {
@@ -1215,55 +1497,6 @@ mod rate_limiter_sweep_tests {
}
/// it.
fn spawn_legacy_cid_sweep(state: &AppState, config: &Config) -> tokio::task::JoinHandle<()> {
let db = state.db.clone();
let repos_dir = config.repos_dir.clone();
let git_bin = state.git_bin.clone();
let git_timeout = std::time::Duration::from_secs(config.git_service_timeout_secs);
let batch = config.pin_repair_sweep_batch;
let delay = std::time::Duration::from_secs(config.pin_repair_sweep_delay_secs);
let mut shutdown_rx = state.subscribe_shutdown();
tokio::spawn(async move {
tokio::select! {
_ = ipfs_pin::run_sweep_rearmed(
&repos_dir, &git_bin, git_timeout, batch, delay,
ipfs_pin::SWEEP_REARM_DELAY, &db,
) => {}
_ = shutdown_rx.changed() => {}
}
})
}
async fn gossip_ping_round(
db: &Db,
client: &reqwest::Client,
failed_once: &mut HashSet<String>,
peers: Vec<db::PeerRecord>,
) {
{
let current_dids: HashSet<&str> = peers.iter().map(|peer| peer.did.as_str()).collect();
failed_once.retain(|did| current_dids.contains(did.as_str()));
}
for peer in peers {
let ok = ping_peer_readiness(client, &peer.http_url).await;
match peer_ping_db_update(failed_once, &peer.did, ok) {
Some(reachable) => {
if let Err(error) = db.mark_peer_ping(&peer.did, reachable).await {
warn!(did = %peer.did, err = %error, "failed to persist peer readiness");
}
}
None => warn!(
did = %peer.did,
"peer readiness probe failed once; preserving previous federation status"
),
}
}
}
/// Decide whether one readiness sample should update the persisted federation
/// gate. A success is authoritative immediately. A failure must repeat on the
/// next gossip tick before it can mark a peer unreachable, so one transient
/// database hiccup does not hide the peer's repositories for five minutes.
fn peer_ping_db_update(failed_once: &mut HashSet<String>, did: &str, ready: bool) -> Option<bool> {
if ready {
failed_once.remove(did);
@@ -1440,6 +1673,32 @@ fn load_or_create_keypair_at(key_path: &std::path::Path) -> Result<Keypair> {
Ok(kp)
}
async fn gossip_ping_round(
db: &Db,
client: &reqwest::Client,
failed_once: &mut HashSet<String>,
peers: Vec<db::PeerRecord>,
) {
{
let current_dids: HashSet<&str> = peers.iter().map(|peer| peer.did.as_str()).collect();
failed_once.retain(|did| current_dids.contains(did.as_str()));
}
for peer in peers {
let ok = ping_peer_readiness(client, &peer.http_url).await;
match peer_ping_db_update(failed_once, &peer.did, ok) {
Some(reachable) => {
if let Err(error) = db.mark_peer_ping(&peer.did, reachable).await {
warn!(did = %peer.did, err = %error, "failed to persist peer readiness");
}
}
None => warn!(
did = %peer.did,
"peer readiness probe failed once; preserving previous federation status"
),
}
}
}
#[cfg(test)]
mod gossip_ssrf_tests {
use super::{
+52
View File
@@ -20,6 +20,9 @@
//! `gitlawb_pack_size_bytes`
//! * a single `gitlawb_info{version, did}` gauge = 1, for joins/dashboards
//! * currently-connected peer count — `gitlawb_peers_connected`
//! * reconciliation sweep gaps found and filled —
//! `gitlawb_reconciliation_gaps_found_total` /
//! `gitlawb_reconciliation_gaps_filled_total`
//!
//! All metrics live in a single process-wide registry initialized by
//! [`init`]. Increment helpers (`record_push`, `record_auth_failure`, ...)
@@ -58,6 +61,8 @@ static GOSSIP_INGEST: OnceLock<IntCounterVec> = OnceLock::new();
static GOSSIP_REPLAY_GUARD_SATURATED: OnceLock<IntCounter> = OnceLock::new();
static PACK_SIZE: OnceLock<Histogram> = OnceLock::new();
static PEERS_CONNECTED: OnceLock<IntGauge> = OnceLock::new();
static RECONCILIATION_GAPS_FOUND: OnceLock<IntCounter> = OnceLock::new();
static RECONCILIATION_GAPS_FILLED: OnceLock<IntCounter> = OnceLock::new();
/// 1 when the node started without its p2p identity because the key storage
/// could not be used. The node keeps serving HTTP in that state and /health
/// stays green, so without this the outage is visible only to whoever reads
@@ -241,6 +246,29 @@ fn init_inner(version: &str, node_did: &str) {
.set(peers_connected)
.expect("set PEERS_CONNECTED once");
let gaps_found = IntCounter::with_opts(Opts::new(
"gitlawb_reconciliation_gaps_found_total",
"Total reconciliation sweep gaps detected (objects that should be pinned but are not)",
))
.expect("gitlawb_reconciliation_gaps_found_total definition");
registry
.register(Box::new(gaps_found.clone()))
.expect("register gitlawb_reconciliation_gaps_found_total");
RECONCILIATION_GAPS_FOUND
.set(gaps_found)
.expect("set RECONCILIATION_GAPS_FOUND once");
let gaps_filled = IntCounter::with_opts(Opts::new(
"gitlawb_reconciliation_gaps_filled_total",
"Total reconciliation sweep gaps successfully filled (objects pinned by the sweep)",
))
.expect("gitlawb_reconciliation_gaps_filled_total definition");
registry
.register(Box::new(gaps_filled.clone()))
.expect("register gitlawb_reconciliation_gaps_filled_total");
RECONCILIATION_GAPS_FILLED
.set(gaps_filled)
.expect("set RECONCILIATION_GAPS_FILLED once");
let p2p_key_load_failed = IntGauge::with_opts(Opts::new(
"gitlawb_p2p_identity_key_load_failed",
"1 if the node is running without p2p because its identity key could not be loaded",
@@ -397,6 +425,20 @@ pub fn set_peers_connected(count: i64) {
}
}
/// Record reconciliation sweep gaps found (objects that should be pinned but are not).
pub fn record_reconciliation_gaps_found(count: u64) {
if let Some(c) = RECONCILIATION_GAPS_FOUND.get() {
c.inc_by(count);
}
}
/// Record reconciliation sweep gaps filled (objects successfully pinned by the sweep).
pub fn record_reconciliation_gaps_filled(count: u64) {
if let Some(c) = RECONCILIATION_GAPS_FILLED.get() {
c.inc_by(count);
}
}
/// Record whether the node is running without its p2p identity.
pub fn set_p2p_key_load_failed(failed: bool) {
if let Some(g) = P2P_KEY_LOAD_FAILED.get() {
@@ -441,6 +483,8 @@ mod tests {
.expect("PUSHES set after init")
.with_label_values(&["alice/repo"])
.inc();
record_reconciliation_gaps_found(7);
record_reconciliation_gaps_filled(3);
let body = encode().expect("encode should succeed after init");
assert!(
@@ -455,6 +499,14 @@ mod tests {
body.contains("gitlawb_pushes_total{repo=\"alice/repo\"} 1"),
"expected the incremented counter to be visible in: {body}"
);
assert!(
body.contains("gitlawb_reconciliation_gaps_found_total 7"),
"expected the reconciliation gaps-found counter to be visible in: {body}"
);
assert!(
body.contains("gitlawb_reconciliation_gaps_filled_total 3"),
"expected the reconciliation gaps-filled counter to be visible in: {body}"
);
}
/// #192 F4: `init` is idempotent and safe to call repeatedly. The panic that
+489 -41
View File
@@ -153,6 +153,26 @@ pub async fn pin_object(
/// `upsert_branch_cid` and the p2p `publish_ref_update` gossip CID. The twin's return is
/// log-only, so it omits a record-failed pin rather than logging a pin the resolver
/// cannot serve. Moving this side to match would need that consumer moved first.
/// Whether the committed `pinned_cids` row already names `repo_id` as
/// first pinner, making a failed redundant `record_pin_source` insert
/// irrelevant to resolvability (`pin_sources_for_oid` unions the
/// primary row). Bounded read; any failure (including its own
/// timeout) answers false — an unproven row never confirms.
/// Used only after the primary `record_pinata_cid` for this call
/// succeeded or verified: callers must not promote a row this call
/// did not establish.
async fn primary_covers_repo(
db: &crate::db::Db,
sha: &str,
repo_id: &str,
deadline: std::time::Instant,
) -> bool {
matches!(
crate::ipfs_pin::db_bounded(deadline, db.provenance_for_oid(sha)).await,
Ok(Some(owner)) if owner.as_str() == repo_id
)
}
// Ten arguments, over clippy's threshold: the three the budget and the git seam add
// (`git_bin`, `git_timeout`, `batch_budget`) plus #173's `repo_id` are what put the read
// under test injection and under a deadline, and grouping them into a struct would only
@@ -170,16 +190,33 @@ pub async fn pin_new_objects(
db: &crate::db::Db,
repo_id: &str,
batch_budget: Duration,
) -> Vec<(String, String)> {
fence: Option<&crate::ipfs_pin::PolicyFence>,
) -> crate::ipfs_pin::PinBatchOutcome {
if jwt.is_empty() {
return vec![];
return crate::ipfs_pin::PinBatchOutcome {
confirmed: Vec::new(),
last_attempted: None,
};
}
let deadline = Instant::now() + batch_budget;
let total = object_list.len();
let mut pinned = Vec::new();
let mut last_attempted: Option<String> = None;
for (attempted, sha) in object_list.into_iter().enumerate() {
// Policy fence (R1-P1): a visibility narrow that lands after the caller
// built this batch must abort it before the next irreversible upload.
if let Some(f) = fence {
if !f.is_current().await {
tracing::warn!(
repo = %f.repo_id(),
unattempted = total - attempted,
"visibility policy changed mid-batch; stopping the Pinata pin loop"
);
break;
}
}
// Top of the iteration, before any of this object's work: an object is never
// started with a remainder too small to cover a bounded read's teardown. The
// gate is shared with the IPFS loop so the two cannot drift apart in how they
@@ -191,6 +228,10 @@ pub async fn pin_new_objects(
{
break;
}
// Attempt progress (not a durability claim): same contract as the
// IPFS twin — the continuation persists this, never the planned
// vector's tail.
last_attempted = Some(sha.clone());
// Every DB call from here to the end of the iteration is bounded by the
// ABSOLUTE batch deadline (F3, #173), through the same `db_bounded` helper the
@@ -395,6 +436,21 @@ pub async fn pin_new_objects(
}
};
// Dispatch fence (R1-P1): re-read the policy epoch immediately before
// the irreversible HTTP POST. The iteration-top check catches a narrow
// that landed before work began; THIS check catches a narrow that landed
// during the has_pinata_cid round-trip or the bounded Git read.
if let Some(f) = fence {
if !f.is_current().await {
tracing::warn!(
repo = %f.repo_id(),
unattempted = total - attempted,
"visibility policy changed during preparation; aborting Pinata upload"
);
break;
}
}
match pin_object(client, upload_url, jwt, &sha, &data).await {
Ok(cid) if !cid.is_empty() => {
// The resolver key (`pinned_cids.cid`) must be the locally-computed
@@ -410,30 +466,109 @@ pub async fn pin_new_objects(
// runs under the shared client's own ceiling, so a successful one can
// return with ~0 of the batch budget left, and an unfloored bound would
// fail a write that today completes in milliseconds. That costs more on
// this side than on the twin: `pinned.push` below is UNCONDITIONAL, so
// `api/repos.rs` builds its `cid_map` from the pair either way and drives
// this side than on the twin: `pinned.push` below feeds
// `api/repos.rs`, which builds its `cid_map` from the pair and drives
// `upsert_branch_cid` plus the p2p `publish_ref_update` gossip from it. A
// dropped record therefore makes the node ADVERTISE a CID whose `/ipfs`
// read 404s. If the floored bound still fires, THIS site's outcome really
// is unknown, and unlike the source record below that is a property of
// the operation: `record_pinata_cid` is a single autocommit upsert, so
// the statement Postgres already started can still land after the client
// future is cancelled. The warn names the arm through the error's own
// Display and the site keeps its existing behavior: the pair is still
// returned, and the row may or may not exist.
if let Err(e) = crate::ipfs_pin::db_bounded(
// read 404s. Only durably recorded pairs are pushed, so the map never
// advertises an unconfirmed row.
//
// Round 10 P2: a closed/failed DB write means the (sha, cid)
// pair is not durable; we suppress the `pinned.push` for this
// sha so the reconcile cannot count a Pinata gap as filled
// when no row exists. The source-record failure arms are also
// hard failures (multi-statement transaction never committed)
// and suppress the push in the same way.
//
// `record_pinata_cid` is an explicit multi-statement
// transaction (row lock + insert + commit), NOT a single
// autocommit upsert, so a timed-out future proves nothing:
// cancellation before COMMIT wrote nothing, during COMMIT is
// unknown. The Elapsed arm therefore verifies by content
// (`verify_pinata_record`: exact row + unchanged fence epoch)
// under its own bound and pushes only on proof; anything
// else stays suppressed and the gap is re-offered.
let fence_epoch = fence.map(|f| f.captured_epoch()).unwrap_or(i64::MAX);
let mut db_record_durable = false;
match crate::ipfs_pin::db_bounded(
crate::ipfs_pin::db_record_deadline(deadline),
crate::ipfs_pin::retry_db_record(|| {
db.record_pinata_cid(&sha, &raw_cid, &cid, Some(repo_id))
// P2 (reviewer round 9): Pinata's POST is
// irreversible exactly like IPFS's, so
// route the record through the fenced
// variant when a fence is in scope. The
// `i64::MAX` sentinel tells the helper
// to skip the lock + comparison (the
// unfenced caller path).
db.record_pinata_cid(&sha, &raw_cid, &cid, Some(repo_id), fence_epoch)
}),
)
.await
{
tracing::warn!(sha = %sha, err = %e, "failed to record pinata_cid in DB");
Ok(()) => db_record_durable = true,
Err(crate::ipfs_pin::BoundedDbError::Elapsed) => {
tracing::warn!(
sha = %sha,
"record_pinata_cid deadline elapsed on a multi-statement transaction; \
verifying the exact row before treating it as durable"
);
match crate::ipfs_pin::db_bounded(
crate::ipfs_pin::db_record_deadline(deadline),
db.verify_pinata_record(&sha, &raw_cid, &cid, repo_id, fence_epoch),
)
.await
{
Ok(true) => {
tracing::warn!(
sha = %sha,
"timed-out pinata record verified present with unchanged epoch; \
treating as durable"
);
db_record_durable = true;
}
Ok(false) => {
tracing::warn!(
sha = %sha,
"timed-out pinata record has no matching row or the epoch moved; \
suppressing the push so the gap is re-offered"
);
}
Err(e) => {
tracing::warn!(
sha = %sha,
err = %e,
"pinata record verification did not complete; \
suppressing the push so the gap is re-offered"
);
}
}
}
Err(e) => {
tracing::warn!(
sha = %sha,
err = %e,
"failed to record pinata_cid in DB; suppressing the (sha, cid) push \
so the reconcile cannot count this gap as filled"
);
// db_record_durable stays false → push suppressed
}
}
// F1 (#173 round 8): also record the first pinner in pin_repo_sources.
// U3: an exhausted retry marks the set incomplete so the resolver keeps
// the scan fallback rather than 404ing a copy it could serve.
//
// Whether the pair stays confirmed depends on what the
// PRIMARY `record_pinata_cid` above established: that row
// already names this repo as first pinner
// (`pin_sources_for_oid` unions it), so the resolver
// reaches this copy with or without the redundant
// `pin_repo_sources` insert. A source-write failure then
// keeps the pair confirmed (but still marks the set
// incomplete, preserving the U3 compensation). If the
// primary row names ANOTHER repo — or is absent — the
// source write was load-bearing and the push stays
// suppressed. (The IPFS twin needs no equivalent: its
// pin and source land in ONE transaction.)
match crate::ipfs_pin::db_bounded(
crate::ipfs_pin::db_record_deadline(deadline),
crate::ipfs_pin::retry_db_record(|| db.record_pin_source(&sha, repo_id)),
@@ -445,9 +580,7 @@ pub async fn pin_new_objects(
// wraps `record_pin_source`, an explicit transaction, so a timed-out
// call definitely never committed and the source is definitely
// missing. Mark the set incomplete rather than leaving it incomplete
// and unmarked. Note the contrast with `record_pinata_cid` a few
// lines up: that one is a single autocommit statement, so its
// timeout genuinely is an unknown outcome and it is warn-only.
// and unmarked.
Err(e @ crate::ipfs_pin::BoundedDbError::Elapsed) => {
tracing::warn!(
sha = %sha,
@@ -464,6 +597,16 @@ pub async fn pin_new_objects(
{
tracing::warn!(sha = %sha, err = %e, "failed to mark pin sources incomplete");
}
if !primary_covers_repo(
db,
&sha,
repo_id,
crate::ipfs_pin::db_record_deadline(deadline),
)
.await
{
db_record_durable = false;
}
}
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "failed to record pin source");
@@ -475,9 +618,21 @@ pub async fn pin_new_objects(
{
tracing::warn!(sha = %sha, err = %e, "failed to mark pin sources incomplete");
}
if !primary_covers_repo(
db,
&sha,
repo_id,
crate::ipfs_pin::db_record_deadline(deadline),
)
.await
{
db_record_durable = false;
}
}
}
pinned.push((sha, cid));
if db_record_durable {
pinned.push((sha, cid));
}
}
Ok(_) => {}
Err(e) => {
@@ -486,7 +641,10 @@ pub async fn pin_new_objects(
}
}
pinned
crate::ipfs_pin::PinBatchOutcome {
confirmed: pinned,
last_attempted,
}
}
// ── Tests ─────────────────────────────────────────────────────────────────────
@@ -688,19 +846,20 @@ mod tests {
&db,
"repo-merge-test",
Duration::from_millis(5500),
None,
),
)
.await
.expect("wedge guard: a 5.5s budget cannot take 30s");
assert!(
(1..=3).contains(&pinned.len()),
(1..=3).contains(&pinned.confirmed.len()),
"the batch must stop partway, not pin all five and not stall on the first: pinned {}",
pinned.len()
pinned.confirmed.len()
);
assert_eq!(
requests.load(std::sync::atomic::Ordering::SeqCst),
pinned.len(),
pinned.confirmed.len(),
"no upload may be issued for an object the budget stopped short of"
);
let text = logs.text();
@@ -728,9 +887,9 @@ mod tests {
})
.unwrap_or_else(|| panic!("the deadline warn must name the unattempted count: {text}"));
assert!(
unattempted >= 1 && unattempted + pinned.len() <= 5,
unattempted >= 1 && unattempted + pinned.confirmed.len() <= 5,
"unattempted={unattempted} with {} pinned is not a partial batch of five",
pinned.len()
pinned.confirmed.len()
);
}
@@ -779,6 +938,7 @@ mod tests {
&db,
"repo-merge-test",
Duration::from_secs(2),
None,
),
)
.await
@@ -789,7 +949,7 @@ mod tests {
let elapsed = started.elapsed();
assert!(
pinned.is_empty(),
pinned.confirmed.is_empty(),
"a git that never answers cannot produce a pinned object: {pinned:?}"
);
assert!(
@@ -868,6 +1028,7 @@ mod tests {
"repo-git-timeout",
// Generous, so a call that ends on time ended on `git_timeout`.
Duration::from_secs(60),
None,
),
)
.await
@@ -878,7 +1039,7 @@ mod tests {
let elapsed = started.elapsed();
assert!(
pinned.is_empty(),
pinned.confirmed.is_empty(),
"a git that never answers cannot produce a pinned object: {pinned:?}"
);
assert!(
@@ -970,6 +1131,7 @@ mod tests {
&db,
"repo-merge-test",
Duration::from_secs(60),
None,
),
)
.await
@@ -979,7 +1141,7 @@ mod tests {
if genuinely_unreadable {
assert!(
pinned.is_empty(),
pinned.confirmed.is_empty(),
"nothing can be pinned through a store that cannot be read: {pinned:?}"
);
assert_eq!(
@@ -1044,6 +1206,7 @@ mod tests {
&db,
"repo-merge-test",
Duration::from_secs(60),
None,
),
)
.await
@@ -1055,7 +1218,7 @@ mod tests {
"an object-scoped fault must not stop the batch: every object must be read"
);
assert_eq!(
pinned.len(),
pinned.confirmed.len(),
4,
"one corrupt object must cost only itself: the other four must still pin"
);
@@ -1095,13 +1258,18 @@ mod tests {
&db,
"repo-merge-test",
Duration::from_secs(60),
None,
),
)
.await
.expect("an immediate endpoint and three healthy objects cannot take 30s");
assert_eq!(pinned.len(), 3, "every healthy object must pin: {pinned:?}");
for (i, (sha, cid)) in pinned.iter().enumerate() {
assert_eq!(
pinned.confirmed.len(),
3,
"every healthy object must pin: {pinned:?}"
);
for (i, (sha, cid)) in pinned.confirmed.iter().enumerate() {
assert_eq!(sha, &oids[i], "the pairs must carry the objects' own oids");
assert_eq!(cid, "QmPinataBatchTestCid");
assert!(
@@ -1124,11 +1292,15 @@ mod tests {
&db,
"repo-merge-test",
Duration::from_secs(60),
None,
),
)
.await
.expect("a fully deduped batch cannot take 30s");
assert!(again.is_empty(), "already-recorded objects must be skipped");
assert!(
again.confirmed.is_empty(),
"already-recorded objects must be skipped"
);
assert_eq!(
requests.load(std::sync::atomic::Ordering::SeqCst),
3,
@@ -1136,6 +1308,63 @@ mod tests {
);
}
/// The outcome reports the last object actually ENTERED, not the planned
/// tail (Pinata twin of the IPFS test above): the first upload takes 6s
/// against a 2s batch budget, so it still completes — the Pinata loop
/// has no per-request timeout — but the loop-top gate then breaks the
/// batch before the second object starts. `last_attempted` must be the
/// first OID even though two OIDs were never visited; the 6s server
/// sleep makes the break deterministic on any box.
#[sqlx::test]
async fn pin_new_objects_reports_last_attempted_not_planned_tail(pool: sqlx::PgPool) {
let db = crate::db::Db::for_testing(pool);
db.run_migrations().await.expect("migrations");
let tmp = tempfile::TempDir::new().unwrap();
let repo_path = tmp.path().join("attempted.git");
let oids = seed_loose_blobs(&repo_path, 3);
let requests = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
let endpoint = delaying_pinata_endpoint(
vec![Duration::from_secs(6)],
std::sync::Arc::clone(&requests),
)
.await;
let client = reqwest::Client::new();
let outcome = tokio::time::timeout(
Duration::from_secs(60),
pin_new_objects(
&client,
&endpoint,
"test-jwt",
&repo_path,
"git",
Duration::from_secs(60),
oids.clone(),
&db,
"repo-attempted",
Duration::from_secs(2),
None,
),
)
.await
.expect("one 6s upload plus bounded gates cannot take 60s");
assert_eq!(
outcome.last_attempted,
Some(oids[0].clone()),
"only the first object was entered; the planned tail was never visited"
);
assert_eq!(
outcome.confirmed.len(),
1,
"the slow first upload still completed and recorded"
);
assert_eq!(
requests.load(std::sync::atomic::Ordering::SeqCst),
1,
"exactly one upload was attempted"
);
}
/// The no-op configuration still short-circuits with the budgeted signature: an
/// empty JWT must return before any git child is spawned and before any request
/// is issued. The `git_bin` here records every invocation, so "git was never
@@ -1177,12 +1406,13 @@ mod tests {
&db,
"repo-merge-test",
Duration::from_secs(60),
None,
),
)
.await
.expect("an unconfigured sink returns immediately");
assert!(pinned.is_empty(), "an empty JWT pins nothing");
assert!(pinned.confirmed.is_empty(), "an empty JWT pins nothing");
assert!(
!log.exists(),
"no git child may be spawned when the sink is not configured"
@@ -1282,6 +1512,7 @@ mod tests {
&db,
"repo-pinata-stalled",
Duration::from_millis(1500),
None,
),
)
.await
@@ -1292,7 +1523,7 @@ mod tests {
let elapsed = started.elapsed();
assert!(
pinned.is_empty(),
pinned.confirmed.is_empty(),
"a stalled pinata-status check cannot produce a pinned object: {pinned:?}"
);
assert!(
@@ -1338,9 +1569,15 @@ mod tests {
// bytes.
let raw_cid =
gitlawb_core::cid::Cid::from_git_object_bytes(b"pinata skip seed").to_string();
db.record_pinata_cid(&sha, &raw_cid, "QmSeedProviderCid", Some("repo-seed"))
.await
.unwrap();
db.record_pinata_cid(
&sha,
&raw_cid,
"QmSeedProviderCid",
Some("repo-seed"),
i64::MAX,
)
.await
.unwrap();
db.record_pin_source(&sha, "repo-seed").await.unwrap();
let requests = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
let endpoint = delaying_pinata_endpoint(
@@ -1372,6 +1609,7 @@ mod tests {
&db,
"repo-pinata-skip-stalled",
Duration::from_millis(1500),
None,
),
)
.await
@@ -1382,7 +1620,7 @@ mod tests {
let elapsed = started.elapsed();
assert!(
pinned.is_empty(),
pinned.confirmed.is_empty(),
"an already-pinned object is skipped, never re-uploaded: {pinned:?}"
);
assert_eq!(
@@ -1426,7 +1664,7 @@ mod tests {
///
/// The lock time is a MARGIN, not a boundary: taking it at 100ms left
/// `has_pinata_cid` racing it on a loaded box, and losing that race makes the read
/// block, time out, and break the batch, which fails on `pinned.len() == 1` for a
/// block, time out, and break the batch, which fails on `pinned.confirmed.len() == 1` for a
/// reason that has nothing to do with the floor. Any time between the
/// `has_pinata_cid` round trip and the upload's 1.7s return proves the same thing.
#[sqlx::test]
@@ -1472,6 +1710,7 @@ mod tests {
&db,
"repo-pinata-spent-budget",
Duration::from_millis(2000),
None,
),
);
let (pinned, ()) = tokio::join!(pin, locker);
@@ -1489,7 +1728,7 @@ mod tests {
record makes api/repos.rs advertise a CID the resolver cannot serve"
);
assert_eq!(
pinned.len(),
pinned.confirmed.len(),
1,
"the uploaded pin must still be returned: {pinned:?}"
);
@@ -1557,6 +1796,7 @@ mod tests {
&db,
"repo-pinata-post-upload",
Duration::from_millis(1500),
None,
),
)
.await
@@ -1570,10 +1810,21 @@ mod tests {
drop(lock);
upload.assert_async().await;
// The primary `record_pinata_cid` committed above naming this
// repo as first pinner, so `pin_sources_for_oid` resolves the
// copy with or without the redundant `pin_repo_sources`
// insert: the pair stays confirmed even though the source
// record timed out. The incomplete marker below is still set
// (compensation preserved); only the suppression is gone.
// Prior to the round 10 fix, the push fired regardless of
// every post-upload outcome; the correction since is that
// confirmation follows the durable primary row, not the
// redundant source write.
assert_eq!(
pinned.len(),
pinned.confirmed.len(),
1,
"the upload succeeded, so this lane still returns the pair: {pinned:?}"
"record_pin_source timed out (pin_repo_sources locked) but the primary row \
covers this repo, so the pair stays confirmed: {pinned:?}"
);
assert!(
elapsed < Duration::from_secs(8),
@@ -1590,6 +1841,203 @@ mod tests {
);
}
/// The other half of the source-write contract: when the committed
/// primary row names ANOTHER repo as first pinner, the failed
/// `record_pin_source` was load-bearing (not redundant) and the
/// push stays suppressed. Fixture: a local-only row owned by
/// repo-first (no Pinata CID yet), then a Pinata loop as
/// repo-second with `pin_repo_sources` locked. The upload lands,
/// the primary upsert keeps repo-first, the source insert stalls,
/// and coverage fails — so no pair, but the incomplete marker
/// still lands and the provider CID is still recorded.
#[sqlx::test]
async fn pinata_post_upload_stalled_source_without_coverage_suppresses_push(
pool: sqlx::PgPool,
) {
let db = crate::db::Db::for_testing(pool.clone());
db.run_migrations().await.expect("migrations");
let tmp = tempfile::TempDir::new().unwrap();
let repo_path = tmp.path().join("pinata_second_source.git");
let oids = seed_loose_blobs(&repo_path, 1);
let sha = oids[0].clone();
// A local-only row owned by repo-first: the writer path below
// (repo-second, Pinata-only so far) must not claim it.
let raw =
gitlawb_core::cid::Cid::from_git_object_bytes(b"pinata loop object 0\n").to_string();
db.record_pinned_cid_with_source(&sha, &raw, "repo-first")
.await
.unwrap();
let mut server = mockito::Server::new_async().await;
let upload = server
.mock("POST", mockito::Matcher::Any)
.with_status(200)
.with_body(r#"{"data":{"cid":"QmSecondSourceProviderCid"}}"#)
.expect(1)
.create_async()
.await;
let (_logs, _log_guard) = capture_logs();
let mut lock = lock_table(&pool, "pin_repo_sources").await;
let client = reqwest::Client::new();
let pinned = tokio::time::timeout(
Duration::from_secs(20),
pin_new_objects(
&client,
&server.url(),
"test-jwt",
&repo_path,
"git",
Duration::from_secs(30),
oids,
&db,
"repo-second",
Duration::from_millis(1500),
None,
),
)
.await
.expect("record + coverage ladders are each floored");
rollback(&mut lock).await;
drop(lock);
upload.assert_async().await;
assert!(
pinned.confirmed.is_empty(),
"repo-second is not covered by the repo-first primary row, so the \
failed source write stays suppressing: {pinned:?}"
);
assert!(
db.pin_sources_incomplete(&sha).await.unwrap(),
"the compensation marker still lands"
);
// The primary upsert itself landed (provider CID recorded under
// the first-pinner row); only the pair was suppressed.
assert!(
db.has_pinata_cid(&sha).await.unwrap(),
"the committed provider row survives the suppressed push"
);
}
/// Hold a row-level lock on one repos row in an open transaction: plain
/// `SELECT`s (fence reads, skip checks) proceed, but any `SELECT ...
/// FOR UPDATE` on the row — exactly what the fenced record takes —
/// blocks until rollback. A table-level lock would also stall the
/// fence's own epoch reads and deadlock the loop under test, proving
/// nothing about the record arm.
async fn hold_repo_row_lock(
pool: &sqlx::PgPool,
repo_id: &str,
) -> sqlx::pool::PoolConnection<sqlx::Postgres> {
let mut conn = pool.acquire().await.unwrap();
sqlx::raw_sql("BEGIN").execute(&mut *conn).await.unwrap();
sqlx::query("SELECT policy_epoch FROM repos WHERE id = $1 FOR UPDATE")
.bind(repo_id)
.fetch_one(&mut *conn)
.await
.unwrap();
conn
}
/// A timed-out `record_pinata_cid` with no verifiable row suppresses the
/// push: `record_pinata_cid` is an explicit transaction, so Elapsed proves
/// nothing, and the verify read finds no row, so nothing is proven that
/// way either. The upload mock still expects the POST — Pinata accepted
/// the bytes — but without a durable row the pair must not be returned,
/// or the reconcile counts a fill that never happened and the push path
/// advertises an unresolvable CID.
#[sqlx::test]
async fn pinata_post_upload_elapsed_record_without_row_suppresses_push(pool: sqlx::PgPool) {
let db = crate::db::Db::for_testing(pool.clone());
db.run_migrations().await.expect("migrations");
let tmp = tempfile::TempDir::new().unwrap();
let repo_path = tmp.path().join("pinata_elapsed_verify.git");
let oids = seed_loose_blobs(&repo_path, 1);
let sha = oids[0].clone();
// A real repo row: the fenced record's `FOR UPDATE` must have a row
// to block on. Against a missing row the predicate takes no lock and
// the record would sail through instead of stalling.
let repo_id = "repo-pinata-elapsed-verify";
let now = chrono::Utc::now();
db.create_repo(&crate::db::RepoRecord {
id: repo_id.to_string(),
name: "elapsed-verify".into(),
owner_did: "did:key:zElapsedVerifyOwner".into(),
description: None,
is_public: true,
default_branch: "main".into(),
created_at: now,
updated_at: now,
disk_path: repo_path.display().to_string(),
forked_from: None,
machine_id: None,
})
.await
.unwrap();
// Capture the fence BEFORE locking: the capture itself is a plain
// epoch read and must succeed for the fenced record path to run.
let fence = crate::ipfs_pin::PolicyFence::capture(&db, repo_id)
.await
.expect("fence captures");
let mut server = mockito::Server::new_async().await;
let upload = server
.mock("POST", mockito::Matcher::Any)
.with_status(200)
.with_body(r#"{"data":{"cid":"QmElapsedVerifyProviderCid"}}"#)
.expect(1)
.create_async()
.await;
let (_logs, _log_guard) = capture_logs();
// Stall only the fenced record's `FOR UPDATE`: fence reads and the
// skip check are plain SELECTs and proceed, the upload lands, then
// the record elapses. The verify read proceeds too and finds no row.
let mut lock = hold_repo_row_lock(&pool, repo_id).await;
let client = reqwest::Client::new();
let pinned = tokio::time::timeout(
Duration::from_secs(30),
pin_new_objects(
&client,
&server.url(),
"test-jwt",
&repo_path,
"git",
Duration::from_secs(30),
oids,
&db,
repo_id,
Duration::from_millis(1500),
Some(&fence),
),
)
.await
.expect(
"record + verify ladders are each floored, so the call returns in seconds, \
never at the lock's lifetime",
);
rollback(&mut lock).await;
drop(lock);
upload.assert_async().await;
assert!(
pinned.confirmed.is_empty(),
"an unverified timed-out record must not return the pair: {pinned:?}"
);
assert!(
!db.has_pinata_cid(&sha).await.unwrap(),
"the cancelled record transaction must not have landed a row"
);
}
#[tokio::test]
async fn test_pin_skipped_when_jwt_empty() {
let client = reqwest::Client::new();
+67 -10
View File
@@ -70,6 +70,14 @@ impl RateLimiter {
self.window.min(Duration::from_secs(1))
}
/// The configured window, exposed so a 429 can carry an honest
/// `retry-after`: a hardcoded value lies when the operator retunes the
/// window, and a client that honors the header would retry into another
/// 429 until the real window elapses.
pub fn window(&self) -> Duration {
self.window
}
pub(crate) async fn check(&self, key: &str) -> bool {
// max_requests == 0 means the limiter is disabled, not "block all".
if self.max_requests == 0 {
@@ -293,12 +301,7 @@ pub async fn rate_limit_by_did(request: Request, next: Next) -> Response {
if let (Some(limiter), Some(did)) = (limiter, did) {
if !limiter.check(&did).await {
return (
StatusCode::TOO_MANY_REQUESTS,
[("retry-after", "60")],
"rate limit exceeded — try again later",
)
.into_response();
return too_many_requests(limiter.window().as_secs());
}
}
@@ -401,18 +404,72 @@ impl<S: Send + Sync> axum::extract::FromRequestParts<S> for PeerAddr {
}
}
/// The message every per-IP brake answers with, shared so the GraphQL surface
/// (which cannot return a 429 status inside a 200 GraphQL envelope) says the
/// same thing as the REST routes.
pub const RATE_LIMIT_MESSAGE: &str = "rate limit exceeded — try again later";
/// The shared 429 response for the per-IP flood brakes. Route-agnostic: this
/// middleware now serves the push path AND the peer-sync routes, so the message
/// stays generic (the offending path is recorded in the warn log below).
pub fn too_many_requests() -> Response {
pub fn too_many_requests(retry_after_secs: u64) -> Response {
(
StatusCode::TOO_MANY_REQUESTS,
[("retry-after", "60")],
"rate limit exceeded — try again later",
[("retry-after", retry_after_secs.to_string())],
RATE_LIMIT_MESSAGE,
)
.into_response()
}
/// The per-IP task-read brake carried as GraphQL request data: the limiter plus
/// the client key the transport already resolved.
///
/// `/graphql` is a single POST endpoint, so this brake cannot sit in middleware
/// the way `task_read_routes` mounts `rate_limit_by_ip`: that would charge every
/// unrelated query and every mutation against the task-read bucket. The two task
/// resolvers debit it instead, which also prices an aliased query honestly — ten
/// aliased `tasks` fields run the visibility gate ten times and pay ten slots,
/// where one request-scoped debit would pay one (#327 review).
/// Default cap on the number of task read fields (across all aliases) permitted
/// in a single GraphQL request before rejecting further field executions (#327 review).
pub const MAX_GRAPHQL_TASK_READS_PER_REQUEST: usize = 5;
#[derive(Clone)]
pub struct TaskReadBrake {
pub limiter: RateLimiter,
/// `None` when no key could be resolved at all (no trusted header and no
/// `ConnectInfo`, e.g. a synthetic test request). Never braked, matching
/// [`rate_limit_by_ip`].
pub key: Option<String>,
pub request_count: Arc<std::sync::atomic::AtomicUsize>,
}
impl TaskReadBrake {
/// Debit one slot for one task-read field. `true` = proceed.
pub async fn check(&self, field: &str) -> bool {
let Some(key) = self.key.as_deref() else {
return true;
};
let count = self
.request_count
.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
if count >= MAX_GRAPHQL_TASK_READS_PER_REQUEST {
tracing::warn!(
key = %key,
field = %field,
count = count + 1,
"per-request GraphQL task field limit exceeded"
);
return false;
}
if self.limiter.check(key).await {
return true;
}
tracing::warn!(key = %key, field = %field, "per-IP rate limit exceeded");
false
}
}
/// Throttle the git push path by resolved client IP. The socket peer address is
/// read from `ConnectInfo` (see `into_make_service_with_connect_info` in
/// `main`). Only skips the limiter when no key can be resolved at all.
@@ -427,7 +484,7 @@ pub async fn rate_limit_by_ip(request: Request, next: Next) -> Response {
if let Some(key) = client_key(request.headers(), peer, limiter.trust) {
if !limiter.limiter.check(&key).await {
tracing::warn!(key = %key, path = %request.uri().path(), "per-IP rate limit exceeded");
return too_many_requests();
return too_many_requests(limiter.limiter.window().as_secs());
}
}
}
File diff suppressed because it is too large. Load diff
+411 -12
View File
@@ -1,5 +1,6 @@
use async_graphql_axum::{GraphQLRequest, GraphQLResponse, GraphQLSubscription};
use axum::extract::DefaultBodyLimit;
use async_graphql::http::ALL_WEBSOCKET_PROTOCOLS;
use async_graphql_axum::{GraphQLProtocol, GraphQLRequest, GraphQLResponse, GraphQLWebSocket};
use axum::extract::{DefaultBodyLimit, WebSocketUpgrade};
use axum::{
extract::State,
middleware,
@@ -23,6 +24,8 @@ use crate::state::AppState;
async fn graphql_handler(
State(state): State<AppState>,
auth: Option<axum::Extension<crate::auth::AuthenticatedDid>>,
headers: axum::http::HeaderMap,
rate_limit::PeerAddr(peer): rate_limit::PeerAddr,
req: GraphQLRequest,
) -> GraphQLResponse {
// `optional_signature` attaches the verified DID when a signature is present.
@@ -32,9 +35,49 @@ async fn graphql_handler(
if let Some(axum::Extension(did)) = auth {
inner = inner.data(did);
}
// The anonymous `tasks`/`task` resolvers run the same #268 visibility gate
// as the REST read routes and cost the node the same queries, so they carry
// the same per-IP brake. It rides as request data rather than a router layer
// because /graphql is one endpoint for every operation — see `TaskReadBrake`
// (#327 review). It debits before the gate runs, but unlike the REST layer
// it sits inside `optional_signature`, so it brakes the gate's query cost
// and not signature verification.
inner = inner.data(rate_limit::TaskReadBrake {
limiter: state.task_read_rate_limiter.clone(),
key: rate_limit::client_key(&headers, peer, state.push_limiter_trust),
request_count: std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)),
});
state.graphql_schema.execute(inner).await.into()
}
async fn graphql_ws_handler(
State(state): State<AppState>,
headers: axum::http::HeaderMap,
rate_limit::PeerAddr(peer): rate_limit::PeerAddr,
auth: Option<axum::Extension<auth::AuthenticatedDid>>,
protocol: GraphQLProtocol,
upgrade: WebSocketUpgrade,
) -> axum::response::Response {
let mut data = async_graphql::Data::default();
if let Some(axum::Extension(did)) = auth {
data.insert(did);
}
data.insert(rate_limit::TaskReadBrake {
limiter: state.task_read_rate_limiter.clone(),
key: rate_limit::client_key(&headers, peer, state.push_limiter_trust),
request_count: std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)),
});
let schema = state.graphql_schema.as_ref().clone();
upgrade
.protocols(ALL_WEBSOCKET_PROTOCOLS)
.on_upgrade(move |stream| {
GraphQLWebSocket::new(stream, schema, protocol)
.with_data(data)
.serve()
})
.into_response()
}
async fn graphql_playground() -> impl IntoResponse {
axum::response::Html(async_graphql::http::playground_source(
async_graphql::http::GraphQLPlaygroundConfig::new("/graphql")
@@ -77,17 +120,14 @@ fn add_auth_layers(
pub fn build_router(state: AppState) -> Router {
// ── GraphQL routes ─────────────────────────────────────────────────────
let schema = state.graphql_schema.as_ref().clone();
let graphql_routes = Router::new()
.route("/graphql", get(graphql_playground).post(graphql_handler))
// Attach the verified DID to /graphql when a signature is present. The
// layer covers only routes added before it, so /graphql/ws (added after,
// read-only subscriptions) stays open.
.route("/graphql/ws", get(graphql_ws_handler))
// Attach the verified DID to /graphql and /graphql/ws when a signature is present.
.layer(middleware::from_fn_with_state(
auth::SignatureBodyLimit(SIGNED_BODY_LIMIT),
auth::optional_signature,
))
.route_service("/graphql/ws", GraphQLSubscription::new(schema));
));
// ── Task routes (write — require HTTP Signature) ───────────────────────
let task_write_routes = add_auth_layers(
@@ -100,10 +140,31 @@ pub fn build_router(state: AppState) -> Router {
SIGNED_BODY_LIMIT,
);
// ── Task routes (read — open) ──────────────────────────────────────────
// ── Task routes (read — open, but scoped) ──────────────────────────────
// `optional_signature` attaches the verified DID when a signature is present
// so the handlers can identify the caller; the routes stay anonymous-reachable,
// but each task/row is gated to its delegator, its assignee, or (for a
// repo-scoped task) whoever can read that repo (#268 — these routes previously
// carried no gate and no identity at all).
// Both routes also carry a per-IP flood brake, mirroring `/ipfs/{cid}`: they are
// anon-reachable and the gate above costs a task lookup plus deduped-repo and
// visibility-rule queries *before* the opaque 404, so a prober pays nothing and
// the node pays per request. The limiter is the outermost layer so a flood is
// rejected before signature verification and the visibility queries run. The
// extension MUST be attached or `rate_limit_by_ip` is a silent no-op.
let task_read_limiter = rate_limit::IpRateLimiter {
limiter: state.task_read_rate_limiter.clone(),
trust: state.push_limiter_trust,
};
let task_read_routes = Router::new()
.route("/api/v1/tasks", get(tasks::list_tasks))
.route("/api/v1/tasks/{id}", get(tasks::get_task));
.route("/api/v1/tasks/{id}", get(tasks::get_task))
.layer(middleware::from_fn_with_state(
auth::SignatureBodyLimit(SIGNED_BODY_LIMIT),
auth::optional_signature,
))
.layer(middleware::from_fn(rate_limit::rate_limit_by_ip))
.layer(axum::Extension(task_read_limiter));
// ── Rate-limited creation routes — require HTTP Signature, plus a per-DID
// throttle AND a per-IP flood brake. The per-DID limiter (inner) caps a
@@ -715,13 +776,351 @@ async fn p2p_info(State(state): State<AppState>) -> Json<serde_json::Value> {
#[cfg(test)]
mod tests {
use super::*;
use crate::test_support::test_state;
use sqlx::PgPool;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
async fn ws_send_text(stream: &mut tokio::net::TcpStream, text: &str) {
let payload = text.as_bytes();
let len = payload.len();
let mut frame = Vec::new();
frame.push(0x81);
let mask = [0x12, 0x34, 0x56, 0x78];
if len <= 125 {
frame.push(0x80 | (len as u8));
} else if len <= 65535 {
frame.push(0x80 | 126);
frame.extend_from_slice(&(len as u16).to_be_bytes());
} else {
frame.push(0x80 | 127);
frame.extend_from_slice(&(len as u64).to_be_bytes());
}
frame.extend_from_slice(&mask);
for (i, b) in payload.iter().enumerate() {
frame.push(b ^ mask[i % 4]);
}
stream.write_all(&frame).await.unwrap();
stream.flush().await.unwrap();
}
async fn ws_recv_text(stream: &mut tokio::net::TcpStream) -> String {
let mut header = [0u8; 2];
stream.read_exact(&mut header).await.unwrap();
let b1 = header[1];
let masked = (b1 & 0x80) != 0;
let mut len = (b1 & 0x7f) as usize;
if len == 126 {
let mut ext = [0u8; 2];
stream.read_exact(&mut ext).await.unwrap();
len = u16::from_be_bytes(ext) as usize;
} else if len == 127 {
let mut ext = [0u8; 8];
stream.read_exact(&mut ext).await.unwrap();
len = u64::from_be_bytes(ext) as usize;
}
let mask = if masked {
let mut m = [0u8; 4];
stream.read_exact(&mut m).await.unwrap();
Some(m)
} else {
None
};
let mut payload = vec![0u8; len];
stream.read_exact(&mut payload).await.unwrap();
if let Some(m) = mask {
for (i, b) in payload.iter_mut().enumerate() {
*b ^= m[i % 4];
}
}
String::from_utf8(payload).unwrap()
}
async fn ws_recv_op_frames(stream: &mut tokio::net::TcpStream, op_id: &str) -> Vec<String> {
let mut frames = Vec::new();
loop {
let text = ws_recv_text(stream).await;
frames.push(text.clone());
if let Ok(v) = serde_json::from_str::<serde_json::Value>(&text) {
if v.get("id").and_then(|id| id.as_str()) == Some(op_id) {
let msg_type = v.get("type").and_then(|t| t.as_str()).unwrap_or("");
if msg_type == "complete" || msg_type == "error" {
break;
}
}
}
}
frames
}
async fn connect_ws(addr: std::net::SocketAddr) -> tokio::net::TcpStream {
let mut stream = tokio::net::TcpStream::connect(addr).await.unwrap();
let req = format!(
"GET /graphql/ws HTTP/1.1\r\n\
Host: {}\r\n\
Upgrade: websocket\r\n\
Connection: Upgrade\r\n\
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\n\
Sec-WebSocket-Version: 13\r\n\
Sec-WebSocket-Protocol: graphql-transport-ws\r\n\r\n",
addr
);
stream.write_all(req.as_bytes()).await.unwrap();
stream.flush().await.unwrap();
let mut buf = [0u8; 1024];
let n = stream.read(&mut buf).await.unwrap();
let resp = String::from_utf8_lossy(&buf[..n]);
assert!(resp.starts_with("HTTP/1.1 101 Switching Protocols"));
// Init connection
ws_send_text(&mut stream, r#"{"type":"connection_init"}"#).await;
let ack = ws_recv_text(&mut stream).await;
assert!(ack.contains("connection_ack"));
stream
}
#[sqlx::test]
async fn graphql_ws_task_query_enforces_per_request_field_limit(pool: PgPool) {
let state = test_state(pool).await;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let app = build_router(state);
tokio::spawn(async move {
axum::serve(
listener,
app.into_make_service_with_connect_info::<std::net::SocketAddr>(),
)
.await
.unwrap();
});
let mut stream = connect_ws(addr).await;
// Query with 6 aliased task fields (exceeding MAX_GRAPHQL_TASK_READS_PER_REQUEST = 5).
// limit: 1 keeps the document under GRAPHQL_MAX_COMPLEXITY so the 6th
// field reaches the brake instead of failing complexity first.
let query = r#"{"id":"1","type":"subscribe","payload":{"query":"query { f1: tasks(limit: 1) { items { id } } f2: tasks(limit: 1) { items { id } } f3: tasks(limit: 1) { items { id } } f4: tasks(limit: 1) { items { id } } f5: tasks(limit: 1) { items { id } } f6: tasks(limit: 1) { items { id } } }"}}"#;
ws_send_text(&mut stream, query).await;
let frames = ws_recv_op_frames(&mut stream, "1").await;
let resp = frames.join("\n");
assert!(
resp.contains("rate limit exceeded"),
"6th task field over WS must be braked: {resp}"
);
}
#[sqlx::test]
async fn graphql_ws_task_query_enforces_per_ip_rate_limit(pool: PgPool) {
let mut state = test_state(pool).await;
// Restrict task read rate limiter to 1 request per 60s
state.task_read_rate_limiter =
crate::rate_limit::RateLimiter::new(1, std::time::Duration::from_secs(60));
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let app = build_router(state);
tokio::spawn(async move {
axum::serve(
listener,
app.into_make_service_with_connect_info::<std::net::SocketAddr>(),
)
.await
.unwrap();
});
let mut stream = connect_ws(addr).await;
// First task query succeeds (or returns valid data)
let query1 = r#"{"id":"1","type":"subscribe","payload":{"query":"query { tasks { items { id } } }"}}"#;
ws_send_text(&mut stream, query1).await;
let frames1 = ws_recv_op_frames(&mut stream, "1").await;
let resp1 = frames1.join("\n");
assert!(!resp1.contains("rate limit exceeded"));
// Second task query on the same connection hits per-IP limiter
let query2 = r#"{"id":"2","type":"subscribe","payload":{"query":"query { tasks { items { id } } }"}}"#;
ws_send_text(&mut stream, query2).await;
let frames2 = ws_recv_op_frames(&mut stream, "2").await;
let resp2 = frames2.join("\n");
assert!(
resp2.contains("rate limit exceeded"),
"exceeded per-IP limiter over WS must return rate limit message: {resp2}"
);
}
#[sqlx::test]
async fn graphql_ws_task_query_resets_field_budget_per_operation(pool: PgPool) {
let state = test_state(pool).await;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let app = build_router(state);
tokio::spawn(async move {
axum::serve(
listener,
app.into_make_service_with_connect_info::<std::net::SocketAddr>(),
)
.await
.unwrap();
});
let mut stream = connect_ws(addr).await;
// Execute 6 sequential 1-field queries on the same connection.
// Each operation must receive its own 5-field budget, so none are braked by the per-request limit.
for i in 1..=6 {
let id = i.to_string();
let query = format!(
r#"{{"id":"{id}","type":"subscribe","payload":{{"query":"query {{ tasks {{ items {{ id }} }} }}"}}}}"#
);
ws_send_text(&mut stream, &query).await;
let frames = ws_recv_op_frames(&mut stream, &id).await;
let resp = frames.join("\n");
assert!(
!resp.contains("rate limit exceeded"),
"operation {id} on the same WS connection must have a fresh field budget: {resp}"
);
}
}
async fn connect_ws_signed(
addr: std::net::SocketAddr,
keypair: &gitlawb_core::identity::Keypair,
) -> tokio::net::TcpStream {
let mut stream = tokio::net::TcpStream::connect(addr).await.unwrap();
let signed = gitlawb_core::http_sig::sign_request(keypair, "GET", "/graphql/ws", b"");
let req = format!(
"GET /graphql/ws HTTP/1.1\r\n\
Host: {}\r\n\
Upgrade: websocket\r\n\
Connection: Upgrade\r\n\
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\n\
Sec-WebSocket-Version: 13\r\n\
Sec-WebSocket-Protocol: graphql-transport-ws\r\n\
content-digest: {}\r\n\
signature-input: {}\r\n\
signature: {}\r\n\r\n",
addr, signed.content_digest, signed.signature_input, signed.signature
);
stream.write_all(req.as_bytes()).await.unwrap();
stream.flush().await.unwrap();
let mut buf = [0u8; 1024];
let n = stream.read(&mut buf).await.unwrap();
let resp = String::from_utf8_lossy(&buf[..n]);
assert!(resp.starts_with("HTTP/1.1 101 Switching Protocols"));
// Init connection
ws_send_text(&mut stream, r#"{"type":"connection_init"}"#).await;
let ack = ws_recv_text(&mut stream).await;
assert!(ack.contains("connection_ack"));
stream
}
#[sqlx::test]
async fn graphql_ws_authenticated_query_accesses_private_task(pool: PgPool) {
let state = test_state(pool).await;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let app = build_router(state.clone());
tokio::spawn(async move {
axum::serve(
listener,
app.into_make_service_with_connect_info::<std::net::SocketAddr>(),
)
.await
.unwrap();
});
let keypair = gitlawb_core::identity::Keypair::generate();
let delegator_did = keypair.did().to_string();
let task = crate::db::AgentTask {
id: "ws-auth-task-01".into(),
repo_id: None,
delegator_did: delegator_did.clone(),
kind: "test".into(),
capability: "read".into(),
status: "pending".into(),
assignee_did: Some(delegator_did.clone()),
ucan_token: None,
payload: None,
result: None,
created_at: "2026-01-01T00:00:00Z".into(),
updated_at: "2026-01-01T00:00:00Z".into(),
deadline: None,
};
state.db.create_task(&task).await.unwrap();
// 1. Unauthenticated WS connection cannot see the private task
let mut unauth_stream = connect_ws(addr).await;
let query = r#"{"id":"1","type":"subscribe","payload":{"query":"query { tasks { items { id } } }"}}"#;
ws_send_text(&mut unauth_stream, query).await;
let unauth_frames = ws_recv_op_frames(&mut unauth_stream, "1").await;
let unauth_resp = unauth_frames.join("\n");
assert!(
!unauth_resp.contains("ws-auth-task-01"),
"unauthenticated WS must not disclose private task: {unauth_resp}"
);
// 2. Forged signature claiming the delegator DID must be rejected at handshake
let attacker_keypair = gitlawb_core::identity::Keypair::generate();
let attacker_signed =
gitlawb_core::http_sig::sign_request(&attacker_keypair, "GET", "/graphql/ws", b"");
let forged_sig_input = attacker_signed
.signature_input
.replace(&attacker_keypair.did().to_string(), &delegator_did);
let mut forged_stream = tokio::net::TcpStream::connect(addr).await.unwrap();
let forged_req = format!(
"GET /graphql/ws HTTP/1.1\r\n\
Host: {}\r\n\
Upgrade: websocket\r\n\
Connection: Upgrade\r\n\
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\n\
Sec-WebSocket-Version: 13\r\n\
Sec-WebSocket-Protocol: graphql-transport-ws\r\n\
content-digest: {}\r\n\
signature-input: {}\r\n\
signature: {}\r\n\r\n",
addr, attacker_signed.content_digest, forged_sig_input, attacker_signed.signature
);
forged_stream
.write_all(forged_req.as_bytes())
.await
.unwrap();
forged_stream.flush().await.unwrap();
let mut forged_bytes = Vec::new();
let mut chunk = [0u8; 1024];
while !forged_bytes.windows(4).any(|w| w == b"\r\n\r\n") {
let n = forged_stream.read(&mut chunk).await.unwrap();
if n == 0 {
break;
}
forged_bytes.extend_from_slice(&chunk[..n]);
}
let forged_resp = String::from_utf8_lossy(&forged_bytes);
assert!(
forged_resp.starts_with("HTTP/1.1 401 Unauthorized"),
"forged WS signature must be rejected with 401: {forged_resp}"
);
// 3. Signed WS connection authenticates caller and retrieves the private task
let mut auth_stream = connect_ws_signed(addr, &keypair).await;
ws_send_text(&mut auth_stream, query).await;
let auth_frames = ws_recv_op_frames(&mut auth_stream, "1").await;
let auth_resp = auth_frames.join("\n");
assert!(
auth_resp.contains("ws-auth-task-01"),
"signed WS must authenticate delegator and return private task: {auth_resp}"
);
}
use axum::body::Body;
use axum::http::{Request, StatusCode};
use sqlx::PgPool;
use tower::ServiceExt;
use crate::db::RepoRecord;
use crate::test_support::test_state;
/// Regression: anonymous callers must not see the pin/anchor index (#121, #134).
#[sqlx::test]
+15
View File
@@ -59,6 +59,10 @@ pub struct AppState {
pub task_event_tx: tokio::sync::broadcast::Sender<TaskEventBroadcast>,
/// GraphQL schema (queries + mutations + subscriptions)
pub graphql_schema: Arc<crate::graphql::GitlawbSchema>,
/// MAC key for task-list continuation tokens, derived from the node
/// keypair seed. Held here (and handed to the GraphQL schema) so REST and
/// GraphQL mint and accept the same tokens.
pub task_cursor_key: crate::api::task_cursor::TaskCursorKey,
/// Fly.io machine ID — used for fly-replay routing in multi-machine deployments
pub machine_id: Option<String>,
/// Centralized repo storage: local disk cache + optional Tigris backend
@@ -326,6 +330,16 @@ pub struct AppState {
/// (`with_default_max_keys`, reject-before-insert) so a source-key farm cannot grow
/// it (INV-15).
pub git_ipfs_walk_per_caller: crate::rate_limit::PerCallerConcurrency,
/// Per-client-IP rate limiter for the task read routes. `GET /api/v1/tasks`
/// and `GET /api/v1/tasks/{id}` are anonymously reachable and run the #268
/// visibility gate (a task lookup plus deduped-repo and visibility-rule
/// queries) before returning the opaque 404, so an unauthenticated prober
/// costs the node real work per request whether or not anything is visible.
/// Keyed on the resolved client IP via `push_limiter_trust`. Layered on
/// `task_read_routes` via `rate_limit_by_ip`.
/// The same per-IP hourly budget applies to GraphQL `tasks` / `task`
/// queries and WebSocket task queries via `TaskReadBrake`.
pub task_read_rate_limiter: RateLimiter,
/// The `git` executable the served-git withheld-blob walk spawns. Production is
/// `"git"` (resolved via PATH); injectable so a fake `git` can drive the walk's
/// process-group teardown in handler tests without mutating the process-global
@@ -354,6 +368,7 @@ impl AppState {
self.ipfs_work_rate_limiter.cleanup().await;
self.sync_trigger_rate_limiter.cleanup().await;
self.peer_write_rate_limiter.cleanup().await;
self.task_read_rate_limiter.cleanup().await;
self.federated_rate_limiter.cleanup().await;
}
+761 -5
View File
@@ -14,6 +14,7 @@
use std::collections::HashMap;
use std::path::Path;
use std::path::PathBuf;
use std::sync::Arc;
use gitlawb_core::identity::Keypair;
@@ -435,9 +436,10 @@ async fn process_batch(
.await;
}
Err(e) => {
warn!(repo = %item.repo, origin = %origin_url, err = %e, "repo sync failed");
let _ = db.mark_sync_failed(&item.id).await;
crate::metrics::record_sync_processed("failed");
// Prune vs terminal classification lives in
// `handle_sync_item_error` (defer = stays pending for the
// next tick; anything else = failed).
handle_sync_item_error(db, &item, &origin_url, e).await;
}
}
}
@@ -691,6 +693,284 @@ async fn existing_promisor_state(repo: &str) -> PromisorProbe {
}
}
/// Byte-prefix check for the two node-managed blob namespaces exempt from
/// the fail-closed commit walk (`visibility_pack.rs`): request markers
/// (`refs/gitlawb/requests/`) and issue records (`refs/gitlawb/issues/`).
/// Bytewise so non-UTF-8 refnames (which git accepts) are classified, not
/// lossy-decoded into a false exempt match. Extracted so the predicate is
/// unit-testable; [`prune_non_exempt_gitlawb_refs`] enforces it on import.
fn gitlawb_ref_is_exempt(raw_refname: &[u8]) -> bool {
raw_refname.starts_with(b"refs/gitlawb/requests/")
|| raw_refname.starts_with(b"refs/gitlawb/issues/")
}
/// Parse one `git show-ref` output line into its refname bytes: `<hex-sha>`
/// (40-hex SHA-1 or 64-hex SHA-256, which a sha256 mirror inherits from its
/// origin), one space, then the refname. Empty lines (trailing newline) yield
/// `Ok(None)`; anything else-shaped is an anomaly the caller fails on rather
/// than skips — silently passing an unclassifiable line could leave an evil
/// ref imported. Extracted so malformed input is unit-testable without git.
fn show_ref_refname(line: &[u8]) -> anyhow::Result<Option<&[u8]>> {
if line.is_empty() {
return Ok(None);
}
let Some(sp) = line.iter().position(|b| *b == b' ') else {
return Err(anyhow::anyhow!("git show-ref emitted an unparsable line"));
};
let (sha, raw) = (&line[..sp], &line[sp + 1..]);
let sha_ok = (sha.len() == 40 || sha.len() == 64) && sha.iter().all(|b| b.is_ascii_hexdigit());
if !sha_ok || raw.is_empty() {
return Err(anyhow::anyhow!("git show-ref emitted an unparsable line"));
}
Ok(Some(raw))
}
/// Delete imported `refs/gitlawb/*` refs outside the two exempt subtrees.
///
/// Mirrors import whatever the origin advertises (`+refs/*:refs/*`), and
/// `transfer.hideRefs` is a serve-side knob: a repo's own `git fetch` never
/// consults local hideRefs, and origins hide only `requests/` anyway. So a
/// planted `refs/gitlawb/<other>` blob ref — creatable on any pre-gate node —
/// lands on every mirror and then trips `assert_all_refs_are_commits` closed,
/// wedging serving on each one. Prune at import instead: after clone/fetch,
/// list `refs/gitlawb` and delete every ref outside the exempt subtrees
/// (negative refspecs cannot re-include them, so post-import deletion is the
/// mechanism that works).
///
/// Callers map a transient prune error to [`PruneFailed`] so the batch
/// defers the row (stays `pending`, retried next tick) instead of failing
/// it: a failed row would leave a possibly-wedged mirror unretried until
/// the next peer announcement. Deterministic refusals (identity mismatch,
/// malformed enumeration) map to [`PruneInvalid`], which fails terminally
/// since no retry can clear them.
///
/// Enumeration is `git show-ref` (no pattern), parsed as raw bytes: refnames
/// cannot contain newlines, so one line per ref is lossless even for
/// non-UTF-8 names, which a lossy decode would miss. The `<sha> SP <refname>`
/// structure (not `for-each-ref --format=%(refname)`, which carries no object
/// id) is what lets the parser width-check 40-hex SHA-1 vs 64-hex SHA-256
/// and reject anything else-shaped instead of skipping it.
///
/// Fail-closed throughout: the exit code is classified (only `1` with empty
/// output means "no refs"; anything else non-zero, e.g. 128 on a missing
/// repo, is fatal, since a silent skip would leave an imported evil ref in
/// place while the sync reports success), and any line that is not
/// `<hex-sha> SP <refname>` — 40-hex SHA-1 or 64-hex SHA-256, which a sha256
/// mirror inherits from its origin — fails the sync rather than being
/// skipped into an unpruned wedge.
/// Bounded git spawn for the prune below: `kill_on_drop` plus a hard timeout
/// so a wedged child cannot stall a mirror import or sweep forever. The
/// refname delete runs on exact bytes, so args arrive as `OsStr`, not `str`.
/// Non-PruneInvalid failures (including a timeout) wrap as PruneFailed and
/// defer for retry, matching the transient classification below.
async fn prune_git(
local_path: &Path,
args: &[&std::ffi::OsStr],
label: &str,
) -> anyhow::Result<std::process::Output> {
let fut = async {
// allow-unbounded-git: bounded by the tokio::time::timeout wrapping
// this future below, plus kill_on_drop.
let mut cmd = tokio::process::Command::new("git");
cmd.kill_on_drop(true);
cmd.arg("-C").arg(local_path).args(args);
cmd.output().await
};
tokio::time::timeout(std::time::Duration::from_secs(30), fut)
.await
.map_err(|_| anyhow::anyhow!("{label} timed out"))?
.map_err(|e| anyhow::anyhow!("{label} failed to spawn: {e}"))
}
async fn prune_non_exempt_gitlawb_refs(local_path: &Path) -> anyhow::Result<()> {
let local_str = local_path.to_str().unwrap_or(".");
// Prove the path is itself a repo before trusting the enumeration:
// `git -C` walks up to the nearest ancestor repo, so inside a non-repo
// child (a partially-created mirror) `show-ref` would exit 0 with the
// ANCESTOR's refs and `update-ref -d` would then run against that repo.
// Require `rev-parse --absolute-git-dir` to resolve to the path itself
// (mirrors are bare repos, where the git dir IS the path).
let git_dir_out = prune_git(
local_path,
&[
std::ffi::OsStr::new("rev-parse"),
std::ffi::OsStr::new("--absolute-git-dir"),
],
"git rev-parse",
)
.await?;
if !git_dir_out.status.success() {
let stderr = String::from_utf8_lossy(&git_dir_out.stderr);
return Err(anyhow::anyhow!(PruneInvalid(format!(
"mirror path is not a git repo (rev-parse failed): {stderr}"
))));
}
let actual_dir = PathBuf::from(
String::from_utf8_lossy(&git_dir_out.stdout)
.trim()
.to_string(),
);
let canonical_path = local_path
.canonicalize()
.map_err(|e| anyhow::anyhow!(PruneInvalid(format!("mirror path does not resolve: {e}"))))?;
let canonical_git_dir = actual_dir
.canonicalize()
.map_err(|e| anyhow::anyhow!(PruneInvalid(format!("git dir does not resolve: {e}"))))?;
if canonical_git_dir != canonical_path {
return Err(anyhow::anyhow!(PruneInvalid(format!(
"mirror path {local_str} is not itself a repo (git dir resolves elsewhere); refusing to enumerate"
))));
}
let out = prune_git(
local_path,
&[std::ffi::OsStr::new("show-ref")],
"git show-ref",
)
.await?;
// Classify on the exit code (the `existing_promisor_state` precedent):
// 0 parses, 1 with empty output is an empty repo, anything else is a
// fatal enumeration failure that must not read as "nothing to prune".
match out.status.code() {
Some(0) => {}
Some(1) if out.stdout.is_empty() => return Ok(()),
_ => {
let stderr = String::from_utf8_lossy(&out.stderr);
return Err(anyhow::anyhow!(
"git show-ref failed with code {:?}: {stderr}",
out.status.code()
));
}
}
for line in out.stdout.split(|b| *b == b'\n') {
// Malformed lines are deterministic (same git, same output every
// tick): fail terminally via PruneInvalid, not transiently.
let Some(raw) =
show_ref_refname(line).map_err(|e| anyhow::anyhow!(PruneInvalid(e.to_string())))?
else {
continue;
};
if !(raw == b"refs/gitlawb" || raw.starts_with(b"refs/gitlawb/"))
|| gitlawb_ref_is_exempt(raw)
{
continue;
}
// Delete by exact bytes (not the lossy display string below): a
// non-UTF-8 refname would otherwise survive the prune.
let refname_os = os_from_raw_refname(raw);
let shown = refname_os.to_string_lossy().into_owned();
let delete = prune_git(
local_path,
&[
std::ffi::OsStr::new("update-ref"),
std::ffi::OsStr::new("-d"),
refname_os.as_os_str(),
std::ffi::OsStr::new("--no-deref"),
],
"git update-ref -d",
)
.await?;
if !delete.status.success() {
let stderr = String::from_utf8_lossy(&delete.stderr);
return Err(anyhow::anyhow!(
"git update-ref -d {shown} failed (mirror still carries a non-exempt refs/gitlawb ref): {stderr}"
));
}
tracing::info!(
gitlawb_ref = %shown,
"mirror: pruned non-exempt refs/gitlawb ref on import"
);
}
Ok(())
}
/// Lossless bytes → OsString for `update-ref -d`: non-UTF-8 refnames must be
/// deleted by exact bytes. Unix is the serving platform; elsewhere fall back
/// to a lossy decode (best effort where exact bytes are unrepresentable).
#[cfg(unix)]
fn os_from_raw_refname(raw: &[u8]) -> std::ffi::OsString {
use std::os::unix::ffi::OsStringExt;
std::ffi::OsString::from_vec(raw.to_vec())
}
#[cfg(not(unix))]
fn os_from_raw_refname(raw: &[u8]) -> std::ffi::OsString {
String::from_utf8_lossy(raw).into_owned().into()
}
/// Marker for mirror-prune failures: the clone/fetch succeeded, so the
/// mirror is on disk, but a non-exempt `refs/gitlawb/*` ref may still be
/// imported. Local-git conditions are transient (a retry can succeed), while
/// `mark_sync_failed` is terminal until the next peer announcement — and a
/// failed row would leave a possibly-wedged mirror unretried in the
/// meantime. The batch error arm downcasts to this and defers the row
/// (stays `pending`, retried on the next 30s tick, where fetch+prune are
/// idempotent) instead of failing it.
#[derive(Debug)]
struct PruneFailed(String);
impl std::fmt::Display for PruneFailed {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "mirror prune failed: {}", self.0)
}
}
impl std::error::Error for PruneFailed {}
/// Marker for deterministic mirror-prune refusals: the identity check (path
/// is not itself a repo) and malformed `show-ref` output cannot clear on
/// retry — re-running fetch+prune every tick would spin forever on a
/// permanently bad path. Unlike [`PruneFailed`], these fail the row
/// terminally so the condition surfaces instead of looping silently.
#[derive(Debug)]
struct PruneInvalid(String);
/// Preserve a deterministic prune refusal through the clone/fetch call
/// sites: `prune_non_exempt_gitlawb_refs` already returns `PruneInvalid`
/// for identity mismatches and malformed enumeration, and re-wrapping
/// every error as `PruneFailed` would stringify that type away before
/// `handle_sync_item_error` downcasts it — leaving a permanently invalid
/// mirror deferred (fetch+prune every tick) instead of failed. Transient
/// failures keep the `PruneFailed` deferral.
fn wrap_prune_error(e: anyhow::Error) -> anyhow::Error {
match e.downcast::<PruneInvalid>() {
Ok(invalid) => anyhow::Error::new(invalid),
Err(other) => anyhow::anyhow!(PruneFailed(other.to_string())),
}
}
impl std::fmt::Display for PruneInvalid {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "mirror prune refused: {}", self.0)
}
}
impl std::error::Error for PruneInvalid {}
/// Classify a mirror sync failure for one queue row. Transient prune
/// failures ([`PruneFailed`]) defer: the mirror is on disk but may still
/// carry a non-exempt refs/gitlawb ref, a transient local-git condition a
/// retry can clear — failing the row would leave it unretried until the
/// next peer announcement, wedging serving in the meantime. Deterministic
/// refusals ([`PruneInvalid`]) and everything else fail the row terminally:
/// retrying a permanently bad path every tick would spin forever instead of
/// surfacing. Extracted so the deferral (not just the prune itself) is
/// unit-testable.
async fn handle_sync_item_error(
db: &Db,
item: &crate::db::SyncQueueItem,
origin_url: &str,
err: anyhow::Error,
) {
if err.downcast_ref::<PruneFailed>().is_some() {
warn!(repo = %item.repo, origin = %origin_url, err = %err, "mirror prune failed; leaving sync row pending for retry");
crate::metrics::record_sync_processed("deferred");
return;
}
warn!(repo = %item.repo, origin = %origin_url, err = %err, "repo sync failed");
let _ = db.mark_sync_failed(&item.id).await;
crate::metrics::record_sync_processed("failed");
}
/// Mirror-clone a repo from a remote URL into a local bare repo.
/// `Promisor` mode adds `--filter=blob:limit=10g`, which marks the repo a git
/// promisor (so a pack with origin-omitted withheld blobs is accepted) while
@@ -715,6 +995,13 @@ async fn clone_repo(remote_url: &str, local_path: &Path, mode: MirrorMode) -> an
let stderr = String::from_utf8_lossy(&out.stderr);
return Err(anyhow::anyhow!("git clone --mirror failed: {stderr}"));
}
// A fresh mirror imports everything the origin advertises, including any
// planted non-exempt `refs/gitlawb/*` ref; prune before serving from it.
// Prune outcome flows through `wrap_prune_error`: transient failures
// defer (stays pending), deterministic refusals fail terminally.
prune_non_exempt_gitlawb_refs(local_path)
.await
.map_err(wrap_prune_error)?;
Ok(())
}
@@ -735,7 +1022,11 @@ async fn fetch_repo(local_path: &Path, remote_url: &str, mode: MirrorMode) -> an
])
.await?;
match mode {
// The fetch imports whatever the origin advertises via the stored
// `+refs/*:refs/*` refspec; prune non-exempt `refs/gitlawb/*` refs after
// every fetch so one planted origin ref cannot wedge serving on the
// mirror (see `prune_non_exempt_gitlawb_refs`).
let fetch_result = match mode {
MirrorMode::Promisor => {
git_run(&["-C", local_str, "config", "remote.origin.promisor", "true"]).await?;
git_run(&[
@@ -775,7 +1066,13 @@ async fn fetch_repo(local_path: &Path, remote_url: &str, mode: MirrorMode) -> an
git_run(&["-C", local_str, "fetch", "--prune", "origin"]).await
}
}
}
};
fetch_result?;
// Prune after every fetch (see above); outcome flows through
// `wrap_prune_error` like the clone path.
prune_non_exempt_gitlawb_refs(local_path)
.await
.map_err(wrap_prune_error)
}
#[cfg(test)]
@@ -2108,4 +2405,463 @@ mod tests {
assert!(object_count(&mirror) > 0, "mirror has no objects");
assert_eq!(sync_status(&pool, "z6Mkfoo/hello").await, "done");
}
/// The import-exemption predicate is bytewise over both node-managed
/// namespaces and nothing else. Non-UTF-8 bytes never match (git accepts
/// them, so a lossy decode could false-exempt). Narrowing either prefix
/// turns these red.
#[test]
fn gitlawb_import_exemption_covers_both_namespaces_by_bytes() {
for exempt in [
&b"refs/gitlawb/requests/550e8400-e29b-41d4-a716-446655440000"[..],
&b"refs/gitlawb/issues/abc123"[..],
] {
assert!(
gitlawb_ref_is_exempt(exempt),
"import must keep {}",
String::from_utf8_lossy(exempt)
);
}
for pruned in [
&b"refs/gitlawb/evil"[..],
&b"refs/gitlawb"[..],
&b"refs/gitlawbfoo"[..],
&b"refs/heads/main"[..],
&b""[..],
// Raw 0xFF byte: never an exempt prefix, always pruned.
&b"refs/gitlawb/\xffvil"[..],
] {
assert!(
!gitlawb_ref_is_exempt(pruned),
"import must prune {}",
String::from_utf8_lossy(pruned)
);
}
}
/// End-to-end prune on a scratch bare repo: a planted non-exempt
/// `refs/gitlawb/*` blob ref is deleted, exempt `requests/`/`issues/`
/// blob refs and content refs survive. Removing the prune call from
/// `clone_repo`/`fetch_repo` leaves the evil ref in place, which is what
/// wedges serving on mirrors — this test plants exactly that state and
/// proves the prune removes it.
#[test]
fn prune_removes_non_exempt_gitlawb_refs_and_keeps_exempt() {
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("test runtime");
rt.block_on(async {
let td = TempDir::new().unwrap();
let work = td.path().join("work");
let bare = td.path().join("m.git");
std::fs::create_dir_all(&work).unwrap();
// A real commit for the content branch (branch refs must point
// at commits; the evil/exempt refs below point at blobs).
std::fs::write(work.join("f.txt"), b"x\n").unwrap();
let run_work = |args: &[&str]| {
assert!(
Command::new("git")
.args(args)
.current_dir(&work)
.status()
.unwrap()
.success(),
"git {args:?} failed"
);
};
run_work(&["init", "-q"]);
run_work(&["config", "user.email", "t@t"]);
run_work(&["config", "user.name", "t"]);
run_work(&["add", "f.txt"]);
run_work(&["commit", "-qm", "init"]);
assert!(
Command::new("git")
.args([
"clone",
"-q",
"--bare",
work.to_str().unwrap(),
bare.to_str().unwrap()
])
.current_dir(td.path())
.status()
.unwrap()
.success(),
"git clone --bare failed"
);
let run = |args: &[&str]| {
assert!(
Command::new("git")
.args(args)
.current_dir(&bare)
.status()
.unwrap()
.success(),
"git {args:?} failed"
);
};
let run_out = |args: &[&str]| {
let out = Command::new("git")
.args(args)
.current_dir(&bare)
.output()
.unwrap();
assert!(out.status.success(), "git {args:?} failed");
out.stdout
};
run(&["init", "-q", "--bare", "."]);
std::fs::write(bare.join("blob-body"), b"marker\n").unwrap();
let blob = String::from_utf8_lossy(&run_out(&["hash-object", "-w", "blob-body"]))
.trim()
.to_string();
// The clone carries the content branch; plant the evil ref and
// both exempt refs (all blob-pointing, like node metadata).
run(&["update-ref", "refs/gitlawb/evil", &blob]);
run(&["update-ref", "refs/gitlawb/requests/r1", &blob]);
run(&["update-ref", "refs/gitlawb/issues/i1", &blob]);
prune_non_exempt_gitlawb_refs(&bare)
.await
.expect("prune succeeds");
let raw_refs = run_out(&["for-each-ref", "--format=%(refname)"]);
let refs = String::from_utf8_lossy(&raw_refs);
let set: std::collections::HashSet<&str> = refs
.lines()
.map(str::trim)
.filter(|l| !l.is_empty())
.collect();
assert!(
!set.contains("refs/gitlawb/evil"),
"non-exempt blob ref must be pruned, still present: {set:?}"
);
for keep in ["refs/gitlawb/requests/r1", "refs/gitlawb/issues/i1"] {
assert!(set.contains(keep), "{keep} must survive the prune: {set:?}");
}
assert!(
set.iter().any(|r| r.starts_with("refs/heads/")),
"content branch ref must survive the prune: {set:?}"
);
});
}
/// `show-ref` line parser: 40-hex SHA-1 and 64-hex SHA-256 lines yield
/// the refname, empty lines yield None, and anything else-shaped errors
/// (the caller fails the sync rather than skipping into an unpruned
/// wedge). Reverting to a 40-only width check turns the sha256 case red.
#[test]
fn show_ref_line_parser_accepts_both_hash_widths_and_rejects_junk() {
// 40-hex SHA-1 line.
let sha1_line = format!("{} refs/gitlawb/evil", "a".repeat(40));
assert_eq!(
show_ref_refname(sha1_line.as_bytes()).unwrap(),
Some(b"refs/gitlawb/evil".as_slice())
);
// 64-hex SHA-256 line (a sha256 mirror inherits the width).
let sha256_line = format!("{} refs/gitlawb/evil", "b".repeat(64));
assert_eq!(
show_ref_refname(sha256_line.as_bytes()).unwrap(),
Some(b"refs/gitlawb/evil".as_slice())
);
// Non-UTF-8 refname passes through as exact bytes.
let mut raw = vec![b'c'; 40];
raw.extend_from_slice(b" refs/gitlawb/\xffvil");
assert_eq!(
show_ref_refname(&raw).unwrap(),
Some(b"refs/gitlawb/\xffvil".as_slice())
);
// Trailing newline residue.
assert_eq!(show_ref_refname(b"").unwrap(), None);
// No space, short sha, non-hex sha, empty refname: all errors.
let bads: Vec<Vec<u8>> = vec![
b"nospaceshere".to_vec(),
format!("{}refs/x", "a".repeat(40)).into_bytes(),
[b'z'; 40]
.iter()
.chain(b" refs/x".iter())
.copied()
.collect(),
format!("{} ", "a".repeat(40)).into_bytes(),
format!("{} refs/x", "a".repeat(41)).into_bytes(),
];
for bad in &bads {
assert!(
show_ref_refname(bad).is_err(),
"malformed show-ref line must error, not skip"
);
}
}
/// Pruning a path that is not a repo errors. Two shapes: a missing path
/// fails `rev-parse` outright (exit 128), while a plain subdirectory of
/// a repo would otherwise enumerate the ancestor's refs — covered by the
/// identity test below. Either way the prune refuses instead of pruning
/// the wrong repo or silently skipping an evil ref.
#[test]
fn prune_on_non_repo_path_errors() {
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("test runtime");
rt.block_on(async {
let err =
prune_non_exempt_gitlawb_refs(Path::new("/tmp/gitlawb-definitely-not-a-repo-xyz"))
.await
.expect_err("non-repo path must fail the prune, not skip it");
assert!(
err.to_string().contains("not a git repo")
|| err.to_string().contains("does not resolve"),
"unexpected error shape: {err:#}"
);
});
}
/// Identity check: pruning a plain subdirectory of a repo refuses instead
/// of enumerating the ancestor. `git -C` on the child exits 0 from
/// `show-ref` with the ANCESTOR's refs, so without the
/// `rev-parse --absolute-git-dir` identity check the delete loop would
/// enumerate and delete the wrong repository's refs. Deleting the check
/// makes the prune succeed and removes the ancestor's evil ref — both
/// assertions go red.
#[test]
fn prune_on_non_repo_child_of_repo_refuses_and_touches_nothing() {
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("test runtime");
rt.block_on(async {
let td = TempDir::new().unwrap();
let ancestor = td.path().join("ancestor.git");
std::fs::create_dir_all(&ancestor).unwrap();
let run_ancestor = |args: &[&str]| {
assert!(
Command::new("git")
.args(args)
.current_dir(&ancestor)
.status()
.unwrap()
.success(),
"git {args:?} failed"
);
};
run_ancestor(&["init", "-q", "--bare", "."]);
// Blob target for the planted ref (mirrors the node-metadata shape).
std::fs::write(ancestor.join("blob-body"), b"marker\n").unwrap();
let out = Command::new("git")
.args(["hash-object", "-w", "blob-body"])
.current_dir(&ancestor)
.output()
.unwrap();
assert!(out.status.success(), "git hash-object failed");
let blob = String::from_utf8_lossy(&out.stdout).trim().to_string();
run_ancestor(&["update-ref", "refs/gitlawb/evil", &blob]);
// A plain child directory: not itself a repo.
let child = ancestor.join("not-a-repo");
std::fs::create_dir_all(&child).unwrap();
let err = prune_non_exempt_gitlawb_refs(&child)
.await
.expect_err("pruning a non-repo child must refuse, not enumerate the ancestor");
assert!(
err.to_string().contains("not itself a repo"),
"refusal must name the identity mismatch, got: {err:#}"
);
// The ancestor's evil ref is untouched.
let refs_out = Command::new("git")
.args(["for-each-ref", "--format=%(refname)"])
.current_dir(&ancestor)
.output()
.unwrap();
let refs = String::from_utf8_lossy(&refs_out.stdout);
assert!(
refs.lines().any(|l| l.trim() == "refs/gitlawb/evil"),
"ancestor evil ref must survive the refused prune"
);
});
}
/// sha256 mirror: a 64-hex `show-ref` line parses and the planted ref is
/// pruned. A 40-only parser skips every line and leaves the evil ref.
#[test]
fn prune_parses_sha256_show_ref_and_prunes() {
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("test runtime");
rt.block_on(async {
let td = TempDir::new().unwrap();
let bare = td.path().join("m256.git");
std::fs::create_dir_all(&bare).unwrap();
let run = |args: &[&str]| {
assert!(
Command::new("git")
.args(args)
.current_dir(&bare)
.status()
.unwrap()
.success(),
"git {args:?} failed"
);
};
let run_out = |args: &[&str]| {
let out = Command::new("git")
.args(args)
.current_dir(&bare)
.output()
.unwrap();
assert!(out.status.success(), "git {args:?} failed");
out.stdout
};
run(&["init", "-q", "--bare", "--object-format=sha256", "."]);
std::fs::write(bare.join("blob-body"), b"marker\n").unwrap();
let blob = String::from_utf8_lossy(&run_out(&["hash-object", "-w", "blob-body"]))
.trim()
.to_string();
assert_eq!(
blob.len(),
64,
"sanity: sha256 repo must mint 64-hex objects"
);
run(&["update-ref", "refs/gitlawb/evil", &blob]);
run(&["update-ref", "refs/gitlawb/issues/i1", &blob]);
prune_non_exempt_gitlawb_refs(&bare)
.await
.expect("prune succeeds on a sha256 repo");
let raw_refs = run_out(&["for-each-ref", "--format=%(refname)"]);
let refs = String::from_utf8_lossy(&raw_refs);
let set: std::collections::HashSet<&str> = refs
.lines()
.map(str::trim)
.filter(|l| !l.is_empty())
.collect();
assert!(
!set.contains("refs/gitlawb/evil"),
"evil ref must be pruned from sha256 mirror: {set:?}"
);
assert!(
set.contains("refs/gitlawb/issues/i1"),
"exempt ref must survive: {set:?}"
);
});
}
/// Prune failures defer the queue row (stays `pending` for the next
/// tick); deterministic prune refusals and any other sync error fail it
/// terminally. Deleting the `downcast_ref::<PruneFailed>` arm restores
/// terminal failure and turns the first assertion red.
/// Prune failures defer the queue row (stays `pending` for the next
/// tick); deterministic prune refusals and any other sync error fail it
/// terminally. Deleting the `downcast_ref::<PruneFailed>` arm restores
/// terminal failure and turns the first assertion red.
///
/// The wrapper below is what the clone/fetch call sites run through:
/// a `PruneInvalid` from the helper must still downcast as
/// `PruneInvalid` afterwards (not stringified into `PruneFailed`),
/// and anything else must arrive as `PruneFailed`.
#[test]
fn wrap_prune_error_preserves_deterministic_type() {
let invalid = wrap_prune_error(anyhow::anyhow!(PruneInvalid("x".to_string())));
assert!(
invalid.downcast_ref::<PruneInvalid>().is_some(),
"deterministic refusal must survive the call-site wrap"
);
assert!(
invalid.downcast_ref::<PruneFailed>().is_none(),
"deterministic refusal must not also read as transient"
);
let transient = wrap_prune_error(anyhow::anyhow!("boom"));
assert!(
transient.downcast_ref::<PruneFailed>().is_some(),
"generic failure must arrive as transient"
);
assert!(
transient.downcast_ref::<PruneInvalid>().is_none(),
"generic failure must not read as deterministic"
);
}
#[sqlx::test]
async fn prune_failure_defers_row_other_failures_fail_it(pool: PgPool) {
let state = crate::test_support::test_state(pool).await;
let item = |id: &str| crate::db::SyncQueueItem {
id: id.to_string(),
repo: "z6Mkfoo/hello".to_string(),
node_did: "did:key:z6MkOrigin".to_string(),
ref_name: "refs/heads/main".to_string(),
new_sha: "0".repeat(40),
cid: None,
status: "pending".to_string(),
enqueued_at: chrono::Utc::now().to_rfc3339(),
};
async fn status_of(pool: &sqlx::PgPool, id: &str) -> String {
let row: (String,) = sqlx::query_as("SELECT status FROM sync_queue WHERE id = $1")
.bind(id)
.fetch_one(pool)
.await
.unwrap();
row.0
}
// Seed three pending rows directly (enqueue_sync mints random ids).
for id in ["sync-prune-defer", "sync-prune-invalid", "sync-other-fail"] {
sqlx::query(
"INSERT INTO sync_queue (id, repo, node_did, ref_name, new_sha, cid, status, enqueued_at)
VALUES ($1, $2, $3, $4, $5, NULL, 'pending', $6)",
)
.bind(id)
.bind("z6Mkfoo/hello")
.bind("did:key:z6MkOrigin")
.bind("refs/heads/main")
.bind("0".repeat(40))
.bind(chrono::Utc::now().to_rfc3339())
.execute(state.db.pool())
.await
.unwrap();
}
handle_sync_item_error(
&state.db,
&item("sync-prune-defer"),
"http://origin.example",
anyhow::anyhow!(PruneFailed("boom".to_string())),
)
.await;
assert_eq!(
status_of(state.db.pool(), "sync-prune-defer").await,
"pending",
"prune failure must leave the row pending for retry"
);
handle_sync_item_error(
&state.db,
&item("sync-other-fail"),
"http://origin.example",
anyhow::anyhow!("boom"),
)
.await;
assert_eq!(
status_of(state.db.pool(), "sync-other-fail").await,
"failed",
"non-prune failure must fail the row"
);
// Deterministic refusal: no retry can clear it, so it fails
// terminally instead of spinning every tick forever.
handle_sync_item_error(
&state.db,
&item("sync-prune-invalid"),
"http://origin.example",
anyhow::anyhow!(PruneInvalid("not itself a repo".to_string())),
)
.await;
assert_eq!(
status_of(state.db.pool(), "sync-prune-invalid").await,
"failed",
"deterministic prune refusal must fail the row, not defer forever"
);
}
}
+319 -156
View File
@@ -86,10 +86,12 @@ fn build_state(db: Arc<crate::db::Db>, pool: PgPool) -> AppState {
let node_did = keypair.did();
let (ref_tx, _) = tokio::sync::broadcast::channel(1);
let (task_tx, _) = tokio::sync::broadcast::channel(1);
let task_cursor_key = crate::api::task_cursor::TaskCursorKey::derive(&keypair.to_seed());
let schema = Arc::new(graphql::build_schema(
db.clone(),
ref_tx.clone(),
task_tx.clone(),
task_cursor_key.clone(),
));
AppState {
config: Arc::new(Config::parse_from(["gitlawb-node"])),
@@ -101,6 +103,7 @@ fn build_state(db: Arc<crate::db::Db>, pool: PgPool) -> AppState {
ref_update_tx: ref_tx,
task_event_tx: task_tx,
graphql_schema: schema,
task_cursor_key,
machine_id: None,
repo_store: crate::git::repo_store::RepoStore::for_testing(PathBuf::from("/tmp"), pool),
rate_limiter: RateLimiter::new(100, Duration::from_secs(60)),
@@ -139,6 +142,7 @@ fn build_state(db: Arc<crate::db::Db>, pool: PgPool) -> AppState {
git_ipfs_walk_per_caller: crate::rate_limit::PerCallerConcurrency::with_default_max_keys(
16,
),
task_read_rate_limiter: RateLimiter::new(1200, Duration::from_secs(3600)),
git_bin: "git".to_string(),
}
}
@@ -186,6 +190,153 @@ pub(crate) async fn silent_http_endpoint() -> String {
endpoint
}
/// Body of the `for-each-ref` arm of a fake-git fixture in the
/// COLUMN SHAPE `blob_paths` phase 2 parses. The caller is
/// responsible for wrapping this in a full `case "$1" in ... esac`
/// shell script and writing it to a tempdir; this is the form
/// used when a fixture has OTHER arms (e.g. `rev-list`,
/// `pack-objects`) that the visibility-pipeline tests also need
/// to fake.
///
/// Two output shapes:
/// - `tag` tip (`peeled_oid` and `peeled_kind` empty) →
/// `echo '<oid> <kind>'` (two tokens).
/// - Annotated tag tip → `echo '<oid> tag <peeled_oid> <peeled_kind>'`
/// (four tokens; the literal `tag` in slot 2 is the parser's
/// "peeled type is `tag`" trigger for the recursive peel).
///
/// An empty `refs` slice emits a single `:` so phase 2 sees no
/// lines (the same as a bare default `*) : ;;` arm).
#[allow(dead_code)] // referenced by `fake_git_with_refs` and the round-9 fixtures
pub(crate) fn fake_git_for_ref_body(refs: &[(&str, &str, &str, &str)]) -> String {
// P2 (reviewer round 9): the previous body emitted
// `echo ... ;;` per ref, which is a `;;` PER REF inside
// a single case arm. The first `;;` closes the arm, and
// every subsequent `echo` parses as a pattern line, which
// `sh -n` rejects with "word unexpected". The right shape
// is one `;;` per arm, emitted once after the last ref.
let mut body = String::new();
if refs.is_empty() {
body.push_str(" : ;;\n");
} else {
for (oid, kind, peeled_oid, peeled_kind) in refs {
if peeled_oid.is_empty() && peeled_kind.is_empty() {
body.push_str(&format!(" echo '{oid} {kind}'\n"));
} else {
body.push_str(&format!(
" echo '{oid} tag {peeled_oid} {peeled_kind}'\n"
));
}
}
body.push_str(" ;;\n");
}
body
}
/// Full fake-git script body for a fixture whose ONLY fake arm is
/// `for-each-ref`. All other `git` subcommands are answered by
/// the default `*) : ;;` no-op, so a real-git repo with matching
/// refs is needed to drive the rest of the walk. Used by tests
/// that want the parser contract enforced without committing to
/// the other arms the smart-HTTP fixture cares about.
#[allow(dead_code)] // referenced by tests in the unit-test mod below
pub(crate) fn fake_git_with_refs(refs: &[(&str, &str, &str, &str)]) -> String {
let mut body = String::from("#!/bin/sh\ncase \"$1\" in\n for-each-ref)\n");
body.push_str(&fake_git_for_ref_body(refs));
body.push_str(" *) : ;;\nesac\nexit 0\n");
body
}
#[cfg(test)]
mod helper_tests {
use super::*;
/// #218 round 9 (guidance #6): the helper emits the column
/// shape `blob_paths` phase 2 parses. Pin the format at the
/// cargo-test level so a parser regression breaks this
/// helper test in addition to the production tests.
/// P2 (reviewer round 9): also execute the generated script
/// through `sh -n` and against a tempdir; the previous
/// substring check passed while the script was a `sh`
/// syntax error (the `;;` was emitted once per ref inside
/// a single case arm, which `sh` rejects).
#[test]
fn fake_git_with_refs_emits_the_column_shape() {
let script = fake_git_with_refs(&[
("commit0000000000000000000000000000000", "commit", "", ""),
(
"tag0000000000000000000000000000000000",
"tag",
"peel000000000000000000000000000000",
"blob",
),
]);
assert!(
script.contains("'commit0000000000000000000000000000000 commit'"),
"non-tag tip must emit two tokens: got\n{script}"
);
assert!(
script.contains("'tag0000000000000000000000000000000000 tag peel000000000000000000000000000000 blob'"),
"annotated tag tip must emit four tokens: got\n{script}"
);
// P2 (reviewer round 9): execute the script through
// `sh -n` to catch the `;;` per-ref syntax error the
// previous test missed. The previous test only checked
// substring presence and was green while the script was
// malformed shell.
let sh_n = std::process::Command::new("sh")
.args(["-n", "-c", &script])
.status()
.expect("sh -n must run");
assert!(
sh_n.success(),
"the generated script must be valid shell; \
`sh -n` exited {sh_n:?}\n----\n{script}\n----"
);
// Also run the script for real against a tempdir. The
// non-tag tip must echo the two-token line, and the
// annotated-tag tip must echo the four-token line.
let td = tempfile::tempdir().expect("tempdir");
let script_path = td.path().join("fake-git.sh");
std::fs::write(&script_path, &script).expect("write script");
std::fs::set_permissions(
&script_path,
std::os::unix::fs::PermissionsExt::from_mode(0o755),
)
.expect("chmod");
let out = std::process::Command::new(&script_path)
.arg("for-each-ref")
.output()
.expect("run script");
let stdout = String::from_utf8_lossy(&out.stdout);
assert!(
stdout.contains("commit0000000000000000000000000000000 commit"),
"the two-token tip must print: stdout={stdout:?}\nscript:\n{script}"
);
assert!(
stdout.contains(
"tag0000000000000000000000000000000000 tag peel000000000000000000000000000000 blob"
),
"the four-token tip must print: stdout={stdout:?}\nscript:\n{script}"
);
}
#[test]
fn fake_git_with_refs_empty_slice_emits_a_zero_refs_marker() {
let script = fake_git_with_refs(&[]);
assert!(
script.contains("for-each-ref)"),
"the helper still owns the for-each-ref arm: got\n{script}"
);
assert!(
script.contains(" : ;;\n"),
"an empty refs slice must emit a single `:` so phase 2 sees zero lines: got\n{script}"
);
}
}
#[cfg(test)]
mod tests {
use super::*;
@@ -1115,7 +1266,7 @@ mod tests {
// repo id afterwards — a 403 that still writes is the injection.
let stored = state
.db
.list_tasks(None, None, 50)
.list_tasks_keyset(None, None, 50, None)
.await
.expect("list tasks");
assert!(
@@ -1474,11 +1625,11 @@ mod tests {
let assignee = "did:key:zTASKASSIGNEEBBBBBBBBBBBBBBBBBBBBBBBBBBBB";
let stranger = "did:key:zTASKSTRANGERCCCCCCCCCCCCCCCCCCCCCCCCCCC";
let state = test_state(pool).await;
state
.db
.create_task(&seed_task("task-1", delegator))
.await
.expect("seed task");
let repo = seed_repo(delegator, "task-pub-repo");
state.db.create_repo(&repo).await.expect("seed repo");
let mut t1 = seed_task("task-1", delegator);
t1.repo_id = Some(repo.id);
state.db.create_task(&t1).await.expect("seed task");
// Assignee claims it: pending -> claimed, assignee_did = assignee.
state
.db
@@ -1497,8 +1648,25 @@ mod tests {
let uri = "/api/v1/tasks/task-1/complete";
let body = || Body::from("{}");
// Stranger (not the assignee) is rejected by the authorization gate, even
// with the empty body that previously bypassed the binding. Exact 403.
// Stranger on an invisible (repo-less) task receives opaque 404 (no existence leak).
let inv_task = seed_task("task-inv", delegator);
state.db.create_task(&inv_task).await.unwrap();
let inv_resp = router()
.oneshot(signed_request_as(
stranger,
Method::POST,
"/api/v1/tasks/task-inv/complete",
body(),
))
.await
.unwrap();
assert_eq!(
inv_resp.status(),
StatusCode::NOT_FOUND,
"an invisible task must 404 so existence is not leaked"
);
// Stranger on a visible task is rejected by the authorization gate with exact 403.
let resp = router()
.oneshot(signed_request_as(stranger, Method::POST, uri, body()))
.await
@@ -1567,9 +1735,11 @@ mod tests {
))
.await
.unwrap();
// The claim gate hides existence: an ineligible claimer gets the same
// 404 as a missing task, not a 403 that would reveal the reservation.
assert_eq!(
resp.status(),
StatusCode::FORBIDDEN,
StatusCode::NOT_FOUND,
"stranger must not steal a pre-assigned task, got {}",
resp.status()
);
@@ -1735,7 +1905,7 @@ mod tests {
);
let listed_bare = state
.db
.list_tasks(Some("claimed"), Some(bare), 10)
.list_tasks_keyset(Some("claimed"), Some(bare), 10, None)
.await
.unwrap();
assert!(
@@ -1744,12 +1914,14 @@ mod tests {
);
let listed_full = state
.db
.list_tasks(Some("claimed"), Some(&full), 10)
.list_tasks_keyset(Some("claimed"), Some(&full), 10, None)
.await
.unwrap();
// The keyset filter normalizes to the did:key short form, so the
// claimer's full DID now matches the bare stored form too.
assert!(
!listed_full.iter().any(|t| t.id == "task-form"),
"exact list by claimer's full DID must miss the bare stored form"
listed_full.iter().any(|t| t.id == "task-form"),
"normalized list by claimer's full DID must match the bare stored form"
);
}
@@ -1786,7 +1958,7 @@ mod tests {
);
let listed = state
.db
.list_tasks(Some("claimed"), Some(&padded), 10)
.list_tasks_keyset(Some("claimed"), Some(&padded), 10, None)
.await
.unwrap();
assert!(
@@ -7180,6 +7352,7 @@ mod tests {
&state.db,
&pub_repo.id,
crate::ipfs_pin::PIN_BATCH_BUDGET,
None,
)
.await;
m.assert_async().await; // asserts /add was NOT called (already pinned)
@@ -7306,6 +7479,7 @@ mod tests {
&state.db,
&pub_repo.id,
crate::ipfs_pin::PIN_BATCH_BUDGET,
None,
)
.await;
m.assert_async().await; // /add NOT called (already pinned)
@@ -7496,6 +7670,7 @@ mod tests {
&state.db,
repo_id,
crate::ipfs_pin::PIN_BATCH_BUDGET,
None,
)
.await;
m.assert_async().await;
@@ -8141,15 +8316,25 @@ mod tests {
let repo = seed_repo(&owner_did, "u3pinata");
state.db.create_repo(&repo).await.expect("seed repo");
// Already carries a pinata_cid, so pin_new_objects takes the skip branch and the
// only DB write under test is the source record.
// Already a Pinata-pinned row so the Pinata pin loop's
// `has_pinata_cid` skip branch fires and the only DB write
// under test is the U3 source record retry. The IPFS pin
// loop's `is_pinned` / `has_ipfs_cid` skip is a separate
// seam (this test exercises the Pinata path, not the IPFS
// path), so a Pinata-only seed is the right shape here.
let (_ty, raw) = crate::git::store::read_object(&bare, &fx.public_oid)
.unwrap()
.expect("object readable");
let raw_cid = gitlawb_core::cid::Cid::from_git_object_bytes(&raw).to_string();
state
.db
.record_pinata_cid(&fx.public_oid, &raw_cid, "QmProvider", Some(&repo.id))
.record_pinata_cid(
&fx.public_oid,
&raw_cid,
"QmProvider",
Some(&repo.id),
i64::MAX,
)
.await
.expect("seed pinata pin");
@@ -8186,6 +8371,7 @@ mod tests {
// never truncates the one object under test: what is being measured
// is the retry backoff, not the budget.
std::time::Duration::from_secs(60),
None,
)
.await;
m.assert_async().await; // the upload is skipped: DB-only path
@@ -8357,6 +8543,7 @@ mod tests {
"repoPinataBound",
// The bound under test.
std::time::Duration::from_secs(2),
None,
),
)
.await
@@ -8424,6 +8611,7 @@ mod tests {
&state.db,
"repoKuboBound",
std::time::Duration::from_secs(2),
None,
),
)
.await
@@ -8486,6 +8674,7 @@ mod tests {
&state.db,
"repoPinataRepair",
crate::ipfs_pin::PIN_BATCH_BUDGET,
None,
)
.await;
m.assert_async().await;
@@ -8544,6 +8733,7 @@ mod tests {
&state.db,
"repoPinataGate",
crate::ipfs_pin::PIN_BATCH_BUDGET,
None,
)
.await;
m.assert_async().await;
@@ -8616,6 +8806,7 @@ mod tests {
&state.db,
"repoPinataWarn",
crate::ipfs_pin::PIN_BATCH_BUDGET,
None,
)
.await;
m.assert_async().await;
@@ -8687,6 +8878,7 @@ mod tests {
&state.db,
"repoPinataNoSkip",
crate::ipfs_pin::PIN_BATCH_BUDGET,
None,
)
.await;
m.assert_async().await;
@@ -8703,7 +8895,7 @@ mod tests {
);
assert_eq!(stashed, None, "and nothing is stashed for it");
assert_eq!(
pinned,
pinned.confirmed,
vec![(fx.public_oid.clone(), "QmPinataUploaded".to_string())],
"the pinata return still carries the provider CID for the announcement cid_map"
);
@@ -9177,7 +9369,7 @@ mod tests {
// raw CID in `cid` with the provider CID in `pinata_cid`.
state
.db
.record_pinata_cid("po1", &raw1, "pcid1", Some("repoA"))
.record_pinata_cid("po1", &raw1, "pcid1", Some("repoA"), i64::MAX)
.await
.unwrap();
assert_eq!(
@@ -9192,7 +9384,7 @@ mod tests {
.into_iter()
.find(|r| r.sha256_hex == "po1")
.expect("po1 row exists");
assert_eq!(po1.cid, raw1, "resolver-key cid is the raw CID");
assert_eq!(po1.cid, Some(raw1), "resolver-key cid is the raw CID");
assert_eq!(
po1.pinata_cid.as_deref(),
Some("pcid1"),
@@ -9208,7 +9400,7 @@ mod tests {
.unwrap();
state
.db
.record_pinata_cid("po2", "rawcid2", "pcid2", Some("repoB"))
.record_pinata_cid("po2", "rawcid2", "pcid2", Some("repoB"), i64::MAX)
.await
.unwrap();
assert_eq!(
@@ -9225,7 +9417,8 @@ mod tests {
.find(|r| r.sha256_hex == "po2")
.expect("po2 row exists");
assert_eq!(
po2.cid, local2,
po2.cid,
Some(local2),
"on conflict the prior local pin's cid is left untouched"
);
@@ -9237,7 +9430,7 @@ mod tests {
.unwrap();
state
.db
.record_pinata_cid("po3", "rawcid3", "pcid3", Some("repoY"))
.record_pinata_cid("po3", "rawcid3", "pcid3", Some("repoY"), i64::MAX)
.await
.unwrap();
assert_eq!(
@@ -9269,7 +9462,7 @@ mod tests {
// Pinata-first: no prior local pin, so this INSERT creates the row.
state
.db
.record_pinata_cid("pfsha", &raw_cid, provider_cid, Some("repoP"))
.record_pinata_cid("pfsha", &raw_cid, provider_cid, Some("repoP"), i64::MAX)
.await
.unwrap();
@@ -9322,10 +9515,11 @@ mod tests {
&state.db,
"repoZ",
crate::ipfs_pin::PIN_BATCH_BUDGET,
None,
)
.await;
assert!(
!pinned.is_empty(),
!pinned.confirmed.is_empty(),
"the object was pinned via the real pin path"
);
m.assert_async().await;
@@ -9386,13 +9580,14 @@ mod tests {
// (PIN_RECORD_ATTEMPTS x PIN_RECORD_BACKOFF), so the batch budget gate
// is never what truncates this run.
std::time::Duration::from_secs(60),
None,
)
.await
})
.await;
assert!(
pinned.is_empty(),
pinned.confirmed.is_empty(),
"a pin with no durable index row must not be reported as pinned, got {pinned:?}"
);
// Exactly two adds: the first record failure did not break the batch.
@@ -9506,6 +9701,7 @@ mod tests {
&db,
"repoWedge",
crate::ipfs_pin::PIN_BATCH_BUDGET,
None,
),
)
.await
@@ -9519,7 +9715,7 @@ mod tests {
not leave it running (which would pin the coalescing key until process death)"
);
assert!(
pinned.is_empty(),
pinned.confirmed.is_empty(),
"a wedged read pins nothing this pass; a later pass/push retries"
);
}
@@ -9636,11 +9832,12 @@ mod tests {
&state.db,
"repoBF",
crate::ipfs_pin::PIN_BATCH_BUDGET,
None,
)
.await;
assert!(
pinned.is_empty(),
pinned.confirmed.is_empty(),
"an already-pinned object is not re-pinned (no bytes returned)"
);
m.assert_async().await; // asserts /add was called 0 times
@@ -9712,9 +9909,17 @@ mod tests {
);
// Legacy-shape row: cid = the PROVIDER CID (raw SQL — the helpers store the
// raw CID). The object itself is public and servable.
// raw CID). The object itself is public and servable: the bytes are
// on local IPFS (so `local_ipfs_provenance = TRUE` under #218 review
// P1's writer-owned contract), only the CID key is wrong. Without
// `local_ipfs_provenance = TRUE`, the new `has_ipfs_cid` check in
// `pin_new_objects` would fall through to the upload path and re-pin
// bytes that are already on IPFS — the test's `expect(0)` mock
// would fail. Setting the flag reflects the real pre-upgrade
// state (a legacy local IPFS pin with the wrong CID).
sqlx::query(
"INSERT INTO pinned_cids (sha256_hex, cid, pinned_at, repo_id) VALUES ($1, $2, $3, $4)",
"INSERT INTO pinned_cids (sha256_hex, cid, pinned_at, repo_id, local_ipfs_provenance) \
VALUES ($1, $2, $3, $4, TRUE)",
)
.bind(&fx.public_oid)
.bind(&provider_cid)
@@ -9758,6 +9963,7 @@ mod tests {
&state.db,
&repo.id,
crate::ipfs_pin::PIN_BATCH_BUDGET,
None,
)
.await;
m.assert_async().await;
@@ -9862,6 +10068,7 @@ mod tests {
&state.db,
"repoCG",
crate::ipfs_pin::PIN_BATCH_BUDGET,
None,
)
.await;
m.assert_async().await;
@@ -9927,6 +10134,7 @@ mod tests {
&state.db,
"repoUR",
crate::ipfs_pin::PIN_BATCH_BUDGET,
None,
)
.await;
@@ -10132,7 +10340,7 @@ mod tests {
.await
.unwrap()
.iter()
.any(|r| r.cid == raw_cid),
.any(|r| r.cid.as_deref() == Some(raw_cid.as_str())),
"the repaired row is advertised"
);
let (st, body) = cid_parts(
@@ -10752,7 +10960,7 @@ mod tests {
.await
.unwrap()
.iter()
.any(|r| r.cid == raw_cid),
.any(|r| r.cid.as_deref() == Some(raw_cid.as_str())),
"the repaired row is advertised again"
);
}
@@ -10945,7 +11153,7 @@ mod tests {
.await
.unwrap()
.iter()
.any(|r| r.cid == low_raw),
.any(|r| r.cid.as_deref() == Some(low_raw.as_str())),
"the repaired row is advertised again"
);
}
@@ -14539,9 +14747,14 @@ mod tests {
/// resolver would withhold. The resolver recomputes the raw CIDv1 from the object
/// bytes and 404s any row keyed on a legacy PROVIDER CID, so advertising that key
/// hands clients a CID this node deliberately refuses. Both states of ONE row are
/// asserted (omitted while legacy, present once repaired) so the test cannot pass
/// by accident. RED before the `is_raw_cidv1` filter lands: the legacy row is
/// advertised.
/// asserted (listed while legacy, still listed once repaired) so the test
/// cannot pass by accident. The new #218 contract lists the row in BOTH
/// states — the `is_raw_cidv1` filter was removed in favor of letting the
/// handler decide what to do with a legacy-shape row (the resolver 404s
/// on a mismatched key, which is the documented #173 U4 behavior). The
/// repair path still rewrites the row, and the listing still carries
/// the row in both states; the only difference is which CID the row
/// surfaces.
#[sqlx::test]
async fn list_pinned_cids_omits_unrepaired_legacy_row(pool: PgPool) {
let state = test_state(pool).await;
@@ -14557,12 +14770,16 @@ mod tests {
.unwrap();
let listed = state.db.list_pinned_cids().await.unwrap();
assert!(
!listed.iter().any(|r| r.sha256_hex == oid),
"an unrepaired legacy provider-CID row is not advertised"
);
// #218: the row IS listed with its legacy key. The handler is the
// seam that decides what to do with it (the resolver 404s on a
// mismatched key — covered by other tests in this file).
let rec = listed
.iter()
.find(|r| r.sha256_hex == oid)
.expect("an unrepaired legacy row is still listed (#218 contract)");
assert_eq!(rec.cid.as_deref(), Some(provider_cid.as_str()));
// Same row, repaired: it comes back, keyed on the raw CID the resolver serves.
// Same row, repaired: it stays listed but the key is now the raw CID.
state
.db
.repair_legacy_provider_cid(&oid, &raw_cid, &provider_cid)
@@ -14574,7 +14791,8 @@ mod tests {
.find(|r| r.sha256_hex == oid)
.expect("the repaired row is advertised again");
assert_eq!(
rec.cid, raw_cid,
rec.cid,
Some(raw_cid),
"the advertised key is the raw-content resolver key"
);
}
@@ -15444,7 +15662,15 @@ mod tests {
"reader's tree body carries the child filename and raw child oid"
);
// Root tree (path "/") stays served to anon who passes the "/" gate.
// Root tree (path "/") is denied to anon: under #218 review
// P1b's recursive structural gate, the root tree's serialized
// bytes name the `secret` entry and the secret subtree's
// child OID, so admitting it would leak the secret subtree's
// existence through the root tree's bytes. The root tree is
// a *path-scoped* deny too — the `/secret/**` rule matches
// `secret` at depth 1. The reader below confirms a structural
// leak: the listed reader can read the *secret subtree* and
// its tree body carries `b.txt` plus the raw secret oid.
let (st, _) = cid_parts(
cid_router(&state)
.oneshot(cid_anon(&root_tree_cid))
@@ -15452,7 +15678,11 @@ mod tests {
.unwrap(),
)
.await;
assert_eq!(st, StatusCode::OK, "root tree stays served (must-serve)");
assert_eq!(
st,
StatusCode::NOT_FOUND,
"root tree denied to anon: its entries name the withheld /secret subtree"
);
// /public subtree tree stays served to anon (allowed path).
let (st, _) = cid_parts(
@@ -15642,106 +15872,18 @@ mod tests {
assert!(body.contains("public bytes"), "owner gets the content");
}
/// Fail-closed walk-error arm: if `withheld_blob_oids` errors (here, a ref
/// pointing at a non-tree-ish blob, which `git ls-tree -r` cannot traverse —
/// the same induction as `visibility_pack::fails_closed_when_a_ref_cannot_be_traversed`),
/// the handler skips the whole repo rather than serving. Asserts no leak of the
/// withheld blob AND that even the *public* blob in that repo is withheld — the
/// latter distinguishes fail-closed-skip from normal per-blob withholding and
/// would serve 200 if the error arm wrongly proceeded. The skip carries no
/// VERDICT (F2), so the response is the retryable truncation 503, not a 404
/// claiming the object is absent — never-serve-unproven and never-404-unproven
/// hold together.
#[sqlx::test]
async fn ipfs_cid_walk_error_fails_closed(pool: PgPool) {
use crate::db::VisibilityMode;
use gitlawb_core::identity::Keypair;
let owner = Keypair::generate();
let owner_did = owner.did().to_string();
let slug = owner_did.replace([':', '/'], "_");
let short = owner_did.split(':').next_back().unwrap().to_string();
let state = test_state(pool).await;
let fx = seed_cid_repos(&slug, &short, &["withhold"]);
let bare = std::path::PathBuf::from("/tmp")
.join(&slug)
.join("withhold.git");
// Recorded pins so get_by_cid resolves each CID to its oid and reaches the
// walk; the 404s below are then the fail-closed skip, not a table miss.
let secret_cid = pin_cid_for(&bare, &fx.secret_oid, &state.db).await;
let public_cid = pin_cid_for(&bare, &fx.public_oid, &state.db).await;
// Force the withheld walk to fail closed: a ref pointing at a blob (not
// tree-ish) makes `git ls-tree -r` error, which `withheld_blob_oids`
// propagates as Err → the handler's `Ok(Err)` arm skips the repo.
std::fs::write(
bare.join("refs/heads/blobref"),
format!("{}\n", fx.secret_oid),
)
.unwrap();
state
.db
.create_repo(&seed_repo(&owner_did, "withhold"))
.await
.expect("seed repo");
let rec = state
.db
.get_repo(&owner_did, "withhold")
.await
.unwrap()
.unwrap();
state
.db
.set_visibility_rule(&rec.id, "/secret/**", VisibilityMode::B, &[], &owner_did)
.await
.expect("deny rule");
// Withheld secret CID under a walk error → the repo is skipped without a
// verdict, so the scan is truncated (503), and nothing leaks.
let (st, body) = cid_parts(
cid_router(&state)
.oneshot(cid_anon(&secret_cid))
.await
.unwrap(),
)
.await;
assert_eq!(
st,
StatusCode::SERVICE_UNAVAILABLE,
"walk error must not serve the withheld blob — the unproven skip sheds 503"
);
assert!(
!body.contains("TOP SECRET"),
"walk-error 503 must not leak the secret"
);
// The PUBLIC blob in the same repo is also not served: the walk error fails
// closed by skipping the whole repo. Without the fail-closed arm this would
// serve 200, so this assertion is the load-bearing discriminator.
let (st, _) = cid_parts(
cid_router(&state)
.oneshot(cid_anon(&public_cid))
.await
.unwrap(),
)
.await;
assert_eq!(
st,
StatusCode::SERVICE_UNAVAILABLE,
"walk error fails closed: repo skipped without a verdict, even the public \
blob is not served and the scan sheds 503"
);
}
/// #173 review (F2): the commit/tag reachability walk must FAIL CLOSED on a git
/// error, exactly like the blob/tree walk. A ref pointing at a nonexistent object
/// makes `rev-list --all` fail, so `reachable_commit_tag_oids` returns Err, which
/// the handler's shared `Ok(Err) => continue` arm turns into a repo skip. The
/// load-bearing discriminator is that the PUBLIC commit is ALSO 404: if the arm
/// fail-OPENed (served on error) it would 200. Drives the commit/tag branch of
/// the shared fail-closed arm specifically (the sibling test covers blob/tree).
/// #218 review P1: the previous `ipfs_cid_walk_error_fails_closed`
/// test relied on a ref pointing at a blob to force
/// `withheld_blob_oids` to error via the pre-fix
/// `assert_all_refs_are_commits` guard. With the guard removed
/// (a ref pointing at a non-commit object is a valid Git shape
/// that `git rev-list --all` silently skips), the trigger is
/// gone. The fail-closed arm is still covered by other tests
/// (e.g. `ipfs_cid_commit_tag_walk_error_fails_closed` below uses
/// a different trigger: a nonexistent ref object makes
/// `rev-list --all` fail, exercising the same shared
/// `Ok(Err) => continue` arm). The shared arm is exercised
/// here; the redundant blob-path trigger is removed.
#[sqlx::test]
async fn ipfs_cid_commit_tag_walk_error_fails_closed(pool: PgPool) {
use crate::db::VisibilityMode;
@@ -16331,10 +16473,29 @@ mod tests {
.await
.expect("path rule");
// Reachable trees at ALLOWED paths must still serve despite the tag-of-tree.
for (cid, want_oid, label) in [
(&root_tree_cid, &fx.root_tree_oid, "root tree"),
(&public_tree_cid, &fx.public_tree_oid, "public subtree"),
// Reachable trees at ALLOWED paths must still serve despite the
// tag-of-tree. Under #218 review P1b's recursive structural
// gate, anon sees ONLY trees whose structural safety holds at
// the caller's path: the public subtree (`/public`) is safe
// (its only entry is a blob at an allowed path). The root
// tree is NOT safe for anon: its entries include `secret/`
// pointing at a denied subtree, so the structural check
// denies the root tree. The test asserts the public subtree
// serves and the root tree is denied (fail-closed on
// subtree metadata leakage).
for (cid, want_oid, label, want_status) in [
(
&root_tree_cid,
&fx.root_tree_oid,
"root tree",
StatusCode::NOT_FOUND,
),
(
&public_tree_cid,
&fx.public_tree_oid,
"public subtree",
StatusCode::OK,
),
] {
let resp = cid_router(&state).oneshot(cid_anon(cid)).await.unwrap();
let served = resp
@@ -16344,15 +16505,17 @@ mod tests {
.map(str::to_string);
let (st, _) = cid_parts(resp).await;
assert_eq!(
st,
StatusCode::OK,
"{label} CID must serve despite a pushable tag-of-tree in the repo"
);
assert_eq!(
served.as_deref(),
Some(want_oid.as_str()),
"{label}: the served object is the reachable tree"
st, want_status,
"{label} CID status under recursive structural gate: root tree is denied \
because its serialized bytes name the withheld /secret subtree, /public is served"
);
if want_status == StatusCode::OK {
assert_eq!(
served.as_deref(),
Some(want_oid.as_str()),
"{label}: the served object is the reachable tree"
);
}
}
// Fail-closed preserved: the DENIED subtree's CID is still withheld — the
+968
View File
@@ -0,0 +1,968 @@
//! Scheduled external-forge fetches for canonical INBOUND mirrors.
//!
//! This worker is deliberately separate from `sync`, which mirrors repositories
//! between Gitlawb peers. External upstream URLs are owner-controlled data that
//! eventually reach `git fetch`, so each cycle resolves DNS itself, validates
//! every address, and pins the approved answers into Git/libcurl with
//! `http.curloptResolve`. Redirects, proxies, credential helpers, submodule
//! recursion, and non-HTTPS protocols are disabled at the command boundary.
use anyhow::{Context, Result};
use std::collections::BTreeSet;
use std::net::IpAddr;
use std::path::Path;
use std::sync::Arc;
use std::time::{Duration, Instant};
use tracing::{info, warn};
use crate::config::Config;
use crate::db::{Db, InboundMirrorTarget, MirrorStatus, MirrorTransitionPhase};
use crate::git::repo_store::RepoStore;
use crate::state::{repo_identity_key, RepoWriteLeases};
const DNS_TIMEOUT: Duration = Duration::from_secs(10);
const MAX_PINNED_ADDRESSES: usize = 32;
const MIN_CURL_OPT_RESOLVE_GIT: (u64, u64) = (2, 37);
#[derive(Debug, Clone, Copy)]
struct EgressPolicy;
#[derive(Debug, Clone, PartialEq, Eq)]
struct PinnedUpstream {
url: String,
curlopt_resolve: Option<String>,
}
impl EgressPolicy {
async fn resolve(&self, url: reqwest::Url) -> Result<PinnedUpstream> {
let raw_host = url.host_str().context("mirror upstream URL has no host")?;
let connect_host = raw_host
.trim_start_matches('[')
.trim_end_matches(']')
.to_ascii_lowercase();
let policy_host = normalize_policy_host(&connect_host);
reject_always_local_hostname(&policy_host)?;
let port = url
.port_or_known_default()
.context("mirror upstream URL has no known port")?;
if let Ok(ip) = connect_host.parse::<IpAddr>() {
self.validate_addresses(&policy_host, &[ip])?;
return Ok(PinnedUpstream {
url: url.to_string(),
// An IP literal performs no DNS lookup, so there is no rebinding
// window to pin. TLS still verifies the literal from the URL.
curlopt_resolve: None,
});
}
let resolved = tokio::time::timeout(
DNS_TIMEOUT,
tokio::net::lookup_host((connect_host.as_str(), port)),
)
.await
.context("mirror upstream DNS lookup timed out")?
.context("resolving mirror upstream host")?;
let addresses: BTreeSet<IpAddr> = resolved.map(|addr| addr.ip()).collect();
if addresses.is_empty() {
anyhow::bail!("mirror upstream host resolved to no addresses");
}
let addresses: Vec<IpAddr> = addresses.into_iter().collect();
self.validate_addresses(&policy_host, &addresses)?;
let pinned = addresses
.iter()
.map(|ip| match ip {
IpAddr::V4(v4) => v4.to_string(),
IpAddr::V6(v6) => format!("[{v6}]"),
})
.collect::<Vec<_>>()
.join(",");
Ok(PinnedUpstream {
url: url.to_string(),
curlopt_resolve: Some(format!("+{connect_host}:{port}:{pinned}")),
})
}
fn validate_addresses(&self, host: &str, addresses: &[IpAddr]) -> Result<()> {
if addresses.is_empty() || addresses.len() > MAX_PINNED_ADDRESSES {
anyhow::bail!(
"mirror upstream host must resolve to 1..={MAX_PINNED_ADDRESSES} addresses"
);
}
for &ip in addresses {
if !address_is_permitted(ip) {
anyhow::bail!("mirror upstream host {host:?} resolved to disallowed address {ip}");
}
}
Ok(())
}
}
fn normalize_policy_host(host: &str) -> String {
host.trim_end_matches('.').to_ascii_lowercase()
}
fn reject_always_local_hostname(host: &str) -> Result<()> {
if host.is_empty()
|| host == "localhost"
|| host.ends_with(".localhost")
|| host.ends_with(".local")
|| host.ends_with(".internal")
{
anyhow::bail!("mirror upstream host is local-only and cannot be fetched");
}
Ok(())
}
fn address_is_permitted(ip: IpAddr) -> bool {
crate::api::peers::mirror_upstream_ip_permitted(ip)
}
fn parse_git_version(output: &str) -> Option<(u64, u64)> {
let version = output.split_whitespace().find(|part| {
part.as_bytes()
.first()
.is_some_and(|byte| byte.is_ascii_digit())
})?;
let mut parts = version.split('.');
Some((parts.next()?.parse().ok()?, parts.next()?.parse().ok()?))
}
fn ensure_safe_git_runtime(git_bin: &str) -> Result<()> {
// Bounded runner like every other git spawn on this path: --version does
// no I/O of its own, but the child still takes a permit and the deadline.
let (status, stdout, stderr) = crate::git::visibility_pack::run_bounded_git_raw_isolated_https(
git_bin,
&["--version"],
&std::env::temp_dir(),
b"",
Instant::now() + Duration::from_secs(10),
)?;
if !status.success() {
anyhow::bail!(
"{git_bin} --version failed: {}",
String::from_utf8_lossy(&stderr)
.chars()
.take(256)
.collect::<String>()
);
}
let stdout = String::from_utf8_lossy(&stdout);
let version = parse_git_version(&stdout).context("parsing git version")?;
if version < MIN_CURL_OPT_RESOLVE_GIT {
anyhow::bail!(
"upstream mirroring requires Git 2.37+ for DNS pinning; found {}.{}",
version.0,
version.1
);
}
Ok(())
}
#[derive(Debug, Clone, PartialEq, Eq)]
struct FetchCommand {
config: Vec<(String, String)>,
args: Vec<String>,
}
fn fetch_command(upstream: &PinnedUpstream) -> FetchCommand {
let http_scope = |name: &str| format!("http.{}.{name}", upstream.url);
let credential_scope = |name: &str| format!("credential.{}.{name}", upstream.url);
let mut config = vec![
("http.proxy".to_string(), "".to_string()),
("http.followRedirects".to_string(), "false".to_string()),
// Reset any inherited multi-valued resolver entries before adding the
// address set validated for this exact fetch.
("http.curloptResolve".to_string(), "".to_string()),
// Never forward operator or repository-scoped HTTP credentials to an
// owner-selected forge. Empty values reset Git's multi-valued headers
// and disable cookie state for this invocation.
("http.extraHeader".to_string(), "".to_string()),
("http.cookieFile".to_string(), "".to_string()),
("http.saveCookies".to_string(), "false".to_string()),
("http.sslVerify".to_string(), "true".to_string()),
// URL-specific configuration outranks generic `http.*` values even
// when the generic value is command-scoped. Repeat every security
// control at the exact validated URL so a repository-local scoped
// setting cannot re-enable redirects/proxies/headers/cookies or disable
// TLS. Key/value environment entries avoid `git -c` splitting a valid
// URL path that contains `=`.
(http_scope("proxy"), "".to_string()),
(http_scope("followRedirects"), "false".to_string()),
(http_scope("curloptResolve"), "".to_string()),
(http_scope("extraHeader"), "".to_string()),
(http_scope("cookieFile"), "".to_string()),
(http_scope("saveCookies"), "false".to_string()),
(http_scope("sslVerify"), "true".to_string()),
];
if let Some(resolve) = &upstream.curlopt_resolve {
config.extend([
("http.curloptResolve".to_string(), resolve.clone()),
(http_scope("curloptResolve"), resolve.clone()),
]);
}
config.extend([
("credential.helper".to_string(), "".to_string()),
(credential_scope("helper"), "".to_string()),
("credential.interactive".to_string(), "false".to_string()),
("core.askPass".to_string(), "false".to_string()),
("fetch.recurseSubmodules".to_string(), "false".to_string()),
// Do not let a smart server or repository-local config introduce a
// second, unvalidated pack/bundle URL outside the pinned upstream.
("fetch.uriprotocols".to_string(), "".to_string()),
("fetch.bundleURI".to_string(), "".to_string()),
("protocol.allow".to_string(), "never".to_string()),
("protocol.https.allow".to_string(), "always".to_string()),
]);
let args = vec![
"fetch".to_string(),
"--atomic".to_string(),
"--force".to_string(),
"--prune".to_string(),
"--prune-tags".to_string(),
"--no-auto-maintenance".to_string(),
"--no-recurse-submodules".to_string(),
"--no-write-fetch-head".to_string(),
upstream.url.clone(),
"+refs/heads/*:refs/heads/*".to_string(),
"+refs/tags/*:refs/tags/*".to_string(),
];
FetchCommand { config, args }
}
fn reject_local_url_rewrite(
git_bin: &str,
repo_path: &Path,
upstream_url: &str,
deadline: Instant,
) -> Result<()> {
let args = ["ls-remote", "--get-url", upstream_url];
let (status, stdout, stderr) = crate::git::visibility_pack::run_bounded_git_raw_isolated_https(
git_bin, &args, repo_path, b"", deadline,
)?;
if !status.success() {
let stderr = String::from_utf8_lossy(&stderr);
anyhow::bail!(
"checking mirror upstream URL rewrite failed: {}",
stderr.chars().take(4_096).collect::<String>()
);
}
let expanded = std::str::from_utf8(&stdout)
.context("Git returned a non-UTF-8 expanded mirror upstream URL")?
.trim_end_matches(['\r', '\n']);
if expanded != upstream_url {
anyhow::bail!("repository-local Git config attempted to rewrite the mirror upstream URL");
}
Ok(())
}
fn reject_sensitive_local_config(git_bin: &str, repo_path: &Path, deadline: Instant) -> Result<()> {
let args = [
"config",
"--local",
"--includes",
"--get-regexp",
r"^(http|credential|url)\.",
];
let (status, stdout, stderr) = crate::git::visibility_pack::run_bounded_git_raw_isolated_https(
git_bin, &args, repo_path, b"", deadline,
)?;
match status.code() {
Some(1) if stdout.is_empty() => Ok(()),
Some(0) => anyhow::bail!(
"repository-local network or credential Git config is not allowed for upstream mirrors"
),
_ => {
let stderr = String::from_utf8_lossy(&stderr);
anyhow::bail!(
"checking repository-local Git config failed: {}",
stderr.chars().take(4_096).collect::<String>()
)
}
}
}
/// Budget for each preflight probe inside [`run_fetch`] (the URL-rewrite
/// check and the sensitive-local-config scan). Both run local git children
/// against the on-disk repo, so a short bound is generous; keeping them on
/// their own deadlines means `upstream_mirror_fetch_timeout_secs` covers
/// only the external pack transfer it documents.
const PREFLIGHT_PROBE_TIMEOUT: Duration = Duration::from_secs(30);
fn run_fetch(
git_bin: &str,
repo_path: &Path,
upstream: &PinnedUpstream,
timeout: Duration,
) -> Result<()> {
let command = fetch_command(upstream);
let args: Vec<&str> = command.args.iter().map(String::as_str).collect();
let config: Vec<(&str, &str)> = command
.config
.iter()
.map(|(key, value)| (key.as_str(), value.as_str()))
.collect();
let rewrite_deadline = Instant::now()
.checked_add(PREFLIGHT_PROBE_TIMEOUT)
.context("mirror URL-rewrite probe timeout is not representable")?;
reject_local_url_rewrite(git_bin, repo_path, &upstream.url, rewrite_deadline)?;
let config_deadline = Instant::now()
.checked_add(PREFLIGHT_PROBE_TIMEOUT)
.context("mirror local-config probe timeout is not representable")?;
reject_sensitive_local_config(git_bin, repo_path, config_deadline)?;
let deadline = Instant::now()
.checked_add(timeout)
.context("upstream mirror fetch timeout is not representable")?;
let (status, _stdout, stderr) =
crate::git::visibility_pack::run_bounded_git_raw_isolated_https_with_config(
git_bin, &args, repo_path, b"", deadline, &config,
)?;
if !status.success() {
let stderr = String::from_utf8_lossy(&stderr);
let stderr = stderr.chars().take(4_096).collect::<String>();
anyhow::bail!("git upstream fetch failed: {stderr}");
}
Ok(())
}
#[derive(Clone)]
struct Worker {
db: Arc<Db>,
config: Arc<Config>,
repo_store: RepoStore,
repo_write_leases: RepoWriteLeases,
git_write_semaphore: Arc<tokio::sync::Semaphore>,
policy: EgressPolicy,
git_bin: String,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum FetchOutcome {
Updated,
SkippedStale,
}
impl Worker {
async fn fetch_target(&self, target: InboundMirrorTarget) -> Result<FetchOutcome> {
let url = crate::db::validate_mirror_upstream_url(&target.upstream_url)?;
let upstream = self.policy.resolve(url).await?;
let repo_identity = repo_identity_key(&target.owner_did, &target.name);
let steal_after = Duration::from_secs(
self.config
.upstream_mirror_fetch_timeout_secs
.saturating_mul(2)
.saturating_add(60),
);
let _lease = self
.repo_write_leases
.acquire(&repo_identity, steal_after)
.await
.context("upstream mirror write-lease waiter cap reached")?;
let _write_permit = Arc::clone(&self.git_write_semaphore)
.acquire_owned()
.await
.context("upstream mirror write semaphore closed")?;
let acquire_timeout = Duration::from_secs(self.config.git_acquire_timeout_secs);
let guard = tokio::time::timeout(
acquire_timeout,
self.repo_store
.acquire_write(&target.owner_did, &target.name),
)
.await
.context("upstream mirror write-lock acquisition timed out")??;
let repo_path = guard.path().to_path_buf();
// A row can leave INBOUND state after selection. Recheck only after
// both the process-local lease and the cluster-wide advisory write lock
// are held. Every future transition path must take these same locks
// before changing authority, so no other node can commit a transition
// between this check and the fetch.
let current = match self.db.get_repo_mirror_state(&target.repo_id).await {
Ok(Some(current)) => current,
Ok(None) => {
let _ = guard.release(false).await;
anyhow::bail!("inbound mirror state disappeared before fetch");
}
Err(error) => {
let _ = guard.release(false).await;
return Err(error);
}
};
// Both stable shapes the CHECK admits: never transitioned, or a
// completed transition whose metadata is retained as audit. Only an
// in-flight transition makes the target stale.
let stable = (current.transition_id.is_none() && current.transition_phase.is_none())
|| (current.transition_id.is_some()
&& current.transition_phase == Some(MirrorTransitionPhase::Completed));
if current.status != MirrorStatus::Inbound
|| !stable
|| current.upstream_url != target.upstream_url
{
let _ = guard.release(false).await;
info!(repo_id = %target.repo_id, "upstream mirror state changed before fetch; skipping stale target");
return Ok(FetchOutcome::SkippedStale);
}
let git_bin = self.git_bin.clone();
let fetch_timeout = Duration::from_secs(self.config.upstream_mirror_fetch_timeout_secs);
let fetch = tokio::task::spawn_blocking(move || {
run_fetch(&git_bin, &repo_path, &upstream, fetch_timeout)
})
.await
.context("upstream mirror fetch task panicked")?;
let success = fetch.is_ok();
let outcome = guard.release(success).await;
// A refused publish (fenced or unknowable upload) must not report
// Updated; surface it as the failure it is before mapping the fetch.
outcome.into_result()?;
fetch.map(|()| FetchOutcome::Updated)
}
async fn scan_once(&self) {
let mut cursor: Option<String> = None;
loop {
let targets = match self
.db
.list_inbound_mirror_targets(
cursor.as_deref(),
self.config.upstream_mirror_page_size as i64,
)
.await
{
Ok(targets) => targets,
Err(error) => {
warn!(err = %error, "failed to list inbound mirror targets");
return;
}
};
if targets.is_empty() {
return;
}
let page_len = targets.len();
cursor = targets.last().map(|target| target.repo_id.clone());
for target in targets {
let repo_id = target.repo_id.clone();
let upstream_url = target.upstream_url.clone();
match self.fetch_target(target).await {
Ok(FetchOutcome::Updated) => {
info!(repo_id, upstream = %upstream_url, "upstream mirror fetch completed")
}
Ok(FetchOutcome::SkippedStale) => {}
Err(error) => {
warn!(repo_id, upstream = %upstream_url, err = %error, "upstream mirror fetch failed")
}
}
}
if page_len < self.config.upstream_mirror_page_size {
return;
}
}
}
}
/// Validate the opt-in runtime and spawn the scheduled worker. The worker runs
/// one scan immediately, sleeps only after a complete scan, and exits between
/// scans when graceful shutdown is signalled. An in-flight Git child is bounded
/// and reaped before the worker observes shutdown.
pub fn start(
db: Arc<Db>,
config: Arc<Config>,
repo_store: RepoStore,
repo_write_leases: RepoWriteLeases,
git_write_semaphore: Arc<tokio::sync::Semaphore>,
mut shutdown_rx: tokio::sync::watch::Receiver<bool>,
) -> Result<()> {
ensure_safe_git_runtime("git")?;
let policy = EgressPolicy;
let worker = Worker {
db,
config: Arc::clone(&config),
repo_store,
repo_write_leases,
git_write_semaphore,
policy,
git_bin: "git".to_string(),
};
tokio::spawn(async move {
info!(
interval_secs = config.upstream_mirror_interval_secs,
"upstream mirror worker started"
);
loop {
worker.scan_once().await;
tokio::select! {
_ = tokio::time::sleep(Duration::from_secs(config.upstream_mirror_interval_secs)) => {}
changed = shutdown_rx.changed() => {
if changed.is_err() || *shutdown_rx.borrow() {
info!("upstream mirror worker stopped");
return;
}
}
}
}
});
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use clap::Parser;
use std::path::PathBuf;
async fn test_worker(
pool: sqlx::PgPool,
git_bin: PathBuf,
page_size: usize,
) -> (Worker, Arc<Db>, tempfile::TempDir) {
let db = Arc::new(Db::for_testing(pool.clone()));
db.run_migrations().await.unwrap();
let repos = tempfile::tempdir().unwrap();
let mut config = Config::parse_from([
"gitlawb-node".to_string(),
"--upstream-mirror-enabled".to_string(),
"--enforce-owner-push".to_string(),
"--upstream-mirror-fetch-timeout-secs".to_string(),
"5".to_string(),
"--upstream-mirror-page-size".to_string(),
page_size.to_string(),
]);
config.validate().unwrap();
let config = Arc::new(config);
let worker = Worker {
db: Arc::clone(&db),
config,
repo_store: RepoStore::for_testing(repos.path().to_path_buf(), pool),
repo_write_leases: RepoWriteLeases::new(8),
git_write_semaphore: Arc::new(tokio::sync::Semaphore::new(4)),
policy: EgressPolicy,
git_bin: git_bin.to_string_lossy().into_owned(),
};
(worker, db, repos)
}
async fn add_inbound_mirror(
worker: &Worker,
db: &Db,
id: &str,
name: &str,
upstream_url: &str,
) -> InboundMirrorTarget {
let now = chrono::Utc::now();
let record = crate::db::RepoRecord {
id: id.to_string(),
name: name.to_string(),
owner_did: "did:key:z6MkMirrorWorker".to_string(),
description: None,
is_public: true,
default_branch: "main".to_string(),
created_at: now,
updated_at: now,
disk_path: format!("/test/{id}.git"),
forked_from: None,
machine_id: None,
};
db.create_repo(&record).await.unwrap();
worker
.repo_store
.init(&record.owner_did, &record.name)
.await
.unwrap();
let state = db
.configure_inbound_mirror(&record.id, upstream_url)
.await
.unwrap();
InboundMirrorTarget {
repo_id: record.id,
owner_did: record.owner_did,
name: record.name,
upstream_url: state.upstream_url,
}
}
#[cfg(unix)]
fn fake_git(repos: &tempfile::TempDir) -> (PathBuf, PathBuf) {
use std::os::unix::fs::PermissionsExt;
let git = repos.path().join("fake-git");
let calls = repos.path().join("fetch-calls");
let script = format!(
r#"#!/bin/sh
if [ "$1" = "ls-remote" ]; then
printf '%s\n' "$3"
exit 0
fi
if [ "$1" = "config" ]; then
# Match real Git when no sensitive repository-local keys are configured.
exit 1
fi
case "$*" in
*fail.git*) printf '%s\n' fail >> '{}'; exit 7 ;;
*) printf '%s\n' ok >> '{}'; exit 0 ;;
esac
"#,
calls.display(),
calls.display()
);
std::fs::write(&git, script).unwrap();
let mut permissions = std::fs::metadata(&git).unwrap().permissions();
permissions.set_mode(0o700);
std::fs::set_permissions(&git, permissions).unwrap();
(git, calls)
}
#[test]
fn git_version_gate_requires_curlopt_resolve_support() {
assert_eq!(parse_git_version("git version 2.37.0"), Some((2, 37)));
assert_eq!(
parse_git_version("git version 2.39.5 (Apple Git-154)"),
Some((2, 39))
);
assert_eq!(parse_git_version("not git"), None);
assert!((2, 36) < MIN_CURL_OPT_RESOLVE_GIT);
}
#[test]
fn every_resolved_address_must_be_public() {
let policy = EgressPolicy;
let public_v4: IpAddr = "8.8.8.8".parse().unwrap();
let private_v4: IpAddr = "10.2.3.4".parse().unwrap();
let private_v6: IpAddr = "fd00::20".parse().unwrap();
assert!(policy
.validate_addresses("public.example", &[public_v4])
.is_ok());
assert!(policy
.validate_addresses("private.example", &[private_v4, private_v6])
.is_err());
assert!(policy
.validate_addresses("mixed.example", &[public_v4, private_v4])
.is_err());
let too_many = (1..=MAX_PINNED_ADDRESSES + 1)
.map(|last| IpAddr::V4(std::net::Ipv4Addr::new(8, 8, 0, last as u8)))
.collect::<Vec<_>>();
assert!(policy
.validate_addresses("oversized.example", &too_many)
.is_err());
for never in [
"127.0.0.1",
"169.254.169.254",
"192.0.2.1",
"198.18.0.1",
"198.51.100.1",
"203.0.113.10",
"::1",
"::ffff:198.18.0.1",
"2002:c612:1::",
"64:ff9b::c612:1",
"100::1",
"100:0:0:1::1",
"2001::1",
"2001:2::1",
"2001:db8::1",
"fec0::1",
"fe80::1",
"3fff::1",
"5f00::1",
] {
let ip = never.parse().unwrap();
assert!(!address_is_permitted(ip), "{never} must stay denied");
}
}
#[test]
fn local_only_hostnames_are_always_rejected() {
for invalid in [
"",
"localhost",
"forge.localhost",
"forge.local",
"ghe.internal",
] {
assert!(
reject_always_local_hostname(invalid).is_err(),
"{invalid:?}"
);
}
assert!(reject_always_local_hostname("github.example").is_ok());
}
#[test]
fn fetch_command_pins_dns_and_only_updates_branches_and_tags() {
let upstream = PinnedUpstream {
url: "https://github.example/org/repo.git".to_string(),
curlopt_resolve: Some("+github.example:443:8.8.8.8,[2606:4700:4700::1111]".to_string()),
};
let command = fetch_command(&upstream);
let has_config = |key: &str, value: &str| {
command
.config
.contains(&(key.to_string(), value.to_string()))
};
assert!(has_config("http.followRedirects", "false"));
assert!(has_config("http.proxy", ""));
assert!(has_config("http.extraHeader", ""));
assert!(has_config("http.cookieFile", ""));
assert!(has_config("http.saveCookies", "false"));
assert!(has_config("http.sslVerify", "true"));
assert!(has_config(
"http.curloptResolve",
"+github.example:443:8.8.8.8,[2606:4700:4700::1111]"
));
assert!(has_config(
"http.https://github.example/org/repo.git.followRedirects",
"false"
));
assert!(has_config(
"http.https://github.example/org/repo.git.proxy",
""
));
assert!(has_config(
"http.https://github.example/org/repo.git.extraHeader",
""
));
assert!(has_config(
"http.https://github.example/org/repo.git.curloptResolve",
"+github.example:443:8.8.8.8,[2606:4700:4700::1111]"
));
assert!(has_config(
"credential.https://github.example/org/repo.git.helper",
""
));
assert!(has_config("protocol.allow", "never"));
assert!(has_config("protocol.https.allow", "always"));
assert!(has_config("fetch.uriprotocols", ""));
assert!(has_config("fetch.bundleURI", ""));
assert!(!command
.config
.iter()
.any(|(key, _)| key.ends_with("sslCert") || key.ends_with("sslKey")));
assert!(command.args.contains(&"--atomic".to_string()));
assert!(command
.args
.contains(&"+refs/heads/*:refs/heads/*".to_string()));
assert!(command
.args
.contains(&"+refs/tags/*:refs/tags/*".to_string()));
assert!(!command.args.iter().any(|arg| arg == "+refs/*:refs/*"));
}
/// Spawn `git` inside a fixture repo with inherited repository-location
/// and config environment scrubbed, so a test run inside an outer Git
/// checkout or custom config can never leak into the fixture.
fn fixture_git(dir: &std::path::Path) -> std::process::Command {
let mut cmd = std::process::Command::new("git");
cmd.current_dir(dir);
for var in [
"GIT_DIR",
"GIT_WORK_TREE",
"GIT_CONFIG",
"GIT_CONFIG_GLOBAL",
"GIT_CONFIG_SYSTEM",
"GIT_CONFIG_NOSYSTEM",
"GIT_CONFIG_COUNT",
"GIT_EXEC_PATH",
"GIT_SSH",
"GIT_SSH_COMMAND",
"GIT_ASKPASS",
"SSH_ASKPASS",
] {
cmd.env_remove(var);
}
cmd
}
#[test]
fn repository_local_instead_of_rewrite_is_rejected() {
let repo = tempfile::tempdir().unwrap();
let init = fixture_git(repo.path())
.args(["init", "--bare"])
.output()
.unwrap();
assert!(init.status.success());
let config = fixture_git(repo.path())
.args([
"config",
"url.https://evil.example/.insteadOf",
"https://github.example/",
])
.output()
.unwrap();
assert!(config.status.success());
let error = reject_local_url_rewrite(
"git",
repo.path(),
"https://github.example/org/repo.git",
Instant::now() + Duration::from_secs(5),
)
.unwrap_err();
assert!(error.to_string().contains("attempted to rewrite"));
}
#[test]
fn repository_local_network_and_credential_config_is_rejected() {
let repo = tempfile::tempdir().unwrap();
let init = fixture_git(repo.path())
.args(["init", "--bare"])
.output()
.unwrap();
assert!(init.status.success());
reject_sensitive_local_config("git", repo.path(), Instant::now() + Duration::from_secs(5))
.unwrap();
let config = fixture_git(repo.path())
.args(["config", "http.sslCert", "/tmp/untrusted-client.pem"])
.output()
.unwrap();
assert!(config.status.success());
let error = reject_sensitive_local_config(
"git",
repo.path(),
Instant::now() + Duration::from_secs(5),
)
.unwrap_err();
assert!(error.to_string().contains("network or credential"));
}
#[cfg(unix)]
#[test]
fn hardened_real_git_reaches_https_transport_for_equals_path() {
let repo = tempfile::tempdir().unwrap();
let init = fixture_git(repo.path())
.args(["init", "--bare"])
.output()
.unwrap();
assert!(init.status.success());
// This is intentionally a loopback test seam around `run_fetch`, below
// the egress-policy layer. Reaching the listener proves the generated
// Git config neither dies on an empty client certificate nor splits the
// valid `=` path at the command-config boundary. The listener closes
// immediately, so the expected final result is a TLS transport error.
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let port = listener.local_addr().unwrap().port();
listener.set_nonblocking(true).unwrap();
let accepted = std::thread::spawn(move || {
let deadline = Instant::now() + Duration::from_secs(3);
while Instant::now() < deadline {
match listener.accept() {
Ok((_stream, _address)) => return true,
Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => {
std::thread::sleep(Duration::from_millis(10));
}
Err(error) => panic!("accepting real-Git transport probe: {error}"),
}
}
false
});
let upstream = PinnedUpstream {
url: format!("https://127.0.0.1:{port}/org/repo=mirror.git"),
curlopt_resolve: None,
};
let error = run_fetch("git", repo.path(), &upstream, Duration::from_secs(5)).unwrap_err();
assert!(
accepted.join().unwrap(),
"real Git must reach the HTTPS transport; got {error:#}"
);
let error = error.to_string();
assert!(!error.contains("credential missing host field"));
assert!(!error.contains("invalid key"));
}
#[tokio::test]
async fn literal_addresses_are_checked_without_dns() {
let policy = EgressPolicy;
let allowed = policy
.resolve(reqwest::Url::parse("https://8.8.8.8/org/repo.git").unwrap())
.await
.unwrap();
assert_eq!(allowed.curlopt_resolve, None);
for denied in [
"127.0.0.1",
"10.2.3.4",
"169.254.169.254",
"192.0.2.1",
"198.18.0.1",
"203.0.113.10",
"[::1]",
"[100::1]",
"[100:0:0:1::1]",
"[2001::1]",
"[2001:db8::1]",
"[fec0::1]",
"[fd00::20]",
"[3fff::1]",
"[5f00::1]",
] {
let url = reqwest::Url::parse(&format!("https://{denied}/org/repo.git")).unwrap();
assert!(
policy.resolve(url).await.is_err(),
"{denied} must stay denied"
);
}
}
#[sqlx::test]
async fn stale_target_is_rechecked_under_the_write_locks(pool: sqlx::PgPool) {
let missing_git = PathBuf::from("/definitely/missing/git");
let (worker, db, _repos) = test_worker(pool, missing_git, 1).await;
let target = add_inbound_mirror(
&worker,
&db,
"00000000-0000-0000-0000-000000000001",
"stale",
"https://8.8.8.8/org/stale.git",
)
.await;
sqlx::query(
"UPDATE repos
SET mirror_status = 'outbound'
WHERE id = $1",
)
.bind(&target.repo_id)
.execute(db.pool())
.await
.unwrap();
let outcome = worker.fetch_target(target).await.unwrap();
assert_eq!(outcome, FetchOutcome::SkippedStale);
assert!(worker.repo_write_leases.is_empty());
}
#[cfg(unix)]
#[sqlx::test]
async fn scan_pages_past_one_failure_and_fetches_the_next_repo(pool: sqlx::PgPool) {
let bootstrap = tempfile::tempdir().unwrap();
let (git, calls) = fake_git(&bootstrap);
let (worker, db, _repos) = test_worker(pool, git, 1).await;
add_inbound_mirror(
&worker,
&db,
"00000000-0000-0000-0000-000000000001",
"fails",
"https://8.8.8.8/org/fail.git",
)
.await;
add_inbound_mirror(
&worker,
&db,
"00000000-0000-0000-0000-000000000002",
"succeeds",
"https://8.8.8.8/org/ok.git",
)
.await;
worker.scan_once().await;
let calls = std::fs::read_to_string(calls).unwrap();
assert_eq!(calls.lines().collect::<Vec<_>>(), ["fail", "ok"]);
assert!(worker.repo_write_leases.is_empty());
}
}
+163 -12
View File
@@ -38,26 +38,151 @@ pub fn fire_event(
event: &str,
payload: serde_json::Value,
) {
let repo_id = repo_id.to_string();
let event = event.to_string();
tokio::spawn(async move {
fire_event_async(db, http_client, &repo_id, &event, payload).await;
});
fire_event_occurrence(db, http_client, repo_id, event, payload, None, None);
}
async fn fire_event_async(
/// Occurrence-keyed variant: at most one delivery per
/// `(request_id, ref_name, hook)` is spawned; concurrent executors
/// collapse via the `webhook_deliveries` ledger. Best-effort on crash
/// (ledger is claimed before spawn), which the PR scope allows.
///
/// For durable outbox usage, call `claim_webhook_delivery_before_spawn` first
/// to claim deliveries synchronously, then call
/// `fire_event_occurrence_with_claimed_hooks` with the claimed hook IDs to
/// spawn HTTP POST only for claimed hooks.
pub fn fire_event_occurrence(
db: Arc<Db>,
http_client: Arc<reqwest::Client>,
repo_id: &str,
event: &str,
payload: serde_json::Value,
request_id: Option<&str>,
ref_name: Option<&str>,
) {
let hooks = match db.list_webhooks_for_event(repo_id, event).await {
Ok(h) => h,
Err(e) => {
tracing::warn!(err = %e, "failed to list webhooks for event {event}");
return;
let repo_id = repo_id.to_string();
let event = event.to_string();
let request_id = request_id.map(|s| s.to_string());
let ref_name = ref_name.map(|s| s.to_string());
tokio::spawn(async move {
fire_event_async_occurrence(
db,
http_client,
&repo_id,
&event,
payload,
request_id.as_deref(),
ref_name.as_deref(),
None,
)
.await;
});
}
/// Variant for durable outbox: only spawn HTTP POST for previously claimed hooks.
#[allow(clippy::too_many_arguments)]
pub fn fire_event_occurrence_with_claimed_hooks(
db: Arc<Db>,
http_client: Arc<reqwest::Client>,
repo_id: &str,
event: &str,
payload: serde_json::Value,
request_id: Option<&str>,
ref_name: Option<&str>,
claimed_hooks: Vec<crate::db::Webhook>,
) {
let repo_id = repo_id.to_string();
let event = event.to_string();
let request_id = request_id.map(|s| s.to_string());
let ref_name = ref_name.map(|s| s.to_string());
tokio::spawn(async move {
fire_event_async_occurrence(
db,
http_client,
&repo_id,
&event,
payload,
request_id.as_deref(),
ref_name.as_deref(),
Some(claimed_hooks),
)
.await;
});
}
/// Claim webhook delivery before spawning HTTP POST.
/// Returns `Some(hook ids)` successfully claimed, or `None` when the hook
/// list itself could not be read (transient DB error, unknown membership).
/// A `None` caller must fall back to best-effort send rather than dropping
/// the delivery: children are still deleted downstream, so an empty claim
/// set on DB error would lose the webhook permanently.
/// This synchronous claim phase must be awaited before child deletion to prevent
/// permanent webhook loss on crash between deletion and claim.
pub async fn claim_webhook_delivery_before_spawn(
db: Arc<Db>,
repo_id: &str,
event: &str,
request_id: Option<&str>,
ref_name: Option<&str>,
) -> Option<Vec<crate::db::Webhook>> {
let mut claimed_hooks = Vec::new();
// Only claim when we have occurrence context (request_id + ref_name)
if let (Some(req), Some(r)) = (request_id, ref_name) {
let hooks = match db.list_webhooks_for_event(repo_id, event).await {
Ok(h) => h,
Err(e) => {
tracing::warn!(err = %e, "failed to list webhooks for event {event}");
return None;
}
};
for hook in hooks {
let delivery_id = crate::db::deterministic_id(&["webhook", req, r, &hook.id]);
match db
.claim_webhook_delivery(&delivery_id, req, repo_id, event)
.await
{
Ok(true) => claimed_hooks.push(hook),
Ok(false) => continue, // Already claimed
// Send fallback (legacy best-effort): a transient claim
// error must not drop the delivery while children are
// still deleted. The in-task claimer this replaced sent
// anyway on claim error; keep that here.
Err(e) => {
tracing::warn!(err = %e, hook_id = %hook.id, "webhook claim failed; sending anyway");
claimed_hooks.push(hook);
}
}
}
}
Some(claimed_hooks)
}
#[allow(clippy::too_many_arguments)]
async fn fire_event_async_occurrence(
db: Arc<Db>,
http_client: Arc<reqwest::Client>,
repo_id: &str,
event: &str,
payload: serde_json::Value,
request_id: Option<&str>,
ref_name: Option<&str>,
claimed_hooks: Option<Vec<crate::db::Webhook>>,
) {
// A caller that already claimed deliveries hands the claimed hook rows
// through; the send path must not re-list, because a list failure here
// would silently drop deliveries already claimed in the ledger.
let already_claimed = claimed_hooks.is_some();
let hooks = match claimed_hooks {
Some(h) => h,
None => match db.list_webhooks_for_event(repo_id, event).await {
Ok(h) => h,
Err(e) => {
tracing::warn!(err = %e, "failed to list webhooks for event {event}");
return;
}
},
};
if hooks.is_empty() {
@@ -76,8 +201,29 @@ async fn fire_event_async(
let client = Arc::clone(&http_client);
let event_name = event.to_string();
let bytes = payload_bytes.clone();
let delivery_id = uuid::Uuid::new_v4().to_string();
// Stable occurrence delivery key when request context exists;
// random UUID otherwise (legacy callers without occurrence).
let delivery_id = match (request_id, ref_name) {
(Some(req), Some(r)) => crate::db::deterministic_id(&["webhook", req, r, &hook.id]),
_ => uuid::Uuid::new_v4().to_string(),
};
// Claim before spawn so concurrent executors collapse to one
// delivery per occurrence. Failures to record fall back to
// sending (legacy best-effort) rather than dropping.
// Skip claim if already done by caller (durable_outbox path).
if !already_claimed && request_id.is_some() {
match db
.clone()
.claim_webhook_delivery(&delivery_id, request_id.unwrap_or(""), repo_id, event)
.await
{
Ok(true) => {}
Ok(false) => continue,
Err(_) => {}
}
}
let db_for_mark = db.clone();
tokio::spawn(async move {
// Policy skip, not a delivery failure: the stored URL predates the
// create-time gate. The "invalid_url" label is documented on
@@ -92,6 +238,7 @@ async fn fire_event_async(
return;
}
let delivery_for_mark = delivery_id.clone();
let signature = hook.secret.as_deref().map(|s| sign_payload(s, &bytes));
let mut req = client
@@ -113,6 +260,10 @@ async fn fire_event_async(
"http_error"
};
crate::metrics::record_webhook_delivery(result_label);
// Any HTTP response proves the delivery fired; mark
// sent so the ledger never claims an unsent delivery.
// Network errors stay pending for stale-reclaim.
let _ = db_for_mark.mark_webhook_sent(&delivery_for_mark).await;
tracing::info!(
url = %hook.url,
event = %event_name,
+577 -27
View File
@@ -416,8 +416,8 @@ fn f3_second_writer_leased_until_reap() {
NOT RepoWriteGuard (which drops at the disconnect instant, reopening F3).",
);
let receive = repos_production
.find("smart_http::receive_pack(")
.expect("F3 gate stale: git_receive_pack no longer calls smart_http::receive_pack");
.find("smart_http::receive_pack_raw")
.expect("F3 gate stale: git_receive_pack no longer calls smart_http::receive_pack_raw");
assert!(
lease_acquire < with_lease && with_lease < receive,
"F3 gate bypassed: the write lease must be acquired, then carried by the \
@@ -509,8 +509,8 @@ fn inv22_ipfs_walk_admission_reaches_every_blocking_site() {
/// too, so an ungated spawn would fire for a push git rejected, pinning and announcing
/// a half-applied repo. Above `release` the `?` on `receive_result` can no longer be
/// what gates it, so the success check is explicit and this gate binds it: the spawn
/// must sit inside `if push_succeeded`, and `release` must consume the same flag so
/// the two cannot drift apart.
/// must sit inside `if disposition.spawn_tail`, and `release` must consume the same
/// disposition (`disposition.release_ok`) so the two cannot drift apart.
///
/// This is an ordering check rather than a cancellation-race test on purpose: it is
/// the companion to `receive_pack_tail_survives_a_disconnect_during_release`, which
@@ -519,7 +519,9 @@ fn inv22_ipfs_walk_admission_reaches_every_blocking_site() {
///
/// MUTATION (RED): move the `tokio::spawn(post_receive_replication_tail` call below
/// `guard.release(` and the ordering assertion fails; take it out of the
/// `if push_succeeded` block and the failed-push assertion fails.
/// `if disposition.spawn_tail` block and the failed-push assertion fails; gate
/// `spawn_tail`/`release_ok` on the outcome commit and
/// `post_git_disposition_replication_carve_out` fails with it.
#[test]
fn inv22_replication_tail_spawns_at_the_durability_boundary() {
let repos = src("api/repos.rs");
@@ -534,41 +536,589 @@ fn inv22_replication_tail_spawns_at_the_durability_boundary() {
.expect("split always yields a first chunk");
let success_flag = production
.find("let push_succeeded = receive_result.is_ok();")
.expect(
"U5 gate missing: the tail's success gate must be bound from receive_result.is_ok()",
.find("post_git_disposition(")
.expect("U5 gate missing: the tail's disposition must be bound before the gate");
// One tail per push, on the proven-landed ref set. Either spawn
// position (carve-out early, or durability-bound at release) must sit
// inside a `disposition.spawn_tail` gate and feed `landed_refs` —
// never the unfiltered `ref_updates`.
let spawn_count = production
.matches("tokio::spawn(post_receive_replication_tail(")
.count();
assert!(
spawn_count >= 1,
"U5 gate missing: the replication tail must be spawned by git_receive_pack"
);
for (i, (pos, _)) in production
.match_indices("tokio::spawn(post_receive_replication_tail(")
.enumerate()
{
let gate = production[..pos].rfind("if disposition.spawn_tail").expect(
"U5 gate missing: every tail spawn must be gated on `if disposition.spawn_tail`",
);
let gate_open = production
.find("if push_succeeded {")
.expect("U5 gate missing: the tail spawn must be gated on the push having succeeded");
// The spawn must sit inside the gate's block: brace depth at the
// spawn position must still be positive measured from the `if`.
// An ungated spawn after the block closes fails this, as does a
// stray `disposition.spawn_tail` mention in a comment.
let mut depth = 0i32;
for ch in production[gate..pos].chars() {
match ch {
'{' => depth += 1,
'}' => depth -= 1,
_ => {}
}
}
assert!(
depth >= 1,
"U5 gate bypassed: tail #{i} sits outside the `if disposition.spawn_tail` block"
);
assert!(
!production[gate..pos].contains("post_receive_replication_tail"),
"U5 gate bypassed: a second tail spawn sits between the gate and its spawn"
);
let call_end = production[pos..].find(");").unwrap() + pos;
let call = &production[pos..call_end];
assert!(
call.contains("landed_refs"),
"U5 gate bypassed: tail #{i} must carry the proven-landed ref set, not unfiltered ref_updates"
);
}
let spawn = production
.find("tokio::spawn(post_receive_replication_tail(")
.expect("U5 gate missing: the replication tail must be spawned by git_receive_pack");
let release = production
.find(".release(push_succeeded)")
.expect("U5 gate stale: release must consume the same success flag as the tail gate");
.find(".release(disposition.release_ok)")
.expect("U5 gate stale: release must consume the same disposition as the tail gate");
let touch = production
.find("state.db.touch_repo(")
.expect("U5 gate stale: git_receive_pack no longer calls touch_repo");
let webhook = production
.find("webhooks::fire_event(")
.expect("U5 gate stale: git_receive_pack no longer fires push webhooks");
// #26 Split PR 1 step 3 — the webhook fan-out moved into
// `durable_outbox::apply_request_effects`, which the live
// handler calls inline (the recovery drain calls the same
// function on the next startup). The gate now pins that the
// handler is wired to the executor, not to a per-ref inline
// webhook call.
let effects_executor = production
.find("apply_request_effects(&state, &request_id)")
.expect("U5 gate stale: git_receive_pack no longer calls apply_request_effects inline");
assert!(
success_flag < gate_open && gate_open < spawn,
"U5 gate bypassed: the tail must be spawned inside `if push_succeeded`, or a \
rejected push spawns a tail that pins and announces a half-applied repo"
);
// Still inside that block: no `}` may close it between the gate and the spawn.
assert!(
!production[gate_open + "if push_succeeded {".len()..spawn].contains('}'),
"U5 gate bypassed: the tail spawn left the `if push_succeeded` block, so a \
rejected push now spawns a tail"
success_flag < spawn,
"U5 gate bypassed: the disposition must be bound before any tail spawn"
);
assert!(
spawn < release && spawn < touch && spawn < webhook,
spawn < release && spawn < touch && spawn < effects_executor,
"U5 gate bypassed: the tail must be spawned BEFORE guard.release, touch_repo \
and the webhook fan-out, so a disconnect in any of those windows cannot drop \
and the effect executor, so a disconnect in any of those windows cannot drop \
this push's pins, recovery copy, and announcements"
);
}
/// #26 Split PR 1 step 3 — drain + handler share a single effect
/// executor. The live handler (api/repos.rs) and the recovery
/// drain (`durable_outbox::drain_receive_pack_requests_with`) both
/// call `apply_request_effects`, so the per-ref effects fan-out
/// lives in exactly one place. The v29 per-ref walk
/// (`derive_one`, `drain_pending_ref_transitions_all`,
/// `lookup_accepted_ordinal`) is dead code: any caller reintroduced
/// would be the per-ref walk the step-3 PR removed. This gate
/// fails if a call site slips back in or the new seam is bypassed.
#[test]
fn inv26_step3_live_and_drain_share_apply_request_effects() {
let repos = src("api/repos.rs");
let outbox = src("durable_outbox.rs");
// The live handler calls `apply_request_effects`. Split the
// file at the test attribute so test code can't satisfy the
// gate by itself.
let production_repos = repos
.split("\nmod tests {")
.next()
.expect("split always yields a first chunk");
assert!(
production_repos.contains("apply_request_effects(&state, &request_id)"),
"live handler must call `apply_request_effects(&state, &request_id)`; \
reverting to a per-ref inline fan-out splits live and recovery"
);
// The durable intent insert closes the pre-outbox crash window this
// PR exists to fix. Wrapping it in `if false` must turn this gate
// red; drain-only tests cannot prove the handler creates the rows.
assert!(
production_repos.contains("insert_receive_pack_request_with_children("),
"live handler must persist durable intent via \
`insert_receive_pack_request_with_children`; disabling it must fail this gate"
);
// The drain's per-request seam calls the same executor. Test
// code lives below `mod drain_tests`, so split there too.
let production_outbox = outbox
.split("\nmod drain_tests {")
.next()
.expect("split always yields a first chunk");
assert!(
production_outbox.contains("drain_receive_pack_requests_with"),
"drain seam `drain_receive_pack_requests_with` must exist; \
removing it forces a per-ref walk back into the drain"
);
assert!(
production_outbox.contains("apply_request_effects"),
"durable_outbox production code must define `apply_request_effects`; \
removing it splits the executor between live and recovery"
);
// The drain's per-request walker is wired to the executor.
// The closure body of `drain_receive_pack_requests_with` is the
// only call site — if a future change calls `derive_one`
// instead, this assertion fires.
let drain_seam_open = production_outbox
.find("pub async fn drain_receive_pack_requests_with<F, Fut>")
.expect("drain seam must be defined in the production half");
let drain_seam_close = production_outbox[drain_seam_open..]
.find("\n}\n")
.expect("drain seam body must close");
let drain_seam_body = &production_outbox[drain_seam_open..drain_seam_open + drain_seam_close];
assert!(
drain_seam_body.contains("apply_request_effects"),
"drain_receive_pack_requests_with must call apply_request_effects; \
wiring it to a per-ref helper reintroduces the v29 walk"
);
// The deleted per-ref drain must have zero call sites in the
// production half. A regression that re-adds a caller would
// bring back the per-ref fan-out.
assert!(
!production_outbox.contains("drain_pending_ref_transitions_all("),
"deleted `drain_pending_ref_transitions_all` must have zero production call sites; \
a per-ref walk is reintroduced"
);
assert!(
!production_outbox.contains("derive_one("),
"deleted `derive_one` must have zero production call sites; \
the per-ref fan-out is reintroduced"
);
assert!(
!production_outbox.contains("lookup_accepted_ordinal("),
"deleted `lookup_accepted_ordinal` must have zero production call sites; \
the per-ref ordinal lookup is reintroduced"
);
assert!(
!production_repos.contains("derive_one("),
"deleted `derive_one` must have zero live-handler call sites"
);
assert!(
!production_repos.contains("drain_pending_ref_transitions"),
"deleted per-ref drain functions must have zero live-handler call sites"
);
}
/// #26 Split PR 1 step 4 — the periodic queue-lifecycle purge is
/// wired in `main.rs` and the contract is enforced by the `idx_receive_pack_requests_completed_at`
/// partial index from v41.
///
/// Assertions:
/// 1. `main.rs` calls `purge_request_queue` once at boot (well, on
/// the spawn-task interval) and the spawn function exists.
/// 2. The DB helpers `purge_completed_receive_pack_requests` and
/// `purge_completed_pending_ref_transitions` exist in `db/mod.rs`.
/// 3. The config knobs `queue_retention_days` and `queue_purge_batch` exist
/// on `Config`.
/// 4. `quarantined` is NOT in the purge WHERE clause (step 4 does not
/// introduce the state, but the invariant holds for the future).
#[test]
fn inv26_step4_queue_lifecycle_purge_is_wired() {
let main_src = src("main.rs");
assert!(
main_src.contains("spawn_queue_lifecycle_sweep"),
"main.rs must spawn the periodic queue-lifecycle purge"
);
assert!(
main_src.contains("purge_request_queue"),
"main.rs must call purge_request_queue on the periodic sweep"
);
let db_src = src("db/mod.rs");
assert!(
db_src.contains("purge_completed_receive_pack_requests"),
"db/mod.rs must expose purge_completed_receive_pack_requests"
);
assert!(
db_src.contains("purge_completed_pending_ref_transitions"),
"db/mod.rs must expose purge_completed_pending_ref_transitions"
);
let config_src = src("config.rs");
assert!(
config_src.contains("queue_retention_days"),
"Config must expose queue_retention_days"
);
assert!(
config_src.contains("queue_purge_batch"),
"Config must expose queue_purge_batch"
);
}
/// #26 Split PR 1 step 5 — the marker gate and the retry-bound
/// quarantine are wired end-to-end. A missing or hash-mismatched
/// marker on disk quarantines the request; a Retry over the bound
/// does too. This gate pins every load-bearing seam, against the
/// production half of each file (the test modules name the same
/// identifiers in their own harnesses).
///
/// Assertions:
/// 1. `reconcile_prepared_page` calls `mark_request_quarantined`
/// and `mark_children_rejected_for_quarantined_parent` on
/// marker-gate failure.
/// 2. The drain's `EffectsOutcome::Retry` arm checks
/// `effects_max_attempts`.
/// 3. `git::store::read_ref` exists in `git/store.rs`.
/// 4. `db::mark_request_quarantined`,
/// `db::mark_children_rejected_for_quarantined_parent`,
/// `db::get_receive_pack_requests_by_ids`, and
/// `db::request_state::QUARANTINED` all exist.
/// 5. The handler writes the marker ref BEFORE calling
/// `smart_http::receive_pack_raw` (the durability window).
#[test]
fn inv26_step5_marker_quarantine_and_bound_are_wired() {
let outbox = src("durable_outbox.rs");
let store = src("git/store.rs");
let db = src("db/mod.rs");
let repos = src("api/repos.rs");
// Split at the TEST MODULE for the production-only assertions.
let production_outbox = outbox
.split("\nmod drain_tests {")
.next()
.expect("split always yields a first chunk");
let production_repos = repos
.split("\nmod tests {")
.next()
.expect("split always yields a first chunk");
// (1) The reconcile's marker gate quarantines via the DB helpers.
assert!(
production_outbox.contains("mark_request_quarantined"),
"reconcile_prepared_page must call mark_request_quarantined on marker-gate failure"
);
assert!(
production_outbox.contains("mark_children_rejected_for_quarantined_parent"),
"reconcile_prepared_page must call mark_children_rejected_for_quarantined_parent \
so quarantined parents cancel their children"
);
// The gate reads the marker ref and compares against the parent's
// `request_bytes_hash` via `git::store::read_ref` /
// `git::store::marker_value_for`. Reverting either reintroduces
// the DoS window the marker gate exists to close.
assert!(
production_outbox.contains("git::store::read_ref"),
"reconcile_prepared_page must read the marker ref via git::store::read_ref"
);
assert!(
production_outbox.contains("marker_value_for"),
"reconcile_prepared_page must compute the expected marker value via \
git::store::marker_value_for"
);
// (2) The drain's `EffectsOutcome::Retry` arm checks the bound.
assert!(
production_outbox.contains("effects_max_attempts"),
"drain_receive_pack_requests_with must consult effects_max_attempts on Retry"
);
// (3) `git::store::read_ref` is the read seam the gate depends on.
assert!(
store.contains("pub fn read_ref("),
"git::store::read_ref must exist; the marker gate reads through it"
);
assert!(
store.contains("pub fn marker_value_for("),
"git::store::marker_value_for must exist; the marker gate computes the expected \
value with it (and the live handler writes the value via the same helper)"
);
// (4) DB-side seams the gate depends on.
assert!(
db.contains("pub async fn mark_request_quarantined"),
"Db::mark_request_quarantined must exist"
);
assert!(
db.contains("pub async fn mark_children_rejected_for_quarantined_parent"),
"Db::mark_children_rejected_for_quarantined_parent must exist"
);
assert!(
db.contains("pub async fn get_receive_pack_requests_by_ids"),
"Db::get_receive_pack_requests_by_ids must exist (avoids N+1 in the marker gate)"
);
assert!(
db.contains("pub const QUARANTINED: &str = \"quarantined\""),
"request_state::QUARANTINED must be defined"
);
// (5) The handler writes the marker ref BEFORE the durability
// boundary (smart_http::receive_pack_raw). Severing the
// ordering re-opens the marker-gate DoS window for live pushes.
let marker_write = production_repos
.find("git::store::marker_value_for")
.expect("U5 gate stale: the live handler no longer computes the marker value");
let receive_raw = production_repos
.find("smart_http::receive_pack_raw")
.expect("U5 gate stale: git_receive_pack no longer calls smart_http::receive_pack_raw");
assert!(
marker_write < receive_raw,
"U5 gate bypassed: the marker ref must be written BEFORE receive_pack_raw so the \
reconcile's gate has evidence of the live push"
);
}
/// Issue #26 split 1 — namespace gate and pre-git refusal lifecycle.
///
/// The push gate (repos.rs) must deny:
/// 1. Any ref-update line that fails UTF-8 decode (the raw body is passed verbatim
/// to git receive-pack, and git accepts refnames with bytes >= 0x80; the gate
/// never sees these refs, no durable child row exists, and under requests/issues/
/// the lossy for-each-ref decode still matches the narrowed exemption).
/// 2. The bare `refs/gitlawb` name (the `starts_with("refs/gitlawb/")` check misses it).
///
/// Every explicit pre-git refusal must call `refuse_receive_pack_before_git` to
/// terminalize the intent aggregate in one transaction, or a dropped future between
/// the intent insert and `receive_pack` strands a `received` + `prepared` aggregate
/// that is never purged, never promotable, and poisons its tuple.
///
/// The visibility exemption (visibility_pack.rs) was narrowed to `requests/` and
/// `issues/` only. The push gate is the premise that narrowing relies on (the comment
/// at visibility_pack.rs:340), so tests must prove:
/// - A `refs/gitlawb/<other>` ref pointing at a blob is denied on push.
/// - A stray `refs/gitlawb/<other>` blob ref planted in the pack path fails closed.
///
/// Source gates below pin the wiring; behavioral proofs live next to the code:
/// `parse_ref_updates_refuses_non_utf8_command_line` and
/// `internal_namespace_gate_covers_bare_name_and_subtree` (api/repos.rs),
/// `planted_non_exempt_gitlawb_blob_ref_fails_closed` (git/visibility_pack.rs),
/// and the mirror import prune
/// `prune_removes_non_exempt_gitlawb_refs_and_keeps_exempt` (sync.rs).
#[test]
fn issue_26_namespace_gate_and_refusal_wiring() {
let repos = src("api/repos.rs");
let vis = src("git/visibility_pack.rs");
// P1: fail the push on ref-update lines the handler cannot decode.
// The gate at parse_ref_updates must refuse when any command line in the
// ref-update section is undecodable UTF-8. The check is an Err arm on
// `from_utf8(data)` that returns BadRequest, so removing it allows non-UTF-8
// refs to bypass the namespace gate.
assert!(
repos.contains("ref-update line is not valid UTF-8; push refused"),
"P1 gate missing: parse_ref_updates must refuse when any ref-update line is \
non-UTF-8 (git accepts bytes >= 0x80; the gate never sees them, no durable \
child exists, and the lossy for-each-ref decode still matches the exemption)"
);
// P1 (cont): the bare `refs/gitlawb` name must also be caught. The
// predicate is extracted as `ref_is_internal_namespace` so it is
// unit-testable; pin the bare-name arm inside it and its use at the
// push gate (a `starts_with("refs/gitlawb/")`-only check misses the
// bare name, and it lands on repos with no internal refs yet).
assert!(
repos.contains("ref_name == \"refs/gitlawb\"")
&& repos.contains(".find(|r| ref_is_internal_namespace(r))"),
"P1 gate missing: ref_is_internal_namespace must match the bare \
'refs/gitlawb' name and the push gate must consult it \
(starts_with('refs/gitlawb/') misses it, and it lands on repos with \
no internal refs yet)"
);
// P1 (cont): every explicit pre-git refusal must call refuse_receive_pack_before_git
// to terminalize the intent. Exactly 6 occurrences: the five inline call
// sites (lease cap, per-source cap, write pool, acquire timeout, acquire
// error) plus the drop-guard's own pre-git spawn. The post-git guard phase
// calls mark_receive_pack_interrupted (a different name), so deleting any
// one inline site drops the count to 5 and turns this red. The parse and
// namespace refusals correctly carry no call: they return before the
// intent insert, so there is no aggregate to terminalize.
assert_eq!(
repos.matches("refuse_receive_pack_before_git(").count(),
6,
"P1 wiring: exactly 6 refuse_receive_pack_before_git calls expected (5 \
inline refusal sites + guard pre-git spawn); deleting any inline site \
must turn this red"
);
// Pin the five inline sites by their reason literals, so a site that stops
// terminalizing (or is renumbered away) fails by name, not just by count.
for reason in [
"repo write-lease waiter cap reached",
"admission per-source cap reached",
"git write pool saturated",
"acquire_write timed out",
"acquire_write failed",
"handler dropped before git started",
] {
assert!(
repos.contains(reason),
"P1 wiring: refusal reason literal missing: {reason}"
);
}
// P1 (cont): the intent drop-guard must be phase-aware. It is armed before
// the insert resolves (covering the cancellation window), flipped to
// git-started immediately before the receive_pack await (a drop from there
// is a mid-git disconnect, terminalized via mark_receive_pack_interrupted
// because the Err arm never runs on a drop), and disarmed once the outcome
// commit resolves. Assert the ordering, not just presence: the phase flip
// must precede the receive_pack call.
assert!(
repos.contains("IntentDropGuard::new") && repos.contains("notify_git_started()"),
"P1 gate missing: IntentDropGuard must exist and flip to git-started phase \
before receive_pack (a dropped future between intent insert and git start \
strands a received+prepared aggregate)"
);
let flip_pos = repos
.find("notify_git_started()")
.expect("notify_git_started call must exist");
let git_pos = repos
.find("receive_pack_raw_with_reflog(")
.expect("receive_pack call must exist");
assert!(
flip_pos < git_pos,
"P1 ordering: notify_git_started() must precede the receive_pack call, \
or a mid-git drop is misclassified as a pre-git shed"
);
assert!(
repos.contains("_intent_guard.disarm()"),
"P1 gate missing: the guard must disarm once the outcome commit resolves, \
or every push spawns a wasteful cleanup txn at scope end"
);
// A sync Drop cannot await: cleanup is spawned via the current runtime
// handle with an off-runtime fallback. A bare tokio::spawn panics
// mid-unwind with no runtime (off-runtime drop, process teardown).
assert!(
repos.contains("Handle::try_current()"),
"P1 gate missing: guard Drop must spawn via Handle::try_current with an \
off-runtime arm, not a bare tokio::spawn"
);
// P2: the post-git drop arm must exist and be wired. The guard's
// GIT_STARTED phase spawns `mark_receive_pack_interrupted` (parent to
// `rejected_at_git`, children to `uncertain` — the Err arm never runs on
// a drop). Deleting that spawn keeps every count above green, so pin the
// method name; its behavior is proved by
// `post_git_interruption_marks_uncertain_and_terminalizes_parent`.
assert!(
repos.contains("mark_receive_pack_interrupted("),
"P2 wiring: the guard's post-git phase must spawn \
mark_receive_pack_interrupted (a mid-git drop otherwise strands \
received+prepared with no Err arm to mark them uncertain)"
);
// The guard must commit computed fates rather than a blanket uncertain
// once the report is parsed, and only disarm on a resolved commit: pin
// both the stash call and the `outcome_commit_ok` gate. Making disarm()
// unconditional strands the commit-failure path and misfiles proven
// rejections as uncertain.
assert!(
repos.contains("set_fates(ComputedFates"),
"P2 wiring: computed fates must be stashed on the guard once parsed, \
or a post-commit drop can only apply a context-free uncertain"
);
assert!(
repos.contains("if outcome_commit_ok"),
"P2 wiring: disarm must be gated on outcome_commit_ok; unconditional \
disarm strands the persistent-commit-failure path"
);
// The live-handler `AwaitingSiblings` arm is unreachable through the
// current outcome builder (documented at the arm itself) and therefore
// has no behavioral test; pin its presence so removing it is conscious.
// The drain consumer IS behaviorally driven (sibling-wait tests).
assert_eq!(
repos.matches("EffectsOutcome::AwaitingSiblings").count(),
1,
"live AwaitingSiblings arm must exist (defensive symmetry with the \
drain consumer); deleting it must turn this red"
);
// P2: pin the mirror prune call sites. The prune helper is called from
// both `clone_repo` and `fetch_repo` (plus its own definition) — deleting
// either call leaves the direct-helper test green while mirrors import
// evil refs again. Count occurrences of the call with its argument.
let sync_src = src("sync.rs");
assert_eq!(
sync_src
.matches("prune_non_exempt_gitlawb_refs(local_path)")
.count(),
2,
"P2 wiring: prune must be called from both clone_repo and fetch_repo; \
deleting either call must turn this red"
);
// Both call sites must route through `wrap_prune_error`, which preserves
// deterministic `PruneInvalid` refusals instead of stringifying every
// error into transient `PruneFailed` (an every-tick deferral spin for
// permanently invalid mirrors). Pin the exact call shape at both sites.
assert_eq!(
sync_src.matches(".map_err(wrap_prune_error)").count(),
2,
"P2 wiring: both prune call sites must use wrap_prune_error; inline \
rewrap must turn this red"
);
// P2: the visibility exemption narrowing removed a serving path. The exemption
// is now scoped to `requests/` and `issues/` only (visibility_pack.rs:371), and
// the comment at :340 states the push gate is the premise. Tests must prove the
// gate works, which the P1 findings above covered (non-UTF-8 line, bare name,
// refusal wiring). This check verifies the exemption narrowing landed.
assert!(
vis.contains(
r#"!r.starts_with("refs/gitlawb/requests/") && !r.starts_with("refs/gitlawb/issues/")"#
),
"P2 gate missing: the visibility exemption must be narrowed to requests/ and \
issues/ only (the push gate is the premise; any other refs/gitlawb/* ref is \
denied on push and must fail closed in the pack path)"
);
}
/// Claimed-webhook delivery must not re-list hooks: the durable path claims
/// synchronously, hands the claimed hook rows to the sender, and a list
/// failure in the send task would silently drop already-claimed deliveries.
#[test]
fn inv26_claimed_hook_delivery_never_relists() {
let webhooks = src("webhooks.rs");
let production = webhooks
.split("\nmod tests {")
.next()
.expect("split always yields a first chunk");
// The claimed entry point exists and passes hook ROWS (not ids) so the
// sender never needs the hook table again.
let entry = production
.find("fire_event_occurrence_with_claimed_hooks(")
.expect("claimed-hooks sender missing");
let claimed_rows = production
.find("claimed_hooks: Vec<crate::db::Webhook>")
.expect("claimed path must carry Webhook rows, not ids");
let list_call = production
.find("db.list_webhooks_for_event(repo_id, event)")
.expect("unclaimed path still lists hooks");
assert!(
entry < claimed_rows && claimed_rows < list_call,
"claimed sender must hand rows through before any list fallback"
);
// Inside the async worker, the claimed branch must bypass the list.
// Inside the async worker (which starts after the claim helper), the
// claimed branch must be selected before the list fallback can run.
let worker = production
.find("async fn fire_event_async_occurrence(")
.expect("occurrence worker missing");
let already_claimed = production[worker..]
.find("let already_claimed = claimed_hooks.is_some();")
.map(|i| worker + i)
.expect("claimed path must short-circuit before re-listing");
let worker_list = production[worker..]
.find("db.list_webhooks_for_event(repo_id, event)")
.map(|i| worker + i)
.expect("unclaimed path still lists hooks");
assert!(
already_claimed < worker_list,
"the claimed branch must be selected before list_webhooks_for_event can run"
);
}
+1
View File
@@ -29,6 +29,7 @@ uuid = { workspace = true }
url = { workspace = true }
percent-encoding = { workspace = true }
urlencoding = "2"
libc = "0.2"
alloy = { version = "1", default-features = false, features = [
"contract",
"provider-http",
+26 -33
View File
@@ -99,6 +99,27 @@ pub fn load_keypair_from_dir(dir: Option<&std::path::Path>) -> Result<Keypair> {
Keypair::from_pem(&pem).context("failed to load keypair from PEM")
}
/// Load keypair from an optional directory override, distinguishing missing default
/// identity (clean anonymous) from an explicit directory error or corrupt PEM.
pub fn load_optional_keypair(dir: Option<&std::path::Path>) -> Result<Option<Keypair>> {
if let Some(d) = dir {
return load_keypair_from_dir(Some(d)).map(Some);
}
let base = match dirs::home_dir() {
Some(h) => h.join(".gitlawb"),
None => return Ok(None),
};
let path = key_path(&base);
match fs::read_to_string(&path) {
Ok(pem) => Keypair::from_pem(&pem)
.context("failed to load keypair from PEM")
.map(Some),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None),
Err(e) => Err(anyhow::Error::from(e)
.context(format!("failed to read identity from {}", path.display()))),
}
}
async fn cmd_new(dir: Option<PathBuf>, force: bool) -> Result<()> {
cmd_new_with_reader(dir, force, &mut std::io::stdin().lock()).await
}
@@ -131,23 +152,13 @@ async fn cmd_new_with_reader(
}
}
fs::create_dir_all(&dir)
crate::secret_file::create_dir(&dir)
.with_context(|| format!("failed to create directory {}", dir.display()))?;
let keypair = Keypair::generate();
let pem = keypair.to_pem()?;
// Write with restricted permissions
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
fs::write(&path, pem.as_bytes())?;
fs::set_permissions(&path, fs::Permissions::from_mode(0o600))?;
}
#[cfg(not(unix))]
{
fs::write(&path, pem.as_bytes())?;
}
crate::secret_file::write(&path, pem.as_bytes())?;
let did = keypair.did();
println!("✓ Generated new identity");
@@ -202,16 +213,7 @@ async fn cmd_backup(out: Option<PathBuf>, dir: Option<PathBuf>) -> Result<()> {
.join("identity.pem.bak")
});
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
fs::write(&dest, pem.as_bytes())?;
fs::set_permissions(&dest, fs::Permissions::from_mode(0o600))?;
}
#[cfg(not(unix))]
{
fs::write(&dest, pem.as_bytes())?;
}
crate::secret_file::write(&dest, pem.as_bytes())?;
println!("✓ Identity backed up");
println!(" DID: {}", keypair.did());
@@ -262,19 +264,10 @@ async fn cmd_restore_with_reader(
}
}
fs::create_dir_all(&base)
crate::secret_file::create_dir(&base)
.with_context(|| format!("failed to create directory {}", base.display()))?;
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
fs::write(&dest, pem.as_bytes())?;
fs::set_permissions(&dest, fs::Permissions::from_mode(0o600))?;
}
#[cfg(not(unix))]
{
fs::write(&dest, pem.as_bytes())?;
}
crate::secret_file::write(&dest, pem.as_bytes())?;
println!("✓ Identity restored");
println!(" DID: {}", keypair.did());
+6 -15
View File
@@ -109,16 +109,16 @@ pub async fn run(args: InitArgs) -> Result<()> {
.dir
.clone()
.unwrap_or_else(|| dirs::home_dir().unwrap_or_default().join(".gitlawb"));
std::fs::create_dir_all(&ucan_dir)?;
crate::secret_file::create_dir(&ucan_dir)?;
let record = json!({
"ucan": ucan,
"node": args.node,
"did": did.to_string(),
"saved_at": chrono::Utc::now().to_rfc3339(),
});
std::fs::write(
ucan_dir.join("ucan.json"),
serde_json::to_string_pretty(&record)?,
crate::secret_file::write(
&ucan_dir.join("ucan.json"),
serde_json::to_string_pretty(&record)?.as_bytes(),
)?;
}
}
@@ -232,22 +232,13 @@ fn generate_identity(dir: Option<&std::path::Path>) -> Result<gitlawb_core::iden
.context("could not determine home directory")?
.join(".gitlawb")
};
std::fs::create_dir_all(&base)?;
crate::secret_file::create_dir(&base)?;
let keypair = gitlawb_core::identity::Keypair::generate();
let pem = keypair.to_pem()?;
let path = base.join("identity.pem");
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
std::fs::write(&path, pem.as_bytes())?;
std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600))?;
}
#[cfg(not(unix))]
{
std::fs::write(&path, pem.as_bytes())?;
}
crate::secret_file::write(&path, pem.as_bytes())?;
Ok(keypair)
}
+178 -8
View File
@@ -127,19 +127,93 @@ async fn cmd_list(node: String, dir: Option<PathBuf>) -> Result<()> {
println!("IPFS pins ({count}) on {node}");
println!();
for pin in &pins {
let cid = pin["cid"].as_str().unwrap_or("?");
let sha = pin["sha256_hex"].as_str().unwrap_or("?");
let pinned_at = pin["pinned_at"].as_str().unwrap_or("?");
// Trim pinned_at to date+time without subseconds
let ts = crate::text::truncate(pinned_at, 19);
println!(" {cid}");
println!(" sha256: {sha}");
println!(" pinned: {ts}");
for line in render_pin(pin) {
println!("{line}");
}
println!();
}
Ok(())
}
/// The rendered lines for one pins-listing entry.
///
/// #218 review round 8 P2 — why this is not just `pin["cid"]`: the wire format
/// gained a provenance split, and `cid` is now NULLABLE. The node stopped
/// aliasing a Pinata provider CID into `cid` (round 3) because `cid` is the
/// key `GET /ipfs/{cid}` resolves against, and a Pinata provider CID is not
/// resolvable there — advertising one would send clients to a guaranteed 404.
/// The truthful representation of a Pinata-only row is therefore `cid: null`
/// plus a `pinata_cid`. This function was reading only `cid`, so such a row
/// printed as a bare `?` — the CLI showed nothing usable for an object that is
/// in fact durably stored, and nothing in the client read `pinata_cid` at all.
///
/// So: the heading is the node-resolvable CID when there is one, and otherwise
/// the Pinata provider CID, labelled as such so nobody feeds it back to this
/// node's resolver. A `backends:` line names where the bytes actually are,
/// taken from the writer-owned `local_pinned` / `pinata_pinned` booleans rather
/// than re-inferred from CID shape (which is what the node's own comment warns
/// against). Older nodes send neither flag; their `cid`-only rows fall back to
/// CID presence and render exactly as they did before.
fn render_pin(pin: &Value) -> Vec<String> {
let sha = pin["sha256_hex"].as_str().unwrap_or("?");
let pinned_at = pin["pinned_at"].as_str().unwrap_or("?");
// Trim pinned_at to date+time without subseconds; truncate() is
// boundary-safe (a raw &s[..19] panics on a split code point).
let ts = if pinned_at.len() >= 19 {
crate::text::truncate(pinned_at, 19)
} else {
pinned_at
};
// `cid` and `local_cid` carry the same raw CID; read both so a node that
// ships only one of them still renders.
let local_cid = pin["cid"]
.as_str()
.or_else(|| pin["local_cid"].as_str())
.filter(|s| !s.is_empty());
let pinata_cid = pin["pinata_cid"].as_str().filter(|s| !s.is_empty());
// Writer-owned flags when present; CID presence is the pre-split fallback.
let local_pinned = pin["local_pinned"]
.as_bool()
.unwrap_or_else(|| local_cid.is_some());
let pinata_pinned = pin["pinata_pinned"]
.as_bool()
.unwrap_or_else(|| pinata_cid.is_some());
let mut lines = Vec::new();
match (local_cid, pinata_cid) {
// Node-resolvable CID present: it leads, as it always has.
(Some(cid), _) => lines.push(format!(" {cid}")),
// Pinata-only: show the provider CID rather than "?", and say plainly
// that this node's resolver will not serve it.
(None, Some(p)) => lines.push(format!(" {p} (pinata provider CID)")),
(None, None) => lines.push(" ?".to_string()),
}
lines.push(format!(" sha256: {sha}"));
lines.push(format!(" pinned: {ts}"));
// A dual row keeps the provider CID visible too; it differs from the local
// CID and is the only key that resolves on Pinata's gateway.
if local_cid.is_some() {
if let Some(p) = pinata_cid {
lines.push(format!(" pinata: {p}"));
}
}
let backends = match (local_pinned, pinata_pinned) {
(true, true) => "local ipfs, pinata",
(true, false) => "local ipfs",
(false, true) => "pinata",
// Neither flag set: the node filters such rows out, so this is only
// reachable from a hand-rolled response. Say so rather than imply
// durability the row does not claim.
(false, false) => "none recorded",
};
lines.push(format!(" backends: {backends}"));
lines
}
/// Automatic resumes attempted after the initial request when the node reports a
/// truncated legacy scan, so at most `MAX_SCAN_RESUMES + 1` node calls per invocation.
const MAX_SCAN_RESUMES: usize = 8;
@@ -633,6 +707,102 @@ mod tests {
dir
}
/// #218 review round 8 P2: a Pinata-only row must render its provider CID,
/// not a bare `?`. The node stopped aliasing that CID into `cid` because
/// `cid` is the key `GET /ipfs/{cid}` resolves on and a provider CID 404s
/// there — so `cid` is now legitimately null for such a row, and a client
/// that reads only `cid` shows the user nothing for an object that IS
/// durably stored. The heading must carry the provider CID and label it,
/// and the row must not claim a local pin.
#[test]
fn render_pin_shows_provider_cid_for_a_pinata_only_row() {
let pin: Value = serde_json::from_str(
r#"{"sha256_hex":"abc123","cid":null,"local_cid":null,
"pinata_cid":"QmProviderOnly","local_pinned":false,
"pinata_pinned":true,"pinned_at":"2026-07-02T12:00:00.123456Z"}"#,
)
.unwrap();
let out = render_pin(&pin).join("\n");
assert!(
out.contains("QmProviderOnly"),
"the provider CID must be shown; got:\n{out}"
);
assert!(
!out.contains(" ?"),
"a Pinata-only row must not render as a bare `?`; got:\n{out}"
);
assert!(
out.contains("backends: pinata"),
"the row must say where the bytes are; got:\n{out}"
);
assert!(
out.contains("pinata provider CID"),
"the heading must be labelled so it is not fed back to this node's \
resolver, which cannot serve it; got:\n{out}"
);
}
/// A dual row leads with the node-resolvable CID (unchanged behavior) and
/// additionally surfaces the provider CID, which is a different key and the
/// only one Pinata's gateway answers for.
#[test]
fn render_pin_shows_both_cids_for_a_dual_row() {
let pin: Value = serde_json::from_str(
r#"{"sha256_hex":"abc123","cid":"bafyLocal","local_cid":"bafyLocal",
"pinata_cid":"QmProvider","local_pinned":true,
"pinata_pinned":true,"pinned_at":"2026-07-02T12:00:00.123456Z"}"#,
)
.unwrap();
let out = render_pin(&pin).join("\n");
assert!(
out.starts_with(" bafyLocal"),
"the resolver key leads; got:\n{out}"
);
assert!(
out.contains("pinata: QmProvider"),
"the provider CID must still be reachable from the listing; got:\n{out}"
);
assert!(out.contains("backends: local ipfs, pinata"), "got:\n{out}");
}
/// Backward compatibility: a pre-provenance node sends `cid` alone, with no
/// `local_pinned` / `pinata_pinned` flags. That row must render as it always
/// did, with the backends line inferred from CID presence.
#[test]
fn render_pin_handles_a_legacy_cid_only_row() {
let pin: Value = serde_json::from_str(
r#"{"sha256_hex":"abc123","cid":"bafyone",
"pinned_at":"2026-07-02T12:00:00.123456Z"}"#,
)
.unwrap();
let out = render_pin(&pin).join("\n");
assert!(out.starts_with(" bafyone"), "got:\n{out}");
assert!(out.contains("sha256: abc123"), "got:\n{out}");
assert!(
out.contains("pinned: 2026-07-02T12:00:00"),
"the timestamp is still trimmed to seconds; got:\n{out}"
);
assert!(out.contains("backends: local ipfs"), "got:\n{out}");
}
/// A node-supplied timestamp whose 19th byte splits a multi-byte code
/// point must not panic the CLI — `&s[..19]` panics there; `truncate`
/// returns the boundary-safe prefix.
#[test]
fn render_pin_does_not_panic_on_a_non_boundary_timestamp() {
let mut ts = "2026-07-02T12:00:0".to_string();
ts.push('é'); // 2-byte char starting at byte 18: byte 19 is its tail
assert_eq!(ts.len(), 20);
assert!(std::panic::catch_unwind(|| &ts[..19]).is_err());
let pin: Value = serde_json::json!({
"sha256_hex":"abc123","cid":"bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4",
"local_cid":null,"pinata_cid":null,"local_pinned":true,
"pinata_pinned":false,"pinned_at":ts,
});
let out = render_pin(&pin).join("\n");
assert!(out.contains("pinned: 2026-07-02T12:00:0"), "got:\n{out}");
}
#[tokio::test]
async fn test_cmd_list_signs_request_and_renders_pins() {
let mut server = mockito::Server::new_async().await;
+1
View File
@@ -26,6 +26,7 @@ mod protect;
mod quickstart;
mod register;
mod repo;
mod secret_file;
mod star;
mod status;
mod sync;
+384 -13
View File
@@ -43,7 +43,6 @@ use std::io::{BufRead, Write};
use std::path::PathBuf;
use crate::http::NodeClient;
use crate::identity::load_keypair_from_dir;
#[derive(Args)]
pub struct McpArgs {
@@ -516,7 +515,8 @@ fn tool_definitions() -> Value {
"properties": {
"status": { "type": "string", "description": "Filter by status: pending, claimed, completed, failed" },
"assignee_did": { "type": "string", "description": "Filter by assignee DID" },
"limit": { "type": "integer", "description": "Max results (default: 50)", "default": 50 }
"limit": { "type": "integer", "minimum": 1, "description": "Total results to return (default: 50). Must be positive; a non-positive limit is rejected rather than answered with an empty list. Values above the node's 200-row page cap are gathered by following continuation tokens.", "default": 50 },
"cursor": { "type": "string", "description": "Resume from a previous call's next_cursor. Must be paired with the same status/assignee_did filter that produced it." }
}
}
},
@@ -640,7 +640,14 @@ async fn call_tool(
node: &str,
dir: Option<&std::path::Path>,
) -> Result<String> {
let keypair = load_keypair_from_dir(dir).ok();
// An explicit dir with no identity file is a clean anonymous caller
// (main's unsigned `webhook_list` path lets the node answer the denial);
// a present but unloadable identity is an error, never a silent
// downgrade to anonymous (this PR's `load_optional_keypair` contract).
let keypair = match dir {
Some(d) if !d.join("identity.pem").exists() => None,
_ => crate::identity::load_optional_keypair(dir)?,
};
let client = NodeClient::new(node, keypair.clone());
match name {
@@ -1114,16 +1121,31 @@ async fn call_tool(
// ── Task tools ────────────────────────────────────────────────────
"task_list" => {
let limit = args["limit"].as_i64().unwrap_or(50);
let mut path = format!("/api/v1/tasks?limit={limit}");
if let Some(s) = args.get("status").and_then(|v| v.as_str()) {
path.push_str(&format!("&status={}", urlencoding::encode(s)));
// Follows the node's opaque `next_cursor` so a limit above the
// 200-row server page cap returns what was asked for instead of a
// silently truncated page, and reports an explicit incomplete
// result when a guard or the node's scan ceiling stops it (#327).
let limit = match args.get("limit") {
None | Some(Value::Null) => 50,
Some(val) => val.as_i64().context("invalid limit: expected an integer")?,
};
let cursor = match args.get("cursor") {
None | Some(Value::Null) => None,
Some(val) => Some(val.as_str().context("invalid cursor: expected a string")?),
};
let result = crate::task::fetch_tasks(
&client,
args.get("status").and_then(|v| v.as_str()),
args.get("assignee_did").and_then(|v| v.as_str()),
limit,
cursor,
)
.await?;
let mut out = result.to_json();
if let Some(warning) = result.truncation_warning() {
out["warning"] = Value::String(warning);
}
if let Some(a) = args.get("assignee_did").and_then(|v| v.as_str()) {
path.push_str(&format!("&assignee_did={}", urlencoding::encode(a)));
}
let resp = crate::http::read_json(client.get(&path).await?, "list tasks").await?;
Ok(serde_json::to_string_pretty(&resp)?)
Ok(serde_json::to_string_pretty(&out)?)
}
"task_create" => {
@@ -1726,7 +1748,7 @@ mod tests {
)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"tasks":[{"id":"t1","kind":"test","status":"pending"}]}"#)
.with_body(r#"{"tasks":[{"id":"t1","kind":"test","status":"pending"}],"has_more":false,"incomplete":false,"next_cursor":null}"#)
.create_async()
.await;
@@ -1740,6 +1762,275 @@ mod tests {
.unwrap();
let parsed: Value = serde_json::from_str(&result).unwrap();
assert_eq!(parsed["tasks"][0]["id"], "t1");
assert_eq!(parsed["complete"], json!(true));
}
#[tokio::test]
async fn test_task_list_via_mcp_uses_loaded_identity() {
let mut server = mockito::Server::new_async().await;
let dir = tempfile::TempDir::new().unwrap();
let kp = gitlawb_core::identity::Keypair::generate();
std::fs::write(
dir.path().join("identity.pem"),
kp.to_pem().unwrap().as_bytes(),
)
.unwrap();
let _m = server
.mock(
"GET",
mockito::Matcher::Regex(r"/api/v1/tasks\?".to_string()),
)
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(
r#"{"tasks":[{"id":"t1"}],"has_more":false,"incomplete":false,"next_cursor":null}"#,
)
.create_async()
.await;
let result = call_tool("task_list", json!({}), &server.url(), Some(dir.path()))
.await
.unwrap();
let parsed: Value = serde_json::from_str(&result).unwrap();
assert_eq!(parsed["tasks"][0]["id"], "t1");
assert_eq!(parsed["complete"], json!(true));
}
/// #327 review: MCP `task_list` issued one request and exposed no
/// continuation, so a limit above the node's 200-row page cap returned a
/// silently truncated result the model had no way to detect. It now
/// follows cursors and, when a guard stops it, says so in the payload.
#[tokio::test]
async fn test_task_list_via_mcp_follows_cursors() {
let mut server = mockito::Server::new_async().await;
let first = server
.mock(
"GET",
mockito::Matcher::Regex(r"^/api/v1/tasks\?limit=200$".to_string()),
)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(
r#"{"tasks":[{"id":"t1"}],"has_more":true,"incomplete":false,"next_cursor":"c1"}"#,
)
.create_async()
.await;
let second = server
.mock("GET", mockito::Matcher::Regex(r"cursor=c1".to_string()))
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"tasks":[{"id":"t2"}],"has_more":false,"next_cursor":null}"#)
.create_async()
.await;
let result = call_tool("task_list", json!({"limit": 500}), &server.url(), None)
.await
.unwrap();
first.assert_async().await;
second.assert_async().await;
let parsed: Value = serde_json::from_str(&result).unwrap();
assert_eq!(parsed["count"], 2);
assert_eq!(parsed["tasks"][1]["id"], "t2");
assert_eq!(parsed["complete"], json!(true));
assert!(parsed.get("warning").is_none());
}
/// A truncated MCP result must carry an explicit warning and a resume
/// cursor, so the model cannot read it as the whole answer.
#[tokio::test]
async fn test_task_list_via_mcp_flags_incomplete_results() {
let mut server = mockito::Server::new_async().await;
let _m = server
.mock("GET", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(
r#"{"tasks":[{"id":"t1"}],"has_more":true,"incomplete":true,"next_cursor":"resume-me"}"#,
)
.create_async()
.await;
let result = call_tool("task_list", json!({"limit": 1}), &server.url(), None)
.await
.unwrap();
let parsed: Value = serde_json::from_str(&result).unwrap();
assert_eq!(parsed["complete"], json!(false));
assert_eq!(parsed["incomplete"], json!(true));
assert_eq!(parsed["next_cursor"], "resume-me");
assert!(
parsed["warning"]
.as_str()
.unwrap()
.contains("result incomplete"),
"{parsed}"
);
}
/// A legacy response without pagination metadata is reported as incomplete via MCP.
#[tokio::test]
async fn test_task_list_via_mcp_legacy_response_is_incomplete() {
let mut server = mockito::Server::new_async().await;
let _m = server
.mock("GET", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"tasks":[{"id":"t1"}],"count":1}"#)
.create_async()
.await;
let result = call_tool("task_list", json!({"limit": 50}), &server.url(), None)
.await
.unwrap();
let parsed: Value = serde_json::from_str(&result).unwrap();
assert_eq!(parsed["complete"], json!(false));
assert_eq!(parsed["incomplete"], json!(true));
assert!(parsed["next_cursor"].is_null());
assert!(
parsed["warning"]
.as_str()
.unwrap()
.contains("node does not support pagination metadata"),
"{parsed}"
);
}
/// A legacy response with limit > 200 stops after 1 page and does not claim completeness via MCP.
#[tokio::test]
async fn test_task_list_via_mcp_legacy_limit_above_page_cap() {
let mut server = mockito::Server::new_async().await;
let ids: Vec<String> = (0..200).map(|i| format!("t{i}")).collect();
let tasks_json: Vec<Value> = ids.iter().map(|id| json!({ "id": id })).collect();
let body = json!({ "tasks": tasks_json, "count": 200 }).to_string();
let m = server
.mock(
"GET",
mockito::Matcher::Regex(r"^/api/v1/tasks\?limit=200$".to_string()),
)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(body)
.expect(1)
.create_async()
.await;
let result = call_tool("task_list", json!({"limit": 500}), &server.url(), None)
.await
.unwrap();
m.assert_async().await;
let parsed: Value = serde_json::from_str(&result).unwrap();
assert_eq!(parsed["count"], 200);
assert_eq!(parsed["complete"], json!(false));
assert_eq!(parsed["incomplete"], json!(true));
assert!(parsed["next_cursor"].is_null());
assert!(
parsed["warning"]
.as_str()
.unwrap()
.contains("node does not support pagination metadata"),
"{parsed}"
);
}
/// A cursor the model passes back must reach the node.
#[tokio::test]
async fn test_task_list_via_mcp_forwards_cursor_argument() {
let mut server = mockito::Server::new_async().await;
let m = server
.mock(
"GET",
mockito::Matcher::Regex(r"cursor=given-cursor".to_string()),
)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"tasks":[],"has_more":false,"incomplete":false,"next_cursor":null}"#)
.expect(1)
.create_async()
.await;
call_tool(
"task_list",
json!({"cursor": "given-cursor"}),
&server.url(),
None,
)
.await
.unwrap();
m.assert_async().await;
}
#[tokio::test]
async fn test_task_list_via_mcp_returns_http_errors() {
let mut server = mockito::Server::new_async().await;
let _m = server
.mock(
"GET",
mockito::Matcher::Regex(r"/api/v1/tasks\?".to_string()),
)
.with_status(500)
.with_header("content-type", "application/json")
.with_body(r#"{"error":"failed"}"#)
.create_async()
.await;
let err = call_tool("task_list", json!({}), &server.url(), None)
.await
.unwrap_err();
assert!(err.to_string().contains("500"));
}
/// Malformed server responses fail visibly in MCP.
#[tokio::test]
async fn test_task_list_via_mcp_malformed_response_errors() {
let mut server = mockito::Server::new_async().await;
let _m = server
.mock(
"GET",
mockito::Matcher::Regex(r"/api/v1/tasks\?".to_string()),
)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"tasks":"not-an-array"}"#)
.create_async()
.await;
let err = call_tool("task_list", json!({}), &server.url(), None)
.await
.unwrap_err();
assert!(
err.to_string().contains("malformed") || err.to_string().contains("invalid JSON"),
"{err}"
);
}
/// #327 review: a model that sends `limit: 0` used to get an empty list
/// marked complete, which reads as "this node has no tasks". The shared
/// helper rejects it, so the model sees an invalid argument instead.
#[tokio::test]
async fn test_task_list_via_mcp_rejects_non_positive_limit() {
let mut server = mockito::Server::new_async().await;
let m = server
.mock(
"GET",
mockito::Matcher::Regex(r"/api/v1/tasks\?".to_string()),
)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"tasks":[],"has_more":false,"incomplete":false,"next_cursor":null}"#)
.expect(0)
.create_async()
.await;
let err = call_tool("task_list", json!({"limit": 0}), &server.url(), None)
.await
.unwrap_err();
assert!(
err.to_string().contains("limit must be a positive"),
"{err}"
);
m.assert_async().await;
}
#[tokio::test]
@@ -2208,6 +2499,86 @@ mod tests {
assert!(err.to_string().contains("no identity found"), "got: {err}");
}
#[tokio::test]
async fn test_task_list_invalid_limit_rejected() {
let server = mockito::Server::new_async().await;
let dir = tempfile::TempDir::new().unwrap();
let kp = gitlawb_core::identity::Keypair::generate();
std::fs::write(
dir.path().join("identity.pem"),
kp.to_pem().unwrap().as_bytes(),
)
.unwrap();
let err = call_tool(
"task_list",
json!({"limit": "50"}),
&server.url(),
Some(dir.path()),
)
.await
.expect_err("string limit must be rejected");
assert!(
err.to_string()
.contains("invalid limit: expected an integer"),
"got: {err}"
);
let err_float = call_tool(
"task_list",
json!({"limit": 50.5}),
&server.url(),
Some(dir.path()),
)
.await
.expect_err("float limit must be rejected");
assert!(
err_float
.to_string()
.contains("invalid limit: expected an integer"),
"got: {err_float}"
);
}
#[tokio::test]
async fn test_task_list_invalid_cursor_rejected() {
let server = mockito::Server::new_async().await;
for invalid in [json!(123), json!({}), json!([]), json!(true)] {
let err = call_tool("task_list", json!({"cursor": invalid}), &server.url(), None)
.await
.expect_err("non-string cursor must be rejected");
assert!(
err.to_string()
.contains("invalid cursor: expected a string"),
"for {invalid}: got {err}"
);
}
}
#[tokio::test]
async fn test_task_list_null_or_absent_cursor_accepted() {
let mut server = mockito::Server::new_async().await;
let _m = server
.mock(
"GET",
mockito::Matcher::Regex(r"/api/v1/tasks\?".to_string()),
)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"tasks":[],"has_more":false,"incomplete":false,"next_cursor":null}"#)
.expect(2)
.create_async()
.await;
call_tool("task_list", json!({}), &server.url(), None)
.await
.unwrap();
call_tool("task_list", json!({"cursor": null}), &server.url(), None)
.await
.unwrap();
}
#[test]
fn test_tool_count_is_42() {
let tools = tool_definitions();
+20 -18
View File
@@ -100,19 +100,29 @@ pub async fn run(args: QuickstartArgs) -> Result<()> {
let payload: Value = resp.json().await.unwrap_or_default();
let ucan = payload["ucan"].as_str().unwrap_or("");
if !ucan.is_empty() {
std::fs::create_dir_all(&dir)?;
crate::secret_file::create_dir(&dir)?;
let record = json!({
"ucan": ucan,
"node": args.node,
"did": did,
"saved_at": chrono::Utc::now().to_rfc3339(),
});
std::fs::write(&ucan_path, serde_json::to_string_pretty(&record)?)?;
crate::secret_file::write(
&ucan_path,
serde_json::to_string_pretty(&record)?.as_bytes(),
)?;
let trust = payload["trust_score"].as_f64().unwrap_or(0.0);
println!(" ✓ Registered successfully");
println!(" Trust score: {trust:.2}");
println!(" UCAN saved to {}", ucan_path.display());
} else {
// A 2xx without a usable ucan is not a registration; the
// response may not even be JSON. Saying success here would
// also leave a stale ucan.json from another node in place.
println!(" ✗ Registration returned no UCAN (unexpected response body)");
// No retry hint here: a `gl register` instruction without
// this invocation's --dir would load a different identity.
}
let trust = payload["trust_score"].as_f64().unwrap_or(0.0);
println!(" ✓ Registered successfully");
println!(" Trust score: {trust:.2}");
println!(" UCAN saved to {}", ucan_path.display());
println!();
}
Ok(resp) => {
@@ -226,23 +236,15 @@ pub async fn run(args: QuickstartArgs) -> Result<()> {
// ── Helpers ───────────────────────────────────────────────────────────────
fn generate_identity(dir: &PathBuf) -> Result<gitlawb_core::identity::Keypair> {
std::fs::create_dir_all(dir).with_context(|| format!("failed to create {}", dir.display()))?;
fn generate_identity(dir: &std::path::Path) -> Result<gitlawb_core::identity::Keypair> {
crate::secret_file::create_dir(dir)
.with_context(|| format!("failed to create {}", dir.display()))?;
let keypair = gitlawb_core::identity::Keypair::generate();
let pem = keypair.to_pem()?;
let path = dir.join("identity.pem");
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
std::fs::write(&path, pem.as_bytes())?;
std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600))?;
}
#[cfg(not(unix))]
{
std::fs::write(&path, pem.as_bytes())?;
}
crate::secret_file::write(&path, pem.as_bytes())?;
let did = keypair.did();
println!(" ✓ Generated new identity");
+5 -2
View File
@@ -72,7 +72,10 @@ pub async fn run(args: RegisterArgs) -> Result<()> {
"did": did.to_string(),
"saved_at": chrono::Utc::now().to_rfc3339(),
});
std::fs::write(&ucan_path, serde_json::to_string_pretty(&record)?)?;
crate::secret_file::write(
&ucan_path,
serde_json::to_string_pretty(&record)?.as_bytes(),
)?;
tracing::debug!("saved UCAN to {}", ucan_path.display());
}
@@ -108,7 +111,7 @@ fn ucan_path(dir: Option<&std::path::Path>) -> Result<PathBuf> {
.context("could not determine home directory")?
.join(".gitlawb")
};
std::fs::create_dir_all(&base)?;
crate::secret_file::create_dir(&base)?;
Ok(base.join("ucan.json"))
}
+180
View File
@@ -0,0 +1,180 @@
//! Writes and directories that hold key material or bearer tokens.
use std::io::Write;
use std::path::Path;
/// Write `contents` to `path` with owner-only permissions.
///
/// On unix the mode is pinned at creation, so the file never exists with a
/// permissive mode between open and chmod. `O_NOFOLLOW` refuses to write
/// through a pre-planted symlink. `mode()` applies only when the file is
/// created, so the mode is pinned on the descriptor before the file is
/// truncated: a pre-existing loose file is tightened before, not after, the
/// new contents land in it.
pub(crate) fn write(path: &Path, contents: &[u8]) -> std::io::Result<()> {
#[cfg(unix)]
{
use std::os::unix::fs::{OpenOptionsExt, PermissionsExt};
let mut f = std::fs::OpenOptions::new()
.write(true)
.create(true)
.mode(0o600)
.custom_flags(libc::O_NOFOLLOW)
.open(path)?;
f.set_permissions(std::fs::Permissions::from_mode(0o600))?;
f.set_len(0)?;
f.write_all(contents)?;
}
#[cfg(not(unix))]
std::fs::write(path, contents)?;
Ok(())
}
/// Create `path` (and missing parents) as an owner-only directory.
///
/// An existing directory is also re-pinned: a `~/.gitlawb` created before
/// this helper stays group- and world-listable otherwise. A symlinked
/// `path` is left alone rather than chmodded through, which would strip
/// group and world access from whatever the link points at.
pub(crate) fn create_dir(path: &Path) -> std::io::Result<()> {
#[cfg(unix)]
{
use std::os::unix::fs::{DirBuilderExt, OpenOptionsExt};
std::fs::DirBuilder::new()
.recursive(true)
.mode(0o700)
.create(path)?;
// Re-pin through the opened directory, not the path: O_NOFOLLOW
// refuses a swapped-in symlink and fchmod acts on the checked inode,
// so a final-component swap between check and chmod cannot retarget
// the mode change at whatever the link points at.
match std::fs::OpenOptions::new()
.read(true)
.custom_flags(libc::O_NOFOLLOW)
.open(path)
{
Ok(dir) => {
use std::os::unix::io::AsRawFd;
let fd = dir.as_raw_fd();
// Re-pin only a leaf this process owns. A --dir the caller
// can write but cannot chmod (foreign-owned 0775 leaf, ACL
// grant) is adopted unchanged: create_dir_all allowed the
// write before, and aborting turns a usable directory into
// a command failure. An owned directory that still refuses
// chmod (EPERM) is adopted the same way.
let owned = unsafe {
let mut st: libc::stat = std::mem::zeroed();
libc::fstat(fd, &mut st) == 0 && st.st_uid == libc::geteuid()
};
if owned && unsafe { libc::fchmod(fd, 0o700) } != 0 {
let e = std::io::Error::last_os_error();
if e.raw_os_error() != Some(libc::EPERM) {
return Err(e);
}
}
}
// ELOOP: the path is a symlink and is left alone per contract.
Err(e) if e.raw_os_error() == Some(libc::ELOOP) => {}
Err(e) => return Err(e),
}
}
#[cfg(not(unix))]
std::fs::create_dir_all(path)?;
Ok(())
}
#[cfg(test)]
mod tests {
use tempfile::TempDir;
#[test]
#[cfg(unix)]
fn create_dir_adopts_writable_leaf_it_cannot_chmod() {
use std::os::unix::fs::PermissionsExt;
// /tmp is root-owned 1777: writable by this euid, not chmod-able.
// The pre-helper create_dir_all allowed the write; the re-pin must
// adopt the leaf unchanged rather than abort on EPERM.
let before = std::fs::metadata("/tmp").unwrap().permissions().mode();
super::create_dir(std::path::Path::new("/tmp")).unwrap();
let after = std::fs::metadata("/tmp").unwrap().permissions().mode();
assert_eq!(before, after);
}
#[test]
#[cfg(unix)]
fn write_creates_file_with_0600() {
use std::os::unix::fs::PermissionsExt;
let dir = TempDir::new().unwrap();
let path = dir.path().join("key.pem");
super::write(&path, b"secret").unwrap();
assert_eq!(std::fs::read(&path).unwrap(), b"secret");
assert_eq!(
std::fs::metadata(&path).unwrap().permissions().mode() & 0o777,
0o600
);
}
#[test]
#[cfg(unix)]
fn write_tightens_preexisting_loose_file() {
use std::os::unix::fs::PermissionsExt;
let dir = TempDir::new().unwrap();
let path = dir.path().join("key.pem");
std::fs::write(&path, b"old").unwrap();
std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o644)).unwrap();
super::write(&path, b"new").unwrap();
assert_eq!(std::fs::read(&path).unwrap(), b"new");
assert_eq!(
std::fs::metadata(&path).unwrap().permissions().mode() & 0o777,
0o600
);
}
#[test]
#[cfg(unix)]
fn write_refuses_symlink() {
let dir = TempDir::new().unwrap();
let target = dir.path().join("target.pem");
std::fs::write(&target, b"planted").unwrap();
let link = dir.path().join("link.pem");
std::os::unix::fs::symlink(&target, &link).unwrap();
assert!(super::write(&link, b"secret").is_err());
assert_eq!(std::fs::read(&target).unwrap(), b"planted");
}
#[test]
#[cfg(unix)]
fn create_dir_modes_0700_and_tightens_existing() {
use std::os::unix::fs::PermissionsExt;
let dir = TempDir::new().unwrap();
let path = dir.path().join("a").join("b");
super::create_dir(&path).unwrap();
assert_eq!(
std::fs::metadata(&path).unwrap().permissions().mode() & 0o777,
0o700
);
std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o755)).unwrap();
super::create_dir(&path).unwrap();
assert_eq!(
std::fs::metadata(&path).unwrap().permissions().mode() & 0o777,
0o700
);
}
#[test]
#[cfg(unix)]
fn create_dir_does_not_chmod_through_symlink() {
use std::os::unix::fs::PermissionsExt;
let dir = TempDir::new().unwrap();
let target = dir.path().join("shared");
std::fs::create_dir(&target).unwrap();
std::fs::set_permissions(&target, std::fs::Permissions::from_mode(0o755)).unwrap();
let link = dir.path().join("link");
std::os::unix::fs::symlink(&target, &link).unwrap();
super::create_dir(&link).unwrap();
assert_eq!(
std::fs::metadata(&target).unwrap().permissions().mode() & 0o777,
0o755
);
}
}
+1364 -34
View File
File diff suppressed because it is too large. Load diff
+1 -1
View File
@@ -90,7 +90,7 @@ async fn cmd_delegate(
let encoded = ucan.encode()?;
if let Some(path) = out {
std::fs::write(&path, &encoded)?;
crate::secret_file::write(&path, encoded.as_bytes())?;
println!("UCAN saved to {}", path.display());
return Ok(());
}
+1
View File
@@ -91,6 +91,7 @@ Required env for on-chain PoS mode:
Optional:
- `GITLAWB_OPERATOR_STRICT_MODE=true` — refuse to start if not registered or not currently active
- `GITLAWB_HEARTBEAT_INTERVAL_HOURS=20` — how often to post heartbeats (must be < 24)
- `GITLAWB_RECONCILIATION_SWEEP=true` — enable the hourly durability sweep that re-pins/backstops missing objects (default `true`; disabled when no IPFS/Pinata backend is configured). Public pin repair runs against any configured backend. Encrypted recovery repair requires local IPFS (`GITLAWB_IPFS_API`); Pinata-only nodes reconcile public pins only. Set `=false` to disable.
## 5. Verify
+221
View File
@@ -0,0 +1,221 @@
#!/usr/bin/env bash
#
# Map a pull request's changed paths to the CI surfaces they touch, so each
# check in pr-checks.yml and codeql.yml runs only when its inputs changed and
# Rust jobs test only the affected crates.
#
# Usage:
# git diff -z --name-only --no-renames BASE HEAD | scripts/ci-surfaces.sh
# scripts/ci-surfaces.sh --all # every surface, whole workspace
#
# Input is NUL-terminated paths (`git diff -z`). Without -z, git C-quotes any
# path with non-ASCII, tab, quote or newline characters ("crates/gl/\303\251.rs"),
# which would match no surface and skip every job; newline-delimited input
# also cannot carry a path that contains a newline. Input that does not end in
# NUL (e.g. a plain `git diff --name-only`) is rejected rather than misread.
# scripts/test-ci-surfaces.sh pins the mapping.
#
# Prints GITHUB_OUTPUT lines (key=value). Each surface runs only for the paths
# listed for it below, so a path no surface lists (docs, README, infra/, any
# other workflow) runs no PR Checks job. Within the Rust surface, a change this
# script cannot attribute to specific crates widens rather than narrows: a
# global Rust input or an unknown crate directory runs the Rust jobs on the
# whole workspace. The CI wiring itself (pr-checks.yml, codeql.yml, this
# script) runs every surface.
# Non-PR events (push to main, merge_group, workflow_dispatch, schedule) are
# expected to pass --all, so main and the merge queue keep full coverage.
#
# Runnable from anywhere in the repo.
set -euo pipefail
ROOT="$(git rev-parse --show-toplevel)"
all=false
if [ "${1:-}" = "--all" ]; then
all=true
elif [ "$#" -ne 0 ]; then
echo "usage: $0 [--all] < NUL-terminated-changed-paths" >&2
exit 2
fi
changed=()
if [ "$all" = false ]; then
path=""
while IFS= read -r -d '' path; do
[ -n "$path" ] && changed+=("$path")
done
# read leaves any unterminated tail in $path.
if [ -n "$path" ]; then
echo "ci-surfaces.sh: input is not NUL-terminated; pipe \`git diff -z --name-only\`" >&2
exit 2
fi
fi
# Workspace crates, derived from crates/*/Cargo.toml so a new crate is picked
# up without editing this script. dir -> package name, and dir -> the crate
# dirs it depends on through `path = "../<dir>"` (normal, dev, or build deps).
declare -A pkg_of=() deps_of=()
for manifest in "$ROOT"/crates/*/Cargo.toml; do
dir="$(basename "$(dirname "$manifest")")"
pkg_of[$dir]="$(sed -n 's/^name *= *"\([^"]*\)".*/\1/p' "$manifest" | head -n1)"
deps_of[$dir]="$(sed -n 's#.*path *= *"\.\./\([^"/]*\)".*#\1#p' "$manifest" | tr '\n' ' ')"
done
# Files outside crates/ that a crate compiles in (include_str!/include_bytes!),
# path -> crate dir. A change to one is a change to that crate, so its tests
# run. Candidates (keep those that resolve outside crates/<dir>/):
# git grep -nE 'include_(str|bytes)!\("(\.\./){2,}' crates
declare -A embedded_in=(
[bootstrap-peers.json]=gitlawb-node
)
# Mark a crate dir as touched; a dir this checkout does not know (removed or
# renamed crate) widens the Rust jobs to the whole workspace.
touch_crate() {
if [ -n "${pkg_of[$1]:-}" ]; then
touched[$1]=1
else
full_rust=true
fi
}
full_rust=false
rust=false
docker=false
audit=false
core_deps=false
release_versioning=false
release_tag=false
cq_actions=false
cq_js=false
cq_rust=false
cq_swift=false
declare -A touched=()
for path in "${changed[@]}"; do
case "$path" in
# The CI wiring itself: validate everything it gates.
.github/workflows/pr-checks.yml | .github/workflows/codeql.yml | scripts/ci-surfaces.sh)
all=true ;;
# Workspace-wide Rust inputs.
Cargo.toml | Cargo.lock | .cargo/* | rust-toolchain | rust-toolchain.toml \
| rustfmt.toml | .rustfmt.toml | clippy.toml | .clippy.toml)
full_rust=true ;;
crates/*/*)
dir="${path#crates/}"
touch_crate "${dir%%/*}" ;;
*)
if [ -n "${embedded_in[$path]:-}" ]; then
touch_crate "${embedded_in[$path]}"
fi ;;
esac
case "$path" in
Dockerfile | .dockerignore | bootstrap-peers.json | Cargo.toml | Cargo.lock \
| .cargo/* | rust-toolchain | rust-toolchain.toml | crates/*/Cargo.toml)
docker=true ;;
esac
case "$path" in
Cargo.lock | .cargo/audit.toml) audit=true ;;
esac
case "$path" in
Cargo.toml | Cargo.lock | crates/gitlawb-core/Cargo.toml \
| ci/gitlawb-core-allowed-deps.txt | scripts/check-gitlawb-core-deps.sh)
core_deps=true ;;
esac
case "$path" in
Cargo.toml | crates/*/Cargo.toml | release-please-config.json \
| .release-please-manifest.json | scripts/check-release-versioning.sh)
release_versioning=true ;;
esac
case "$path" in
scripts/resolve-release-tag.sh | scripts/test-resolve-release-tag.sh)
release_tag=true ;;
esac
case "$path" in
.github/workflows/* | .github/actions/*) cq_actions=true ;;
esac
case "$path" in
npm/* | *.js | *.mjs | *.cjs | *.jsx | *.ts | *.mts | *.cts | *.tsx) cq_js=true ;;
esac
case "$path" in
macos-app/* | *.swift) cq_swift=true ;;
esac
done
if [ "$all" = true ]; then
full_rust=true
docker=true
audit=true
core_deps=true
release_versioning=true
release_tag=true
cq_actions=true
cq_js=true
cq_swift=true
fi
# Affected crates: the touched ones plus everything that depends on them,
# transitively, so a gitlawb-core change still tests every consumer.
declare -A affected=()
for dir in "${!touched[@]}"; do affected[$dir]=1; done
grew=true
while [ "$grew" = true ]; do
grew=false
for dir in "${!deps_of[@]}"; do
[ -n "${affected[$dir]:-}" ] && continue
for dep in ${deps_of[$dir]}; do
if [ -n "${affected[$dep]:-}" ]; then
affected[$dir]=1
grew=true
break
fi
done
done
done
if [ "$full_rust" = true ] || [ "${#affected[@]}" -gt 0 ]; then
rust=true
cq_rust=true
fi
rust_packages=""
windows_packages=""
if [ "$full_rust" = true ]; then
rust_packages="--workspace"
windows_packages="-p gl -p git-remote-gitlawb"
elif [ "$rust" = true ]; then
for dir in $(printf '%s\n' "${!affected[@]}" | sort); do
rust_packages+="${rust_packages:+ }-p ${pkg_of[$dir]}"
case "${pkg_of[$dir]}" in
gl | git-remote-gitlawb) windows_packages+="${windows_packages:+ }-p ${pkg_of[$dir]}" ;;
esac
done
fi
windows=false
[ -n "$windows_packages" ] && windows=true
languages=()
[ "$cq_actions" = true ] && languages+=('{"language":"actions","build-mode":"none","runner":"ubuntu-latest"}')
[ "$cq_js" = true ] && languages+=('{"language":"javascript-typescript","build-mode":"none","runner":"ubuntu-latest"}')
[ "$cq_rust" = true ] && languages+=('{"language":"rust","build-mode":"none","runner":"ubuntu-latest"}')
[ "$cq_swift" = true ] && languages+=('{"language":"swift","build-mode":"autobuild","runner":"macos-latest"}')
codeql_matrix="{\"include\":[$(IFS=,; echo "${languages[*]}")]}"
codeql=false
[ "${#languages[@]}" -gt 0 ] && codeql=true
cat <<EOF
rust=$rust
rust_packages=$rust_packages
windows=$windows
windows_packages=$windows_packages
docker=$docker
audit=$audit
core_deps=$core_deps
release_versioning=$release_versioning
release_tag=$release_tag
codeql=$codeql
codeql_matrix=$codeql_matrix
EOF
+121
View File
@@ -0,0 +1,121 @@
#!/usr/bin/env bash
# Fixture test for scripts/ci-surfaces.sh, which alone decides which PR Checks
# jobs and CodeQL languages run. Pins at least one must-not-skip case per
# surface row, the crate fan-out and widening rules, and paths that git would
# C-quote without -z. Run from pr-checks.yml before the mapper is used.
set -euo pipefail
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
mapper="$repo_root/scripts/ci-surfaces.sh"
failures=0
# map PATH... : run the mapper on the paths as `git diff -z` would emit them.
map() {
if [ "$#" -gt 0 ]; then printf '%s\0' "$@"; fi | "$mapper"
}
# check CASE OUTPUT KEY=VALUE... : each KEY must have exactly VALUE.
check() {
local name="$1" out="$2" want key
shift 2
for want in "$@"; do
key="${want%%=*}"
if ! grep -qxF -- "$want" <<<"$out"; then
printf 'FAIL %s: want %s, got %s\n' "$name" "$want" \
"$(grep -E "^${key}=" <<<"$out" || echo '<missing>')" >&2
failures=$((failures + 1))
fi
done
}
# has_packages CASE OUTPUT PKG... : rust_packages must include each -p PKG.
has_packages() {
local name="$1" out="$2" pkgs pkg
shift 2
pkgs=" $(sed -n 's/^rust_packages=//p' <<<"$out") "
for pkg in "$@"; do
if [[ "$pkgs" != *" -p $pkg "* ]]; then
printf 'FAIL %s: rust_packages lacks -p %s (got%s)\n' "$name" "$pkg" "$pkgs" >&2
failures=$((failures + 1))
fi
done
}
nothing=(rust=false rust_packages= windows=false windows_packages= docker=false
audit=false core_deps=false release_versioning=false release_tag=false
codeql=false 'codeql_matrix={"include":[]}')
everything=(rust=true rust_packages=--workspace windows=true
'windows_packages=-p gl -p git-remote-gitlawb' docker=true audit=true
core_deps=true release_versioning=true release_tag=true codeql=true)
all_languages='codeql_matrix={"include":[{"language":"actions","build-mode":"none","runner":"ubuntu-latest"},{"language":"javascript-typescript","build-mode":"none","runner":"ubuntu-latest"},{"language":"rust","build-mode":"none","runner":"ubuntu-latest"},{"language":"swift","build-mode":"autobuild","runner":"macos-latest"}]}'
rust_only='codeql_matrix={"include":[{"language":"rust","build-mode":"none","runner":"ubuntu-latest"}]}'
# Docs-only and empty input run nothing.
check "docs only" "$(map README.md docs/x.md infra/x.tf)" "${nothing[@]}"
check "empty input" "$(map)" "${nothing[@]}"
# A crate change tests that crate and its reverse deps, nothing else.
check "gl source" "$(map crates/gl/src/main.rs)" \
rust=true 'rust_packages=-p gl' windows=true 'windows_packages=-p gl' \
docker=false audit=false "$rust_only"
out="$(map crates/gitlawb-core/src/lib.rs)"
check "gitlawb-core source" "$out" rust=true windows=true \
'windows_packages=-p git-remote-gitlawb -p gl' "$rust_only"
has_packages "gitlawb-core fan-out" "$out" gitlawb-core icaptcha-client \
gitlawb-attest git-remote-gitlawb gl gitlawb-node
check "icaptcha-client source" "$(map crates/icaptcha-client/src/lib.rs)" \
'rust_packages=-p gl -p icaptcha-client' 'windows_packages=-p gl'
# Anything the Rust surface cannot attribute to known crates widens.
check "unknown crate dir" "$(map crates/no-such-crate/src/lib.rs)" \
rust=true rust_packages=--workspace docker=false
check "root Cargo.lock" "$(map Cargo.lock)" \
rust=true rust_packages=--workspace docker=true audit=true core_deps=true \
release_versioning=false
check "toolchain file" "$(map rust-toolchain.toml)" rust=true rust_packages=--workspace
# Files compiled into a crate from outside crates/.
check "bootstrap-peers.json" "$(map bootstrap-peers.json)" \
rust=true 'rust_packages=-p gitlawb-node' windows=false docker=true
# One must-not-skip trigger per non-Rust row.
check "Dockerfile" "$(map Dockerfile)" docker=true rust=false
check "crate manifest" "$(map crates/gl/Cargo.toml)" \
docker=true release_versioning=true 'rust_packages=-p gl'
check "audit config" "$(map .cargo/audit.toml)" audit=true rust_packages=--workspace
check "core allowlist" "$(map ci/gitlawb-core-allowed-deps.txt)" core_deps=true rust=false
check "release-please config" "$(map release-please-config.json)" \
release_versioning=true rust=false
check "release tag resolver" "$(map scripts/resolve-release-tag.sh)" release_tag=true rust=false
check "other workflow" "$(map .github/workflows/release.yml)" "${nothing[@]:0:9}" \
codeql=true 'codeql_matrix={"include":[{"language":"actions","build-mode":"none","runner":"ubuntu-latest"}]}'
check "npm" "$(map npm/package.json)" codeql=true \
'codeql_matrix={"include":[{"language":"javascript-typescript","build-mode":"none","runner":"ubuntu-latest"}]}'
check "swift" "$(map macos-app/Sources/App.swift)" rust=false codeql=true \
'codeql_matrix={"include":[{"language":"swift","build-mode":"autobuild","runner":"macos-latest"}]}'
# Editing the CI wiring runs every surface, exactly like --all.
for wiring in .github/workflows/pr-checks.yml .github/workflows/codeql.yml scripts/ci-surfaces.sh; do
check "wiring $wiring" "$(map "$wiring")" "${everything[@]}" "$all_languages"
done
check "--all" "$("$mapper" --all </dev/null)" "${everything[@]}" "$all_languages"
# Names git C-quotes without -z (non-ASCII, tab, quote, newline) still map.
for name in $'\xc3\xa9.rs' $'a\tb.rs' 'a"b.rs' $'a\nb.rs'; do
check "quoted name $(printf %q "$name")" "$(map "crates/gl/src/$name")" \
rust=true 'rust_packages=-p gl'
done
check "non-ASCII swift" "$(map $'macos-app/Sources/Caf\xc3\xa9.swift')" codeql=true \
'codeql_matrix={"include":[{"language":"swift","build-mode":"autobuild","runner":"macos-latest"}]}'
# Newline-delimited input (git diff without -z) is rejected, not misread.
if printf 'crates/gl/src/main.rs\n' | "$mapper" >/dev/null 2>&1; then
echo "FAIL newline-delimited input: mapper accepted it" >&2
failures=$((failures + 1))
fi
if [ "$failures" -ne 0 ]; then
echo "$failures ci-surfaces mapping check(s) failed" >&2
exit 1
fi
echo "ci-surfaces mapping: all checks passed"