* fix(gl): make doctor exit non-zero on Fail-class checks (#357)
gl doctor printed per-row status and 'Some checks failed' prose, but its
exit code was always 0. A diagnostic whose exit status cannot express
failure is a trap waiting for the first 'gl doctor && ...' to land
somewhere it should not (#357).
Three changes:
1. Re-tier the 'GITLAWB_NODE unset' check from Fail to Warn. The CLI's
--node flag defaults to https://node.gitlawb.com (PUBLIC_NODE) and the
CLI works fine without the env var, so an unset env is advisory, not
a failure. Without this re-tier the obvious 'return non-zero on any
failure' change would have flipped gl doctor to exit 1 on a stock
working install, which is exactly the regression the issue warns
against.
2. Route the exit-code decision through a new has_failures(&[Check])
helper: exit 1 if and only if at least one row is Fail-class.
Warn-class rows (iCaptcha offline, version drift, shell-alias
shadowing, GITLAWB_NODE unset) keep the process at exit 0 because
those conditions are still printed to the user and the obvious
'&& pipeline' pattern stays valid. Use std::process::exit(1) rather
than returning Err so anyhow's error frame does not duplicate the
user-facing summary that already prints.
3. Three unit tests pin the new predicate against the three regimes:
all-Ok, warn-only, and a single Fail row tripping the exit. The
warn-only case is the regression guard for the re-tier: if anyone
flips GITLAWB_NODE back to Fail, exit_predicate_is_false_for_warn_only
fails too, so the two halves of the fix cannot drift apart.
Verified end-to-end against the public gitlawb node:
* unset GITLAWB_NODE, missing identity/registration -> exit 1
* unset GITLAWB_NODE, full healthy install -> exit 0 (warn)
* missing git-remote-gitlawb -> exit 1
* fix(gl): pin doctor exit wiring and GITLAWB_NODE Warn tier
- Extract GITLAWB_NODE env classification into gitlawb_node_env_check()
so unit tests can pin the Warn tier directly; the previous
has_failures-only tests built their own Check::warn rows and could not
see a re-tier back to Fail.
- Add three unit tests for the extracted helper (unset/empty is Warn,
set is Pass, loopback is Pass).
- Add integration probe crates/gl/tests/doctor_exit.rs that drives the
real binary with a temp --dir and --node http://127.0.0.1:1 and asserts
exit 1. Deleting the std::process::exit(1) gate keeps the unit suite
green but breaks this probe, closing the gap reviewer noted.
Co-authored-by: review fix for #391
* fix(gl): align GITLAWB_NODE operator copy with PUBLIC_NODE
* fix(gl): dial the public node when GITLAWB_NODE is empty
Check 3 groups an empty env with unset (Warn), but clap still binds
args.node to "" and the connectivity probe dialed it verbatim, so a
GITLAWB_NODE="" install showed a Fail row and exited 1. effective_node_url
normalizes empty or whitespace-only input to PUBLIC_NODE before the
probe so empty matches unset on check 4.
---------
Co-authored-by: Ayush7614 <[email protected]>
* fix(node): bound federated repository aggregation
* fix(node): propagate federated peer truncation
* fix(node): Bound federation latency and preserve truncated peer pages.
* fix(node): complete federation review follow-ups
Register the bounded peer-query fixture in the writer ledger, name federation bounds, and share the state-owned limiter across routers and the cleanup sweep. Cover exhausted-bucket rejection before database access.
* fix(node): pin federated response contract end to end (#466 review)
Address beardthelion's P2: the federated envelope was only exercised at
unit level, so hardcoding "truncated": false, deleting the failed-peer
truncated flag, or dropping node_did from enrich_federated_repo each left
all nine federated tests green.
Add four route tests driving build_router and asserting on the real HTTP
200 body:
- envelope fields (count, nodes_queried, truncated) and local annotations
(node_url, node_did, local) for a seeded public repo
- truncated == true when the peer table overflows MAX_FEDERATED_PEERS
(unreachable peer URLs, no live peer needed)
- truncated == true on the failed-fetch path with the table under the bound
- peer annotations (node_url, node_did, local: false) via a mockito peer
* fix: strengthen federated response bound tests per review
- db/mod.rs: add writer-ledger entries and disposition docs for the
three federated route tests touching the peers table
- api/repos.rs: seed the envelope test repo under a non-node owner DID
so node_did must equal the actual node DID; pin the outbound peer
query to exact limit=200&offset=0; make the peer-table-overflow test
use a reachable stub for every peer so the count bound is the only
possible reason for truncated=true, and assert nodes_queried
---------
Co-authored-by: euxarist <[email protected]>
Co-authored-by: Marlowe <[email protected]>
created= is attacker-supplied i64 with no range clamp. now - created
overflows on i64::MIN (panic in debug, wrap in release), and the wrap
case made abs() return i64::MIN, which is not > 300, so a skew of 2^63
seconds read as fresh. Treat an unrepresentable difference as the
largest possible skew.
* fix(node): bound Identify-derived Kademlia addresses.
* fix(node): preserve refreshed Identify addresses.
* fix(node): Log rejected known-peer addresses for diagnosis.
* fix(node): bound Identify report processing and refresh eviction
Avoid retaining peers with empty or rejected address reports, bound canonicalization to a fixed input prefix, and move refreshed addresses behind stale addresses in global eviction order. Add regressions for each bound and below-cap growth.
* fix(node): address CodeRabbit majors on Identify address bounds
- Reject Identify address admissions once the global address budget
is full instead of evicting another peer's entry. Cross-peer
eviction let a flood of new identities push honest peers'
addresses out of the book and, when the evicted address was their
last one, out of the Kademlia routing table. Per-peer eviction of
the reporting peer's own addresses is unchanged.
- Re-add refreshed Identify addresses to Kademlia, not just newly
added ones. Kademlia may have dropped a retained address after a
failed dial or never kept it for a full k-bucket, and a TTL
refresh alone would never re-admit it.
* fix(node): address #462 review findings from beardthelion
- remove the write-only insertion_order bookkeeping (field,
remove_insertion_token, and its call sites) and collapse the repeated
peers.get/get_mut lookups in IdentifyAddressBook::update into a single
entry binding
- drop a peer row at the end of update() when every candidate was
rejected, so zero-address rows no longer linger until the expiry tick
- add a regression test proving per-peer eviction still applies at full
global budget (fails if the global check is hoisted above eviction)
- extract the explicit-address retention predicate into the tested
is_explicit_address helper used by both Kademlia removal sites
* fix(node): pin row-removal and expiry guards in Identify address book
Address beardthelion's round-two P3 findings on #462:
- Add test that a fully-rejected fresh peer at full global budget
leaves no peers row (pins the empty-row drop in update()).
- Add test that update() past IDENTIFY_ADDRESS_TTL reports the stale
address in changes.removed (pins expiry reporting).
- Add test that expire() drops rows emptied by expiry (pins the
sibling drop in expire_peer).
- Distinguish per-peer from cross-peer eviction in the saturation
test: insert a sentinel peer before the victim and assert its
address survives, so a global-oldest eviction policy would fail.
* fix(node): pin refresh predicate negative direction per beardthelion round 3
---------
Co-authored-by: euxarist <[email protected]>
Co-authored-by: Marlowe <[email protected]>
Co-authored-by: cairn-intern <[email protected]>
* fix(node): gate /ipfs/pins and /arweave/anchors behind authentication (#134)
Add auth rejection to list_pins and list_anchors handlers. The pin index
spans the entire node and would expose metadata for every object ever
pushed; anonymous callers must not see it.
- Require AuthenticatedDid extension in both handlers, return 401 when absent
- Add server.rs regression test for anonymous rejection through build_router
- Fix closed-pool tests to pass authenticated requests (test 503 path)
* fix(node): enforce authentication on /api/v1/ipfs/pins and /api/v1/arweave/anchors routes
* fix(node): format arweave routes for consistency in build_router function
* fix(node): gate scoped /api/v1/arweave/anchors reads on repository visibility
Closes the review follow-up on #134: a signed non-reader could pass the
auth-layer check on `?repo=<owner>/<name>` and obtain private-repo anchor
metadata (ref names, old/new SHAs, CIDs, irys tx ids). The auth check
proved only that the signature was valid; it did not bind the request to
the canonical repo-read policy.
* Parse the slug through `validate_repo_slug` (same helper the sync path
uses) and call `authorize_repo_read` before the SQL query. Missing
repos, quarantined mirrors, and signed non-readers all collapse to the
standard 404 (`repo_not_found`).
* The unscoped global listing stays auth-only (#121 narrowing; the wider
permissionless-identity enumeration is the #136 class, out of scope).
* Clamp the limit to `[0, 200]` so `?limit=-1` no longer reaches Postgres
as `LIMIT -1`.
Five new `build_router` regressions cover the full production contract
through the real `optional_signature` middleware: scoped unsigned 401,
scoped owner 200, scoped non-reader 404 with no anchor metadata leaked
in the body, scoped missing-repo 404 indistinguishable from non-reader,
and `?limit=-1` clamped to an empty 200.
* fix(node): keep /api/v1/ipfs/pins out of the CID resolver's per-IP flood brake
Follow-up to #134: the previous reordering put `/api/v1/ipfs/pins` and
`/ipfs/{cid}` under the same `rate_limit_by_ip` + `IpRateLimiter` layer
stack, so both routes shared the resolver's flood brake bucket. That
bucket is documented as the once-per-request brake for `GET /ipfs/{cid}`
because the resolver can drive bounded-but-expensive repo walks. The
pin listing is a single `list_pinned_cids()` call and has no reason to
share the bucket. Sharing meant /ipfs/{cid} traffic could exhaust the
bucket and 429 the pins endpoint, and signed pin polling could exhaust
the bucket and 429 legitimate CID reads.
Compose `ipfs_routes` from two sub-routers with independent policies,
then merge:
* `/ipfs/{cid}` keeps `optional_signature` + `rate_limit_by_ip` + the
`IpRateLimiter` extension. Same behavior as before #134.
* `/api/v1/ipfs/pins` gets only `optional_signature`. Auth-required
(handler denies with 401), but no rate limit — it has nothing to
flood.
Two new `build_router` regressions exercise the bucket independence
through `TrustedProxy::XForwardedFor`:
* 5 signed pin requests against a size-2 CID bucket all 200, then a
`/ipfs/{cid}` from the same IP also 200 (pins did not debit the
bucket).
* A `/ipfs/{cid}` exhausts a size-1 CID bucket (200, then 429), then
a signed pin from the same IP is 200 (the bucket does not gate
pins).
All 65 tests in `server::tests::` + `test_support::tests::ipfs_*`
still pass, including the existing 429 tests for the CID resolver.
* fix(node): normalize scoped anchor SQL filter to stored repo slug
After authorize_repo_read succeeds, the SQL filter used the raw ?repo=
string (format!("{owner}/{name}")). Anchor rows are stored with
normalize_owner_key(owner_did)/name (short key), so a caller using
?repo=did🔑.../name passed authz but got anchors: [] — a false
empty page that broke callers that legitimately pass the full DID
form. Build the filter from record.owner_did normalized and
record.name instead, matching the shape the seeder uses.
* All three findings addressed on bug_fix_2:
P2 — signed_scoped_anchors_full_did_form_matches_stored_slug_through_build_router now seeds
a second repo under a different owner and asserts irys-other-repo-tx is absent. Negative
control: flipping the production filter to None made the test go RED with count: 2 showing
the cross-repo leak.
P3 — let short = owner_did.split(':').next_back().unwrap() replaced with let short =
crate::db::normalize_owner_key(&owner_did).to_string(). Reader and writer now share one
convention.
P3 — PR #134 description updated: pins bullet now describes the actual pre-merge layering
(and cites the axum-layer-vs-merge-pitfall memory entry), and round 2's stored-slug
normalization is added as its own bullet with the regression's coverage.
5/5 anchor-related tests pass, cargo fmt --check clean.
* fix(node): pass SignatureBodyLimit state to optional_signature layers
* test(node): complete listing-denial regressions for the new gates
---------
Co-authored-by: Gravirei <[email protected]>
* fix(node): owner-gate create_task repo_id against hosted repos (#496)
* fix(node): address review on #496 gate — opaque lookup errors, no-persist asserts, GraphQL arm pins, honest oracle comment
* fix(node): address P3 review on #496 gate — shared denial text, honest oracle wording, db-fault pins
- Give the REST and GraphQL copies of the create_task owner gate one denial
sentence: add REPO_OWNER_ONLY_MESSAGE, used by require_repo_owner and the
REST forbidden() envelope, and pin it from both surface tests.
- Carry the corrected one-bit-oracle wording into the GraphQL gate comment and
the REST test docstring (drop the "oracle-free" claim).
- Fault-inject the gate's repo lookups: drop machine_id (get_repo_by_id) for
REST and GraphQL, and quarantined (is_repo_quarantined) for REST, asserting
the fixed opaque db_error envelope and no schema text.
* docs(node): stop claiming caller-scoped task reads in #496 gate comments
The create_task gate is a write gate. Unknown/quarantined repo ids are stored
verbatim, and the REST/GraphQL task-list paths are caller-independent on this
base, so the comments must not imply those ids are read-scoped to the
delegator/assignee. Describe only the enforced ownership check; note the
one-bit oracle split without asserting a read contract.
* fix(node): mirror AppError's 503 split and opaque INSERT in create_task
- db_error now sends pool-class failures to 503 db_unavailable like
AppError::Db (db_unavailable made pub(crate)); query faults stay the opaque
500. Unit tests cover the split and context-wrapped sqlx errors.
- Route the create_task INSERT through db_error instead of serializing
e.to_string(); a REST fault-injection test pins the opaque envelope.
- The deny/allow matrix reads back the unknown and quarantined tasks to prove
the id is stored verbatim, and drops no-leak asserts on the fixed forbidden
envelope that could never carry the repo id.
* test(node): pin stored-verbatim repo_id on GraphQL gate arms
The REST deny/allow matrix reads each accepted task back and asserts the
stored repo_id is the supplied id; the GraphQL unknown/quarantined arms only
asserted no errors, so a GraphQL regression that dropped, normalized, or
rebound the id stayed green. Read those tasks back by the mutation's returned
id and assert the stored label matches.
* test(node): pin createTask repo-ownership marker in the GraphQL authz fence
---------
Co-authored-by: Gravirei <[email protected]>
* fix(api,sync): Delimit positional arguments in git clone mirror invocations
Pass "--" before positional path/URL arguments in Command invocations
for "git clone --mirror" across repos.rs, sync.rs, and gl mirror.rs.
Fixes#374
* style(gl): Format args array in mirror.rs
Fix cargo fmt line-length break for git clone --mirror arguments.
Refs #376
* fix(node): Pin the git clone delimiter against remote-argument injection
The delimiter change had no test, so nothing stopped it being reverted as
cosmetic. What it guards is real: git parses a remote beginning with a
dash as an option, and --upload-pack=<cmd> is one git hands to a shell.
No path reaches that today. Peer URLs are gated by is_public_http_url, so
the origin clone_repo composes can only begin with http:// or https://;
fork paths are joins under repos_dir; and clap rejects a leading-dash
positional in gl mirror. The delimiter is what removes git's dependence
on those three unrelated gates staying correct.
Assert git reports the whole argument as a repository it cannot find,
which is only true once the delimiter forces it to be a path.
Undelimited, git consumes it as an option and fails against the
destination instead, never naming the injected string, so the assertion
is red before the fix and green after.
Refs #374
* fix(sync,gl): Delimit positional arguments in fetch_repo and gl clone
Carry the positional argument delimiter into fetch_repo when setting the
remote origin URL and into setup_partial_clone across all clone branches.
Correct the error string quoted in the clone regression test docstring.
Refs #374
* test(gl,node): Pin the argv delimiter on every git sink this PR changed
The delimiter contract is duplicated across five direct `git` argv
constructions, but only `sync::clone_repo` had a test that went red when
`--` was deleted. Removing or misplacing the delimiter in any other sink
left the suite green, so the hardening rested on review rather than on
the tests.
Each changed sink now has a subprocess-facing regression driven through
real git: the three `setup_partial_clone` shapes (plain, `--branch`, and
the sparse/promisor arm), `fetch_repo`'s `remote set-url`, `gl mirror`,
and the fork clone.
Two of these could not carry the obvious assertion:
- `git_global` and `git_run` both format failures as
`git {args:?} failed: {stderr}`, so the injected value is already in
the message via the debug-printed argv. Asserting that the error
merely contains it passes with `--` deleted. The clone tests assert on
git's own single-quoted `repository '<value>'` instead, which only
appears when git read the value as a repository.
- `remote set-url` does not fail on a delimited option-shaped value, it
stores it. The `fetch_repo` test asserts the stored URL rather than an
error: undelimited, git exits 129 with `unknown option` and leaves the
previous URL in place.
`gl mirror` and `fork_repo` needed their argv extracted to be testable.
`mirror::run` loads a keypair and contacts a node before reaching the
clone, and bails with a message that interpolates the source whether or
not the delimiter is present. The extraction keeps the production
`.status()` call, so a large mirror clone still streams git's progress
to the terminal instead of being captured.
Every one of the five was confirmed red with its delimiter removed and
green with it restored.
Refs #374
* fix(gl): Route the last three clone warnings through emit_warning
`emit_warning` sanitizes a line before writing it, which strips ANSI
escape and bidi characters from untrusted text. Three warning sites in
the blob-recovery paths still used a bare `eprintln!` while interpolating
`oid`, which comes from node JSON or an Arweave gateway manifest, so
those escapes reached the user's terminal raw.
Refs #374
* style(gl): elide the lifetime in mirror_clone_args
* test(gl,node): pin LC_ALL=C on git-subprocess stderr assertions
---------
Co-authored-by: euxarist <[email protected]>
* docs: state that GITLAWB_ENFORCE_OWNER_PUSH defaults to true
PR #330 flipped the code default to true and updated .env.example, the
README env-var table, the AWS guide, the macOS app, and docs/RUN-A-NODE.md,
but two prose mentions in operator-facing docs still describe the
opposite. SECURITY.md line 57 ("Repository write authorization defaults"
section) and README.md line 69 (Known limitations) both say the flag
"defaults to `false`", which contradicts the live runtime and the rest
of the same files. Reviewer 2 closed#224 with this as the P2 finding
to fix in the config-and-policy split.
The correct text already exists in the README env-var table; the prose
restatements were missed. Bring the prose into line with the code.
- SECURITY.md: rewrite the section to lead with the default being true
and describe `false` as the explicit rolling-upgrade override.
- README.md: rewrite the known-limitations bullet to state the default
is true and the `false` opt-out is a rolling-upgrade only.
- Add `crates/gitlawb-node/tests/owner_push_default_docs.rs`: a
regression guard that reads the four surfaces (clap default, .env.example,
README known-limitations prose, README env-var table, SECURITY.md) and
asserts each one says the default is `true`. Reverting either doc edit
turns one named test red; flipping `default_value_t` back to `false`
turns the env example test red. Each surface is checked independently
so a stale doc lies alone.
No code change. No new migration. No dependency change. Compiles clean,
clippy clean under `-D warnings`.
Refs: closes the P2 finding from the review of #224.
* fix(docs): clarify default value consistency for GITLAWB_ENFORCE_OWNER_PUSH
* docs(readme): drop the default-open roadmap line and the em-dash punctuation
---------
Co-authored-by: Gravirei <[email protected]>
* fix(gl): truncate node-supplied display strings at a char boundary (#350)
Every timestamp/id/SHA the CLI prints was cut with &s[..N] or
&s[..N.min(s.len())]. The first panics on a short value, the second
still panics when byte N is inside a multi-byte char. A shared
text::truncate cuts at a char boundary and covers all sites, including
two the issue did not name (node events, agent did) and the same shape
in bounty, changelog, ipfs pin listing, and the merge-commit print.
* fix(gl): bound truncate by bytes, not chars (#460)
Address PR review feedback: char_indices().nth(max) capped characters,
so a multibyte char near the cut could exceed the byte limit. Start at
max.min(len) and walk back to the nearest char boundary instead.
* fix(gl): bound the bounty leaderboard did suffix at a char boundary
gitlawb-core added the same guard in 73bf132c; the second copy of the
parser in gitlawb-attest never got it, so an untrusted signer string of
arbitrary length reaches a decoder whose cost is quadratic in input.
* fix(icaptcha-client): parse GITLAWB_ICAPTCHA_INSECURE as truthy only
Presence checks treated =0/=false/empty as enabled. Only explicit 1/true
now enable loopback HTTP trust relaxation (#227).
* fix(icaptcha-client): address #246 review (trim test, fmt, lock)
Pin trim with a whitespace truthy case, recover from a poisoned
ICAPTCHA_ENV_LOCK, document trim + warn visibility, and cargo fmt.
* fix(icaptcha-client): do not log raw ICAPTCHA_INSECURE value
The non-truthy warn path previously recorded the full trimmed env
value. That is arbitrary deployment input; report only that the
setting is invalid (#246 review).
* fix(icaptcha-client): pure insecure flag helpers for #246 review
Warn on present non-Unicode GITLAWB_ICAPTCHA_INSECURE (fail closed).
Pass insecure mode into is_https_with / resolve_solver_url_with so
tests no longer mutate process-global env (removes InsecureEnv race).
* fix(icaptcha-client): cover real env read + fix warn spacing
Add a serialized test that drives icaptcha_insecure_enabled() against
the process env so presence-check / typo / always-true mutations fail.
Collapse the mid-sentence space run in the non-truthy warning (#246).
* fix(icaptcha-client): put the non-truthy warning on one literal line
---------
Co-authored-by: Ayush7614 <[email protected]>
Co-authored-by: Kevin Codex <[email protected]>
* fix(attest): make verify_all lenient under RequireAll (#359)
Registry::verify_all propagated every per-attestation error via ?, so a
single malformed or unknown attestation aborted the whole batch. Under
Policy::RequireAll that is a DoS vector: any third party can attach an
attestation, so an unrelated attacker/spam/v1 could block an otherwise
valid cert.
Under RequireAll, drop entries that fail to verify instead of aborting;
the required_types check remains the gate and only counts fully_verified
entries, so a required type present only as a malformed attestation still
fails with RequiredMissing. AcceptKnown and RejectUnknown stay strict.
Adds tests for both: a malformed extra is dropped without aborting, and a
malformed-only required type is still rejected.
Closes#359
* test(attest): lock the strict batch policies with an error-propagation test
AcceptKnown and RejectUnknown were only exercised per-attestation, so a
cleanup applying RequireAll's filter_map(Result::ok) shape to the strict
arms would fail open with every test green. Drive both policies through
verify_all with a wrong-cert-hash member and assert the batch surfaces
the underlying error. Also correct the RequireAll doc: unknown valid
types are retained as unverified, not dropped — only real Err paths are.
---------
Co-authored-by: andreolf <[email protected]>
Co-authored-by: Kevin Codex <[email protected]>
* fix(node): canonicalize and 404-on-no-match in remove_label (#344)
add_label stored labels trimmed and lowercased, but remove_label passed the
path segment through as-is, so a label stored as 'bug' did not match
DELETE /repos/{owner}/{repo}/labels/Bug. Worse, remove_label's DB call
discarded the row count and the handler always returned 200 with
{removed: true}, including when the delete matched nothing — the
no-op-rendered-as-success shape the rest of the API rejects.
Move trim+lowercase and charset/length validation into a shared
canonicalize_label helper used by both handlers, and have
db::remove_label return the affected row count so the handler can return
404 when no label was deleted. add_label's behavior is unchanged
(idempotent semantics preserved); the only externally observable change is
that mixed-case and surrounding-whitespace path segments now match the
stored label, and a delete that matches no row returns 404 instead of
{removed: true}.
Tests:
* remove_label_canonicalizes_and_404s_on_no_match exercises the
mixed-case path, the trimmed path, the empty-after-trim rejection,
the disallowed-charset rejection, and the no-op-returns-404 case.
* remove_label_denies_non_owner re-pins the 403 owner-gated shape
(matches AGENTS.md: owner-gated mutations return 403 after the repo
lookup, distinct from the not-found shape).
* test(node): pin len>50 arm in remove_label canonicalize
The helper's length branch had no coverage — 51 chars must 400 and
50 chars must pass validation (404 when absent). This pins the arm
lifted into canonicalize_label and decouples the no-match 404 from the
double-delete above, as requested in review of #390.
---------
Co-authored-by: Ayush7614 <[email protected]>
* fix(node): gate id-keyed bounty mutations on repo read before status checks
submit, approve, cancel, and dispute on /api/v1/bounties/{id}/* looked the
bounty up and ran status and participant checks without gating on the
bounty's repo read visibility. A signed stranger holding an id got a
distinguishable 400/403 on a private-repo bounty where an absent id gets
404, so the routes were an existence and lifecycle oracle.
Insert the same authorize_repo_read block get_bounty and claim_bounty
already carry, right after the lookup, so a denied caller sees the
not-found shape either way. Participant checks are unchanged; on a public
repo a stranger still reaches them.
Closes#341
* Address PR review feedback (#448)
- assert the exact not_found body shape for both held and absent bounty
ids in the mutation-gate helper, so a denial carrying lifecycle or
participant detail fails the test
* fix(node): fail pin_git_object when the add response carries no Hash
A 200 from /api/v0/add without a parseable Hash fell back to the
caller-computed raw CID, recording an address the backend never
confirmed stored. For payloads above the backend's chunk threshold
(262144 bytes on Kubo) that raw CID is not even a stored block, so
every later /api/v0/cat on it fails permanently: get_encrypted_blob
errors on every fetch and mirrors retry the envelope on every sync.
A Hash-less 200 is now an add error carrying a truncated body excerpt.
The delaying_endpoint test fixture, which relied on the fallback by
returning an empty 200 body, now answers with a conformant NDJSON line.
* fix(node): reject a non-CID Hash in the add response
Address PR review feedback (#453): a string-valued Hash that does not
parse as a CID would be recorded into encrypted_blobs.cid and fail on
every later cat, the same failure class as a missing Hash. Validate the
returned Hash before accepting it and cover the malformed case with a
regression test.
* test(node): return real CIDs from the add-response test mocks
The pin_git_object CID validation rejects non-CID Hash strings, and a
dozen test_support fixtures answered 200 with placeholder values like
"bafyprovider" or "x". Replace them with a valid CID so the fixtures
model a healthy backend; no test asserts on the literal value.
* fix(node): bound the invalid Hash echoed in the add error
* fix(#477): scope bounty aggregates to anonymously-listable repos
bounty_stats and agent_bounty_stats ran unfiltered aggregates over all
bounties, leaking private-repo bounty activity to anonymous callers.
Mirror the stats() pattern from server.rs (#104):
1. Batch-load all deduped repos + visibility rules (2 SQL round-trips)
2. Filter to listable_at_root(rules, is_public, owner_did, None)
3. Pass visible (owner, name) pairs into SQL aggregates via EXISTS/unnest
with owner key normalization on both sides (BOUNTY_OWNER_CASE_SQL).
4. Normalize claimant DID in agent_bounty_stats_visible and group by
normalized claimant in bounty_leaderboard_visible (BOUNTY_CLAIMANT_CASE_SQL).
5. Remove orphaned unfiltered aggregate methods to pass dead_code lints.
6. In tests, seed completed bounty claimant state via the real claim/approve
path, and verify root-rule-decided visibility overrides.
Aggregates are intentionally anonymous-scoped for all callers (matching the
#104 stats pattern).
Fail-closed: DB errors collapse the visible set to empty, so all counts
return 0 — an under-count never leaks existence.
Fixes#477
* style(node): apply rustfmt to the bounty visibility change
* test(node): pin bounty aggregate visibility for mirror rows
* docs(node): record why mirror rows are handled in the bounty visibility filter
* perf(node): keep the claimant index usable in per-agent bounty stats
* fix(node): return DB errors from bounty stats instead of zero counts
---------
Co-authored-by: Mystic <[email protected]>
* fix(node): resolve issue ids from git refnames, not request input
resolve_issue_id answered an exact match by echoing the request-path id
back to callers, so get_issue and close_issue spliced caller input into
`git cat-file blob refs/gitlawb/issues/<id>`. #521 restricted the input
to [A-Za-z0-9-], but the value handed to git was still the request
string (CodeQL rust/command-line-injection #83).
Resolve exact and prefix lookups through one for-each-ref listing and
return the refname git reports. An exact refname still wins over longer
ids that share it as a prefix; unique-prefix, ambiguous, not-found, and
revision-syntax behavior is unchanged.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(node): keep a dangling exact issue ref from winning over longer ids
for-each-ref lists a ref even when its blob is missing, so the exact-id
precedence added in the previous commit picked a dangling exact ref over a
readable longer sibling: resolve_issue_id returned the dangling id where
main reported an ambiguous prefix, get_issue then returned None, and
close_issue hit its expect().
Let the exact id win only when `git cat-file -e` finds its object, run on
git's own refname so no request input returns to the argv. Otherwise fall
through to the unchanged prefix rules, matching main for every layout.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
Resolve Dependabot alerts for the ANS-104 golden-vector tooling in
scripts/:
- follow-redirects 1.16.0 -> 1.16.2 (lockfile only, within axios's
^1.16.0 range): GHSA-mrwp-f3q8-3g9p, GHSA-q3jh-wrc6-jgq8,
GHSA-8r9p-f939-6h3c.
- ws 8.18.0 -> 8.22.1 via an npm override. @ethersproject/providers
5.8.0 (latest v5) pins ws to exactly 8.18.0, so a lockfile bump
alone cannot move it: GHSA-96hv-2xvq-fx4p, GHSA-58qx-3vcg-4xpx.
elliptic (GHSA-848j-6mx2-7j84, low) has no patched release and is
left as is.
Co-authored-by: Claude Opus 5.5 <[email protected]>
* ci(release): gate publish jobs on the release environment, anchor the tag check, pin npm
Closes#234. workflow_dispatch runs the selected ref's YAML and npm trusted
publishing matches on the workflow filename, not the ref, so a dispatched
non-main ref could reach the OIDC publish with no approval. Declare
environment: release on all three publish jobs (npm-publish, docker,
docker-manifest) so the environment's deployment-branch rule gates every
publish surface, not just the OIDC one.
Also close the two residuals the issue carries: the release-tag case glob
was right-unanchored (accepted v1.2.3.4.5, v1_x, v1-any-branch-name, bare
v1), now an extglob v+([0-9]).+([0-9]).+([0-9]); and the OIDC publish ran
npm@^11.5.1, a floating range, now pinned to [email protected].1.
The environment itself must exist with a deployment-branch rule limited to
main (and required reviewers if desired) for the gate to bite; npmjs.com
can also scope each package's trusted publisher to environment "release"
for a registry-side check that survives a workflow edit.
* ci(release): qualify tag checkouts and keep moving tags forward-only
Closes#271, closes#238.
actions/checkout resolves an unqualified ref as a branch before a tag, so
a branch named v0.7.1 shadows the tag and the release builds unreviewed
content. Qualify all four release-tag checkouts as refs/tags/..., which
takes checkout's unconditional tag path.
docker-manifest applied :latest and :X.Y unconditionally, so a backfill
of an older tag repointed them at older code. The manifest now always
publishes the immutable :X.Y.Z and applies the moving tags only when the
resolved version is not older than the registry's latest release. The
same shape existed on the npm side: publish moves the latest dist-tag to
whatever it publishes, so a backfilled older version would have stolen
it; older backfills now publish under the backfill dist-tag instead.
The manifest also asserted nothing about the digest set it stitches, so
a wrong-arity or misnamed artifact would ship under :latest. Assert two
digests of 64 lowercase hex chars before imagetools create.
Extend the resolver test to pin both publish steps byte-for-byte and to
reject any unqualified tag-checkout ref, so these guards cannot regress
silently.
* ci(release): gate every job on the release environment and verify tag provenance
Two residuals from review of the dispatch-publish surface:
- environment: release only covered the three registry-publish jobs. Declaring
it on all eight jobs means a dispatch of unmodified YAML on a non-main ref
is denied by the environment's deployment rules before any step runs.
- The resolver shape-checked vX.Y.Z but did not prove the tag was produced by
the release flow. A write-access user could plant a v99.99.99 tag on
unreviewed content and dispatch a backfill against it. In CI (GH_TOKEN
present) the resolver now requires a GitHub release for the tag and a
main...tag compare status of identical or behind, so the tag's commit must
already be reachable from main.
* ci(release): qualify the provenance compare and fail closed on npm lookup errors
Second adversarial pass found two residuals:
- The provenance check compared main...<tag> with an unqualified refname,
the same branch-shadows-tag ambiguity the refs/tags/ checkout prefix was
added to avoid. Resolve the tag through commits/refs/tags/<tag> to a
commit SHA and compare that instead.
- npm view ... || true mapped every lookup failure to "no latest", so a
registry outage during an older backfill would publish under --tag
latest and move the dist-tag backward. E404 still means first publish;
any other failure now aborts before a dist-tag is chosen.
* ci(release): check release author, decide npm dist-tag per package, close the test gap
Third adversarial pass:
- Provenance now also requires the release to be authored by the release
automation account, so a hand-created release on a valid tag does not
qualify for backfill.
- The npm dist-tag decision is per package. A partial publish can leave
platform packages ahead of the wrapper, and a single lookup on
@gitlawb/gl would then downgrade an ahead package's latest tag.
- The environment gate test now emits one line per job and marks jobs with
no declaration MISSING, so a job added without environment: release can
no longer pass silently.
* ci(release): anchor docker moving tags to the registry floor, not the release pointer
GitHub's latest-release pointer is operator-mutable and is not a bound on
what :latest serves. List the package's versions on ghcr and refuse to
move :X.Y or :latest below the newest X.Y.Z tag there. A 404 means the
package does not exist yet, so the first publish applies all tags; any
other lookup failure aborts. The version filter is grouped with || true so
an empty list does not trip pipefail.
* ci(release): require backfill release assets to be automation-uploaded
The npm backfill republishes GitHub release assets, which are mutable for
anyone with contents: write. Require every asset on the release to have
been uploaded by github-actions[bot], matching the release-author check.
Not a hard bound, but it removes the direct edit path and leaves a review
signal when a hand-uploaded asset blocks a backfill.
* ci(release): pin the verified commit and the exact asset bytes end to end
The provenance checks ran at resolve time but the consumers re-resolved
mutable names afterward, leaving a move-the-tag or replace-the-asset race:
- Checkouts now use the commit SHA the resolver emitted (tag_commit),
falling back to the qualified tag ref.
- The resolver captures the release's name/id asset pairs once, checks all
uploaders against them, and the lay-in step downloads by asset id. An
asset deleted and reuploaded between resolve and download gets a new id
and fails closed.
- The npm-publish comment now names the third trusted-publisher component
(environment), which is what binds the OIDC token to the gate.
* ci(release): run the resolver on the producer path and hash homebrew inputs locally
- release-binaries and web-sync now resolve the release tag and check out
the verified commit SHA, closing the tag-move race on the job that
produces the release assets, not just the jobs that consume them.
- homebrew-bump resolves the tag, downloads the tarballs by captured asset
id, and computes sha256 locally instead of trusting the mutable
*.tar.gz.sha256 sidecar assets.
- The step pin now covers the lay-in, formula, and release-tag checkout
blocks so edits to the trust boundary force a deliberate fixture update.
* ci(release): bind the tag to the commit its release was created against
Reachable-from-main alone accepts a tag moved to any other main commit.
The release object records its creation-time target_commitish, so require
tag_commit == release.targetCommitish: a moved tag now fails closed, and
the SHA emitted for checkouts is provably the released commit.
* ci(release): force bash for the release-binaries resolver step
The matrix includes windows-latest where run: defaults to pwsh, and the
resolver body is bash.
* ci(release): base64 release-asset names in the resolver map
Release asset names are attacker-controlled text. A crafted name
carrying whitespace could smuggle a fake uploader into the third
positional field and pass the automation-only check. Emitting names
base64-encoded keeps each record in three real fields and makes the
name-to-id lookup immune to injection.
* Address PR review feedback (#447)
- floor the docker minor-line tag at its own X.Y max so an
in-line-newest backfill advances :X.Y without moving :latest
* Address PR review feedback (#447)
- serialize all release runs under one concurrency group; the
ref-keyed group let dispatches from different refs interleave
* ci(release): freeze docker moving tags on dispatch, retire the registry floor
* ci(release): make the resolver test's gh stub discriminate on the full call
* ci(release): assert the pinned npm version took effect
* ci(release): surface missing-asset errors from the formula sha() subshell
* ci(release): consume resolver outputs instead of release-please outputs post-resolve
* ci(release): document the concurrency group's pending-slot replacement
* ci(release): share the image output and bound the step scanners to jobs
The docker-manifest rel step now emits image= the same way the docker
job's does, and both manifest steps consume steps.rel.outputs.image
instead of recomputing the lowercase ghcr path.
The three whole-workflow awk scanners now ignore everything before
jobs: and flush at job boundaries, so an `on.push.branches` list item
can no longer open a pseudo-step that swallows top-level YAML, and a
step record can no longer absorb the next job's header lines.
* ci(release): close the review-found evasions in the release checks
Review of the follow-up diff found the structural assertions could be
satisfied or evaded without the behavior they guard:
- the gh stub now pipes a fixture response through real jq with the
caller's own -q on every arm, so a resolver that hardcodes its
expected fields is detected
- the moving-tag scan keys on the outcome (a push mechanism plus a
mutable-tag reference in decommented step text) and requires the
event+ref guard on the carrier's if: line, instead of binding one
argv spelling and accepting comment-borne guard text
- a new assertion pins the on.push.branches / workflow_dispatch
triggers, which no step-level check could see
- the npm check detects install/i and -g/--global variants, requires a
non-empty install set, and reads the pin comparison from code, not
comments
- the stale-reads scan covers first step lines, job-level env/if lines,
and jobs without a resolver step
- the resolver test asserts the GHAE heredoc terminator, clears ambient
GH_TOKEN/GITHUB_REPOSITORY on non-stub calls, and prints the captured
resolver stderr when a pass-expected case fails
- sha() exits on a failed download, since errexit is off inside its
command substitution and a truncated file would otherwise be hashed
* ci(release): floor the moving tags against the releases list
The event gate alone cannot bound a stale run: a re-run of an older
push run replays its stored needs outputs under the original event and
ref, and would move :latest and :X.Y backward to that run's version.
The gated step now derives the newest published release from the live
releases list (contents:read, drafts and prereleases excluded) and
refuses when this run's version is not the maximum. The check reduces
to VERSION == max, because the run's own release already exists by the
time the step runs.
The all-steps assertion now requires the carrier to carry the floor as
well as the event+ref guard, and the red-check spec gains a
gated-but-floorless carrier mutation to prove the clause binds.
* ci(release): pin the rebranded Homebrew formula text in the resolver-steps test
* ci(release): keep npm stderr out of the dist-tag lookup
The Publish step captured `npm view ... dist-tags.latest` with 2>&1, so a
warning on stderr from a successful lookup became part of registry_latest and
could make a new release compare as older than the registry latest, publishing
it under the backfill dist-tag instead of latest.
Send stderr to a temp file and read it only when the lookup fails. E404 still
means the package was never published, and any other failure still fails
closed. The pinned copy of the block in the resolver test is updated to match,
and a stubbed-npm test covers a warning on success, a clean lookup, a genuine
backfill, E404 and a non-E404 failure.
* ci(release): bound the moving tags against every release page
The Move floating tags step floored a stale re-run against the semver maximum
of the first page of the releases list only. That list is ordered by creation,
not semver, so once history passes one page the true newest version could sit
on a later page and a stale re-run would move :latest and :X.Y backward.
Read every page with gh api --paginate and take the semver maximum in bash.
The gh call stays its own command substitution so a failure on any page still
fails closed, and an empty list still yields no floor. The pinned copy in the
resolver test is updated to match, and a stub-gh test serving two pages covers
a stale run whose newer release is only on page 2, the run that is the newest,
a gh failure, and an empty list.
* ci(release): fail closed when a release asset has no uploader login
The resolver ran its automation-only asset check only when the collected
uploader list was non-empty. A null uploader login made jq emit an empty third
field, the list collapsed to nothing, and provenance passed although no asset
was shown to be automation-uploaded.
Substitute a sentinel for a missing login so every asset row carries a value
and a null, missing, or login-less uploader fails the existing comparison. The
base64 name and id map is unchanged, and a release with no assets behaves as
before. The resolver test covers an all-null uploader, a null login, a missing
uploader key, a bot asset mixed with a null one, and the all-bot release.
* fix(node): ANS-104 transport, three-outcome probe, verify endpoint (#26 split 2/4)
Reviewer 2 closed PR #224 on 2026-08-28 with a directive: split
into four narrow PRs. This is Split PR 2 (Arweave transport and
verification).
The P1 finding the reviewer assigned to this split: the legacy
`anchor_item_present` mapped `BAD_REQUEST` and `GONE` to `Ok(false)`,
and the recovery code interpreted false as permission to reclaim the
row and pay for another upload. Neither 400 nor 410 proves a
previously paid item was never accepted: 410 can describe an
artifact that existed but is no longer served, and 400 can be
produced by gateway, proxy, or routing failure. The concrete
failure sequence the reviewer named: bundler accepts the item,
the response or terminal DB write is lost, recovery probes the
persisted item_id, the gateway returns 400 or 410, and the node
pays for a second immutable item for the same transition.
Fix: model the probe result as three outcomes, and authorize a
re-upload only on a trustworthy, protocol-defined absence.
NEW MODULE crates/gitlawb-node/src/ans104.rs
- DataItem: the on-wire ANS-104 shape (signature, owner, target,
anchor, tags, data — all base64url-encoded except target/anchor).
- deep_hash: the canonical signing input per the spec. The
signature is over a SHA-256 mix of the dataitem/list/map
discriminators, the signature type, owner, target, anchor, the
recursively-hashed tag list, and the data payload. A
hand-rolled sha256 of the JSON body would produce a hash no
Arweave gateway would recognize; the deep-hash is the only
correct form.
- sign_data_item: Ed25519 over deep_hash.
- verify_data_item: parses the owner, checks the signature against
an expected pubkey, and reports a specific failure reason on
each branch.
- 7 unit tests covering round-trip, flipped signature, wrong
expected key, mutated data, wire-shape round-trip, deep-hash
stability, and the empty-tags edge case.
NEW MODULE crates/gitlawb-node/src/arweave_v2.rs
- ProbeOutcome: Present (2xx, body parses as ANS-104, sig
verifies), DefinitivelyAbsent (404 with a known JSON body
shape), Indeterminate (everything else). The reviewer's
three-outcome model.
- ProbeOutcome::permits_reupload: the recovery policy. ONLY
DefinitivelyAbsent authorizes a paid re-upload. Indeterminate
keeps the outbox non-terminal; Present skips re-payment.
- probe_anchor_item: the gateway probe. 2xx bound to a different
item id is Indeterminate, not Present. 2xx with a body that
does not parse as ANS-104 is Indeterminate. Oversized bodies
are Indeterminate (defense against body-stuffing).
- read_capped_body: enforces PROBE_MAX_BODY_BYTES (1 MiB) on the
full body, with a Content-Length fast-path that rejects
before reading.
- verify_anchor: the full path the public verify endpoint takes.
Fetches the data item, parses it, verifies the Ed25519
signature against the persisted node_did, decodes the data
payload as JSON, and reports the result.
- 15 unit tests pinning each classification boundary:
400/410/5xx/transport/2xx-non-json/2xx-bad-sig/2xx-different-
owner are all Indeterminate; 404 with empty or known-JSON
body is DefinitivelyAbsent; 2xx with a valid signed item is
Present; the re-upload policy is exhaustive.
NEW DB METHOD crates/gitlawb-node/src/db/mod.rs
- get_arweave_anchor_by_item_id: looks up an existing
arweave_anchors row by its id (which is the Irys tx id, the
same value the Arweave gateway uses to serve the data item).
Used by the public verify endpoint to fetch the persisted
node_did so the envelope signature can be verified against it.
NEW CONFIG crates/gitlawb-node/src/config.rs
- GITLAWB_ARWEAVE_GATEWAY_URL: the Arweave gateway the probe
reads from. Defaults to https://arweave.net; operators can
point it at a private mirror.
NEW HTTP ENDPOINT crates/gitlawb-node/src/api/arweave.rs
- GET /api/v1/arweave/anchors/verify/{item_id}: the public
verify endpoint. Returns {item_id, status, verified,
owner_did, data_payload, error} where status is one of
"verified" / "definitively_absent" / "indeterminate".
- 3 endpoint-level tests covering the verified, 404, and 400
shapes via the live router, asserting the public surface
surfaces the three outcomes.
WIRE-UP crates/gitlawb-node/src/server.rs
- The verify route is registered alongside the existing
/api/v1/arweave/anchors list route.
NOT IN THIS SLICE (the bundler upload, the next commit):
- The handler that builds a DataItem from a ref-cert and POSTs
it to the bundler is not in this PR. The ANS-104 module is
ready to be called by it; the cert/push-event flow in PR 1
persists the row that the bundler upload would consume.
- PR 1's recovery drain produces an `anchor_jobs` row keyed on
the (repo, ref, old, new) tuple; the next slice reads that row,
builds the ANS-104 data item, POSTs to the bundler, and writes
the resulting tx id to arweave_anchors.
Compiles clean, 1113 tests pass with 0 regressions, clippy clean
under -D warnings, fmt clean.
Cross-PR overlap (declared in the PR description):
- #134 (anchors auth): composes. The verify endpoint reads from
arweave_anchors; #134's auth layer applies unchanged. The
probe model is server-side and unauthenticated by design.
- #285 (advisory-lock session affinity): independent. The probe
is read-only.
- #306 (Content-Digest on signed requests): independent. The
probe does not sign requests.
- #314 (small-order Ed25519): composes. The verify path calls
Did::to_verifying_key which already enforces the small-order
check from #314.
- #324 (libp2p keypair persistence): independent.
- #325 (gossip ref-update auth): independent. The verify path
is for the public Arweave gateway, not gossip.
- #382 (replication withheld-subtree trees): independent.
* fix(node): address reviewer findings on #26 split 2/4 (transport)
- P1 (DB lookup): get_arweave_anchor_by_item_id now filters on
irys_tx_id (the externally-routable item id) rather than the
internal UUID primary key. The production writer stores a fresh
UUID in `id` and the Irys response / ANS-104 derived id in
`irys_tx_id`; the old filter would always 404. New
integration tests exercise the production writer path with
distinct values for the two columns.
- P1 (verify-gate): verify_anchor now gates on repo read via
authorize_repo_read. The route is split in server.rs: the
public list_anchors stays ungated (#134 tracks that surface),
the verify route gets optional_signature per the team memory
`axum-layer-vs-merge-pitfall.md`. Denials on row-lookup, gate,
and missing repo all collapse to the same opaque 404 so the
public endpoint does not leak anchor-row existence.
- P1 (ANS-104 deep-hash): rewritten to the verified SHA-384
recursive list/blob algorithm (matches `arbundles` JS
reference). deep_hash returns [u8; 48]; signature is over the
raw 48-byte digest. Three reference vectors from
`Irys-xyz/arbundles/src/__tests__/deepHash.spec.ts` are
bit-exact-asserted as the interop canary per the team memory
`self-roundtrip-tests-do-not-prove-interop.md`.
- P1 (artifact-identity): v2 verify now derives the protocol id
via DataItem::id() (base64url(SHA256(signature))) and requires
equality with the requested URL item_id. A node key signs
many data items, so a valid signature only proves who signed
the response — not that the served item is the one the caller
asked to verify. The team memory
`verify-against-artifact-id-not-signer.md` is the policy.
- P1 (verifier can't read this head's writes): verify_anchor
now accepts BOTH the v2 ANS-104 format and the v1 raw-JSON
shape the live path on this branch actually writes. The v1
path does a field-equality check on repo, ref_name, old_sha,
new_sha, node_did against the persisted row (no signature in
v1; Irys storage plus the JSON parse are the integrity
guarantee).
- P2 (ProbeOutcome in result): AnchorVerifyResult now carries
outcome: ProbeOutcome. The HTTP handler maps the structured
field to the status string instead of parsing the human-
readable error message — any future wording change in
arweave_v2.rs can no longer silently reclassify a
DefinitivelyAbsent as Indeterminate.
- P2 (empty 404 body): classify_404 returns Indeterminate for
bodyless 404s. The team memory
`distinguish-unknown-from-empty.md` is the policy: a
bodyless 404 from a proxy is not a proof of absence, and the
recovery policy (permits_reupload -> true on
DefinitivelyAbsent) authorizes a paid, irreversible re-upload.
- P2 (streaming cap): read_capped_body no longer uses
resp.bytes() (which buffers the full response before checking
size). It now reads chunk-by-chunk and aborts the moment the
cumulative byte count crosses the cap, so a chunked response
without Content-Length cannot force unbounded allocation.
* fix(node): address three reviewer findings on #26 split 2/4 (round 2)
* fix(node): address five reviewer findings on #26 split 2/4 (round 3)
* fix(node): address reviewers round-4 on #26 split 2/4 (interop)
* fix(node): address coderabbit on #26 split 2/4 (raw tags, vint overflow, golden key)
* fix(node): enforce artifact-id in probe_item on #26 split 2/4
* fix(node): require expected owner for probe Present on #26 split 2/4
* fix(node): narrow #26 split 2/4 to codec and policy types
* fix(node): silence dead-code for deferred slice API on #26 split 2/4
* fix(node): doc and bounds-check cleanup in ans104 on #26 split 2/4
* docs(node): point tests doc at Ed25519 golden script on #26 split 2/4
* fix(node): strict owner lengths, sign-before-hash on #26 split 2/4
* fix(node): keep ans104 tags and field widths consistent on hash and encode
DataItem keeps both the decoded `tags` and the original Avro `raw_tags`, both
public. After a parse, changing `tags` left deep_hash and verify_data_item
using the stale raw bytes, and to_binary wrote a header tag count that no
longer matched the payload, so the frame failed to parse back.
Take the tag block for hashing and encoding from one helper that decodes
`raw_tags` and requires it to equal `tags`, and fail closed otherwise. The
header count comes from the same validated pairs. Parsed multi-block and
size-prefixed items still re-encode byte-exact and keep their signatures.
deep_hash now also rejects a non-empty target or anchor that is not 32 bytes,
as to_binary already does, so sign_data_item fails instead of signing an item
that cannot be encoded. The legacy non-base64url anchor fallback is kept.
The module and signature-type docs now say binary parsing accepts sigtypes 1
to 4 while signing and verifying are Ed25519 only on this slice.
* build(scripts): declare the golden script's direct npm dependencies
ans104_golden_ed25519.mjs imports bs58, @noble/ed25519 and base64url, and
ans104_golden.mjs imports base64url, but package.json listed only arbundles, so
they resolved only through hoisting. Declare them at the versions already
locked, update the lockfile root entry (no resolved version or hash changes),
and note the regeneration command in the golden output file.
---------
Co-authored-by: Gravirei <[email protected]>
* fix(node): validate ref-update SHAs are hex, not just length 40
parse_ref_updates accepted any 40-byte string as an object id, so a
peer-supplied pkt-line with non-hex content was forwarded into
RefUpdateEvents on gossip, branch_cid rows, and sync-notify JSON as a
canonical-looking identifier. Require [0-9a-fA-F] on both fields.
Closes#398
* chore: refresh PR head
* fix(gl): pass the mirror temp path to git as an OS string
The clone destination went through to_str().unwrap(), so a TMPDIR with
non-UTF-8 bytes panicked before git started. Pass the Path directly and
add a Unix regression test cloning into a non-UTF-8 directory name.
Closes#417
* test(gl): cover a non-UTF-8 TMPDIR end to end via a child process
* chore: refresh PR head
* fix(node): reconcile public-read messaging with per-repo enforcement
GITLAWB_PUBLIC_READ is read nowhere except a startup warning, while
read enforcement actually happens per repository through is_public and
path-scoped visibility rules. The flag's help text, the startup
warning, .env.example, and the readiness audit all still claimed
private-read enforcement was unwired.
The help and warning now state the flag is reserved and inert, and
point at the real per-repo control. A config test asserts the rendered
help neither claims enforcement is missing nor omits that the flag
does nothing.
Closes#338.
* docs: drop the stale private-read priority entry
* chore: refresh PR head
* fix(git-remote): stop waiting on EOF for a delete-only push
* fix(git-remote): consume negotiated push-options and push-cert sections
* fix(git-remote): frame signed pushes correctly, cert before options and no command list
* chore: refresh PR head
* ci(triage): match test as a name segment, not just a whole component
#202 landed the namespaced-attribute fix, so main already detects
#[tokio::test] and #[sqlx::test]. What it still misses is a third-party
harness whose final path component carries `test` as an underscore
segment. #[test_case(1)] and #[wasm_bindgen_test] both fall through and
wrongly earn the PR a needs-tests label.
Widen the final component to (?:[A-Za-z0-9]+_)*test(?:_[A-Za-z0-9]+)*,
keeping the leading ::, the r# form, and the whitespace tolerance #202
added. Differential over 972 real file-patches from 400 commits: no
verdict changes either way, so this is a forward-looking net for
harnesses the tree does not use yet, not a fix for a current mislabel.
Segment rather than substring, on purpose. addsInlineTest suppresses the
label when it matches, so a false positive is the quiet failure:
#[contest] or #[latest] would clear needs-tests on a PR that added no
tests and nobody would notice. The cost is the harness names with no
underscore, such as #[rstest], which stay unmatched and produce the loud,
author-correctable failure instead.
The segment classes are [A-Za-z0-9]+ rather than \w+ so that `_` is only
ever the literal delimiter. \w+ contains `_`, which makes the repetition
ambiguous and backtracks exponentially. That matters here specifically
because the workflow triggers on pull_request_target, so f.patch is
fork-controlled text on a privileged runner: with \w+, `#[test` followed
by a long `_a` run took 71ms at 22 characters and roughly quadrupled
every two more, which stalls the triage job for any contributor who can
open a PR. The character-class form measures linear across every
adversarial input tried.
* fix(ci): allow Rust token separators before test attribute delimiters
The needs-tests inline detector required ] or ( immediately after horizontal
whitespace, so valid spellings like #[test /* rationale */] or a path split
across added patch lines still triggered needs-tests. Match block comments,
line breaks, and split-line closers instead.
* ci(triage): scan attribute closers token-aware, bound to adjoined added lines
The two-regex fallback matched a path-only attribute line and a ]/( line
anywhere else in the patch, so unrelated records could complete each other
and silently clear needs-tests; the mandatory separator regex also dropped
legal //-to-newline and nested block comments, relabeling real tests.
Replace both with one forward token scan that starts on the attribute's
added line, continues only through immediately adjoined added lines under
a hard bound, and answers 'no inline test' on anything uncertain.
Fence the detector for extraction and add scripts/test-pr-triage-detect.mjs,
run by a new pr-checks job, since pull_request_target executes the target
branch's workflow and can never exercise the proposed body.
* fix(ci): skip test attributes inside raw strings, block comments, and strings
patchAddsInlineTest had no Rust lexical state, so a #[test_case] inside a
raw string (r#"..."#), block comment, or regular string literal, followed
by ] or ( on an adjoining added line, falsely cleared needs-tests. That is
the silent failure direction the detector is supposed to fail loud against,
and it is newly reachable because the underscore-segment regex introduced
by this branch matches forms the old test\b did not.
A pre-pass now walks every patch line (context and added, not removals)
tracking raw strings (r#*", br#*", cr#*"), block comments (/* */ with
nesting), regular strings (which can span newlines in Rust), and line
comments. Added lines that start inside any of these are skipped as
candidates. The pre-pass resets at hunk headers, is O(n) in patch length,
and only sees what the patch shows: a non-code region opened before the
first context line of a hunk is invisible, and the candidate is still
scanned (conservative, fails toward the loud label).
Also corrects the failure-direction comment in test-pr-triage-detect.mjs:
false positives suppress needs-tests (they set touchedTests, which clears
the label), while false negatives apply it (the loud, corrigible
direction). The previous text had them backwards.
Adds 7 regression cases: raw-string, block-comment, no-hash raw-string,
and multiline regular-string false positives (reject), plus raw-string,
block-comment, and regular-string closing before a real test attribute on
the next added line (accept). Revert-check confirmed the pre-pass is
load-bearing for both the nonCodeStart skip and the inString carry:
disabling either makes the corresponding reject cases go RED, restoring
makes all 31 pass.
* fix(ci): recognize char literals so their contents cannot open string state
The lexical pre-pass in patchAddsInlineTest missed real tests when an
earlier added line contained a double-quote char literal like '"' or
b'"'. The scanner saw the " inside the char token and set inString,
which persisted to the next line and suppressed a following #[test].
Add bounded char-literal recognition: on seeing ', look ahead 2-4 chars
(or to } for \u escapes) for a closing '. If found, skip the whole
literal. If not, it is a lifetime or label, and the apostrophe is
skipped as a regular char. Not a skip-to-next-' rule; lifetimes and
labels fall through unchanged.
* fix(ci): bound the \u{ escape scan to prevent quadratic stall
The char-literal pre-pass scanned to the next } with no bound when it
saw \u{, so a fork-controlled patch repeating '\u{ thousands of times
made the triage job quadratic (21s for 30000 reps). Bound the scan to
20 chars past the { (a valid \u{...} holds at most 6 hex digits). Add a
runtime probe for the malformed-escape shape.
* chore: refresh PR head
---------
Co-authored-by: Kevin Codex <[email protected]>
* fix(icaptcha): return unsolved instead of blocking or overflowing
interactive_prompt waited on stdin().read_line even when stdin was an
open, silent non-TTY (an orchestrator holding the pipe open), which
blocked obtain_proof forever. Check IsTerminal first and return None so
the caller surfaces the "couldn't auto-solve" error.
The deterministic solvers evaluated prompts with unchecked i64
arithmetic; service-controlled literals could push add/sub/mul/div/abs
past the range, panicking in debug builds or wrapping to a wrong answer
in release. Use checked arithmetic throughout and return None on
overflow.
Fixes#345
* chore: refresh PR head
* fix(node): scope the refs listing by the canonical repo slug
`list_refs` built its `branch_cids` filter from the raw URL path segments
while the only production writer stores the canonical
`normalize_owner_key(owner_did)/name` slug. `get_repo` resolves both
`did:key:zX` and bare `zX` to the same record, so a request in the full
DID form passed the read gate and then matched zero rows, returning 200
with an empty ref list. A denial shaped like a success is worse than an
error: a caller cannot tell an empty repository from one it queried under
a form the table does not carry.
Bind the record `authorize_repo_read` already returns and build the filter
from it, so the queried resource is the one that was authorized. This is
the existing pattern in `api/events.rs` and in this file's own push path.
Two regressions, each proven to fail without the line it guards. The first
drives both owner-DID forms through `build_router` and asserts they return
the same refs, with a second repo whose refs must not appear; widening the
query to match any owner turns that half red. The second covers the deny
direction, which the endpoint had never had: a private repo answers an
anonymous caller with the same 404 a missing repo gets and puts no CID in
the body. Without it, removing the read gate outright left the suite green.
* chore: refresh PR head
* fix(node): gate replica register/unregister on repo read visibility
list_replicas already applied authorize_repo_read, but the two
mutations looked the repo up directly: a non-reader of a private repo
could register or remove their own replica and receive the replica
count in the response. Route both handlers through authorize_repo_read
so a denied caller gets the same 404 as a missing repo and nothing is
written. Self-registration and self-removal are unchanged for
authorized readers.
Fixes#435
* test(node): assert denial bodies leak no replica or repo data
CodeRabbit nitpick: the new replica-mutation denial tests only checked
the 404 status. The path instructions for this file require non-leaking
body assertions on denied responses, matching the sibling
list_replicas_is_read_visibility_gated test. Assert the denied PUT and
DELETE bodies contain neither the submitted replica URL nor the repo id,
in both the handler-level test and the real-signature e2e.
* chore: refresh PR head
* test(node): add a completeness fence over the GraphQL mutation surface
Gate the GraphQL mutation surface the way REST is gated, as a SEPARATE guard with
its own bucket table (a guard over a heterogeneous set must classify per member,
INV-1). every_graphql_mutation_has_its_gate asserts the mutation set is complete
(every mutation classified) and that each carries its gate marker.
Completeness is derived from async-graphql's own introspection: the field names
come from the engine as structured data, so no source-text or SDL-format shape
(a brace or async fn in a comment or string, a whitespace/generic/skip variant, a
brace in a rustdoc description) can hide a live mutation. A new unregistered
mutation, including a repo-write, fails the build.
The marker check (does each resolver body call its gate) is a best-effort static
backstop, bounded to each resolver's own comment-masked, string-aware body. It is
explicitly not the completeness guarantee and not load-bearingness: a marker inside
a string literal is a documented vacuity residual, and the adversarial sqlx tests in
graphql/mutation.rs are the real proof the gates reject.
Verified by execution: an unregistered mutation reddens (unclassified), a removed
did_matches gate reddens (missing marker), the clean tree is green; no DB, no
feature flag.
* docs(node): reconcile GraphQL mutation-auth claims with the shipped state
README's known limitations, the OSS-readiness live-network blockers, and the
maintainer roadmap still listed GraphQL mutation auth as an open item, which
contradicts #87: GraphQL mutations already require a verified signer bound to the
acting DID. Reconcile each to the shipped reality while keeping the genuinely-open
nuance: the GraphQL mutation surface is the agent-task queue, repo writes stay
REST-only, and no owner-gated repo-write mutation is exposed over GraphQL. The
every_graphql_mutation_has_its_gate fence forces any new mutation to be classified
in its table; owner-gate and REST parity stay the INV-1 review check.
* fix(node): source the GraphQL mutation fence from SDL, not introspection
Resolves jatmn's #219 P1. The every_graphql_mutation_has_its_gate completeness
fence enumerated mutation fields via async-graphql introspection, which filters
out `#[graphql(visible = false)]` fields. Those fields stay executable, so a
future hidden mutation would need no registry row and no marker while both
set-equality checks stayed green -- defeating the fence's guarantee that no live
mutation goes unclassified. (Confirmed by execution: a synthetic visible=false
mutation is absent from introspection, executable, and present in schema.sdl().)
Source completeness from the engine's own SDL instead. A new pure helper
mutation_field_names_from_sdl parses schema.sdl() with async-graphql's parser
AST (async_graphql::parser::parse_schema) and reads the mutation type's fields;
SDL includes visible=false fields and the AST parse is immune to a brace in a
field description. Introspection is kept as a cross-check: any field present in
SDL but hidden from introspection now fails named as a forbidden hidden
mutation. The stale doc comments (which claimed introspection could not hide a
field) are corrected.
A self-checking version-safety test builds a synthetic schema with a
visible=false mutation and asserts the SDL-sourced set includes it while
introspection omits it -- so if a future async-graphql ever filters SDL by
visibility, that test goes RED rather than the gap reopening silently.
Load-bearing verified by execution: stubbing the SDL enumeration to empty turns
four tests RED, including the fence itself; restoring returns green. Only
api/mod.rs changed; the mutation resolvers are untouched. `cargo test -p
gitlawb-node --bins` green (495); clippy and fmt clean.
* fix(node): make the SDL mutation-fence wiring load-bearing; admit deprecated
An adversarial review of the SDL switch found three gaps:
- The fence's completeness source (SDL) and its hidden-mutation cross-check were
not load-bearing: because the real MutationRoot has no visible=false field,
reverting `fields` back to introspection and dropping the cross-check left the
whole suite green (INV-21). Extract the sourcing + cross-check into a generic
`assert_mutation_fence<Q,M,S>` that the real fence AND a synthetic hidden schema
both drive. A new #[should_panic] test (fence_rejects_a_hidden_mutation) feeds a
visible=false mutation that IS registered, so only the cross-check can fire;
reverting the SDL source or deleting the cross-check now changes/removes that
panic and turns the test RED (verified).
- The introspection cross-check used bare `fields { name }` (GraphQL default
includeDeprecated:false), so a `#[graphql(deprecation = "...")]` mutation --
executable and present in SDL -- would false-red as a hidden visible=false
mutation. Use `fields(includeDeprecated: true)`; a new test
(fence_admits_a_deprecated_mutation) pins it, going RED without the flag.
- schema_lists_exactly_the_registered_mutations still sourced from introspection
and its comment called introspection "the source of truth", contradicting the
fix. Retargeted at the SDL set with a corrected comment.
Also folds the two introspection navigations into one generic helper
(introspection_mutation_fields) and points the version-safety test at it, so
there is a single introspection query. cargo test -p gitlawb-node --bins green
(497); clippy and fmt clean. Only api/mod.rs changed.
* fix(node): rebase #219 and require signer gate in the GraphQL fence
Rebase onto current main and skip the obsolete 0.5.1 lockfile commit.
The mutation fence now requires both require_signer(ctx)? and did_matches(
in every registered resolver, and each_mutation_requires_and_binds_signer
covers all four task mutations for unsigned and DID-mismatch denial.
* docs(node): reconcile GraphQL fence docs with SDL discovery
Restore the historical OSS audit blocker text and add a dated status
update for #219. README and the maintainer roadmap now describe the
SDL-sourced fence and signer-plus-binding markers.
* ci: trigger the PR checks on a new head
* fix(gl): status-check the client read/write surfaces so a node denial surfaces as an error, not a fake result (#123)
Convert the gl CLI + MCP read/write surfaces to status-check the node's
response before parsing, so a denial (401/403/404) or a degraded 5xx surfaces
as an error instead of being rendered as an empty list or a fake success. The
shared http::read_json helper checks the status first and bounds the error-body
read (a hostile node can't stream an arbitrarily large error body to exhaust
the client), replacing the prior parse-before-status pattern across agent,
bounty, cert, changelog, issue, mcp, peer, pr, protect, repo, star, status,
sync, task, visibility, and webhook. A no_parse_before_status regression gate
locks the converted set in.
Rebased onto current main (squash of the prior branch). main had independently
gated several of these same surfaces, so the overlapping arms are reconciled to
keep BOTH fixes rather than let either regress:
- repo::cmd_label_list keeps main's get_maybe_signed (read-visibility gating:
public repos stay anon-listable, a private-repo owner signs) carrying
read_json on top; the PR's get_authed is dropped so the visibility gate holds.
- mcp webhook_list keeps main's get_signed and its signing-keypair-DID owner
resolution (NOT resolve_owner's node DID) — list_webhooks is owner-gated AND
auth-required, so the PR's plain get()+resolve_owner would have 401'd then
403'd — with read_json layered for the bounded error read.
- webhook create/list/delete keep main's get_signed + resolve_owner_repo_pair
structure and its tests, with read_json swapped in for the manual
resp.json() so the error body is bounded. main's superseded resolve_owner
helper and its duplicate tests are dropped.
gl: 388 unit tests + the no_parse_before_status gate green; fmt + clippy clean.
Rebased again onto current main (2026-08-11), which had moved 238 commits.
main independently landed read_body_capped and sanitize_node_msg into http.rs,
so this branch's duplicate definitions in sync.rs are dropped and read_json is
layered on main's helpers instead. main's peer.rs also grew its own announce
handling (remote_announce_failure, local_add_report/local_add_refusal, capped
read, sanitized message) which supersedes this branch's read_json conversion
there; main's side is kept whole and only peer::cmd_list, which main had not
gated, still takes read_json.
fix(gl): report why the node-DID comparison was skipped in cert show
The rebase onto main dropped did_check_report, since main's cmd_show had
been rewritten to do real Ed25519 verification and collapsed every
node-info failure into one "could not fetch current node info" NOTE.
Put the tri-state reporting back on top of main's structure: the signature
verdict and the --verify issuer anchoring are untouched, but the node-DID
comparison now names the reason it could not run and points at the offline
verification path, and "carried no DID" stays distinct from "the lookup
failed". The lookup goes through read_json so a denial yields a reason
rather than an opaque None, and it stays fail-soft: a node-info hiccup must
not turn a successfully displayed certificate into an error exit.
The DID extraction is split out as did_from_node_info so its guard is
testable. An empty did rejects rather than flowing into the comparison,
where it would print a mismatch WARNING against a DID the node never
claimed. That guard had no coverage: removing it leaves every pre-existing
test green and only the new one goes red.
fix(gl): close two completeness-gate escapes; surface the node's error key
Review of the rebased branch found the #186 completeness gate could not fail
on the two shapes that matter most. Both were reproduced by seeding the
regression and watching the gate stay green.
An unguarded parse escaped entirely. The rule flagged a parse followed by
.is_success(), which describes one bad ordering and says nothing about a read
with no status check at all -- the more dangerous revert, since it renders a
denial body as a result. Inverted it to require a status check above the parse,
which also covers a check spelled as_u16() >= 400. The lookback is 24 lines
because sync.rs's hand-rolled Trigger arm puts its .status() 19 lines up; it
should shrink if that arm is ever routed through read_json.
Membership in the scanned set keyed on the text "read_json" rather than a call
site, so a handler fully reverted off read_json kept its membership on a
leftover comment. Still-derived meant the pinned-vs-derived equality never
fired and the deconversion shipped green, dropping the cap and the sanitizer.
Now counts call sites on non-comment lines.
read_json read only `message`, but the task API answers with `error` alone
({"error":"task not found"} in gitlawb-node/src/api/tasks.rs), so every gl task
denial rendered as the generic "request failed". Falls back to `error` through
the same sanitizer and cap; `message` still wins when both are present, which
is the shared AppError envelope. sync.rs already read both.
Also covers cmd_show --verify, which had no test at all: the issuer anchoring
is the security-bearing half of the command, since a valid signature only
proves the certificate is self-consistent and a hostile node can self-sign one.
The certificates carry real signatures, otherwise --verify would bail before
reaching the anchoring and the assertions would be vacuous.
Each fix verified by reverting it: both gate escapes go red, dropping the error
fallback fails the two new read_json tests, and removing the anchoring block
fails three of the five --verify tests.
* test(gl): pin the stats mock hits and the unsigned MCP webhook_list denial
Closes the two items left open on the last review round.
The bounty-stats denial tests were already load-bearing: breaking either
mock's route regex turns both red, because mockito answers an unmatched
route with 501 and neither test's asserted status is 501. That proof is
incidental to mockito's fallback status, though, so a reworded assertion
would silently drop it. Assert the hit directly with expect(1) +
assert_async.
MCP webhook_list keeps get_maybe_signed rather than aligning with the CLI's
get_signed. An agent driving the MCP tool may legitimately have no local
identity, and issuing the request so the node's own denial reaches the
caller is more useful than failing client-side before the node is asked.
That is only defensible if the denial actually surfaces, so pin it: no
identity plus an explicit owner must issue an UNSIGNED request and turn the
node's 401 into an error carrying the status and the node's reason, not an
empty webhook list.
The new test points at an empty temp dir instead of passing None. None
falls back to the ambient ~/.gitlawb/identity.pem, which exists on a dev box
and not on CI, so the request would have been signed here and unsigned
there; the first run of this test failed exactly that way before the dir was
pinned.
Verified by execution: swapping get_maybe_signed for get_signed reds the new
test on "must name the status", and breaking either stats mock route reds
its test. gl: 412 unit tests + the no_parse_before_status gate green; fmt
and clippy clean.
* docs(gl): describe the client-contract rules in plain English
The comments carried internal shorthand identifiers that resolve to nothing
for a reader outside this repo. Replace each with the property it names: a
denial must surface as an error rather than render as an empty or fabricated
result, and an error body must be read under a cap and sanitized before it
reaches the terminal. Comment-only; no behavior change.
Verified after the edit because the completeness gate reads source text to
decide which handlers it scans: reverting a converted handler to a bare
.json().await still turns it red. gl: 412 unit tests green, fmt and clippy
clean, cargo check --workspace --locked clean.
* test(gl): accept only a full non-success guard above a raw parse
A 404-only equality upstream counted as status evidence, so reverting a
read_json call behind agent.rs's special-404 hint to resp.json().await?
kept the fence green while a 500's denial body rendered as a result.
Require .is_success() or an as_u16() range comparison in the lookback,
and pin the distinction with a mutation regression that performs that
exact revert on the real agent.rs source.
* test(gl): drop the traced result from the cert assertion diagnostics (#186)
CodeQL reported seven high-severity "Cleartext logging of sensitive
information" alerts at cert.rs:497, 540, 584, 667, 708, 803 and 862, and the
aggregate check has been red on the branch since. All seven are #[cfg(test)]
assertions that format a whole Result<()> through {result:?} or {got:?}.
CodeQL traces a value associated with resolve_cert_id into the command error
and treats the assertion's panic diagnostic as a logging sink.
The alerts are test-only. The single mod tests opens at cert.rs:328 and runs to
EOF, so all seven sit inside it, and the traced certificate id is already
printed unconditionally in production at cert.rs:145 because it is a public
identifier. That makes these repeated instances of one test-diagnostic pattern
rather than seven distinct vulnerabilities.
Each message now carries static context naming the scenario instead. The
failing test name and source line still identify what broke, and every
assertion predicate is unchanged, so nothing is weakened into vacuity. The
production cleartext-logging query stays enabled: nothing is suppressed or
reconfigured.
One {got:?} at cert.rs:900 is deliberately left alone. CodeQL did not flag it,
its value comes from did_from_node_info over hardcoded literal JSON with no
resolve_cert_id taint, and the interpolation is load-bearing there because it
names which of three loop cases failed.
Verified: cargo test -p gl --bin gl cert, 20 passed. fmt clean.
* test(gl): restore the nine peer denial tests dropped in the rebase (#186)
The 2026-08-11 rebase squashed the branch and lost nine tests. Counting
`async fn cmd_add_` in peer.rs per head: 6 at 06863da0, 3791c40c, 9ed2947d and
ad0e35ad, then 0 at 21ba974c and 0 at fba90038. The production fail-soft at
peer.rs:173-180 survived; only the tests pinning it went.
That matters because the rule this PR's own round 3 established is that a
carve-out with no test that goes red when someone makes it fatal is an
incomplete sweep, not a documented decision. The peer-add public-URL lookup has
been in exactly that state for three weeks.
Recovered from e179cd22, not 5eff6060: the earlier commit carries only four of
the seven items, while e179cd22 has all of them plus three same-class denial
tests for cmd_list, cmd_ping and cmd_resolve that were lost the same way.
Neither commit is an ancestor of HEAD.
Nothing needed adapting. cmd_add's signature and its node-info block are
byte-identical between the two heads. The announce arm has since moved to
remote_announce_failure, which still formats the status the denied-announce
test asserts on, and the local-add arm has moved to local_add_report, which
still returns Ok either way, so both local-add tests hold. The named import
list gains cmd_list, cmd_ping and cmd_resolve.
Proven load-bearing rather than assumed: rewriting the fail-soft match into a
fatal `?` turns cmd_add_announces_fallback_url_when_node_info_denied and
cmd_add_falls_back_on_malformed_node_info red, and leaves the pre-existing
a_peer_failure_past_the_read_cap_still_reports_its_status green. That last
point is the reason these are not redundant with what was already there: the
surviving test drives a 503 whose body is past the read cap, so it pins the
cap and ordering interaction, not the fallback.
Verified: cargo test -p gl --bin gl peer, 25 passed. fmt clean.
* fix(gl): route the status repo sections through read_json and pin the file (#186)
`gl status` was hardened in this PR with section_unavailable_line and
trust_line, so a denied section prints "unavailable ({status})" instead of
rendering as empty. But it was hardened by hand rather than through read_json,
so status.rs carried no marker, never entered the completeness gate's derived
set, and was never scanned. Deleting the helper call would have restored the
denial-as-empty bug with the gate still green: the one file written to fix this
bug was the one converted file the gate did not cover.
The two repo-section parses now route through read_json and status.rs is
pinned in CONVERTED_IN_186. Exactly two source lines change.
section_unavailable_line takes the response by reference and reads only its
status, so read_json sits underneath it with no restructuring, and the outer
if-let shape still stops one denied section from aborting the others. All 35
status tests pass unchanged, including the eight denial tests and their exact
rendered strings.
trust_line stays hand-rolled on purpose. It needs a three-way split (2xx,
exactly 404, other non-2xx) that read_json collapses into a single Err, and it
already carries a literal status check the scan can see.
A second pinned const naming hand-rolled guards was considered and rejected. It
would enforce only that a symbol is spelled somewhere in the file: deleting the
println while keeping the call would stay green. A structural anchor already
exists, which is read_json, and the repo's own guard-design note is explicit
that a name-keyed filter must never be the membership predicate when one does.
Proven, not assumed. Pinning status.rs before converting it turned the gate red
naming the file. Reverting even one of the two read_json calls afterwards turns
it red again, on the unguarded check rather than the pin, because the issues
parse has no full status guard in its lookback window. The one gap worth
naming: deleting both section_unavailable_line call sites while keeping
read_json still scans green, and the denied section would then print nothing
rather than a false "no open pull requests". Probed it, and clippy catches that
case as a dead function under -D warnings, so CI would fail. That backstop
holds only while the helper has no other caller.
Verified: gate 2 passed, cargo test -p gl --bin gl status 35 passed. fmt clean.
* fix(gl): let gl pr view finish rendering when a sub-fetch is denied (#186)
Converting cmd_view's sub-fetches to read_json turned a tolerated 404 into a
mid-render abort. The header prints first, then the reviews fetch is bare `?`,
so a denial leaves a half-rendered PR, exit 1, and the comments section never
runs. Before the conversion the error body parsed, the section was skipped and
the rest of the command still rendered, so this is a regression this PR
introduced rather than pre-existing behavior.
Both sub-fetches now degrade: a denial prints "Reviews unavailable ({status})"
or "Comments unavailable ({status})" and rendering continues. That is the shape
this PR already invented one file over for gl status's denied sections, so the
two commands now behave the same way. read_json stays in use, so pr.rs stays in
the gate's derived set and the pinned list is untouched.
The header fetch stays fatal. With no header there is nothing to render
partially, and softening it would resurrect denial-as-success for the whole
command. cmd_view_header_denial_stays_fatal pins that, and it passed before
this change as well as after, which is the point: it is the must-not case.
mcp.rs's pr_view arm has the same fetch coupling and is deliberately left
alone. It emits a single JSON result, so there is no partial-output problem, a
whole-call error is the correct fail-closed answer for a machine consumer, and
inventing a partial-result shape is a design change nobody asked for.
Test-first and observed: cmd_view_continues_when_reviews_are_denied and
cmd_view_continues_when_comments_are_denied both failed before the change and
pass after. Each asserts the comments mock was hit with expect(1), which is the
load-bearing part: the mock can only be hit if rendering continued past the
denied fetch.
Verified: cargo test -p gl --bin gl, 473 passed. Gate 2 passed. fmt and clippy
clean.
* ci: trigger the PR checks on a new head
---------
Co-authored-by: Kevin Codex <[email protected]>
* fix(node): hold fork to the same repo-name rules as every other creation route
fork_repo built its clone destination with store::repo_disk_path, a raw
join with no validation, while create_repo and the sync mirror path both
go through validated_repo_disk_path. Its own name check is
`.chars().all(|c| c.is_alphanumeric() || c == '-' || c == '_')`, which is
Unicode-aware and has no length or leading-character rule, so it admits
four shapes validate_repo_name refuses: the empty string (all() is
vacuously true on it), a non-ASCII alphanumeric such as "cafe" with an
acute e, a leading '-', and a name over the 100-char bound.
The empty name was the reachable defect. Driven through the production
router with a real Ed25519 signature, a `{"name":""}` fork returned 201
Created with a repo row whose name is "" and a clone_url ending in
`/.git`, materialized at `<repos_dir>/<owner_slug>/.git`. It now returns
400 and creates neither the row nor the directory.
This is a deliberate narrowing, not only an empty-name fix: the other
three shapes above are now 400 on fork where they previously reached the
raw join. None of them could ever have been created through create_repo,
which fails at repo_store.init on the same validator, and no existing row
can carry such a name (create_repo validates before its DB insert, and
mirror rows go through validate_repo_slug), so no repo becomes
unforkable. A fork that defaults to source.name is unaffected.
The traversal shape of the same join is NOT reachable and is not what
this changes: forker_did comes from AuthenticatedDid, which auth/mod.rs
inserts only after Did::to_verifying_key resolves a multibase-decodable
did:key, so it can carry neither ".." nor "/".
repo_disk_path is now cfg(test). It has no production caller left, and
the attribute turns the next one into a compile error rather than a
silent bypass. Tests keep it because a fixture that must escape repos_dir
cannot be built with the validated form.
Two tests. A unit test on the mounted handler, and an end-to-end test
through server::build_router with real signatures that covers all four
refused shapes plus a positive control asserting an ordinary name still
forks to disk and to a row. Both are mutation-verified in both
directions: reverting to the raw join reddens the refusals, and forcing
fork to always refuse reddens the control.
Closed#272 fixed this class on the sync route and pointed at
repo_disk_path as the sanitizing convention; it is in fact the
unvalidated join, and the fork route was never scoped.
* fix(node): validate the fork name before spending the proof
The barrier landed after `proof.consume` and after `repo_store.acquire`, so a
name the character allowlist admits but `validate_repo_name` refuses burned a
valid iCaptcha proof and paid a source-repo download before its 400. The
handler's own header comment promises the opposite: a fork rejected for a bad
name never burns a proof. Reported by CodeRabbit on the review of a7257dce.
It now runs with the other admissibility checks, above both.
The regression test pins the ORDER without iCaptcha plumbing. Seed a repo row
whose bytes are not on disk: late validation lets the acquire fail first and
the caller sees a 500 from git, so only early validation can produce the 400
the test asserts. Verified load-bearing by degrading the early call to
`unwrap_or_else`, which reddens it.
Both name rules are kept. CodeRabbit also suggested deleting the character
allowlist as a strict subset of `validate_repo_name`, and that is not the
relation between them: the allowlist rejects a dot, so `v1.2.3` and `my.repo`
are refused today while `validate_repo_name` accepts both, and the allowlist
accepts a non-ASCII alphanumeric that `validate_repo_name` rejects. Removing
it would newly admit dotted fork names, which is a behaviour change and not a
cleanup. A comment now records why the two coexist.
The empty-name fixture also generates its owner DID per run. It writes to a
fixed `/tmp/<owner_slug>`, so a shared constant let two concurrent `cargo
test` processes delete each other's repo mid-test (reported by Greptile).
* fix(node): bound the request body the signature middleware buffers before it verifies
`require_signature` buffered the whole request body with an unlimited
`body.collect()` before parsing a single header, and nothing upstream capped it.
On the git write routes the 2 GB `RequestBodyLimitLayer` is applied to the inner
router while `add_auth_layers` wraps it, and a later `.layer()` is the outer
service, so the middleware ran first and saw the raw body. The other signed
groups carry no limit layer at all and lean on axum's 2 MB `DefaultBodyLimit`,
which the extractors enforce through a request extension this middleware never
consults, so that default was bypassed too. `optional_signature` forwards to the
same code whenever a caller merely presents signature headers, so the reach
extended to `/graphql`, `/ipfs/{cid}` and the other optional-signature groups.
Measured against the real router: an unsigned push with `GITLAWB_MAX_PACK_BYTES`
set to 64 KiB drained the full 4 MiB body and only then returned 401. No DID, no
signature, and no repo were required, and in front of it sits only a per-IP
arrivals-per-hour counter, no timeout layer and no connection cap.
The collect now runs through `http_body_util::Limited`, so reading stops at the
limit instead of after the fact, and an over-limit request gets 413 rather than
the 400 reserved for a genuinely unreadable body. The limit is per route group:
`add_auth_layers` takes it and passes it to the middleware as state, the git
groups keep `max_pack_bytes` so large pushes are unaffected, and the rest get a
constant equal to axum's default because that is already what their extractors
enforce. Verification, content-digest, clock-skew and the request reconstruction
are untouched.
The regression test asserts bytes actually polled out of a streaming body, not
the status code, because a status assertion passes whether or not the body was
drained. Neutralizing the limit turns it red; handing every group the git limit
turns the non-git test red; returning 400 instead of 413 turns the status test
red.
This bounds per-request buffering only. Total resident memory is still
O(connections x limit); a read timeout and a connection cap are what close that
and are deliberately not in this change.
* test(node): prove a signed push under the limit lands on disk, and an over-limit one writes nothing
The body-limit guard was covered only by tests that reject. Nothing proved it
still admits what it should, so the fix could have passed every test by refusing
everything, and nothing would have caught a middleware that truncated or
corrupted the body it hands downstream.
These drive a real push: a source repo with a 300 KiB incompressible blob, an
empty bare repo at the path the store resolves, and the byte-for-byte
receive-pack request body captured off the wire from an actual `git push`. The
body is captured rather than generated, because `git send-pack --stateless-rpc`
emits remote-curl's extra RPC framing (a wrapping length plus a stray flush
before PACK) and `git receive-pack --stateless-rpc` rejects it with a pack
signature mismatch.
The positive case sets the cap just above the body and asserts the ref moved and
the tree resolves, not merely that the status was 200. That distinction is
load-bearing: under a truncation mutation the status stays 200, because git
reports the failure inside report-status, and only the on-disk assertion goes
red. Running the whole suite under that mutation leaves exactly one test failing,
so this is the only thing standing between a corrupted body and a green build.
The negative case sets the cap just below the body and asserts 413 with neither
the ref nor the commit present afterwards, so a rejected push cannot half-write.
Its existence check peels to ^{commit}, since rev-parse echoes any well-formed
40-hex string back and would have passed vacuously.
* test(node): give the unresolvable-DID middleware test the signature body limit state
---------
Co-authored-by: Kevin Codex <[email protected]>
* fix(core): reject small-order Ed25519 keys at DID resolution and before X25519
A did:key can encode a small-order (weak) Ed25519 point. VerifyingKey::from_bytes
only decompresses, so such a key resolved cleanly, and x25519_public converted it
to the all-zero Montgomery u. X25519 against u = 0 is the all-zero shared secret
for every scalar, so the per-recipient wrap in seal_blob could be rebuilt with no
secret at all. Since one content key is wrapped per recipient, a single weak
recipient exposed the blob for every recipient.
Reject in Did::to_verifying_key, which is the choke point every consumer that
resolves a DID shares, so a weak key becomes unresolvable and the existing
fail-closed recipient handling catches it without a new code path. Guard
x25519_public as well so the primitive is safe for a caller that obtained the key
some other way.
The attacker test performs the real attack rather than asserting an absence: with
either guard removed it recovers the plaintext and fails printing it.
* fix(attest): mirror the small-order key rejection in its own did:key parser
gitlawb-attest parses did:key itself in verifying_key_from_did_key and never
routes through gitlawb-core's Did. It cannot: gitlawb-core is a dev-dependency
here, so the choke-point rejection is structurally unreachable from this crate
and has to be mirrored rather than inherited.
Signature verification is already strict, so this is defense in depth against a
future consumer of this parser that does not verify strictly, not the closing of
an exploitable path.
One consequence worth naming: verify_rejects_weak_key_signature now observes
Error::Did rather than Error::Signature, because the weak signer is refused at
resolution before verify_strict is reached. That test is retargeted accordingly.
It no longer exercises verify_strict, which remains the second layer for
malleability that does not involve a weak public key (a small-order R under an
honest key); this fixture cannot construct that case.
* test(node): pin that a weak recipient DID falls into the existing fail-closed arm
Because a small-order did:key no longer resolves, it lands in the unresolved set
and plan_seal returns the #47 SkipUnresolvable arm that already refuses to seal
to a partial recipient set. Nothing is sealed for the blob and the existing
bounded operator warning fires. No new branch, no new SealPlan variant, no new
log site.
The must-not-over-reject control asserts an all-legitimate set still seals.
* fix(gl): stop telling a did:key that did:key is unsupported
did_resolve reported every resolution failure as "only did:key is supported
without a resolver". That is right for a method with no local resolver and
misleading for a did:key that parsed fine and failed for its own reason, which
now includes a small-order key.
Split the message on whether the DID is a did:key and surface the underlying
error when it is. Extracted as a pure helper so both branches are unit-testable
without an MCP client harness.
* fix(review): restore verify_strict coverage the parser guard displaced
Refusing a small-order key at DID resolution means a weak signer can no longer
reach attestation.rs's verify_strict through verify_signature, which left that
line with no test at all: downgrading it to the non-strict verify kept the whole
attest suite green. Split the signature check into verify_sig_with_key so a test
can drive it directly with a weak key. Check order in verify_signature is
unchanged; only the seam is new.
Also from review:
- auth/mod.rs sent a did:key that failed on its own key material away with
"only did:key is supported in alpha", the same misleading hint the gl MCP
tool was just fixed for. Made it conditional on is_did_key so the two
parallel surfaces say the same thing.
- The guard comments claimed a small-order point converts to the all-zero
Montgomery u. That holds for order 1 and 2 only; order 4 and 8 are
annihilated by the scalar clamping instead. is_weak covers the whole torsion
set, and mixed-order points are correctly still accepted.
- did.rs doc comment now states the new rejection.
- encrypt.rs: extracted the repeated weak-key fixture into a test helper and
switched to the file's existing Err(anyhow!(..)) rejection idiom.
- gl tests derive the did:key discriminator from a real parsed Did instead of
hardcoding the bool, so inverting it at the call site now fails.
* fix(attest): cap the did:key method-id before decoding, mirroring core
The parser here mirrors gitlawb-core's small-order rejection because it cannot
call into it (gitlawb-core is a dev-dependency of this crate). It was mirroring
only one of the two guards that sit three lines apart in the function being
copied: core also caps the method-id at 64 bytes BEFORE the multibase decode,
because base58 decoding is quadratic in its input.
signer comes off an attacker-supplied attestation and verify_signature reaches
this parser after computing the JCS bytes, so an oversized method-id bought a
large decode for a short request. An ed25519 did:key method-id is a fixed 48
characters, so the bound is slack rather than a behavior change.
The test asserts the length error specifically rather than any error: a
multibase error there would mean the cap ran after the decode and bought
nothing.
* fix(node): surface stored envelopes sealed to a now-unresolvable recipient
plan_seal short-circuited on a matching stored tag before resolving the
recipient set, so a blob sealed to a small-order did:key before the
choke-point guard landed kept its old CID and was silently treated as
unchanged. Resolve the set on the match and surface a now-unresolvable
reader as SkipUnresolvableStored so the operator sees the exposure.
* fix(attest): anchor VerifiedAttestation.signer to the key that verified
Registry::verify discarded the VerifyingKey returned by verify_signature
and copied the artifact's signer string into the result. The parser is
canonical so the strings agree for honest artifacts and a mismatched field
already fails verification, but the consumer-facing value now provably
comes from the key that verified rather than a raw artifact field.
* fix(node): bound the echoed DID in the unresolvable-did error and pin the hint branches
The keyid is unauthenticated request input and was reflected into the
response message at full length. Bound the echo to 96 chars and extract the
two-branch hint selection into a testable helper with both branches pinned.
* test(gl): build the weak did:key fixture from its identity-point bytes
The hardcoded WEAK_DID_KEY constant trips secret scanners, which cannot
tell a public DID from an API key. Deriving the DID from the compressed
identity point states why the fixture is weak and matches the house pattern
used in the core, attest, and node crates. ed25519-dalek is added as a
dev-dependency (it was already a workspace dependency).
* fix(core): skip a low-order ephemeral on the envelope open side
open_blob built the X25519 box from the header-supplied eph with no
small-order check, so a crafted entry forced the all-zero shared secret and
unwrapped for any reader. Skip entries whose eph decompresses to a
small-order point; honest entries in the same envelope still open.
* fix(core): test the exchange result, not the ephemeral's encoding
The open-side guard decompressed the ephemeral to Edwards and asked
is_small_order, treating a failure to decompress as safe. That reports safe
for any low-order input that is not a valid Edwards encoding. Measured across
the seven standard low-order encodings: six caught, u = p-1 missed. Its
exchange with any reader is still the all-zero shared secret, so an entry
built on it unwraps for everyone and the attacker controls the plaintext.
Check the result instead: an ephemeral is rejected when the exchange with
this reader's own scalar is all-zero. That is the property the guard exists
to enforce, and it holds for every low-order input regardless of encoding.
The new must-not test needed two attempts. The first passed with the guard,
without it, and with the original pre-fix guard alike, because a blanket
`use chacha20poly1305::aead::Aead` in the test resolved the wrap's encrypt
through a different aead version than open_blob decrypts with, building a
ciphertext that could never open for any reason. Every call in it now names
its trait.
* docs(core): anchor open-side guard comments to exchange-result invariant
The seal-side comment still named is_low_order_montgomery and the
low-order vector test doc claimed a decode-path filter that does not
exist. Both now describe yields_all_zero_shared_secret.
* test(core): use generated keys in poisoned-envelope fixtures
Replace hard-coded test scalars that CodeQL flags as cryptographic
material. The all-zero shared-secret attack does not depend on which
scalar builds the attacker box.
* test(core): use Default::default() for zero-point fixtures flagged by CodeQL
CodeQL flags [0u8; 32] literals that flow to a cryptographic key
argument, even in test fixtures where the value is the all-zero
Montgomery point (a public low-order test vector, not a secret).
Default::default() produces the same bytes without a literal, clearing
the alert without changing behavior.
* test(core): generate body nonces with OsRng in poisoned-envelope fixtures
CodeQL flags hard-coded byte arrays used as nonces, even in test
fixtures where the value is a fixed test vector. Generate with OsRng
instead, matching the production seal path.
* chore: sync Cargo.lock after merging main (ed25519-dalek edge for gl)
* feat(node): add the freshness check and replay guard, not yet wired
Builds the two layers that bound replay of an authenticated ref-update, with
nothing calling them yet; the ingest path is wired in the next commit.
The freshness check is deliberately two-directional rather than an absolute
delta: an abs() window admits an event stamped up to the window ahead and pins
its seen-set slot until the clock catches up. Producers were enumerated before
settling the unparseable arm; the sole production publish site emits RFC-3339,
so an unparseable timestamp is refused rather than admitted.
The replay guard keys on SHA-256 of the canonical signing bytes, not the raw
wire bytes, because one signature verifies against many encodings and only the
canonical form collapses them to a single key. A golden digest is frozen for
the pre-version artifact, and the same constant is asserted for its v-injected
twin, so the collapse is pinned rather than described. Reservations settle
through a drop guard so only a confirmed entry outlives the ingest call, which
keeps a transient write failure from permanently burning an event's slot.
Replayed and StaleTimestamp are separate outcomes because they diagnose
different conditions, a mesh replay against a broken clock or a healing
partition, and folding them would be the same observability lie the unsigned
shed variant already exists to avoid.
* fix(node): bound replay of authenticated gossip ref-update events
Wires the freshness check and the replay guard into ingest, immediately after
signature verification and above the author debit. The lower placement, just
above the writes, stops the duplicate row and the duplicate sync but still lets
every replay drain the victim author's budget, which is the harm the defect
names: a captured signature replayed 500 times empties the victim's window and
their next genuine push is refused.
The guard runs on the verified path only. Unsigned event bytes are predictable,
so applying it there would let an attacker pre-send a victim's expected event
and have the genuine one dropped as a replay, which is a censorship primitive
rather than a defense.
A replay flood still costs a parse and one Ed25519 verify, because the guard
has to sit below verification for the reason above. What it removes is the
peer_exists round trip, the victim's author debit, the ref-update row and the
sync enqueue.
The existing author-budget test signed one event and ingested the same bytes
five hundred times, so it had to re-sign per iteration to keep exercising the
budget under a shared guard. Giving it a fresh guard per call would have kept
it green while gutting the property it exists to prove. Its over-budget probe
needed the same treatment, since the replay gate sits above the author gate and
would have refused the burst's last bytes before the budget assertion ran.
Both saturation tests now take a shared lock so each keeps an exact assertion
on a process-wide counter; a lower bound would stay green if the fail-open
branch ever double-counted.
* fix(review): bound the saturated guard's rescan and the log it writes
At capacity the inline sweep ran on every event, so a full O(capacity) retain
happened under the lock once per message and reclaimed nothing when nothing had
expired. The guard that bounds replay became a CPU amplifier in exactly the
state an attacker drives toward. The sweep is now rate limited to once a second;
the periodic sweep still reclaims on its own cadence, so only the redundant
rescans go away.
An unparseable timestamp was echoed into the refusal detail verbatim. That value
is attacker-controlled and arbitrary length, so it reached a warn! as both a
log-injection and an unbounded-size sink. Only that arm needs sanitizing; the
other two ran through the parser first.
The capacity rationale cited a count of registered DIDs, which this same file
says elsewhere an attacker mints freely through the announce path, so it was not
a bound at all. It now cites the bound that is real: reaching saturation costs a
hundred thousand durable rows and as many sync enqueues inside one retention
horizon, which the database makes loud.
ingest_now was read twice per ingest while its own doc comment claimed the two
layers share one reading, which is the invariant the retention derivation rests
on. Now read once and passed to both.
Also: the Unparseable outcome is driven through ingest rather than only as a
pure function, the periodic sweep is observable without a live swarm, the
restart exposure is written down where the rest of the tradeoffs already are,
and the saturation-counter lock covers every test that can reach Saturated
rather than the two I first found.
* test(node): close the replay guard's remaining proof gaps, and fix the sibling window
Seven tests for the six gaps a review found. The reservation's release path was
reasoned rather than executed: the drop guard was only ever proven through an
early refusal, while the case its own doc comment names, a transient write
failure burning the event's slot, was never driven. Both write directions now
are. An expired entry at capacity must be replaced in place rather than answer
Saturated, confirm on an already-swept entry is pinned as a deliberate no-op,
and the single-critical-section shape is now driven concurrently rather than
asserted sequentially. Restart behavior was documented but untested; a fresh
guard readmits a seen event and the freshness window still bounds it, which is
the composition that makes the restart exposure finite.
check_created in gitlawb-core used a symmetric abs() window, so a request
stamped 299 seconds ahead was accepted and a signer could roughly double a
signature's effective validity by stamping forward. It is now two comparisons
like the gossip path, 300s late and 60s early, with the error naming the
direction so a fast peer and a slow one need different operator action. There
was no future-direction test at all; there are now five covering both. Every
caller was checked and none depended on the symmetry.
Adding a freshness window obliges auditing the siblings, which is how this one
surfaced: the repo argued both ways in two files for a week.
* fix(core): stop an attacker-supplied created timestamp from overflowing the window check
created is parsed as an unrestricted i64 straight from the Signature-Input
header, so the sender picks it. At i64::MIN the past-side subtraction overflows,
which panics a debug build and wraps a release one into a value that can read as
inside the window. Confirmed by execution before the fix: 'attempt to subtract
with overflow'.
Saturating subtraction answers correctly at both ends, since a timestamp that
far out is refused by whichever side it saturates toward. A test drives all four
extremes of the type and asserts each is refused by a named direction; reverting
to plain subtraction reddens it on the overflow.
The symmetric abs() form this replaced had the same hazard, so splitting the
window into two comparisons did not introduce it, but it did not remove it
either. Found by a cross-family review pass after six same-family reviewers and
I had all read the line.
* fix(review): reconcile the replay guard with the unsigned-admission split
Integration only, from rebasing this branch onto #325's current head. #325 now
returns IngestOutcome::UnsignedAdmitted where it used to return Accepted for an
unsigned rolling-upgrade admission, and this branch predates that split.
Three sites. The seen-set bypass test asserted Accepted on both unsigned
deliveries and on the stale-timestamp case; both now expect UnsignedAdmitted,
and the properties under test are unchanged, that unsigned bytes are admitted
twice rather than deduplicated and that the freshness window does not reach
them. The warn-only-on-admission test's budget-spent case gained the
ReplayGuard argument the signature now takes, with a fresh guard because that
case drives an unsigned event the replay block skips.
The ingest match also grows an arm rather than changing one: `None if unsigned`
returns UnsignedAdmitted without settling a reservation, since the replay block
is gated on `verified` and an unsigned admission never holds one.
* fix(node): render rejected gossip timestamps bounded before logging
DateTime::parse_from_rfc3339 accepts arbitrarily many fractional-second
digits, so a signed timestamp can parse successfully and still carry
megabytes of attacker-chosen wire bytes into the TooOld / TooFarFuture
refusal details, which the swarm loop logs verbatim. Only the
parser-error branch sanitized. The root cause was treating successful
parsing as proof the wire representation is safe to expose.
The TooOld and TooFarFuture variants now carry the PARSED instant and
the detail renders its canonical to_rfc3339() form - bounded by this
build's formatter and control-free by construction - while the
unparseable arm keeps sanitize_for_log on the wire string. All three
arms go through one freshness_refusal_detail helper so the invariant
has a single home, plus a regression test driving an overlong but
parseable fractional stamp through both directions.
* test(node): pass the replay guard to the post-fork shed tests
The two ingest tests added on the target branch after this PR forked
(an_over_budget_author_is_shed_without_a_peer_lookup and
the_early_author_check_does_not_track_an_unseen_did) still call
ingest_ref_update with the pre-guard signature. Thread a fresh
ReplayGuard through each call, matching every other test site.
* test(node): drive the live-mesh publish and refusal path end to end
Two real swarms on QUIC loopback with real signatures and real Postgres:
a signed ref-update from one node is admitted and stored by the other, a
re-delivery of the same signed content under a malleated wire encoding is
refused by the replay guard, and an unsigned event is refused under
require_signed. Both refusals are asserted in the warn stream, and a
control event landing after them proves the mesh still delivers.
This closes the seams ref_update_publish_args names as needing a live
swarm: the PublishRefUpdate select! arm dispatching to sign+publish, and
require_signed reaching the receive path. It needs a test-only command
variant, PublishRawRefUpdate, since the production publish path signs and
serializes itself and can never put a malleated or unsigned event on the
wire.
Still uncovered and named in the test's doc comment: the run_ingest_sweep
select! arm on its 300-second interval, and the main.rs half of the flag
wiring above p2p::start.
* fix(node): harden the gossip rejection path against unauthenticated bytes
Two findings on the pre-auth reject path:
- An unsigned event under require_signed was resolved and slug-validated
before the flag refused it, so a hostile flood bought DID resolution and
a rejection reason built from attacker-chosen fields under only the
loose pre-parse brake. The unsigned refusal now runs immediately after
the version gate, before any field is read.
- The swarm loop logged IngestOutcome::Rejected and WriteFailed reasons
verbatim, and several producers embed wire fields (the claimed node_did
in resolve failures, for one). A signed event carrying a control-bearing
node_did reached the operator log with escape sequences intact and no
length bound. Both warn sites now render through sanitize_for_log, so
every present and future Rejected producer is bounded and control-free
at the one sink.
The live-mesh test now also publishes a signed event with a 4096-char
control-bearing node_did and asserts the warn stream shows the refusal
with no escape byte and no run past the sanitizer ceiling. A unit test
pins the new ordering by asserting the exact fixed refusal for an
unsigned event with a hostile node_did.
* fix(node): dedupe the ref-update row on the replay key, not on a random id
The reservation releases on a failed write so a republish can repair the
half that never landed. But each ingest minted a fresh row id, so a retry
after a partial failure (row stored, queue write failed) stored the same
signed content a second time. A replay in a degraded state could repeat a
landed side effect until a budget refused it.
The row id for a verified event is now derived from the replay key, so the
retry's insert dedupes on the existing ON CONFLICT instead of duplicating.
The queue entry still retries, which is the repair the release exists for.
Unsigned events keep a random id: they carry no replay key by design, and
deduplicating unauthenticated content would let an attacker pre-send a
victim's expected event and have the genuine one dropped.
Two existing tests pinned the duplicate write and now assert the deduped
shape: the partial-failure republish repairs the queue entry while the
row stays at one, and the restart readmission's residual effect is a
duplicate queue entry (a refetch), not a second record. The new test
asserts both the retry reaches the writes (WriteFailed, not Replayed) and
the row count stays at one.
* test(node): pass a libp2p identity to p2p::start in the gossip delivery tests
* ci: trigger the PR checks on the retargeted base
---------
Co-authored-by: Kevin Codex <[email protected]>
* fix(node): reject revision syntax in issue ids and drop service passthrough
resolve_issue_id spliced the request-path issue id straight into a git
revision (`cat-file -e refs/gitlawb/issues/<id>`) and a for-each-ref
pattern, and on an exact match echoed the raw input back as the
canonical id. Revision and glob syntax such as `<uuid>^{blob}`, `~0`, or
`*` therefore resolved to an existing issue and was returned to callers
(and stored by create_issue_comment) as if it were the id. Accept only
ASCII alphanumerics and '-' and report anything else as not found.
service_to_command fell through to returning the caller's service string
for an unknown service. Every caller validates first, but the mapping
now yields only the two fixed subcommands and errors otherwise, so an
unrecognized value can never reach git's argv.
Addresses CodeQL rust/command-line-injection alerts #83 and #80.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* test(node): cover service_to_command allowlist and passthrough rejection
Direct unit test ensures unknown services and option-shaped inputs cannot
map to git argv even if HTTP-layer validate_service is bypassed in future.
Co-authored-by: Cursor <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
Co-authored-by: Cursor <[email protected]>
* chore: rebrand user-facing gitlawb references to Twigpine
Rename the product name in docs, README, CLI help/output, MCP tool
descriptions, macOS app strings, installers, npm metadata and license
headers from gitlawb/Gitlawb to Twigpine. Point GitHub links at the
renamed Twigpine org (and fix the release workflow's repository guard,
which still matched Gitlawb/node), and switch the public node hosts and
website/install URLs to twigpine.com.
Protocol and operator identifiers are intentionally unchanged
(did:gitlawb, gitlawb://, signing domains, GITLAWB_* env vars, crate
and binary names, X-Gitlawb-* headers, icaptcha and peer-network URLs);
they need a follow-up migration.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs: point image pulls at ghcr.io/twigpine/node and fix macOS DMG name
The release workflow publishes the container image to
ghcr.io/${GITHUB_REPOSITORY,,}, which is ghcr.io/twigpine/node since the
org rename; the old ghcr.io/gitlawb/node path no longer resolves for
anonymous pulls. Point the RUN-A-NODE docker run example, the OSS
readiness audit's claim about it, and the macOS app's generated compose
file at the published path.
The README listed the macOS DMG as "Twigpine Node.dmg", but
build-macos-app.sh writes GitlawbNode-macOS.dmg (kept on purpose), so
document the real file name.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(infra): keep the AWS security group description to avoid replacement
`description` on aws_security_group is ForceNew, so renaming it forced a
replace of the in-use group on the next apply of an existing deployment.
Restore the original text and note why it must stay until a separate
infra migration.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(macos): regenerate the app's compose file instead of treating it as a repo's
Config.repoPath fell back to the data directory when no repo clone was
found, and DockerCompose then picked up the compose file it had generated
there as if it belonged to a repo. The generator therefore only ran on
first launch, so upgraded installs kept their original compose, including
the old ghcr.io/gitlawb/node image, which no longer serves anonymous pulls.
repoPath now returns nil when there is no clone, and the app regenerates
its compose file whenever no repo compose is in use. Project directory and
named volumes are unchanged.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(infra): default the AWS node image to ghcr.io/twigpine/node
release.yml publishes to ghcr.io/${GITHUB_REPOSITORY,,}, which is now
ghcr.io/twigpine/node; ghcr.io/gitlawb/node denies anonymous pulls, so the
default made new deployments fail to start and the upgrade command fail
on existing ones. On an existing deployment `terraform apply` only updates
the upgrade SSM document in place (user_data drift is ignored); the new
image is pulled on the next upgrade_command.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(macos): keep operator edits to the generated compose, migrate only the image
e04efe4 regenerated the app's compose file on every access, overwriting
edits operators make to pass settings the generated environment block
omits (e.g. GITLAWB_OPERATOR_STRICT_MODE). Generate the file only when it
is missing; when it exists, keep it and rewrite only references to the
retired ghcr.io/gitlawb/node image, keeping tags and digests.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(macos): only migrate complete ghcr.io/gitlawb/node references
Without a left boundary the migration also rewrote the retired path when
it appeared inside a longer reference, such as a registry mirror
(registry.example.com/ghcr.io/gitlawb/node), pointing it at a path the
mirror may not have.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
* fix(node): close the advisory-lock probe's session if it is dropped mid-acquire
A cancelled .await does not cancel an already-sent SQL statement, so a
pg_try_advisory_lock whose future is dropped still takes the lock
server-side while the caller abandons the result, leaving nothing to
release it. The connection then returns to the pool holding the lock and
wedges that repo until sqlx recycles the session.
Introduce LockProbe, which owns the connection across the in-flight
try-lock and closes it in its own Drop if it is still held. close_on_drop
is a one-way setter, so the arming lives in Drop rather than being set up
front and cleared on success; disarming is Option::take, which is what
into_conn does once an acquire is actually observed. This is now the only
place that issues pg_try_advisory_lock.
The committed gate drops a probe without taking its connection, which is
the state a cancellation leaves behind, and polls a standalone observer
until the lock frees. Deterministic on purpose: the timing sweep that
found this window leaks roughly 1 in 600, which is not something a CI
gate can rest on. Observed RED before this change with the lock still
held for the full 10s window.
Refs #279
* fix(node): add a dedicated, lazily-connected advisory-lock pool
Pinning a connection for the lock's lifetime is only safe if those
connections come from somewhere other than the pool serving ordinary
request handlers, otherwise a push burst starves every other query. Add
GITLAWB_DB_LOCK_POOL_MAX_CONNECTIONS (default 32) and a Db::lock_pool
builder, with the sizing tradeoff documented on the field and in
.env.example: every in-flight write pins one connection here, so the
value is a hard ceiling on simultaneous writes node-wide.
The pool connects lazily on purpose. The main pool must connect eagerly
because it runs migrations, which is why it needs connect_db_with_retry's
backoff and degraded-server handoff; that function is not a generic retry
helper and the lock pool is built well after the db-ready handoff has
already resolved. A lazy pool has no startup work, so it adds no new way
for the process to fail to boot and needs no second copy of that
machinery. If Postgres is unreachable when the first write arrives, that
write fails on the pool's own acquire timeout, like any other
database-backed request.
Pure configuration, so no proof-first cycle: the knob is covered by a
parse/default/reject-zero test. Db::lock_pool has no caller until the
guard wiring lands, hence the temporary dead_code attribute.
Refs #279
* fix(node): hold the lock-owning connection in RepoWriteGuard (#279)
Postgres advisory locks are session-scoped: only the backend that took one
can release it. acquire_write took the lock through fetch_one(&pool) and
release unlocked through execute(&pool), two independent checkouts, so the
unlock usually landed on a session that held nothing and returned false.
Measured on main: two writers on one node and the same repo BOTH acquired,
50 of 50 sequential cycles leaked, and 100 writes left 100 orphaned
advisory locks on the server.
The guard now owns the PoolConnection that took the lock, drawn from the
dedicated lock pool, and releases on that same session. The retry loop
probes through LockProbe so a cancellation mid-acquire cannot strand the
lock, and hands the connection back before each backoff so a spinner on a
contended repo does not pin a slot while idle.
Pool exhaustion is deliberately not retried. It is a different condition
from lock contention, and retrying it would spend all 60 attempts on a
capacity problem unrelated to this repo while reporting it as someone else
holding the lock.
Both #279 acceptance tests were observed RED first: the exclusion test
admitted the second writer, and the leak test reported 1 lock held where 0
was required. Both GREEN after. Full crate suite 516 passed.
Db::pool() is removed because this change was its only caller.
Refs #279
* fix(node): free the advisory lock when a write guard dies without releasing
A guard can exit without reaching release(): an early ? on the pre-write
download, a panic, or an axum handler future cancelled when the client
disconnects. Its session still holds the lock, so returning that
connection to the pool would block every future write to the repo until
sqlx recycles it, which on the 0.8.6 defaults is ten minutes idle or
thirty minutes lifetime. Close the session instead and let Postgres free
the lock at session end.
One hazard found by writing the teardown test rather than by reasoning:
PoolConnection::drop spawns onto the runtime for both closing and
returning, and panics outright when no runtime handle exists. That panic
would fire inside a Drop and abort the process during unwind. It is not
introduced by this commit, it comes with owning a PoolConnection at all,
but this is where it becomes reachable. So Drop checks for a runtime
first and, with none, leaks the handle deliberately rather than panicking:
the process is already exiting and socket teardown ends the session, which
is what frees the lock at exit anyway.
Observed RED before the fix, with the lock still held for the full 10s
poll window against a standalone observer on a no-reap pool, and the
teardown case panicking in sqlx-core connection.rs:208. Proven
load-bearing after: neutering the close_on_drop call turns the drop test
RED again. The must-not case (a released guard reuses its backend pid
across four writes on a pool sized 1) passes in both states, so the
signal is specific to the abandoned-guard path.
Full crate suite 519 passed.
Refs #279
* fix(node): read the advisory unlock's result instead of discarding it
pg_advisory_unlock reports "you did not hold this lock" as a false RETURN
VALUE plus a server WARNING, never an error, so let _ = execute(...) could
not distinguish a real release from a no-op. Three blindnesses stacked in
those four lines: execute discards the row, the boolean lives in the row,
and let _ discarded the Result too.
Read it through fetch_one into (bool,). A false means this session's lock
state is not what we believe it is, so the connection stays in the guard
for Drop to close rather than being handed back to the pool as clean. Only
a confirmed unlock returns it. A query error gets the same treatment, since
the lock must not outlive a session we can no longer reason about.
Note this is the only unlock site: the pre-write download's error path
returns through the guard, so Drop covers it and there is no second place
to keep in sync.
RED before: the connection came back on the same backend pid after an
unlock that returned false. GREEN after, and proven load-bearing by
treating false as success, which turns it RED again. The must-not case (a
normal release still reuses its backend) passes in both states, so this
does not over-close the happy path.
Full crate suite 520 passed.
Refs #279
* fix(node): bound the object-storage transfers that run under the write lock
Two transfers happen while the per-repo advisory lock is held: the archive
download inside acquire_write, which runs after the lock is taken and
before the guard exists, and the upload inside release. Both were free
before the guard pinned a lock-pool connection, because the lock's
connection went back to the pool immediately. Now an unbounded stall holds
a lock-pool slot for its whole duration, so enough concurrent stalls deny
every write on the node with no reaping path. Add
GITLAWB_LOCK_HELD_TRANSFER_TIMEOUT_SECS (default 300) and route both
through it.
A timed-out upload is UNKNOWABLE rather than failed, so the timeout arm
takes no compensating action: the PUT may well have landed. The lock is
released either way, which trades a narrow last-writer-wins window for not
wedging the repo behind a stalled transfer. Deliberate, and stated here
rather than discovered later.
Note what is NOT bounded by this knob: acquire_fresh's download. It runs
before any lock is taken, so it holds nothing. An earlier revision of this
change put the bound there by mistake, because both functions contain an
identical download call and the first match won.
Two disclosures.
Coverage: the committed tests cover the bound mechanism, not the wiring.
Driving a genuinely stalled transfer through acquire_write needs either the
object-store abstraction (out of scope) or a process-global
AWS_ENDPOINT_URL_S3 mutation, which would make the suite order-dependent
under the concurrent runner. That a stalled transfer is bounded is
therefore verified by reading, not by execution.
Behavior: on timeout with a local copy present, the download falls into the
pre-existing self-healing fallback and the write proceeds against that
local copy. This widens the conditions reaching that path from
corrupt-or-unreachable to include merely-slow, so a stale tree could now be
written and re-uploaded on a slow link. Kept consistent with the existing
failed-download behavior rather than inventing new semantics here; changing
it is its own decision.
Full crate suite 522 passed.
Refs #279
* fix(node): authorize before taking the write lock in close_issue
close_issue acquired the per-repo advisory lock, then ran the
owner-or-author check, then returned 403 with the lock still held. That was
harmless only because the lock excluded nothing. Making it work, as the
preceding commits do, turns this ordering into a denial primitive: any
caller with repo read access could take the write lock on demand and be
refused the write, while a legitimate writer burned its 60-attempt retry
budget against a lock held by someone with no write authorization. On a
public repo that is every permissionless identity. This series creates the
exposure, so it closes it in the same series.
The owner check is cheap and moves above the lock outright. The author
fallback needs the issue's git-JSON blob, since there is no author column,
so it now reads through acquire() rather than acquire_write(): an issue's
author is set at creation and never changes, so reading it outside the lock
races nothing, and only the mutation needs exclusion.
Two deliberate behavior choices. A non-owner whose authorship cannot be
established gets 403 rather than 404, so this route does not tell an
unauthorized caller whether an issue exists. And the owner's existing
404-for-a-missing-issue path is preserved by re-reading under the guard,
which the mutation wants anyway.
Swept the other three acquire_write sites: merge_pr owner-gates at :200
before acquiring at :214, the repo write path checks did_matches at :916
before :933, and create_issue's read gate is legitimate because that caller
IS authorized for the action it performs. close_issue was the only one with
the wrong order.
RED before: with an independent session holding the repo's lock, a
stranger's request sat in the retry loop until the 3s deadline fired
(Elapsed), which is the wedge itself. GREEN after, refused immediately.
Proven load-bearing: disabling the pre-lock refusal restores the Elapsed.
Full crate suite 523 passed.
Refs #279
* fix(node): address the review findings on the advisory-lock series
Seven fixes from a seven-reviewer pass. Eight of the nine substantive
findings were defects introduced by this branch, and two contradicted
claims made in its own earlier commit messages.
P0, the worst of them. The under-lock download's timeout was folded into
the same Err as a corrupt archive, so it fell through to the
use-the-local-copy fallback and the write proceeded. Two ways that
destroys data: we do not know whether we hold the latest tree, so
re-uploading can overwrite another node's newer archive; and the abandoned
download's extraction runs in an uncancellable spawn_blocking that ends in
remove_dir_all + rename over the same path, so git would have been running
against a directory a background task was about to delete. Now the
Option is branched rather than collapsed: a genuine fetch error keeps the
local-copy fallback, a timeout refuses the acquire and lets the guard's
Drop free the lock.
LockProbe closed its connection on ordinary contention, so a 60-attempt
spinner tore down 60 backends -- the exact behavior an earlier commit
claimed it avoided. The test written to guarantee that asserted the
HOLDER's lock count, which cannot observe the probe's own connection, so it
passed throughout. The correct predicate turned out to be narrower than the
first attempt at this fix: not "we saw an answer" but "we positively know
nothing was acquired," because a true answer means the lock IS held and
dropping without handoff leaks it exactly as a cancellation would. The
first attempt reopened U1's leak and U1's gate test caught it.
tigris.exists() ran unbounded inside the lock-held span, so the knob's
promise was not kept and the docs were wrong about the shape. The HEAD and
the download now share one budget, so worst-case occupancy is one budget
per span rather than two.
close_issue's pre-lock author read used acquire(), whose fast path returns
on a cached directory and never contacts object storage, so on a multi-node
deploy an issue author would be refused their own issue. Now acquire_fresh,
which refreshes without taking the lock -- the property the comment
already claimed. That comment also asserted authorship is immutable, which
a reviewer disproved by force-pushing a forged blob at refs/gitlawb/**;
reworded to the real justification (a pre-check whose mutation re-reads
under the guard). The underlying pushability is pre-existing and larger
than this branch.
The retry loop had no wall-clock cap, reaching ~360s against a 120s proxy
idle timeout that was itself lowered from 600s after held connection slots
caused a production outage. Capped at 90s, with the bail message naming the
real bound.
All four acquire_write call sites stringified their error, destroying the
anyhow downcast that error.rs documents explicitly ("without this, every
database outage surfaces as a 500 instead of a 503"). PoolTimedOut is in
that 503 set, so a saturated lock pool told clients not to retry something
transient. Now propagated.
Tests: six that the plan required and the first pass never wrote. The R4
pool-isolation test (named in the plan as proving the pool split, entirely
absent), close_issue's owner and non-owner-author twins (INV-21c, a
two-principal gate with only its deny arm covered), a waiter-does-not-block-
an-unrelated-repo case, and a config test for the transfer knob. The
runtime-teardown test no longer returns green when DATABASE_URL is unset.
Drop's no-runtime branch now detaches via leak() instead of mem::forget:
PgConnection has no Drop impl, so the socket closes synchronously and the
lock frees immediately rather than at process exit.
Evidence. The probe predicate is proven load-bearing in BOTH directions:
forcing always-close reddens the churn regression, forcing never-close
reddens the cancellation gate, and only the correct predicate satisfies
both. The author twin reddens when the fallback is removed.
One honest limit: the author twin does NOT redden when acquire_fresh is
reverted to acquire. With tigris disabled in tests the two calls are
identical, so that fix is correct by reading, not by execution -- the same
seam that leaves the transfer bound's wiring untested.
Full crate suite 529 passed, clippy -D warnings clean.
Refs #279
* fix(node): log lock-pool saturation in the request path, not via readiness
Completes the shed-in-path half of the availability story. F7 already made a
pool timeout a retryable 503 by letting the sqlx downcast through; this adds
the operator-facing half, with the pool's own size/idle counters so an
incident can distinguish "the pool is full" from "the database is gone"
without reproducing it.
Deliberately NOT a readiness probe. /ready gates Fly routing with no
fail-open, so failing it on a saturated pool would pull this node's READS
out of service too and push its write load onto peers carrying the same
load. That is the downward spiral AWS's health-check guidance names and the
SRE Book documents independently. A lock-pool readiness probe would also add
nothing on the reachability axis, because both pools are built from the same
database_url, so the existing app-pool ping already answers it.
The error is still returned via .context() rather than replaced, because
anyhow preserves downcastability through context layers and the 503 mapping
depends on it.
Cost is bounded by construction: one line per failed acquire, and a failed
acquire has already paid a multi-second pool timeout.
Refs #279
* fix(node): refuse a write when object storage is unknowable, and shed contention as 503
Four defects a seven-reviewer pass found in the previous two commits, three of
them cases where one arm of a match contradicted its neighbour.
The under-lock refresh read a failed Tigris HEAD as "no archive"
(`exists().await.unwrap_or(false)`), skipped the refresh, and wrote against a
possibly-stale tree, then re-uploaded over it. That is the outcome the timeout
arm twenty lines below explicitly refuses. A failed HEAD and a failed download
leave us knowing different things, so they no longer share a branch: only a
download failure after a successful HEAD establishes that the local copy is a
sound thing to fall back to and re-upload. A HEAD failure now refuses the write.
Lock contention that ran out the acquire deadline surfaced as a 500 carrying the
owner slug and repo name in the client-visible body, while pool exhaustion
fifteen lines up returned a deliberate 503. Contention is transient and ordinary,
so it now maps through a typed `RepoBusy` to a retryable 503 with a fixed body;
the detail stays in the log at the raise site.
`lock_not_taken` was assigned after the try-lock answered, so an error left a
previous `true`-derived value standing and could return a lock-holding session to
the pool. It is now cleared before the statement is sent.
`leak()`'s no-panic property in the guard's no-runtime Drop branch depended on
`min_connections == 0` without saying so; a future tuning change would have
silently re-armed a panic inside Drop. Made explicit with the reason.
Two tests that did not bind:
- `waiter_on_one_repo_does_not_block_another` proved only that two lock keys do
not collide. It now samples the pool counters across more than two backoff
cycles. RED at 0/50 samples with `drop(probe)` moved after the sleep, the
exact defect it names, which the previous version survived.
- the exhaustion test's `Ok(Err(_)) | Err(_)` was satisfied by its own outer
timeout. It now requires `PoolTimedOut` and asserts the slots are accounted
for. RED with the pool sized N+1.
`contended_acquire_sheds_as_repo_busy_not_internal_error` is new and drives the
deadline path for real; the deadline became a field so it does not wait 90s. RED
at 500-vs-503 with the downcast removed.
The acquire deadline's docstring claimed a total under the 120s proxy idle
timeout, which the 300s under-lock transfer bound in the same function
contradicts. It bounds the wait only, and now says so. The backoff is clamped to
the remaining budget.
* fix(node): re-authorize close_issue under the guard instead of only re-checking existence
The guarded re-read matched `Ok(Some(_))` and discarded the blob, so the
authorization decision rested entirely on the pre-lock read. A comment above
claimed the opposite. That is not a narrow window: `acquire_write` re-downloads
the archive after locking, so the tree that gets mutated is routinely not the one
the author was read from. With owner-push enforcement defaulting to false and
branch protection covering only `refs/heads/*`, `refs/gitlawb/issues/*` is
pushable, so a forged author blob landing between the two reads was honored.
The blob is already in hand under the guard, so re-asserting owner-or-author
costs a deserialize. A non-owner whose issue has vanished gets 403 rather than
404, matching the pre-check's existing refusal to reveal existence.
`owner_can_still_close_after_the_reorder` seeded the owner as their own issue's
author, which made it unable to fail: with the owner check disabled the author
fallback granted the close and the test stayed green. It now seeds a third party,
so only the owner arm can grant. RED with `is_owner = false`.
Also drops the claim that the author twin covers the acquire-vs-acquire_fresh
distinction. `RepoStore::for_testing` hardcodes `tigris: None`, so the two calls
are identical in every test here and reverting that line leaves the twin green.
Separating them needs an object-storage seam. Stating the gap beats asserting
coverage that does not exist.
* style(node): rustfmt the new contention test
Whitespace only; cargo fmt --check gates the push.
* fix(node): shed an unrefreshable write as a retryable 503 with a fixed body
The two refusal arms added for the under-lock transfer bound returned bare
anyhow errors, so AppError's From impl (which downcasts only sqlx::Error and
RepoBusy) landed them in Internal: a 500 internal_error whose body was the
error string. Two defects in one. A transient object-storage failure told the
client not to retry, and the timeout arm's message interpolated the owner slug
and repo name straight into the response body, contradicting the fixed-body
policy the RepoBusy arm sets six lines above it.
RepoUnavailable follows RepoBusy exactly: a fieldless type raised with the
operator detail in a context string, downcast to its own rung, mapped to a 503
whose body interpolates nothing. The detail stays in the log at the raise site.
The timeout arm logs at error! where its sibling logs at warn!, deliberately: a
300s stall pinned a lock-pool slot for five minutes and is the condition the
transfer bound exists to surface, so it must keep paging when the handler
classifier demotes the ordinary blip case.
* fix(node): refuse a fresh-copy write when the archive HEAD cannot be read
acquire_fresh collapsed a failed HEAD into "no archive exists" via
unwrap_or(false) and served the local copy, while the sibling path under the
lock already refused on the same condition. A push that hit a storage blip got
an advertisement built from a possibly-stale tree, uploaded a whole pack, and
was then refused by acquire_write for the reason the advertisement had already
swallowed. The authorship pre-check on close_issue read the same way, which is
an infrastructure failure resolving toward a denial.
A failed HEAD now raises RepoUnavailable, so both callers surface it as the
retryable 503 rather than a stale success. The download-failure fallback is
unchanged: a present-but-unreadable archive is still self-healed from local.
info_refs keeps its map_err for every other failure so the read path's error
vocabulary does not move; only the typed error takes the From chain.
A test-only TigrisClient constructor pointed at a closed port makes both
refusals executable, so this is no longer verified by reading. It also proves
the under-lock arm end to end, which the earlier draft had recorded as an
untestable gap.
* fix(node): log expected transient acquire failures at warn, not error
Both acquire call sites logged every failure at error, including RepoBusy,
which the raise site already logs at warn. Ordinary write contention paged.
The previous commit widened the problem: info_refs now raises RepoUnavailable
on a storage blip, so that site would have started paging on the condition
this series just classified as transient and retryable.
A classifier over the two typed refusals picks the level at both sites,
mirroring the startup path's permanent-vs-transient split. Anything it cannot
classify still logs at error, so an unknown failure keeps paging.
* fix(node): log the read failure the close_issue pre-check folds into a denial
The authorship pre-check treated a get_issue I/O error and a genuinely absent
issue as the same None, with no log line. The client answer is deliberately
identical, since a caller who cannot write must not learn whether the issue
exists, but the two are not the same event and an operator had no way to tell
a real authorization denial from a filesystem or parse failure behind it.
Splitting the arm leaves the 403 exactly where it was and makes the read
failure visible in the log.
* test(node): poll for the closed session instead of sleeping a fixed 300ms
The release-invariant test slept 300ms for a Drop-spawned close before
comparing backend pids, which is a coin flip on a loaded CI runner. It now
polls pg_stat_activity for the captured pid on a standalone connection, the
same discipline poll_until_free documents: a pooled observer would be handed
the session under measurement and hide the effect.
The conversion was checked against the failure it exists to catch rather than
assumed. Pooling the session on an unlock that returned false makes the test
fail after the poll deadline, not hang, and no_reap_pool disables idle timeout
and max lifetime so nothing but the close under test can retire that backend.
A generous deadline would have been the same defect as the sleep.
* fix(node): raise a failed fresh download with no local copy as RepoUnavailable
acquire_fresh already refused a failed Tigris HEAD as RepoUnavailable, so the
handler layer mapped it to a retryable 503. A failed GET on the same path still
returned a bare anyhow error, which the info/refs map_err closure stringified to
AppError::Git and answered 500. One endpoint therefore told the client a
transient object-storage blip was permanent or retryable depending on which call
failed, and close_issue's pre-check inherited the same split through its bare ?.
Raise it at the source instead of at each consumer: From<anyhow::Error> for
AppError already downcasts RepoUnavailable out of the context chain, so both
callers pick up the retryable mapping without touching either.
The new test drives HEAD 200 with GET 500, which is the exact state the refusal
is for: archive present per HEAD, GET failed, no local copy to fall back on.
* test(node): assert the second writer sheds as RepoBusy instead of timing out
two_writers_on_the_same_repo_are_not_both_admitted wrapped the second
acquire_write in an 8-second outer timeout and only checked the future had not
finished. That passes for any stall, including lock-pool saturation or a slow
CI box, so it could not tell a working shed from an unrelated hang, and it cost
8 seconds on every suite run.
Use with_lock_acquire_deadline and assert the typed RepoBusy downcast while the
first guard is still held, matching what contended_acquire_sheds_as_repo_busy_
not_internal_error already does. It now also fails loudly if a second writer is
admitted, which the timeout version could not distinguish.
* fix(node): read the close_issue author pre-check from a non-mutating snapshot
The non-owner author fallback refreshed through acquire_fresh, which publishes
into the live repo directory: its extract step removes the existing directory
and renames the new one into place. That runs with no write lock held, so any
signed non-owner could trigger a directory swap underneath an in-flight
guarded write on the same path.
read_snapshot downloads to a throwaway temp dir and hands back a RepoSnapshot
that removes it on drop, so the pre-check still sees fresh data and the live
path is never touched. download_to grows a publish flag to serve both shapes
from one path.
The wedge invariant still holds: a stranger is refused without waiting on the
write lock.
* fix(node): bound every await in the lock-acquire loop by the deadline
The remaining budget was checked before the pool checkout but bounded neither
the checkout nor the pg_try_advisory_lock query that follows it. A checkout
starting just under the deadline could wait out the pool's own acquire timeout,
and a slow query could be accepted after the budget was spent, so the advertised
wall-clock cap held only on the fast path.
Both awaits now run under the remaining budget and shed as RepoBusy when it runs
out. The probe's Drop closes its session, which cannot hold a lock it never
confirmed taking, so the query-timeout arm is a plain shed.
* test(node): add an S3 mock that really enforces conditional writes
The fence work landing next is only as good as what the tests can observe, and
a mock that answers 200 to every PUT would make the whole suite vacuous. This
one holds the object and its ETag, refuses a mismatched If-Match and an
If-None-Match "*" over an existing object with 412, and mints a fresh ETag per
successful PUT so two byte-identical archives never share a token.
Capture-then-replay is deliberate rather than parking a handler and hoping it
resumes: when tokio drops an SDK future the client can tear the connection down
and cancel the server task with it. Replaying what arrived models the arm that
matters (body fully transmitted, commit decided later) with no timing in it.
Six tests pin the semantics in both directions so a hollowed mock cannot hide.
* fix(node): stop pretending a held lock fences a late upload
The timeout arm claimed that keeping the advisory lock protected a successor
from an abandoned PUT. It does not. release takes mut self, so the guard drops
the moment it returns and Drop closes the session; measured on this branch, a
successor took the same repo's lock 5ms after release returned while the PUT
was still in flight.
The comment and warn now say what is actually true: the outcome is unknowable,
the PUT may still land, the lock releases normally, and a conditional upload is
what keeps a late publish from overwriting a successor's archive.
Two tests replace the claim. Session disposition is the observable that
separates the two shapes, so the unlock is pinned to run and be confirmed on the
guard's own session with the connection returned to the pool, checked by backend
pid. Successor admission is pinned too, but noted as not what proves the point,
since the lock frees within milliseconds either way.
* feat(node): let the storage client express and detect a conditional write
upload takes an UploadPrecondition (IfMatch, IfAbsent, Unconditional) rather
than an Option<String>, so the absent case and the deliberate no-fence case are
distinguishable at the type level and a caller cannot lose the fence by passing
None. head_etag reads the current ETag alongside exists, which keeps exists and
its other callers untouched.
A failed precondition has to be classified off the raw HTTP status: PutObjectError
models no PreconditionFailed variant, so a 412 arrives as Unhandled with nothing
useful on it. 412 is always a lost precondition and 409 is one under IfAbsent.
404 deliberately is not: no archive delete exists on this line, so a 404 on a
conditional PUT means a wrong bucket or endpoint, and reporting that as retryable
would send clients into a loop against a permanent fault.
The three background uploads outside the write guard now publish with IfAbsent.
They fire only where the archive is expected absent, and leaving them
unconditional would defeat the fence from the side: init uploads an empty bare
repo, so a push landing just before it could have its archive replaced by that
empty one. A refusal there means someone else already published the key, which is
logged as the correct outcome rather than a failure.
* fix(node): fence the release publish so an abandoned upload cannot win
acquire_write now reads the archive's ETag under the advisory lock and the guard
carries it, so release publishes conditionally on the generation it actually
refreshed from. An upload abandoned by an earlier writer no longer overwrites a
successor's archive: the store rejects it, because the ETag it was written
against is gone.
A refused precondition gets exactly one supersede-retry, never a loop. The
distinction that makes this sound is that a 412 is a definite answer, unlike the
timeout arm where nothing is knowable, and the retrying writer still holds the
lock, so whatever landed underneath was written without one and its tree is not
the authority. Two losses in a row refuse instead of escalating.
That retry is what keeps ordinary pushes working now that init publishes
create-only: a first push racing init's upload of the empty repo loses once and
then wins, rather than surfacing a 503 on the most common operation there is.
release returns a must-use outcome and the four publishing handlers propagate it
before any trust bump, webhook, or success body, so a publish the store refused
reads as a retryable 503 instead of a 201. The three release(false) sites
deliberately do not map it: they publish nothing, and a 503 there would shadow
the 403 or 404 the route means to return.
The download-failure fallback also publishes fenced now. That arm knows the
stored generation (its HEAD succeeded, only the GET failed), so publishing
unconditionally from it would reintroduce the same overwrite.
* test(node): drive the abandoned-upload race end to end, and probe the real backend
The headline test is the one that had to exist. Writer A's release is parked past
its transfer bound and returns with the outcome unknowable, B acquires and
publishes, and A's captured PUT is then replayed: the store answers 412 and B's
archive survives. The create-only variant covers the arm whose real-world failure
is silent rather than loud.
The control is what makes those attributable. With no interleaved B, an
abandoned-then-replayed PUT whose generation still matches lands. Without it the
headline would only show that replays get rejected, not that staleness is what
rejects them.
Header assertions come last in all three, so a lost fence reds on the outcome it
is about rather than on a wire-format check.
A mock cannot prove Tigris honors any of this, and the vendor requires a
Single-region or Multi-region bucket for conditional operations, so against a
Global or Dual-region bucket the fence is a silent no-op. The credentials-gated
probe checks both arms against the real endpoint and cleans up unconditionally,
including when an assertion fails, which is the case it exists to catch. It
accepts 412 or 409 on the create-only arm because both mean the precondition was
enforced and both are already classified as a loss.
* fix(node): reconcile F-series tests and test seams with the rebased guard shape
The rebase onto main carried main's #174 F-series guard tests forward. The
branch's guard replaces main's locked/released/test_pre_unlock_gate Drop
mechanics with close-on-drop (runtime) and leak (off-runtime), so two tests
that asserted the replaced mechanics are reconciled: the off-runtime disposal
test now asserts the leak observable (the slot never returns to idle) and the
detached-unlock-returns-connection test is dropped, its invariant covered by
the U-series drop-frees-the-lock gates. The pre-unlock gate seam is restored
(test-only) so the mid-unlock cancellation tests keep their deterministic
park. ipfs.rs tests are updated to the branch's sync test-client constructor
and 4-arg RepoStore::new.
Refs #279
* fix(node): address the 2026-08-10 review round on the advisory-lock series
- P1: serve the receive-pack advertisement from a non-mutating snapshot
instead of acquire_fresh, so the unlocked advertisement cannot delete or
swap the live repo directory under a concurrent guarded write (the stale-ETag
success ordering). acquire_fresh loses its only production caller and is
removed.
- P1: read-gate, rate-limit, bound, and cancellation-clean the close_issue
pre-lock snapshot, so a signed non-owner cannot drive unbounded Tigris
downloads and blocking extractions, and an abandoned extraction no longer
leaks its temp dir.
- P1: classify raw 409/412 responses as a lost conditional write via
SdkError::raw_response() in both upload and the publish supersede-retry, so
an unparsable error body cannot acknowledge a write that was never published.
- P2: wrap the cold-cache under-lock download failure in RepoUnavailable so it
sheds as a retryable 503 like the HEAD arm, not a permanent 500.
- P2: surface a refused create-only fork upload as PreconditionLost and refuse
the fork, so an orphan archive cannot shadow a fork's DB record.
Refs #279
* fix(node): record why the close-issue read gate cannot carry a synced row
The gate added for the pre-lock snapshot reads a repo row's own visibility
rules and public flag. A row synced from a peer is stored public and carries
none of the owner's rules, so for that class the gate can only return allow.
Refusing instead would deny the repo's real owner and the issue's real author
on any node whose only copy is a synced one, and it would not buy the
protection it appears to, because a synced row's recorded owner comes from the
peer that sent it. Every other read gate in the API reaches the same verdict
for such a row, so this is left as is and stated at the call site rather than
special-cased here.
The test that covered the gate seeds a locally created repo and cannot observe
this. Adds one that pins the property directly: the gate allows an arbitrary
caller on a synced row, and the handler refuses that caller anyway, because the
owner-or-author check is what decides this route.
* fix(node): address jatmn review on merged advisory-lock head
Align .env.example db pool with validate floor, move snapshot cleanup into
the blocking extract task, remove fork clones on PreconditionLost, restore
upload_site_reached test plumbing, and use owner DIDs in receive-pack tests
after #330 owner-push default.
* fix(node): cap close_issue snapshot work on the read pool
Take a git_read_semaphore permit before the non-owner author snapshot so
parallel close attempts cannot each drive a full archive extraction while still
under the hourly rate bucket. Format owner-DID test helpers.
* fix(node): mark close_issue mirror visibility gate for push hook
Annotate list_visibility_rules with mirror-rows-handled so the pre-push
surface detector accepts the synced-row read gate.
* fix: unblock pre-push after main merge
Read CappedBody.text in gl whoami (#381 shape), restore legacy CID sweep
wiring dropped in the merge, route rate-limiter cleanup through AppState,
and allow build_lock_pool for test-only callers.
* fix(node): carry transient acquire failures through every write handler
Extend acquire_write_app_error so RepoBusy and RepoUnavailable map to
fixed-body 503s on issue and pull paths, matching receive-pack. Fix the
flaky u2 drain exhaustion test via a drain_faults seam and add handler
regressions for contention and storage unavailability on create_issue.
* fix(node): drop unused logcap helper after drain_faults seam
* fix(node): route fork disk paths through validated_repo_disk_path
CodeQL flagged remove_dir_all on a user-derived path in fork_repo. Use the
same three-layer barrier as RepoStore::local_path and reject names the
validator refuses before clone.
* fix(node): harden release-upload outcomes and replication tail gating
Treat timed-out or failed archive uploads as non-durable in into_result,
gate pinata/gossip on confirmed publish durability while keeping inv22 spawn
order, split lock-pool vs main DB pool validation, map LockPoolBusy through
From<anyhow::Error>, and attach Retry-After on repo write contention errors.
* fix(node): close jatmn round on publish authority and durability gating
Revoke under-lock refresh swaps when the advisory lock ends, fail closed on
unrecorded release outcomes before Pinata/gossip, compensate fork archives when
create_repo fails, recompute acquire backoff from the live deadline, and route
publish swaps through validated_repo_disk_path for CodeQL.
* fix(node): harden issue close/create and fork publish paths on #285
Move close_issue rate limiting after the read gate so denied readers see 404
not 429, roll back local issue refs when publish refuses, re-check swap
authority under publish_lock, propagate fork upload failures before DB insert,
and update lock-pool config help text.
* fix(node): close write-attempt ownership gaps on advisory-lock release
Make publish swaps claim-exclusive via CAS, invalidate the local read cache
on definite publish refusals, harden fork create compensation and clone
cleanup, and keep snapshot temp dirs owned through the async handoff.
* fix(node): gate replication tail on publish durability and split close_issue limits
Wait for a confirmed release outcome before post_receive_replication_tail runs
any walks or coalescing work. Give close_issue its own per-IP rate bucket so
it cannot drain receive-pack quota. Add regression tests for both paths.
* fix(node): add ValidatedRepoDiskPath for CodeQL and reconcile tail durability
Introduce a validated-path newtype with the same inline join and component
walk main uses, so path-injection sinks only accept sanitised repo paths.
Reconcile the publish-durability gate with the disconnect-safe tail spawn via
PublishDurabilitySlot and explicit UploadUnknowable handling.
* fix(node): close three jatmn findings on advisory-lock PR
Retry fork create confirmation before compensating or returning, schedule
background recovery when the lookup stays unavailable, install publish
durability outcomes before the recorded flag, and size the lock pool for
non-push mutations (default 40).
* fix(node): verify adversarial findings on advisory-lock PR
Fail closed when publish durability never records, bound pg_advisory_unlock
with the same transfer budget as upload, and cap close_issue pre-lock snapshots
at git_acquire_timeout_secs instead of the write-lock transfer bound.
* fix(node): compensate under the write lock and harden tail durability handoff
Run create_issue rollback through release_compensating so issue refs are
deleted only while the advisory lock is still held and only on definite
publish refusals, not UploadUnknowable. Replace the tail durability slot's
async try_lock drop with a std mutex so cancellation during polling cannot
skip installing UploadUnknowable.
* fix(node): refuse stale under-lock refresh and gate tail durability on release
Under-lock GET failures now always shed as RepoUnavailable, even when a local
copy exists, so a transient download error cannot publish over a newer stored
generation. PublishDurabilitySlot only synthesizes UploadUnknowable after
release starts, and receive-pack marks that boundary before spawning the tail.
Adds regression tests for both paths; inv22 F4 gate accepts the release wrapper.
* style(node): rustfmt advisory-lock gap-closure commits
* fix(node): satisfy clippy on stale-local download test path join
* feat(node): add a publish attempt/stage boundary
The four lifecycle findings on this branch share one root cause: a resource
was tracked by a coarse state ("release started", "the path exists", "a row
with this name exists") when the safety question was about the IDENTITY and
the PUBLICATION STAGE of one specific write attempt.
This is the vocabulary that makes those questions answerable:
- `PublishAttemptId` — minted before the request is built, carried with the
bytes as object user metadata, read back off the store to decide whether
what is published is THIS attempt's work. That turns "did my request
succeed", which a lost response makes undecidable, into "are the published
bytes mine", which the store can answer.
- `PublishStage` / `PublishStageCell` — how far one attempt got, observable
from OUTSIDE the future doing the work. A cancelled handler never returns
an outcome, so the stage it had reached is the only thing that can classify
it.
- `UploadError` split by what a failure ENTITLES A CALLER TO DO rather than
by which HTTP status came back. Destructive compensation is licensed only
by `proves_not_published()`.
Deliberately free of any object-storage type. #79 deletes `git/tigris.rs` and
replaces it with a BlobStore layer; this module is what survives that swap,
with each backend supplying only its own error classifier.
* feat(node): stamp uploads with an attempt id and preserve dispatch ambiguity
`upload` now reports its progress into a `PublishStageCell` and stamps every
PUT with an attempt id in object user metadata, so both of the questions a
cancelled or unanswered publish raises become answerable:
- `PreparingArchive` is marked before the blocking compression and
`PutDispatched` immediately before `send()`. The conditional PUT is not
constructed until compression returns, so a caller abandoned in that window
definitely never attempted publication — a fact nothing could previously
observe, because the only report was the return value of a future that no
longer existed.
- `attempt_landed()` HEADs the key and compares the stored attempt id, which
is what lets a client whose response was lost recover its own committed
write instead of guessing.
`UploadError::Other` is replaced by `NotPublished` (proven never committed)
and `Ambiguous` (may have been dispatched and may have committed). The single
backend-aware classifier is `classify_put_failure`: a 4xx is an answer the
server gave before storing anything and proves non-publication; a 5xx does
not, and neither does a timeout, a dispatch failure, or a response the SDK
could not read. `SdkError` is `#[non_exhaustive]`, so the fallback arm is the
cautious one.
`delete` is replaced by `delete_if_attempt_matches`, which reads the attempt
off the object and fences the DELETE with `If-Match` on the generation it
came from. A second name lookup before an unconditional delete would only
narrow the window in which a successor's object can be erased; the
conditional delete closes it.
* fix(node): key durability, cache validity and fork cleanup on the attempt
Closes the four lifecycle findings by consuming the attempt/stage boundary at
the sites that were deciding on coarse state.
P1 — cancellation before PUT dispatch is not publish durability.
`PublishDurabilitySlot` is armed on the guard's publish STAGE rather than on
a "release started" flag, and its `Drop` classifies an abandoned handler by
how far the attempt actually got: a cancellation during compression records a
definite refusal, a cancellation after dispatch records unknowable, and a
cancellation after the store acknowledged records `Released` (which the flag
could never say). `publish_durability_confirmed` now requires `Released`, so
the detached tail no longer does IPFS/Pinata/P2P/GraphQL/Arweave/peer work
for refs that exist only in a rejected local write. A guard with no storage
backend seeds `NoBackend`, keeping Tigris-less deployments' tails running.
P1 — an unknowable generation is quarantined, not served as ordinary cache.
`release` writes a sidecar marker naming the unresolved attempt beside the
live tree (never inside it: it would be tarred into the next archive), and
`acquire`'s existing-path fast path reconciles it before serving. Only the
store confirming it holds that attempt lifts the quarantine; anything else is
a retryable `RepoUnavailable`. The tree is deliberately NOT deleted — the PUT
may have landed and this can be the only local copy. A confirmed publish, an
under-lock refresh, and cache invalidation each clear the marker, so the
quarantine is a bounded refusal rather than a standing outage.
The release-side timeout arm also stops treating every stall alike: a bound
that expires at `PreparingArchive` is a definite non-publication, and a bound
that expires after dispatch spends one short, separately bounded HEAD trying
to reconcile the attempt before falling back to unknowable.
P1 — fork confirmation and cleanup bind to the creating attempt. The DB row
id is minted up front and used as the attempt id, so the object's metadata,
the clone's sidecar stamp and the row all name one attempt.
`confirm_fork_repo_row` looks the row up BY ID and reports Ours / Foreign /
Absent, so a concurrent create, mirror registration or retry can no longer be
returned as this request's own commit. Every destructive step —
`compensate_fork_archive`, its background retries, and `ForkCloneGuard::drop`
— is conditional on the resource still belonging to this attempt, so recovery
that observed `None` before a successor committed can no longer erase that
successor's archive or directory afterwards.
P2 — response-loss ambiguity is preserved rather than compensated. Guarded
writes map an ambiguous publish to `UploadUnknowable` + quarantine instead of
`UploadFailed` + cache invalidation + the caller's undo, so `create_issue` no
longer deletes an issue ref whose archive is durable. Fork creation
reconciles by attempt id: a create-only PUT that committed and lost its
response is RECOVERED and the row inserted, and an unresolved one keeps its
only local clone and refuses retryably. The fork name is no longer fenced
behind the attempt's own orphan.
Tests, each RED-checked by reverting its guard:
- slot_drop_during_compression_records_a_definite_non_publication
- slot_drop_after_dispatch_records_unknowable
- slot_drop_after_the_store_acknowledged_records_released
- slot_drop_with_no_storage_backend_records_released
- publish_durability_confirmed_accepts_only_released
- publish_durability_confirmed_refuses_quickly_after_unrecorded_slot_drop
- receive_pack_cancelled_during_compression_publishes_nothing_and_runs_no_tail
(+ receive_pack_that_completes_its_publish_still_runs_the_tail as control)
- a_bound_that_expires_before_dispatch_is_a_definite_failure
- an_unresolved_publish_quarantines_the_tree_and_a_later_read_refuses
- a_quarantined_tree_is_served_once_the_store_confirms_the_attempt
- the_next_write_clears_an_inherited_quarantine
- a_confirmed_publish_leaves_the_tree_readable (control)
- a_guarded_write_whose_response_is_lost_is_not_compensated
- a_put_that_commits_and_loses_its_response_is_ambiguous_and_reconcilable
- a_closed_response_with_no_http_status_is_ambiguous
- upload_classifies_404_as_a_definite_non_publication
- upload_classifies_500_as_ambiguous_not_definite_failure
- a_generation_that_moves_between_the_head_and_the_delete_is_not_deleted
- an_attempts_own_object_is_deleted_and_a_foreign_one_is_not
- fork_confirmation_never_claims_a_concurrent_attempts_row
- fork_confirmation_recognizes_this_attempts_own_committed_row
- fork_confirmation_reports_absent_when_nothing_owns_the_name
- fork_clone_guard_leaves_a_successors_mirror_alone
- fork_recovery_resuming_after_a_successor_deletes_neither_object_nor_path
- fork_compensation_removes_what_this_attempt_still_owns
- fork_publish_that_loses_its_response_stays_recoverable
- a_fork_whose_publish_lost_its_response_is_recovered_not_fenced
- a_fork_whose_publish_is_unresolved_keeps_its_clone_and_refuses_retryably
- an_ordinary_fork_publishes_and_commits (control)
- mock_round_trips_the_attempt_metadata_a_put_stamped
The S3 mock gains attempt metadata, conditional DELETE, a commit-then-lose-
the-response mode, a delivered-but-not-committed mode, and a per-key object
store (fork creation touches the source and fork keys in one request, and a
single slot would have let an assertion about one silently read the other).
The compression seam is a condvar gate so no test holds a guard across an
await.
* fix(node): settle the cache contract on the abandoned path and the attempt
The publication boundary this branch introduced was applied at call sites
that each had to be remembered, and five were not. Close the eight gaps
that survived, grouped by what actually failed rather than by symptom.
The marker's lifecycle was answered by the wrong event in three places.
The swap that replaces a live tree now owns the clear, so a cache-miss
download no longer leaves a stale marker that fences the archive it just
installed, and the under-lock refresh no longer lifts a quarantine on a
HEAD that downloaded nothing. init clears a predecessor's marker.
A cancelled handler never reached release, so it applied the tail
contract and not the cache contract. RepoWriteGuard::drop now settles the
tree it abandoned, keyed on how far the publish actually got, and a guard
that handed out its path for writing marks the tree whatever stage it
reached. A marker naming no attempt is a definite non-publication rather
than a permanent refusal: it invalidates and the next read serves the
stored generation, so an abandoned first push cannot wedge a repo at 503.
Only an unreadable sidecar still fails shut.
Both of the store's live-path hand-outs now go through one confirming
helper, so read_snapshot can no longer return an unconfirmed tree as an
ordinary snapshot. The gate binds ahead of the lazy-migration spawn,
because a quarantined tree backfilled to object storage by an IfAbsent
PUT would publish exactly the refs the marker exists to withhold. The
marker no longer rests on a single fs::write; an unwritable sidecar falls
back to an in-memory record that reads consult.
Fork cleanup claims ownership by renaming the stamp before it touches the
tree, so a successor that claims the same path cannot lose its directory
to a predecessor's check-then-unlink, and the directory-absent path no
longer unlinks a stamp it does not own. A create-only precondition loss
whose reconciliation HEAD fails is refused retryably instead of reported
as a permanent name conflict, and its clone is released so the retry can
re-clone rather than finding the destination occupied.
The pin sweep skips a tree whose publish is unresolved, so provider CIDs
are not rewritten from refs that may never become durable. That is the
sidecar marker, distinct from the operator-level quarantined row flag.
26 tests, each written and observed failing before its fix. Suite 1211
passed, 0 failed, 1 ignored; fmt, clippy -D warnings and --locked clean.
* fix(node): settle the abandoned tree at the disconnect, not at reaper exit
The settlement that marks an abandoned write ran from RepoWriteGuard's
Drop, and on a client disconnect that guard is held by the Arc clone
riding the admission guard into the detached reaper, so it ran only once
the process group had been reaped. Reads take no advisory lock and
consult only the sidecar, so for the width of that window a concurrent
fetch was served the abandoned refs off the live tree with no marker
beside it. Confirmed by execution before the fix: no marker on disk,
acquire returning the live path, and the abandoned ref still on the tree
it served.
Only the lock release has to wait for the reaper; the settlement does
not. TreeSettlement is a token carrying the path, the publish stage and
the hand-out flag, holding the exhaustive stage match that used to live
in Drop. The handler takes it out of the guard before the guard is
shared with the reaper, so on a disconnect the handler future drops
first and the marker lands at the disconnect instant while the reaper
still holds the lock. A successful release disarms it. Drop still
settles a guard nobody took the token from, so the arms have one home.
The regression test reads inside the window and asserts the abandoned
refs are not served. It samples pg_locks from an independent session
first, so a run where the reaper had already finished reports itself
inconclusive rather than passing for the wrong reason.
Suite 1212 passed, 0 failed, 1 ignored. The five settlement mutations
are load-bearing, including one that reverts this split and reddens the
window test.
* fix(node): let exactly one classifier settle an abandoned tree
Moving settlement into a token the handler holds left two classifiers
able to run for one write. release() classifies the tree and only then
awaits the unlock, while the handler disarms the token after that await
returns, so a cancel in the unlock gap dropped a still-armed token and
settled a second time. After a definite refusal that is a quarantine
marker written beside the tree release had just deleted, which sends
every later read down the refuse-and-reconcile path for a repo whose
state was already resolved. Observed before the fix: the tree gone, the
marker there.
The arm is now an AtomicBool shared between the guard and its token.
release stores false at the classification point, above the unlock
await, and settle takes the arm with compare_exchange, so whichever
classifier gets there first is the only one that acts.
tree_settled stays, and now has a test that says why: after the handler
takes the settlement the arm is still true, so tree_settled is the only
thing stopping the guard's own Drop, which on a disconnect runs in the
detached reaper, from settling again at reaper exit.
The reap-window test was passing for weaker reasons than it claimed. Its
fake git took SIGTERM and let the group die, so the reaper could free
the advisory lock before the probe read; it now traps TERM and re-parks
so only SIGKILL ends it. It asserts the marker exists at the disconnect
instant rather than only mentioning it, and the served-refs half no
longer passes when acquire fails for an unrelated reason.
Suite 1214 passed, 0 failed, 1 ignored. Nineteen mutations load-bearing,
including one that reverts the release-side disarm and one that reverts
the settlement split.
* test(node): use the test's own pool now that Db::pool is gone
---------
Co-authored-by: Kevin Codex <[email protected]>
* fix(node): drop withheld-subtree trees from the replication pin set
A tree reachable only under a denied path survived both pin filters because
trees were never in the blob universe, so public IPFS still learned child
filenames and blob oids. Filter trees the same way as blobs on the delta
withheld set and the full-scan allow-list.
Closes#172.
* chore: re-trigger code quality check
* fix(node): require Content-Digest on a signed request
require_signature rebuilds the RFC 9421 signing string from the request
itself, and it read the Content-Digest header with a .unwrap_or("")
fallback. When the header was absent, the covered content-digest component
came out as the empty string on both the signing side and the verifying
side, so the Ed25519 check passed over a string that committed to nothing
about the body. The comparison that would have caught it was guarded on the
presence of that same absent header, so it was skipped. The two defects
compose: a signed request could carry any body at all, and the handler
behind the middleware saw it as authenticated.
The header is now required. An absent Content-Digest is a 400 with
missing_content_digest, and the digest-versus-body comparison is
unconditional, since presence is established before it runs. There is no
longer a path through the middleware on which the body goes uncompared.
Refusing is safe for every client that already works. sign_request computes
the digest with no branch and always returns it in SignedHeaders, and all
eight production call sites attach it as a Content-Digest header: the three
in gl's HTTP client, the fetch and push requests in git-remote-gitlawb, the
node's peer announce, the sync push in sync.rs, and the sync notify in
api/repos.rs. So no conforming client regresses. What the refusal removes is
the case where a caller signed an empty digest and sent a body no signature
covered.
HttpSignature::parse now also refuses a covered component that appears more
than once, per RFC 9421 section 2.1. A repeated identifier says nothing the
single one did not, and it lengthens the signing string a verifier has to
build from a list the caller chooses.
* fix(core): linear duplicate check and reject component parameters
Use a HashSet for covered-component duplicate detection so hostile
Signature-Input headers cannot force quadratic work before rejection.
Reject RFC 9421 component parameters during parse since only bare names
are supported end to end.
* chore: re-trigger code quality check
* fix(node): validate git branch refs to close option injection
PR branch refs and a repo's default_branch were stored from request bodies
with no ref validation, then interpolated into single git argv elements:
git diff {target}...{source} (branch_diff / branch_diff_names), and
git worktree add ... {target} / git merge {source} (merge_branch). A value
beginning with '-', e.g. --output=/tmp/x, is parsed by git as an option
rather than a revision, so it becomes an arbitrary file write. get_pr_diff
takes an optional identity, so on a public repo the trigger is unauthenticated;
planting the PR needs only read access, and the write happens at the withhold
check before the visibility gate.
Defense is applied at two layers:
- Storage boundaries: create_pr validates source_branch and the resolved
target_branch; create_repo validates default_branch (which becomes a PR's
target when the PR omits one). These fail fast with 400 and keep junk out
of the DB.
- The sink: branch_diff, branch_diff_names, and merge_branch reject an
option-shaped ref before building the git argv, so the property holds for
every caller and every row, including legacy rows and any future writer
that skips the boundary check.
The shared validator is crate::git::store::validate_git_ref (git
check-ref-format rules, leading-dash rejection as the core), re-exported as
crate::api::validate_git_ref for the boundary handlers. No -- delimiter is
used: the arguments are revisions, and -- there reinterprets them as
pathspecs.
Both boundary guards and the sink guard are mutation-proven load-bearing.
resolve_head is unaffected (it prefixes refs/heads/); fork_repo takes no
branch from the request.
* fix(node): delegate branch ref validation to git check-ref-format
Rebase onto current main and tighten validate_git_ref to call
git check-ref-format --branch so symbolic names like HEAD and
trailing-dot components are rejected at storage boundaries and
git sinks. Add boundary and legacy-row tests for HEAD, feature.,
and feature/x.
* fix(node): reject fully qualified refs and close review test gaps
Reject refs/ prefixes after check-ref-format so tags and qualified
head names cannot be stored as PR branch fields. Add sink tests for
option-shaped merge source, HEAD target, and poisoned default_branch
resolution; assert get_pr_diff fails on poisoned diff rows.
* fix(node): allow-unbounded-git for check-ref-format validation
Stateless ref-name validation holds no repo concurrency permit.
* style: cargo fmt
* fix(node): reject revision shorthands and split ref-validation errors
git check-ref-format accepts @ and pseudoref names that git later treats as
symbolic revisions, so merge could succeed without merging the source branch.
Reject those explicitly at the shared validator and add sink/boundary tests.
Spawn failures from check-ref-format now surface as git_error (500) instead of
bad_request (400) at create_pr/create_repo boundaries.
* test(node): close ce-review gaps on git ref validation
Add sink tests for @{-1} and pseudoref poisoned rows at diff/merge,
HTTP tests that spawn failure returns git_error (500) not bad_request,
and expand API boundary cases for remaining pseudorefs and qualified refs.
Introduce GITLAWB_TEST_VALIDATE_GIT_BIN test seam with a mutex so parallel
sqlx tests do not race on the override.
* fix(node): prefix sink git argv with refs/heads and close shorthand gap
Pass stored branch short names to diff/merge/worktree as refs/heads/{name}
so revision shorthands cannot retarget another ref namespace at the sink.
Reject heads/, tags/, and remotes/ prefixes at storage boundaries, extend the
pseudoref denylist, and route boundary validation through AppState::git_bin
instead of a process-global test env override that raced parallel tests.
* fix(node): attach merge worktree to the target branch
git worktree add with refs/heads/{name} checks out detached HEAD, so a
successful merge did not advance refs/heads/{target}. Use the branch name
for the worktree and keep refs/heads/ qualification on merge revisions.
Add an end-to-end regression for diverged branches.
* chore: cargo fmt on merge ref-advance test
* fix(node): bind merge worktree to the exact local target branch
validate_git_ref proves branch-name grammar only; git's revision DWIM can
resolve the same short name in another namespace (refs/tags/{name}), so a
PR targeting a tag-shadowed name checked out a detached HEAD, merged onto
it, discarded the commit at cleanup, and the unchecked plain rev-parse
echoed refs/heads/{target} back as a bogus merge SHA — marking the PR
merged and firing pull_request.merged with no branch advanced.
Close the resolution class instead of denylisting names: (1) require the
exact refs/heads/{target} via show-ref --verify before creating the
worktree, (2) verify the worktree's symbolic HEAD is attached to exactly
that ref before merging, (3) accept the merge only if rev-parse --verify
{target}^{commit} exits zero. Add end-to-end regressions for a same-named
tag with no local branch (merge fails, PR stays open, no merged event) and
for a tag/branch coexistence (the local branch advances, the tag stays).
* fix(node): recover a leftover merge worktree instead of wedging the endpoint
merge_branch cleaned up a leftover _merge_worktree only when the directory
still existed. The state git worktree add actually refuses is the opposite: a
registration under $GIT_DIR/worktrees whose directory is gone. It fails with
"missing but already registered" (exit 128), and once a repo is in that state
every later merge on it fails, permanently.
The state is self-inflicted. remove_worktree runs git worktree remove and then
removes the directory unconditionally, so one failed removal leaves the
registration behind.
Two changes. Run the cleanup unconditionally at entry, which also makes the
bail-without-cleanup paths self-healing on the next merge. And pass --force
twice: a single --force refuses a LOCKED worktree ("cannot remove a locked
working tree"), and git worktree prune skips locked registrations too, so with
one flag a locked leftover survives the whole cleanup and git worktree add
then fails with "missing but locked worktree".
Enumerated the leftover states that can reach this code (registered or not, x
directory present / gone / locked / corrupt admin dir, x gc.worktreePruneExpire
default and never). remove -f -f plus the directory removal clears all of them,
and adding a separate git worktree prune changes no outcome in any of them, so
there is no prune call here.
Four tests, one per leftover shape that reaches this code. Against
merge_branch: the plain stale registration, the locked variant, and an
unregistered leftover DIRECTORY (git worktree remove refuses that one with "is
not a working tree", so the directory removal is what clears it). Plus an
end-to-end pass through the merge endpoint on the locked state, asserting the
PR row reaches "merged", the reported merge_sha IS the new refs/heads/{target},
and the merge leaves no worktree behind for the next one. The e2e fixture
asserts git really refuses the state before asserting recovery, so a future git
cannot make it vacuous.
* chore: re-trigger code quality check
* fix(node): close three host-class gaps in the public-URL gate (#339)
is_public_http_url is the shared gate for every operator-supplied outbound
URL: peer announce, webhook creation, and the boot-time peer prune. Three
host forms got past it that the checks around them plainly meant to exclude.
A name with no dot never matched the `.local` / `.internal` suffix rules, so
`http://internal/` and `http://wpad/` were accepted and left to the resolver
to complete from its search domain. The dotless rule sits on the branch where
the host did not parse as an IP literal, which keeps bracketed IPv6 (also
dotless) on the accepting path; the existing 6to4 and NAT64 accept cases pin
that.
fec0::/10 fell through the IPv6 arm because the link-local mask does not
cover it: 0xfec0 & 0xffc0 is 0xfec0, not 0xfe80. RFC 3879 deprecated the
range, which argues for rejecting it rather than ignoring it.
Trailing root dots were stripped once, so `localhost..` reduced to
`localhost.` and matched neither the equality nor the suffix check. Stripping
to fixation closes that.
Each of the three is covered by a test that reddens when its own production
line is reverted, and the dotted forms stay as negative controls so the
suffix checks remain proven.
* fix(gl): consume CappedBody text in whoami error path after main merge
* chore: re-trigger code quality check
---------
Co-authored-by: Kevin Codex <[email protected]>
* feat(node): add a persistent libp2p identity key file
Add p2p_key_path (--p2p-key-path / GITLAWB_P2P_KEY, default
~/.gitlawb/p2p.key) with a resolver mirroring resolved_key_path, and
load_or_create_p2p_keypair, which generates an Ed25519 keypair on first
start, persists it 0600, and loads it thereafter. Mirrors the existing
load_or_create_keypair idiom for the node identity PEM.
A corrupt or unreadable key file is a hard error naming the path rather
than a silent regeneration, so a disk problem cannot quietly rotate the
node's network identity.
Not yet wired into p2p::start; that follows.
* fix(node): load the libp2p identity from the persistent key file
p2p::start now takes the Ed25519 keypair loaded by
load_or_create_p2p_keypair instead of computing one from the node DID,
so a node's network identity is generated once from the OS RNG and kept
on disk rather than recomputed from a public value on every start.
The node DID parameter is gone from start; the call site loads the key
first and continues without p2p if the key file cannot be read, matching
how a swarm-start failure is already handled.
The gossipsub message_id_fn is untouched and keeps its own hasher.
* fix(node): create the p2p key with 0600 at creation and reject a loose one
Open the key file with create_new and the mode set at creation, then
fsync, instead of writing it and narrowing the mode afterwards. The
secret is never on disk under a wider mode, an interrupted start cannot
leave it readable, and the exclusive open also refuses a pre-existing
entry at the path and makes a concurrent start take the key that landed
rather than clobber it.
Refuse to load a key file whose mode grants group or other access, and
name the observed mode so the operator can fix it. Report an empty key
file as empty rather than surfacing a protobuf decode error that blames
a missing rsa feature.
Pin GITLAWB_P2P_KEY onto the mounted volume in the Docker and fly
configs and document it, so the key does not depend on home-directory
resolution to land on persistent storage.
* fix(node): publish the p2p key atomically and pin its directory
Write the key to a scratch file in the same directory and hard-link it
onto the final path. The bytes are durable before any name points at
them, so a crash cannot leave a partial key that fails to load on the
next start and takes the node off the network until someone reads the
logs. A concurrent reader can no longer observe a half-written file
either, since the final name appears complete or not at all.
hard_link rather than rename: rename replaces its destination silently,
so refusing to clobber an existing key would depend on a check followed
by a separate rename, and a concurrent start can land in that gap.
hard_link is atomic and refuses an occupied path, including a symlink,
which it does not follow.
Create the key directory 0700 and tighten it when an existing one grants
group or other access. A 0600 key under a writable directory can still be
replaced or unlinked. Tightening rather than refusing to start, because
existing installs already have 0755 there and refusing would take p2p
down on all of them through a path that only warns.
Formatting on the branch is swept up here; it was already failing
cargo fmt --check before this change.
* style(node): comma instead of a dash in the key-load warning
House style avoids em dashes in text we write. The swarm-failure warning
beside it predates this branch and is left alone.
* fix(node): require the p2p key path to name a directory
A bare filename in GITLAWB_P2P_KEY put the key in whatever directory the
process started from, and the directory guard was skipped entirely on that
path: Path::parent returns Some("") for a bare filename, which the caller
filtered out before ever reaching ensure_key_dir. The key file was created
0600 inside a directory that kept whatever mode it already had.
Config::validate now rejects a p2p key path that names no directory, so the
node says so at boot instead of starting with a key it cannot protect. That
placement is the point: an error raised in the p2p start path is logged and
stepped over, leaving the node running without p2p and reporting healthy.
The check is lexical on the tilde-resolved path. canonicalize would fail on a
parent that does not exist yet, which is the shipped ~/.gitlawb default and
every container's first boot, and comparing against the working directory
would reject /data/p2p.key under the image's WORKDIR, an absolute directory
the operator did name.
Three sites answered the parent question differently, which is how the gap
arose: one filtered the empty case out, one already normalized it, and one
opened "" and silently skipped its fsync. They now share key_parent, and
Config::validate calls it rather than adding a fourth answer.
load_or_create_p2p_keypair also refuses a path naming no directory. That is a
backstop behind the config gate, not the gate, so a later caller that skips
validation cannot quietly restore the old behaviour.
* test(node): run the key-permission probe in its own process
The probe zeroes the umask so the assertion means something: under a
restrictive ambient umask the bits are masked to 0600 regardless of whether
the code pins the mode, and the check passes either way.
Zeroing it in the shared test process is the problem. umask is process-global
and cargo runs these tests on threads, so any test creating a file in that
window inherits 000. Measured before this change: an unrelated concurrent
test's file was created 0666.
The probe now runs in a child process, where the zeroed umask cannot reach a
sibling and dies with the child. The parent is an ordinary test that runs
concurrently with everything else. Double-gated with #[ignore] plus an env
check so a bare --ignored sweep does not zero the umask in the shared process
after all.
The parent asserts the child ran exactly one test and that it passed, not
just that it exited 0. A libtest filter matching nothing runs zero tests and
still exits 0, so without that assertion a renamed fixture would read as a
green permission check while asserting nothing. Verified by pointing the
filter at a name that does not exist and watching the parent fail.
* docs(node): scope the p2p key permission claim and note the rotation
The old wording said the key file is "created with owner-only permissions"
without qualification, which is only true on Unix: every permission path in
p2p/mod.rs is cfg(unix), so on other platforms the file inherits whatever the
directory gives it and nothing is enforced. Say what is actually enforced and
where.
Also document what operators now have to do rather than leaving them to
discover it:
- GITLAWB_P2P_KEY must name a directory, since a bare filename is refused at
startup.
- The PeerId rotates once on the first start after upgrading, so a
GITLAWB_P2P_BOOTSTRAP multiaddr pinning a peer's old id with a /p2p/<PeerId>
suffix needs updating or dropping. Suffix-less addresses and the HTTP seed
list are unaffected.
- If the node reports tightening a loose key directory, the key that was in it
should be treated as possibly exposed and deleted so a fresh one is
generated.
* fix(review): close a `..` escape from the p2p key-path check
Two reviewers found the same hole independently: the check rejected a path
naming no directory, but a relative parent that walks back out through `..`
named one and still landed in the working directory. `a/../p2p.key` and
`./keys/../p2p.key` resolve to the cwd itself and `../p2p.key` resolves above
it, so all three put the key exactly where the check exists to keep it out of,
and had ensure_key_dir chmod that directory to 0700 on the way. Verified by
running the paths through the predicate and printing where each parent lands.
The rule is now that a relative key path must name a directory and must not
walk back out: at least one Normal component, no ParentDir. `..` inside an
absolute path stays accepted, since it cannot depend on where the process
started. The predicate moves into names_no_usable_directory next to key_parent,
and the config gate and the load_or_create_p2p_keypair backstop both call it, so
they cannot drift apart.
Also fixes two smaller gaps found in the same pass:
- The permission fixture could report "1 passed" while asserting nothing. Its
env gate returns early, and an early return is a passing test, so a renamed
variable would look green. It now prints a sentinel after its assertions and
the parent requires it. Confirmed by pointing the child at a different
variable and watching the parent fail.
- A GITLAWB_P2P_KEY starting with `~/` is refused when no home directory
resolves, instead of creating a literal `~` directory relative to wherever
the node happened to start.
The backstop had no test, so it has one now, along with a both-directions test
for the predicate. That test cleans up after itself: with the guard removed it
really does write a key next to the source, which broke a later run once.
* fix(review): reject `..` in an absolute p2p key path too
The previous commit closed this for relative paths and exempted absolute ones,
reasoning that an absolute path cannot depend on the working directory. That is
true and it is not the hazard. `key_parent` hands `ensure_key_dir` the lexical
parent, so `/data/keys/../p2p.key` chmods `/data` rather than the `keys`
directory the path appears to name, and `/data/../p2p.key` run as root would
try to tighten `/` to 0700. The exemption also had a test asserting the first of
those was fine, so the gap was written down as intended behaviour.
`..` is now rejected wherever it appears. An absolute path's root counts as
naming a directory, so `/p2p.key` still validates and `/data/keys/p2p.key` is
unaffected.
Found by a second-model review pass after the in-process reviewers had cleared
the relative half.
* fix(node): scrub the serialized p2p private key from memory
Two buffers held the private key in its protobuf form and dropped without
scrubbing: the encoding produced when a new identity is generated, and the file
contents read back on every subsequent start. Both are now Zeroizing, matching
what gitlawb-core already does for its own key material.
Scope worth being honest about: this scrubs our copies of the serialized form,
not the libp2p Keypair itself, which owns the secret for the process lifetime
and exposes no way to zeroize it. The gain is that the encoded bytes do not
outlive the write and the read.
zeroize was already in the tree through gitlawb-core, so this promotes it to a
direct dependency of gitlawb-node and adds no packages; the lockfile change is
the one line recording that.
* fix(node): validate the p2p key path before securing its parent
Reject filesystem-root parents, directory-valued paths, and symlinks before
any directory is created or chmodded. Create missing ancestors at the ambient
mode and pin only the nominated key directory to 0700. Refuse symlink reads
with O_NOFOLLOW, propagate directory durability sync failures, and document
the loose-key rejection path for operators.
* fix(node): satisfy clippy single-element-loop on p2p path test
* fix(node): tighten the p2p key storage contract before mutation
Anchor parent-directory checks on symlink_metadata, reject ~/ escapes and
non-regular existing keys with a bounded read, treat directory AlreadyExists
as a first-boot race, and skip key-path validation when P2P is disabled.
* fix(node): refuse foreign-owned p2p key paths and cap reads on all platforms
Check directory and key ownership before chmod or load, walk ancestors for
foreign control, bound non-Unix reads to MAX_P2P_KEY_BYTES, and gate unix-only
fixture tests without panicking on other targets.
* fix(node): hold the p2p key directory open across its trust checks
Resolve the key path once, then do every check and mutation through an
O_DIRECTORY|O_NOFOLLOW handle on the parent (openat/fstat/fchmod/linkat),
so nothing can be swapped between the ownership check, the tighten, and
the key open. Key reads go through one bounded, non-blocking loader on
every platform; new keys are generated in an unlinked scratch file and
published atomically, with concurrent first boots converging on the
winner's identity. The storage contract is indexed by two matrix tests:
filesystem shapes in p2p (each refusal proven side-effect free against a
recursive snapshot) and path spellings plus the p2p-disabled guarantee
in config.
* fix(node): walk the p2p key ancestor chain descriptor-anchored
Replace the path-following metadata walk (foreign_ancestor_error) and the
post-walk create_dir_all gap with a single descriptor-anchored walk that
verifies and creates the chain together. Missing components are created at
0700 relative to the verified parent and re-verified; AlreadyExists is a race
the winner must pass. Non-unix builds keep create_dir_all.
* fix(node): refuse group-writable and non-directory p2p key ancestors
The ancestor walk now refuses any component with group or world write bits
unless sticky, and any intermediate component that is a symlink or another
object type. Tests drive the walk through real boots: 0770/0775 ancestors
refused before any key IO, 0755 and 1777-sticky ancestors accepted with stable
reload, and an intermediate symlink refused with its target untouched. Both
guards proven load-bearing by neuter revert-checks.
* fix(node): verify the cwd anchor for relative p2p key paths
A bare-relative key path is now walked from a verified working directory:
the cwd is opened as '.' with O_NOFOLLOW|O_DIRECTORY, fstat-verified for
ownership and write bits before any relative component is touched, and a
writable or foreign-owned cwd is refused before anything is created. The
refusal names the working directory path. Child-process fixtures prove both
directions without mutating the shared process cwd.
* test(node): stop rendering uids in p2p assert-message strings
CodeQL flags the uid-interpolating assert messages as cleartext logging to a
console sink. Every assert message in the p2p test module now avoids
interpolating uid values or the uid-bearing error text, while the substantive
assertions (the refusal names both uids, the ownership check fires before the
chmod) stay intact and are verified by the same assertions.
* test(node): add walk rows to the p2p key-storage contract matrix
The matrix now proves the ancestor-walk refusals table-driven: a
group-writable (0775) ancestor, a world-writable non-sticky (0777) ancestor,
and an intermediate symlink component are each refused with the no-mutation
snapshot discipline, while the existing boot rows still reload to a stable
PeerId. The fchmod-on-create path pins newly created ancestors to 0700
regardless of the ambient umask.
* docs(node): note the verified working directory for relative p2p key paths
README, .env.example, and the clap help for GITLAWB_P2P_KEY now state that a
relative key path is anchored at a verified working directory, and that an
unsafe cwd or ancestor is refused with p2p off while HTTP stays up. Also
normalizes two pre-existing em dashes in the README settings table.
* fix(node): pin and verify the mode of every key object this process creates
POSIX applies the process umask to every requested creation mode, so the
0700 passed to mkdirat and the 0600 passed to openat were requests rather
than results. Under a mask that strips owner bits the created directory
landed 0000 and the no-follow reopen failed EACCES before the repairing
fchmod was reached, so that half failed loudly. The scratch key was worse:
it was published unreadable, the boot that created it succeeded on its
already-open descriptor, and only the next boot lost p2p, behind a health
check that still reported healthy.
Every object this process creates now goes through one rule: create, pin
the mode on the object just created, reopen, and verify the achieved mode
by fstat. The directory pin is issued by name off the verified parent
before the reopen, because a directory that landed 0000 cannot be opened
at all, so a pin that waits for the reopen is unreachable in exactly the
case it exists for. A race winner is verified as-is and never chmodded,
which is the rule the ancestor walk already applied.
Pinned's field is private to the pin module and no constructor is
exported, so a descriptor that skipped the pin cannot be turned into a
key-directory handle or reach publication. That is what keeps the rule
from depending on review.
Two adjacent corrections the same defect exposed. The leaf mode predicate
was supersets-only: mode & 0o077 != 0 asks whether anything is granted
beyond the owner, so 0000, 0100 and 0400 all passed it while being
unusable, and an inherited setgid bit was judged on the wrong bit width. A
loose directory is still tightened; an over-closed one is now refused with
its remedy rather than widened, matching how an over-closed key file is
already handled. And an unreadable key or key directory now names its own
cause: every open failure previously reported a refused symlink, which is
what an operator saw for a key masked at creation.
The matrix runs first boot and fresh-process reload across umask 0000,
0022 and 0777, crossed with missing and existing ancestors and leaf, with
concurrent creators and an injected write failure. It runs in child
processes because umask is process global, and it requires at least one
successful concurrent creator so a row where every creator fails cannot
pass. All 21 rows were RED under umask 0777 before this change.
* fix(node): create and publish the node identity key through the pinned path
The node identity PEM had its own storage flow: create_dir_all with no
mode, then write, then set_permissions. That is the exists-then-write-then-
chmod sequence INV-23 prohibits, and it sits in the same ~/.gitlawb the p2p
key uses, so the umask defect the previous commit fixed for one key was
still live for the other.
Measured on the current code, as a non-root uid, all three RED:
umask 0000 directory landed 0777, world-writable, holding the key
umask 0022 directory landed 0755, world-traversable
umask 0777 directory landed 0000 and the write failed with EACCES
The third is the one that mattered most: load_or_create_keypair runs before
the listener binds, so the node exited there and no p2p code was reached at
all. The umask-independence guarantee could not be demonstrated on the
shipped default while this stood.
The directory is now created through the same pin helper, at a verified
0700, and the PEM is published through the same scratch-then-link path at a
verified 0600. An existing directory that grants access beyond the owner is
tightened, which closes the other half of the same gap: a 0600 key inside a
0755 directory is still replaceable by anyone who can write that directory.
An existing key is loaded untouched and never chmodded.
Deliberately not the full ensure_key_dir. That carries the ancestor trust
walk, and importing its refusals onto a path that never had them would turn
an unsafe but currently booting deployment into a boot failure on upgrade.
Only the immediate parent goes through the pin helper, which verifies the
grandparent it is about to chmod a child of; ancestors above that keep the
existing create_dir_all behavior. The one new refusal is a group or world
writable non-sticky grandparent, which is the case where another local user
can replace the node's identity outright.
* fix(node): split p2p key config validation from live storage validation
The key-path validator answered two different questions and its callers
disagreed about what to do with the answer. Bare names, `..`, a trailing
separator and the filesystem root are properties of the configured value,
and Config::validate refuses them before the listener binds. The same
function also stat'd the key path and its parent, so a symlinked parent, a
non-directory parent or an unreadable parent exited the node as invalid
configuration, while the identical class of fault found one layer later in
load_or_create_p2p_keypair only logged and left HTTP serving.
That split was invisible from the outside and the docs described only one
half of it. Measured against the binary before this change: a symlinked
parent, a regular-file parent and an unreadable parent all exited 1 before
bind, which is the opposite of what README and .env.example promise.
Validation is now lexical only, behind a P2pKeyConfigError so the two
domains cannot be confused at a call site. Every live storage fact is left
to the load path, which already re-establishes each one on a descriptor it
opened rather than a pathname it stat'd: O_NOFOLLOW on the key open, a
regular-file check by fstat, and ELOOP or ENOTDIR at the leaf. The verdicts
are unchanged; what changes is that one policy now decides all of them.
Two supporting changes. The whole p2p port gate moves ahead of the database
connect, both arms together, because connect_db_with_retry retries forever
and left the disabled arm unreachable without a database; only p2p::start
still needs the pool. And a failed key load is now logged at error with a
stable event name and mirrored into a gauge, because the policy this commit
settles on is to keep serving HTTP with a green health check while the node
is off the p2p network, and that is only defensible if the outage is
visible to something other than a human reading startup logs.
A new integration test drives the real binary with no database and proves
both domains at the process boundary: nine lexical spellings exit before
binding, ten storage faults serve HTTP with the failure logged, and
GITLAWB_P2P_PORT=0 reaches its disabled log without touching the key tree.
Every row asserts a before/after snapshot, because a refusal that mutates
storage on its way out is its own defect and a returned error cannot show
it. Three of those rows were RED before this change.
A symlinked key directory also reports as a symlink again. Linux returns
ENOTDIR rather than ELOOP when O_NOFOLLOW meets O_DIRECTORY, so the errno
alone cannot separate a symlink from a regular file and the message had
regressed to naming the wrong cause.
* docs(node): state the two-domain key-path policy and how to see a degrade
README and .env.example promised that p2p stays off while HTTP keeps
serving when the key cannot be loaded, including for an unsafe ancestor or
working directory. Measured against the binary, that was false for a
symlinked parent, a non-directory parent and an unreadable parent, each of
which exited before binding. The preceding commit made the code match the
promise; this makes the promise precise.
All three operator surfaces now name the split the same way. A value that
cannot name a securable key file is refused before the node binds. Anything
wrong with the storage itself degrades, and the sentence says how to notice:
the node logs p2p_identity_key_load_failed at error and sets the matching
metric, /health stays green in that state, and fixing the storage and
restarting restores the same PeerId. A node quietly off the network is the
cost this policy accepts, so the docs point at the signal rather than
leaving an operator to find it in startup logs.
Two corrections while here. The key directory is tightened only when it
grants access beyond its owner; one closed too far is refused with a
chmod 700 remedy rather than widened, so the sentence no longer implies the
node will open up a directory an operator locked down. And the existing
advice to delete a key after a tightening warning is now scoped to the
loosening case it was written for: an over-closed directory is refused, not
tightened, and nothing in it was ever exposed, so following that advice
would have meant deleting a safe key and taking a second PeerId rotation
that breaks every pinned bootstrap multiaddr.
* test(node): pin the degrade signal and wait for both startup lines
Two gaps in the boundary test, both found by extending the mutation spec to
cover this commit's own guards.
The degrade rows asserted only the prose message, so renaming the event
field or dropping the level would have passed while every alert built on
the operator docs broke and the node still looked healthy. The rows now
pin the error level and the p2p_identity_key_load_failed name the docs tell
operators to watch.
The rows also waited on a single log line, which is a race: the degraded
server logs "ready" from a spawned task while the p2p gate logs its verdict
from the main task, so whichever the reader stopped at first left the other
uncaptured. It passed when the test binary ran directly and when the row ran
alone, and failed only once other work shifted the scheduling. Reads now
wait for every needle before returning.
The fix stays on the test side on purpose. Logging the p2p verdict before
the degraded server is spawned would also make the order deterministic, but
that is production sequencing changed for a test's convenience.
* fix(node): keep key publication transactional and path-policy split
Bare GITLAWB_KEY filenames publish into the working directory without
chmodding it. A directory this process created is removed if pin or
verify fails. Ancestor walks use search descriptors so a safe 0111
component is traversable. Scratch unlink is part of publish success and
is fsynced. Rotation advice is limited to writable or readable exposure,
not ordinary 0755 tightening.
* fix(node): refuse a writable cwd for a bare identity key
Publishing identity.pem into cwd must not chmod cwd, but a 0600 key is still replaceable if that directory is group or world writable. Apply the same trusted-parent check used for nominated key directories before the handle is returned.
* fix(node): load identity without following pathnames
Create already published through a directory handle. Load still used exists-then-read, which followed a symlink at the key and opened the grandparent by pathname. Open both no-follow, treat /identity.pem like a cwd publish so it reaches the filesystem, and open a search-only grandparent with the walk flags.
* fix(node): parse key paths, leave identity parents, repair special modes
A terminal `.` in GITLAWB_P2P_KEY was accepted then retargeted by Path onto
the parent directory. Identity creation imported p2p's 0700 pin onto any
existing named parent. A setgid-only 2700 key directory skipped repair and
disabled P2P. Darwin ancestor walks still required directory-list permission.
* fix(node): type-check the non-unix p2p scratch writer
The shared scratch-then-link path names pin::Pinned on every target, but
the pin module was unix-only, so gitlawb-node did not compile off Unix.
* fix(node): refuse identity path spellings that Path would retarget
GITLAWB_KEY had the same terminal-dot hole as GITLAWB_P2P_KEY, and its
tilde expansion did not refuse a suffix that escaped home. Bare filenames
stay legal.
* fix(node): create every missing identity parent on one pinned walk
The identity key path split one creation transition across two policies.
`create_dir_all` built every ancestor above the immediate parent at the
ambient umask, then `create_dir_pinned_at` ran `verify_trusted_parent` on
the last of those and refused it. Under umask 0002 that ancestor lands
0775, so the phase that made it handed it to the phase that rejects it:
the node exits before binding and the failed boot leaves the directory
behind at 0775, so the next attempt starts from state the first one made.
Replace the split with one descriptor-anchored walk. Pass 1 walks up from
the key's parent until a component opens; that descriptor is the anchor, and
it is the same fd the next step creates into. Pass 2 walks back down calling
`create_dir_pinned_at` per component, each one pinned 0700 and verified at
its achieved mode before it becomes the parent of the next, so from the
anchor down no pathname is resolved twice. Pre-existing ancestors are
adopted as-is and never chmodded, since GITLAWB_KEY is a file path rather
than a dedicated-directory setting; write-authority is still required on the
directory each component is created in.
A boot that fails past the walk now removes exactly what it created,
deepest first, through unlinkat with AT_REMOVEDIR against the descriptor
each entry was made in. A race-adopted directory is never in that list, and
AT_REMOVEDIR refuses a non-empty directory, so a leaf another boot has
published into survives. The accumulator is an out-parameter so a mid-walk
failure and a post-walk failure reach the same single rollback arm.
Two consequences worth naming. A symlink at the deepest existing ancestor is
refused when the missing suffix reaches it, where `create_dir_all` followed
it. Ancestors ABOVE that anchor are still resolved by pathname and their
symlinks still followed, unchanged from before this walk, because O_NOFOLLOW
binds only the final component of a path-based open; closing that would mean
an openat chain from the root and the full ancestor policy that GITLAWB_KEY
paths deliberately do not get. The doc comment now says so rather than
claiming more than the code does. And a losing concurrent first boot can
remove an intermediate another boot adopted but has not yet written into,
turning that boot's start into an ENOENT failure; both still fail closed.
Also replace the root-adjacent test. It proved a pathname property by
running the full mutating path against the real filesystem root and
assuming the runner lacks permission. Under euid 0, common in build
containers, `/` is accepted (root-owned, 0755, no group or other write
bit), the production path returns Ok, the test's panic arm fires before its
cleanup assertion, and a real private key is left at /identity.pem, after
which the leading exists() check makes every later run a silent skip. The
routing is now asserted lexically through a named predicate, with no
filesystem access and no dependence on euid.
The matrix covers one, two and three missing levels under umasks 0000,
0002, 0022 and 0777, a pre-existing 0755 ancestor that must stay 0755, a
relative multi-level path, a symlinked deepest ancestor, refusal without
partial mutation, and a populated leaf whose removal must be refused and
reported. Created-directory modes are asserted by exact equality, and each
success row reloads in a fresh process and requires the same DID.
* test(node): race real concurrent first boots of the identity key
The rollback added with the pinned walk can remove an intermediate that a
concurrent boot has already adopted, and until now that was reasoned about
rather than executed. Nothing in the suite started two real first boots
against the same key path.
Add a `concurrent-first-boot` layout to the self-exec fixture and a driver
that races several children through `load_or_create_keypair_at` on one
missing multi-level path. A two-phase file rendezvous puts every child
inside a spin loop before any of them is released, so the spread across the
racing call is one loop iteration rather than process-startup jitter. Three
arms: unaided boots, one injected write failure against one unaided boot,
and four injected failures against one, which is what actually opens the
window.
The assertions are properties of every legal outcome, never one
interleaving, so the test is deterministic even though the race is not. A
boot that claims success has a parseable 0600 key on disk, all successes
agree on the DID, surviving directories are exactly 0700 with no scratch
residue, and if no key was published then nothing claimed success and a
retry must still work. Observation counters are printed, not asserted, so a
run that never hits the window says so rather than passing as proof.
Two things the race showed that reading could not. The documented window is
real: a boot was refused with "No such file or directory" on its publish
after a loser rolled back the leaf it had adopted. But the more common shape
is one hop earlier, the loser removing an adopted intermediate so the ENOENT
lands on the next component's creation instead. And under enough concurrent
rollbacks every boot can fail, not merely the losers, so several nodes
starting against a shared first-boot path can all need a restart. Two-child
racing never opened the window at all in 500 iterations.
* fix(node): resolve identity key path components one at a time
`O_NOFOLLOW` binds only the final component of a path-based `open(2)`, which
is all the kernel offers for a multi-component pathname. Three sites resolved
a whole configured path in one open and so guarded only the last position:
the load path, the existing-named-parent fast path, and the walk that creates
missing parents. Every interior symlink was followed, and `verify_trusted_parent`
then attested the directory that was reached rather than the path taken to it.
That is enough to substitute the node's whole identity, not merely misplace
its key. Load and create resolve the configured pathname identically, so
repointing one interior link redirects both. Point it at a directory holding
another PEM and the node boots as that DID while its real key sits untouched
on disk and nothing is logged as wrong. Point it at an empty directory and
the node mints a fresh identity and reports a first boot. Neither needs the
attacker to read anything: ownership and mode checks still pass, because they
run on the object at the end of the path.
Resolve each component with `openat` and `O_NOFOLLOW` against the descriptor
of the one before it, from `/` for an absolute path or the working directory
for a relative one, so a symlink is refused at every position. Interior
components open search-only and the final component keeps the caller's flags,
so a legitimate 0111 ancestor still resolves and the leaf descriptor is still
the one that gets fsynced and chmodded.
Components are adopted with no ownership or mode judgment. The only new
refusal is a symlink on the key path: a shared 0775 volume, a foreign-owned
ancestor and `/etc` all keep working, because `GITLAWB_KEY` is a file path
rather than a dedicated-directory setting. `verify_trusted_parent` stays
exactly where it was, on the descriptor a child is created in. The p2p key
path is untouched; it already walked component by component.
This is pre-existing rather than a regression: the same probe against the
branch point produces the same result. It is fixed here because closing it
replaces the walk this branch just added, so splitting it would mean
reviewing that code twice.
Covered by rows for an interior symlink with the next level present, an
absolute target escaping the base, a two-link chain, the fast path, a
relative configured path, and both load-path variants asserting on the DID
the node ends up presenting rather than on a mode or an error string. The
cases that were already refused are pinned too, including a symlink at the
immediate parent and a dangling link, neither of which had a test before.
---------
Co-authored-by: Kevin Codex <[email protected]>
hickory-proto/hickory-resolver 0.25.2 were pulled in only by libp2p-dns
0.44. libp2p-dns 0.45 moves to hickory 0.26 but needs libp2p-core 0.44
and libp2p-identity 0.3, so the node's direct libp2p crates move to the
matching release line (core 0.44, gossipsub 0.50.1, identify 0.48,
identity 0.3, kad 0.49, quic 0.14, swarm 0.48, dns 0.45). multiaddr goes
to 0.19 to match libp2p-core and avoid compiling two copies. No source
changes were needed.
Lockfile: hickory-proto/resolver 0.25.2 -> 0.26.3 (GHSA-q2qq-hmj6-3wpp /
RUSTSEC-2026-0119, GHSA-6f2x-v7q7-m7m5), serde_with 3.18.0 -> 3.22.0
(GHSA-7gcf-g7xr-8hxj).
With the fix in, drop the RUSTSEC-2026-0118/0119 ignores from
.cargo/audit.toml as #76 requires.
Closes#76
Co-authored-by: Claude Opus 5.5 <[email protected]>
* feat(node): sign and verify gossip ref-update events
Add an optional sig field to RefUpdateEvent carrying a base64 Ed25519
signature over the event's own fields, plus signing_bytes, the single
producer of signing input on both sides, and sign/verify helpers.
The field is optional and skipped when absent, following the owner_did
precedent from #144, so an event from a signing node still deserializes
on a peer that predates this change. A nested envelope would have broken
that and is deliberately not used.
Verification resolves the key from the claimed node_did, so a signature
that does not bind the DID it claims is refused. Non-did:key and
unresolvable DIDs are refused with the peers-table denial sentences,
built from PeerWriteDenied rather than retyped so the two surfaces
cannot drift apart.
A golden test pins the signing input: a field reorder changes what every
peer signs, which no round-trip test can see.
Helpers are not wired into the ingest or emit paths yet.
* fix(node): authenticate gossip ref-update events before writing them
The gossipsub ingest path accepted a ref-update from any peer and wrote
it, while the HTTP route carrying the same payload checks the signature
against the claimed DID, requires a known peer, and validates the repo
slug. Extract the path into ingest_ref_update and apply the same three
gates in that order, so both transports answer the same input the same
way.
A present signature must verify in either mode. An absent one is refused
when GITLAWB_REQUIRE_SIGNED_PEER_WRITES is set and otherwise accepted
with a warning naming the flag, so peers that predate signing keep
federating until the fleet is ready. The DID-method refusal runs right
after parse in both modes and reuses the peers-table denial sentences.
Rejection writes nothing: neither a ref-update row nor a sync-queue
entry. Those are separate sinks and the tests assert them separately.
from_peer's comment now says it records the forwarder, not the author.
* fix(node): bound gossip ingest per source and sign outbound events
Brake the ingest path at 60 events per minute per forwarding peer,
checked ahead of the parse and the signature work. Verifying a signature
is the expensive step, so a brake behind it would sell an unauthenticated
flood the CPU the brake exists to protect. Rejection writes nothing to
either sink.
Sign events on the way out, so a fleet running with enforcement on can
verify what its peers publish. A signing failure warns and skips the
publish rather than emitting something a verifying peer would drop.
The emit-to-ingest round trip is asserted end to end, which is what
proves both sides agree on the signing input by execution rather than by
inspection.
* test(node): cover the did-method gate and correct overclaiming comments
The did-method gate had no load-bearing test: deleting it left the suite
green, because the only non-did:key case was signed, so verification
resolved the DID internally and returned the same refusal. Add the case
only the gate decides, an unsigned non-did:key event from a registered
peer with enforcement off, and correct the comment that claimed the
other test covered it.
Assert the limiter's source bound, which no test pinned, and make the
backward-compat test able to see an added field by parsing under a
deny_unknown_fields reader as well as a permissive one.
Say plainly at the peers check that authentication is not authorization:
a self-registered did:key passes it, so what the gate buys is blocking an
unregistered DID and, with the signature, blocking impersonation of an
existing peer. Drop the claim that the unsigned-acceptance warning is
covered by a test, because it is not.
An outcome of Accepted now means the row was stored; a failed write
reports itself instead of reading as success.
* fix(node): charge the ingest budget to the author, not the mesh edge
A single 60-per-forwarder budget was both too tight and pointed at the
wrong principal. Pushes emit one gossip event per ref, so a push touching
more than 60 refs lost its tail with no signal to the publisher, and
because gossipsub re-shares mesh-wide, a flood routed through an honest
neighbour spent that neighbour's budget on every receiver.
Split it. A loose pre-parse bound on the forwarding peer still caps the
CPU spent before anything is parsed or verified, which is all a
free-to-mint identity can honestly buy. A tighter bound on the
authenticated node_did sits after verification and the peer gate, so the
two durable writes are charged to whoever actually authored them, and it
is sized to admit a large push whole.
Both rejections now name the budget and its window, so a dropped
ref-update says which bound refused it rather than vanishing.
Replace the per-event full peers scan with a keyed lookup on the same
column; upsert_peer shares it.
A push above the author bound still loses its tail, now loudly. The two
transports continue to disagree about events per push: gossip publishes
per ref while sync/notify batches.
* docs(node): the did-method gate runs after the parse, not before it
Two test comments called it the pre-parse did-method gate. The parse is
above it and the gate is below; what is true is that it runs before the
signature branch, in both enforcement modes. The pre-parse label belongs
to the forwarder brake, which is genuinely the first statement, and the
nearby correct usage is where the wrong one came from.
* fix(node): charge the gossip write budget only to a proven author
An unsigned ref-update event's node_did is asserted, not proven. The author
budget was debited on that key in both signature arms, so a flood of unsigned
events naming a registered peer drained that peer's budget and its own genuine
signed updates were shed. The attacker chose the victim.
The author budget is now charged only when the signature verified. Unsigned
compatibility traffic charges a separate forwarder-keyed bucket sized at
GOSSIP_UNSIGNED_SOURCE_MAX_EVENTS (1500), above the per-author cap because a
forwarder aggregates many authors, and below the pre-parse brake; the
compile-time assert now pins all three in one chain. The shed carries its own
IngestOutcome variant so a forwarder is never named as an author.
validate_repo_slug moves above both the signature match and peer_exists, so a
malformed event costs no database round trip rather than merely being charged
to nobody. A test-only peer_exists tally proves that placement, which an
outcome assertion cannot distinguish.
* fix(node): version the signed ref-update format and refuse unknown versions
The signed payload carried no version and verify_ref_update knew one
representation, so the next signed-field addition would partition the mesh
mid-rollout: an upgraded publisher signs bytes an older receiver cannot
reproduce, and the older receiver reports a genuine event as a bad signature.
RefUpdateEvent now carries v, skipped when zero, so a v0 event's wire and
signing bytes stay byte-identical to the versionless form. GOLDEN_SIGNING_BYTES
is unchanged and every signature already issued still verifies. A frozen
artifact signed before the field existed is checked into the suite and verified
without being re-signed, since a signature a test just produced is
self-consistent by construction and proves nothing about compatibility.
An event whose version this build cannot interpret is now refused in its own
words, ahead of every other gate: no field below an unknown version has known
meaning. Without that, a v1 event was admitted outright rather than merely
mislabelled.
* docs(node): describe the real signing handoff at the publish site
The comment claimed signing was future work and the event went out unsigned.
It does not: the swarm loop signs via p2p::signed_publish_bytes immediately
before publishing, and skips the publish entirely when signing fails. Leaving
sig None here is correct, and the comment now says why rather than describing
a state the code left behind.
* docs: GITLAWB_REQUIRE_SIGNED_PEER_WRITES spans both transports
The flag gates gossip ref-update ingest as well as the HTTP announce and sync
routes, but README and .env.example still scoped it to those two routes. An
operator who upgraded the HTTP peers and then enabled it would silently drop
gossip from every publisher still running an old build, with no error visible
on the publisher's side because gossip carries no response.
Documents the cross-transport behavior and the one ordering requirement that
follows: upgrade every gossip publisher before enabling.
* fix(review): count gossip ingest outcomes and sweep the ingest limiters
Every IngestOutcome reached tracing and nothing else, so gossip-ingest health
was invisible on /metrics, the only surface this daemon exposes to a dashboard
or an alert. Adds gitlawb_gossip_ingest_events_total{outcome} alongside the
existing sync-queue and webhook counters, with the label derived from an
exhaustive match so a seventh variant fails to compile rather than shipping
uncounted. The reason strings stay in the log line; they are sender-shaped and
would make the label set unbounded.
The three ingest limiters were built as locals in the swarm task, so
sweep_rate_limiters could not reach them and their keys stayed resident until
the 200k ceiling. The swarm loop now sweeps them on its own interval, and
cleanup iterates a destructured array so a fourth limiter breaks the build
instead of being silently missed, which is the failure sweep_rate_limiters'
own comment records against the /ipfs limiter.
* test(node): close four gaps the review found in this PR's own guards
The frozen signing constants pinned only the all-Some optional shape, but the
sole publish site always emits cert_id None, so no constant covered the
encoding every real event uses. Proven during review: adding
skip_serializing_if to cert_id left the suite green while the signing input
changed for 100% of production events. A second constant now pins the
null-carrying form.
The legacy artifact's provenance assertion claimed re-capturing it from current
code was what the assertion refused. It was not: skip-when-zero means a
regenerated artifact also carries no version key, so the check passed and the
claim was unfounded. A SHA-256 over the frozen bytes does the job the message
promised, and the message now states only what it checks.
WriteFailed was the one outcome variant no test constructed, so the
independent-writes property it exists to express was unexercised. Both
directions are now driven by failing a real write.
The publish arm's body is extracted so the exact bytes handed to gossipsub are
asserted signed and accepted by ingest under enforcement. The select! dispatch
and the require_signed threading from main.rs still need a live swarm and are
named as uncovered rather than implied covered.
* fix(review): close the remaining findings from the round on this PR
The unsigned-acceptance warning fired before two gates that could still drop
the event, so a message the operator reads as an admission could be followed by
a forwarder-budget shed or an unknown-peer refusal. It now fires only once the
event is genuinely admitted, and a test pins that it fires exactly once, which
nothing checked before.
The publish site hardcoded the format version while the ingest gate compared
against the constant, so emitter and gate could drift apart silently. The
publish site now reads the constant.
The known-peer gate changed a federation precondition without documenting it: a
gossip publisher must also be a known peer on the receiving node, via bootstrap
or a prior announce, or its events are dropped in both flag modes. That is now
in the README beside the rollout ordering, derived from what the ingest path
does rather than from another docstring.
Six tests each hand-rolled the same both-modes rejection loop; they now share
one helper. The log-capture helper was left under a tmp_ prefix from the round
that added it and is renamed, since scaffolding vocabulary should not ship.
* fix(review): report unsigned gossip admissions as their own outcome and metric (#325)
An unsigned event that survives the rolling-upgrade window was returned as
IngestOutcome::Accepted, so `gitlawb_gossip_ingest_events_total{outcome="accepted"}`
mixed signature-verified admissions with unauthenticated legacy traffic. An
operator could not tell whether the fleet still relied on the compatibility
allowance or authenticated gossip was healthy.
Add IngestOutcome::UnsignedAdmitted with its own `unsigned_admitted` metric
label, returned when an unsigned event passes every gate and the writes land.
`Accepted` is now reserved for signature-verified events. Propagate through the
swarm-loop logging and the tests that asserted `Accepted` for unsigned
admissions.
Both guards proven load-bearing by execution: reverting the return-site change
goes RED on flag_off_unsigned_known_peer_event_is_accepted ("got Accepted"),
and giving UnsignedAdmitted the "accepted" label goes RED on
every_ingest_outcome_carries_a_distinct_metric_label.
* fix(node): shed an over-budget gossip author before the peer lookup (#325)
The author brake sat below `db.peer_exists`, so a signed author already past its
budget still paid a Postgres round trip before being refused. That is a brake
behind the work it bounds: the lookup is done for a request that was never going
to be admitted, and a signed author can drive it at the pre-parse source rate,
further with peer-id rotation.
The brake now answers first, and the answer is a READ rather than the charge.
`check` inserts a window for a key it has never seen, so probing with it here
would let a flood of self-minted signed DIDs occupy the bounded author map before
`peer_exists` could refuse them, and legitimate new authors would be shed once it
filled. That would have made the remedy worse than the finding. `is_over_budget`
allocates nothing, reports an unseen key as within budget, and leaves the charge
itself below the lookup, unmoved. The only behaviour that changes is WHEN an
already-over-budget author is refused.
Two guards, both proven load-bearing by execution. Removing the early shed makes
`an_over_budget_author_is_shed_without_a_peer_lookup` red on the call counter,
which observes the lookup not happening rather than restating the ordering.
Swapping the read for `check` makes `the_early_author_check_does_not_track_an_unseen_did`
red on the map size, which is the must-not that keeps the brake from becoming its
own memory-fill surface.
Found by a cross-family refute pass on the pushed head.
* chore: retrigger checks after a CodeQL infrastructure failure
The swift analysis job on run 32041989078 failed in "Set up job", before
checkout or any analysis step. GitHub refuses a rerun on default-setup
(dynamic) runs, so a new head is the only way to get a fresh scan.
* fix(deps): bump h2 to 0.4.19 for RUSTSEC-2026-0258
The branch lockfile still pinned h2 0.4.13, so cargo audit failed on
every push while main had already moved on. This is lockfile-only; no
source change in the gossip auth work.
* chore: re-trigger code quality check
---------
Co-authored-by: Kevin Codex <[email protected]>
Cargo.lock resolved aws-smithy-json 0.61.9 (via aws-sdk-s3 1.119.0) and
0.62.5 (via aws-config and the sso/ssooidc/sts clients). Both are in the
range affected by GHSA-8ffr-xgwf-xj56 / CVE-2026-18140 (<= 0.62.6):
uncontrolled recursion when skipping unknown keys in deeply nested JSON.
0.62.7 requires aws-smithy-runtime-api ^1.12.3, so the AWS SDK moves as
a set to the newest releases that still build on our 1.91 MSRV (they
declare rust-version 1.91.1; later releases need 1.94.1):
aws-config 1.8.18, aws-sdk-s3 1.137.0 (which itself requires
aws-smithy-json ^0.62.7), aws-sdk-sso 1.102.0, aws-sdk-ssooidc 1.104.0,
aws-sdk-sts 1.107.0, aws-sigv4 1.4.5, and matching aws-smithy-* crates.
aws-sigv4 1.4.5 moves to p256 0.13, which drops crypto-bigint 0.4.9 from
the lockfile. Update the RUSTSEC-2026-0387 note in .cargo/audit.toml to
match; the ignore is still needed for crypto-bigint 0.5.5.
Lockfile and audit-config comment only; no manifest or source changes.
Co-authored-by: Claude Opus 5.5 <[email protected]>
* fix(deps): resolve cargo audit failures on main
Update quinn-proto 0.11.19, rustls 0.23.46, rustls-webpki 0.103.16,
webpki-roots 1.0.11, zstd 0.13.4, zstd-safe 7.4.0 and zstd-sys 2.1.2 in
Cargo.lock to clear RUSTSEC-2026-0338..0342, 0346..0349, 0381, 0382,
0391, 0398, 0403 and 0410.
Ignore RUSTSEC-2026-0387 (crypto-bigint): the fix is only in 0.7.x, and
no dependent compatible with our tree uses it. alloy-signer (through
2.5.0) pins elliptic-curve 0.13, and aws-sigv4 >=1.5 requires Rust
1.94.1, above the 1.91 MSRV.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs(audit): document RUSTSEC-2026-0387 reachability and graph-based removal
Correct the aws-sigv4 boundary (1.6.4 is the first release on
crypto-bigint 0.7; 1.5.x-1.6.3 still need ^0.5.4), mark the ignore as
reachable through the k256 PrivateKeySigner and SigV4a signing paths,
and tie removal to Cargo.lock having no crypto-bigint below 0.7.6.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
On a resumed request whose visit budget was already spent by the provenance
phase, the scan's top-of-loop visit arm sealed pager.cursor, which at that
moment is the position the caller just sent. The node returned the caller's own
token, verbatim, rung after rung. Three rungs were observed returning an
identical position.
A token looks like progress, so the client keeps going: gl retries to its
resume cap, and every one of those requests re-runs the full provenance phase,
up to seventeen repo acquires and cat-file subprocesses, advancing nothing
before it errors. That is roughly nine anonymous requests worth of work for
none, and it is worse than shedding nothing, because a caller who is told the
ladder is over stops immediately.
A seal now has to be strictly ahead of where the request itself started: a
different candidate is ahead by construction, since only the gated advance can
name one, and the same candidate needs a row past the start row. A request that
started at the front is before everything, so its seals pass untouched.
When the proposer settled at least one row this rung, the existing ceiling arm
already seals that row, and it is strictly ahead because a resumed scan only
walks rows past its cursor. Only a rung that settled nothing sheds without a
token, and that is honest: the spender is the provenance phase, which runs the
same way every rung, so no retry can do better.
The filter sits at the single mint site, where a future call site cannot bypass
it, and it logs the drop as a boolean. record_scan_truncation has already
logged that a position was sealed by then, and a 503 carrying no token next to
that line is the confusion that log exists to prevent.
The compose reinstall added last round is not safe under overlap, and `main`
never wrote compose from this document, so the race arrives with that commit.
Two executions — a double run, a retry while the first is still pulling, two
operators — shared the fixed `/opt/gitlawb/compose.yaml.new`. The second
truncates the file the first is about to install; and once the first renames,
the second's open descriptor keeps writing into the live `compose.yaml` inode.
Both outcomes stay valid YAML, so the `up -d --remove-orphans` on the next line
accepts a blended or truncated service set and deletes whatever fell off — on a
live node, the local postgres or the TLS terminator. Both reviewers reproduced
it independently; one measured a short file in 12 of 12 trials.
The whole step now runs under `flock -n` on /opt/gitlawb/.upgrade.lock and
exits non-zero if the lock is held, so a second run cannot restart against a
file the first is mid-install. The temp file comes from `mktemp`, so concurrent
writers cannot truncate each other, with a trap to clean it up on failure.
The compose body is base64-encoded rather than written through a nested
heredoc. The rendering carries `$${VAR}` passthroughs that must reach the file
unexpanded, which needs a quoted delimiter, and a shell heredoc nested inside an
indented Terraform heredoc depends on the dedent landing that delimiter at
column 0. Decoding one line has neither failure mode and no delimiter can
collide with the content.
Not validated locally: terraform is not installed on this machine, so this
change has had no `fmt`, `validate`, or `plan` run against it.
A CID that maps to several oids shared one resume slot across every candidate,
and the slot kept the maximum position. An earlier candidate could spend the
probe budget walking past a repo that holds the object for a later one, seal a
position beyond it, and the next rung would resume past a row that candidate
never examined, wrap, and shed tokenless. The client reads an absent token as
the ladder being over, so the object became permanently unretrievable, at stock
config, deterministically on every retry.
The token now names which candidate it is resuming, and the rules that keep
that sound are narrower than they first look:
Only one candidate per request may seal, and which one depends on where the
REQUEST started. On a resumed request it is the resumed candidate alone, since
the shared pager holds only the table suffix from the caller's cursor, so a
later candidate walked a suffix and never saw the front. On a front-started
request it is the first unfinished candidate, since there every candidate walks
from the front and a later candidate's stop is honest coverage. Silencing later
candidates unconditionally would remove the only thing that mints rung 1 when
the first candidate wraps untruncated.
A candidate is finished when its row loop walked every fetched row, or when it
owed no scan at all. Both matter: a properly provenanced candidate never wraps,
so without the second arm the ladder dies every rung. The wrap is witnessed per
candidate at the row loop's own two exits, never by reading the shared pager
flag at the tail, which any short page sets and which would let a candidate that
truncated mid-page look finished and strand the rows it refused.
Finishing a non-final candidate advances the seal to the next one at a front
sentinel and taints, because the tail emits a continuation only when something
tainted; sealing without tainting would suppress the taint and return a
definitive 404 while discarding the token it had just minted.
The keep-the-maximum comparison is gone. With one proposer per request the slot
is written at most once, so an assertion states that directly instead.
The pager stays shared per request. A per-candidate pager would restore the
fan-out the paging exists to remove.
A CID can map to several git oids, and the ladder needs to name which one it is
resuming. The sealed position gains the candidate's oid hex so a rung resumes
that candidate rather than a position in a list: oids_for_cid is a sorted set,
so an ordinal silently repoints at a different candidate when a pin that sorts
earlier arrives between rungs, while an identity degrades safely to "not found,
restart at the front".
The field is length-prefixed and padded to 64, matching the framing the row
fields already use, because production oids are 40 hex, not 64: repos are
created with --object-format=sha1 and only the test fixtures are sha256. A
fixed 64-byte field would fail every seal on a real deployment and shed a
tokenless 503, which the client reads as the ladder being over. Both widths are
exercised, and a zero-length candidate is rejected at decode so it cannot be
confused with the front-of-table sentinel, which is empty row fields with a
real candidate.
VERSION goes to 3 and the plaintext to 527 bytes, so a token minted under the
old layout opens to None and the caller restarts at the front. Nothing has
minted one outside tests.
The slot carrying the position is a struct rather than a widened tuple on
purpose: a 3-tuple would have pulled the hex into the existing keep-the-maximum
comparison, changing behavior this commit is meant to leave alone.
Token length stays invariant across both oid widths, since length would
otherwise be a side channel for the withheld row it names. That is asserted on
a real seal in gitlawb-core, not on the gl fixtures: nothing in gl seals or
opens a token, so its width constant cannot detect a wrong layout.
oids_for_cid ran a bare SELECT with no ORDER BY, so Postgres was free to return
the candidates in physical heap order. get_by_cid walks those candidates under
one shared probe budget, visit budget and pager, so whichever comes back first
is the one that spends the request's budget: two nodes holding identical data,
or one node before and after an unrelated write, could resolve the same CID
differently and one could shed a 503 where the other serves.
The instability is not hypothetical. An unpin and re-pin of a single object,
which is an ordinary production sequence, moves that row to the end of the heap
and rotates the list.
The sibling pin_sources_for_oid already orders its union for exactly this
reason, and the handler comment next to it leans on that determinism.
Code review of the previous commit turned up three things worth fixing in place.
record_scan_truncation centralized the taint and the seal but logged nothing, so
centralizing actually made a truncation less visible than the scattered inline
taints it replaced: only the visit ceiling logged, and it logs from inside the
gate, before the caller decides whether a position gets sealed. Two identical log
lines could therefore mean "the ladder continues" or "the caller is stranded".
One debug line now carries the reason and whether a position was sealed. It logs
only whether one exists, never its value, since the position names a withheld
row's created_at and id.
The doc comment claimed to be "the one site that records a scan truncation" while
eight other taint sites bypass it. The distinction is real but it is not the one
the comment drew: a ceiling stops the scan and owes the caller a position, while
a transient skip refuses one row and the rows behind it are still walked. Says
that now.
The forward-only rule's stated justification was wrong. A later candidate's
position is still ahead of the token the caller echoed, so letting it win would
not move the ladder backwards; it would shrink each rung toward a single row.
Also records that the comparison is Rust byte order while the pager's keyset runs
under the database collation, which can disagree on a non-C collation, and why
that costs a replay rather than a skipped row.
The probe and visit ceilings taint inside gate_and_serve and returned Skip, so
the row they refused and every row behind it were walked past without a verdict
while the resume position was sealed from pager.cursor, the end of the fetched
page. Two ways that stranded content:
- On the final page, `pager.exhausted` breaks ahead of every mint arm, so the
shed carried no continuation at all. A tokenless search_incomplete is the
wrapped-scan answer, which tells `gl ipfs get` its ladder is over, so a holder
on that page was unreachable on every retry.
- Mid-page, the sealed cursor sat past the refused rows, so the resume skipped
them. The shipped ladder tests all set page_rows == ceiling, which puts the
break exactly on a page boundary and hides both.
The ceiling now returns GateOutcome::CeilingStop rather than tainting on its way
out, and the scan loop stops there and seals the row in front of the one that was
refused. record_scan_truncation is the single site that records a truncation:
it taints and seals together, and only ever moves the position forward, so a
later oid candidate re-walking the same rows on a spent budget cannot hand back a
token the caller already echoed. The wrapped-scan tail no longer clears a seal,
since a ceiling can stop a resumed scan part way through the last page.
The lifecycle comment still listed image_tag among the values needing a deliberate instance replacement. It is rendered into compose.yaml, which the upgrade command now reinstalls, so apply plus upgrade rolls it out. Only the values user-data writes into .env at first boot stay replacement-only.
All four are in the deployment half added last round; none is in the Rust.
The AWS guide told operators two different things
--------------------------------------------------
"Changing configuration" still said editing `image_tag` does not affect a running
instance and routed to `-replace`, while the upgrade section thirty lines above
now reinstalls the rendered compose, which carries `image_tag`. The same
paragraph told operators to edit `.env` and restart — which the new "Why the
upgrade rewrites the compose file" section explains is silently ineffective for
any key the installed compose does not already name.
Split by which file carries the change rather than deleted, because the
distinction is real: values rendered into `compose.yaml` (`image_tag`, ports,
`domain_name`, `db_host`, `icaptcha_*`) roll out with `apply` + `upgrade_command`;
values user-data writes into `.env` at first boot (`public_url`,
`bootstrap_peers`, integrations) do not propagate from tfvars at all and need an
edit on the instance or a replacement. The third case is the one that bites: a
setting the installed compose does not yet name is read from `.env` and dropped,
so it needs the upgrade first.
`--remove-orphans` stopped being inert
---------------------------------------
The flag is not new, but until the upgrade rewrote `compose.yaml` it reconciled
against the instance's own file, so the only orphans were hand-started
containers. It now reconciles against a fresh rendering whose service set is
conditional — `postgres` only when `db_host` is empty, `caddy` only when
`domain_name` is set. Setting `use_rds` or clearing `domain_name` and then
running what the README calls a pull-and-restart stops and removes the local
postgres or the TLS terminator. That is what those variables mean, but it happens
on the upgrade rather than on apply, so it is now stated next to the
Terraform-owned warning. Data survives either way; both bind-mount under
/mnt/data.
The macOS setting had no writer
--------------------------------
`enforceOwnerPush` was persisted, loaded, emitted into `.env`, and forwarded by
the generated compose — and nothing set it. `SettingsWindow` had no state and no
assignment, so the only way to reach the opt-out on that launch path was hand
editing config.json in Application Support, which no document mentions. Editing
`.env` does not work there either: `writeEnvFile()` rebuilds it from `Config` on
every start. It now has a Toggle beside "Sync repos from peers", following the
pattern already in that view.
Three lines the change made untrue
-----------------------------------
The handler test asserted "with no configuration at all" while setting
`enforce_owner_push` explicitly — the right thing for the test, the wrong claim
for the message. The cross-tenant docstring still described three distinct pusher
DIDs after A and B were given `n1`'s owner. And OSS-READINESS-AUDIT listed
unprotected branches not enforcing owner checks as a live blocker; owner checks
now cover every branch, and only the UCAN half of that sentence still stands.
Also dropped the `#[allow(dead_code)]` on `test_state_with`, redundant since the
same commit added its call site.
Code review found the doc comments carrying claims the code does not
support and two coupling points documented from one side only.
The floor and the resolver's walk_cap are not textual twins: the floor
reads the constant, the resolver reads the AppState seam, and they agree
only because every construction seeds one from the other. Say that, and
give walk_cap the back-reference it lacked. The lifted node_verifies
helper no longer named the middleware it mirrors, so an edit to
require_signature would not find either copy.
The request-target clause pins @path, not @method or content-digest: a
301, 302 or 303 still rewrites a signed POST to a bodyless GET while the
signature headers ride along. Record that rather than implying the hop is
safe. Add the query-removed and fragment-only matrix rows, and a mutant
pinning that scan-phase walks are not charged to the work bucket, which
nothing covered.
The floor reserved one complete legacy search per window, probes plus page
tolls, but the provenance visibility walk debits the same bucket before the
fallback runs and its cap is charged per phase. With a route limit set below
the floor the provenance phase spent from the budget the search was promised,
the fallback 429d short of its configured reach, and the retry re-paid the
same walk charges, so a readable holder past that point stayed unreachable.
Add the walk term, min(MAX_HISTORY_WALKS_PER_REQUEST, ipfs_max_repos_walked),
which is textually the resolver's own walk_cap so the two move together. The
ladder fixture pins repos-walked to 1 to keep the page toll, not the new term,
as the thing binding it.
The verifier already rebuilds @path from the URI it was sent, which is why
the client-side redirect bug surfaced as a 401 rather than as a bypass.
Nothing pinned that, so a change to the reconstruction could drop the query
half or collapse it to a constant and only the clients would notice.
Drive the production router fixture with a signature made over one path and
a request on another, and again with a query mismatch, plus an identically
signed control. No pre-fix RED is obtainable here by construction, so each
case is proven load-bearing by injecting the defect it names.
The two same-origin follow tests drove exactly the path-changing hop the
predicate now refuses, and their mocks only proved a target was reached,
never that the signature verified there.
Rewrite both into refusal tests and run the real gitlawb-core verification
over the request the target actually received, so the recorded verdict is
what the assertion speaks about. Each refusal is paired with a positive
control that verifies, so an empty verdict slot is attributable to the
refusal rather than to a dead harness.
may_follow compared host, port and scheme only, so a same-origin hop that
normalized a trailing slash or a query was followed. The signature binds
@path as the client sent it and the node rebuilds it from the URI it
received, so that hop left the signature covering a target nobody asked
for and the read 401d.
Add the request-target clause and repoint the seven matrix rows that rode
on a path change, so each keeps pinning the host or port property it
exists for rather than going false on the path alone.
The previous round forwarded GITLAWB_ENFORCE_OWNER_PUSH in three compose
templates. Two of those are not the file their runtime actually reads, so the
documented rolling-upgrade opt-out did not work where it mattered.
Existing AWS instances could not opt out
-----------------------------------------
`aws_instance` ignores user_data drift, so an instance keeps the compose file it
was created with, and the published SSM upgrade only ran `docker compose pull &&
up -d`. Compose passes only the variables named in a service's `environment:`
block, so a key added to the template never reached a running instance: an
operator following our own instructions wrote GITLAWB_ENFORCE_OWNER_PUSH=false
to /opt/gitlawb/.env, upgraded, and got the new default anyway — silently, with
their CI pushers refused.
The upgrade document now reinstalls the rendered compose file before restarting.
It is idempotent (identical bytes on an already-current instance) and it makes
the upgrade a migration rather than a restart. compose.yaml is Terraform-owned
and overwritten; per-instance settings belong in .env, which is untouched.
infra/aws/README.md documents that `terraform apply` must precede the upgrade,
since the SSM document embeds the rendering.
The packaged macOS app read a different compose file
-----------------------------------------------------
The bundled Resources/docker-compose.yml is not what a packaged app runs: with
no source checkout selected, `generateComposeFile()` writes its own template,
whose environment block did not carry the key. `Config.writeEnvFile()` also
regenerates .env on every start, so a hand-added value was discarded. Between
them, a packaged-app user had no way to opt out at all.
`enforceOwnerPush` is now a persisted app setting, emitted into .env on every
start, and `generateComposeFile()` forwards it with the same interpolation as
the bundled resource — one definition, three consistent sites.
A test inherited the environment it was supposed to assert about
-----------------------------------------------------------------
`default_config_refuses_a_non_owner_push` built state through `test_state`, which
parses `Config` from the process environment — 47 fields come from GITLAWB_*
variables. On a host exporting GITLAWB_ENFORCE_OWNER_PUSH=false, which is exactly
what the rolling-upgrade guidance tells operators to set, it constructed a
disabled state, let the push through to git, and failed for a reason unrelated to
the code under test.
Renamed to `enforced_owner_push_refuses_a_signed_non_owner` and given an explicit
config through a new `test_state_with`, so it states the configuration it is
about. Verified by running it both with and without the variable exported. The
shipped default keeps its own env-independent proof off the parser declaration,
in config::tests — a parser question and an authorization question should not
share a failure mode. `test_state` now carries a warning about the hazard, since
any of those 47 fields can skew a test the same way.
Every item below was verified against the code before changing it; two of them
were confirmed by running the probe the reviewer described.
A test the fixture sweep missed, and that could not fail
--------------------------------------------------------
`f1_write_cap_is_inert_without_a_resolvable_source_key` seeded `z6f1none` and
pushed as `z6MkF1NoKeyPusher…`, so the owner gate rejected it before it reached
the per-caller cap the test exists to measure. Its assertion was
`!matches!(r, Err(Overloaded(_)))`, which a `Forbidden` satisfies, so it reported
ok while measuring nothing — and it is not `#[cfg(unix)]`, so it did that inside
every green run. Confirmed by inverting the assertion to expect `Forbidden`: it
passed, proving the request stopped at the gate.
The pusher is now the owner, and the assertion is positive: reaching git on the
missing on-disk repo (`Err(Git(_))`) is proof the request cleared both the gate
and the cap. The sibling `f1_write_cap_key_is_per_source_not_per_repo` carried
the same weak negative in its second assertion and is tightened the same way.
The default was barely tested, and what tested it read the environment
---------------------------------------------------------------------
`Config::parse_from` consults the process environment, so on a host exporting
`GITLAWB_ENFORCE_OWNER_PUSH=false` the old `enforce_owner_push_defaults_to_true`
reported the operator's setting rather than what this crate declares — it failed
outright, which would also break any contributor who sets the variable. It is
replaced by two tests that are env-independent and behavioural:
* `enforce_owner_push_is_declared_true_independent_of_the_environment` reads
the declared default off `Config::command()`.
* `default_config_refuses_a_non_owner_push` drives the handler on an untouched
`test_state` config and asserts 403 — authenticated by a real RFC 9421
signature, so 403 rather than 401 is what makes it a statement about
authorization.
Both were checked by reverting `default_value_t` to `false`: the handler test
reports 500 where it wants 403, and the declared test reports `["false"]`.
The cross-tenant test lost its second tenant
--------------------------------------------
The sweep collapsed `u1_parked_push_does_not_shed_another_pusher_behind_the_same
_ip` onto one identity, leaving the test named for cross-tenant behaviour with a
single tenant and two docstrings describing a scenario it no longer ran. `n2` is
now owned by `z6u1nat2` and its push signs as that owner, so two distinct tenants
share one edge IP again — which is the property under test, since the per-source
key is the resolved peer address and never the DID.
Four comments outlived their premise
-------------------------------------
`repos.rs` (3 sites) and `.env.example` still said owner enforcement defaults
off. The IP-keying rationale they support is unchanged and still correct — a DID
farm defeats a DID key — so only the dead premise is rewritten: the cap is
load-bearing because the rate limiter bounds arrival rate rather than occupancy,
because the gate can be turned off for a rolling upgrade, and because one owner
can open many concurrent pushes to their own repo.
Operators on the container path got none of this
-------------------------------------------------
`GITLAWB_ENFORCE_OWNER_PUSH` is now forwarded in `docker-compose.yml`, the macOS
app's bundled compose, and `infra/aws/compose.yaml.tftpl`, defaulting to `true`
to match the node. Without it the rolling-upgrade opt-out this PR documents in
three places was unreachable for anyone running compose.
A cutover this project never promised
--------------------------------------
The README paragraph announcing a 15 September 2026 flip of
`GITLAWB_REQUIRE_SIGNED_PEER_WRITES` is removed. That flag is not changed here,
the date appears nowhere on main, and a PR about a different flag is not where
that commitment would be made. The table row for `GITLAWB_ENFORCE_OWNER_PUSH`
carries the part that belongs.
A docstring that credited the wrong thing
------------------------------------------
The field and test docs claimed the value-taking action is what makes
`GITLAWB_ENFORCE_OWNER_PUSH=false` resolve to `false`. It is not: the env form
resolved under the presence-only declaration too. What the change buys is
`--enforce-owner-push false` parsing at all, rather than erroring as an
unexpected argument. Both docstrings now say so.
`p2_push` signed every push as one shared `P2_PUSHER` constant, but its two
callers own different repos — `z6p2tail` and `z6p2fail`. Pinning the constant to
either owner leaves the other rejected by the owner-push gate, so the identity
now follows the `owner` argument the helper already takes, the same way the
burst test derives it.
The rejection surfaced as `async fn resumed after completion` rather than a 403:
a denied push completes on the first poll, `p2_logged` never observes
`receive-pack`, and the polling loop then polls the finished future again.
Fourteen concurrency tests seed a repo through `f4_state_with_repo` or
`p2_parked_release_state` and then push under an unrelated DID, which the
owner-push gate now rejects before the machinery each test asserts on is ever
reached. They are transport and lease tests, not authorization tests, so each
now signs as the owner of the repo it targets.
Two needed more than a swapped constant:
- `receive_pack_burst_scans_serialized_and_both_pushes_succeed` pushes to two
repos with different owners, so the identity is derived per push from the
owner already threaded through the closure.
- `u1_parked_push_does_not_shed_another_pusher_behind_the_same_ip` used three
distinct pusher DIDs, which owner-only push makes impossible. The distinction
was narrative: the per-source key is the resolved peer IP and never the DID,
so three concurrent pushes from one edge IP contend exactly as before. The
comment now says so rather than leaving the reader to infer it.
The gate keeps its own coverage in `enforced_rejects_non_owner_with_forbidden`
and `enforced_rejects_missing_did_with_forbidden`, so nothing here removes the
assertion that a non-owner push is refused.
These tests are `#[cfg(unix)]`, so a Windows checkout cannot run them; they were
missed in the previous round because the Windows suite runs 717 of the 825 tests
Linux runs.
A `did:key` signature is authentication, not authorization: the method is
self-certifying, so any party can generate a keypair, derive its DID and sign.
With `GITLAWB_ENFORCE_OWNER_PUSH` defaulting to `false`, `owner_push_rejection`
short-circuited and every `git-receive-pack` carrying any valid signature was
accepted — including pushes to a repository the signer does not own, and
including private ones, on every branch not explicitly protected.
The gate itself already worked when switched on. What was missing is that no
reachable default configuration switched it on, so a node started with nothing
configured accepted a push from anyone. That default is what this changes.
The argument now takes a value so there is a way back: `--enforce-owner-push
false` and `GITLAWB_ENFORCE_OWNER_PUSH=false` both disable it, while the bare
`--enforce-owner-push` form still means `true`. Without this the flag would be
presence-only (`ArgAction::SetTrue`), which with a `true` default would leave
operators no way to opt out during a rolling upgrade.
Three concurrency tests pushed as non-owners in order to reach the shedding
logic they actually assert on. They now push as the repo owner, so they keep
exercising the shipped default rather than opting out of the gate.
Docs updated in README.md, .env.example and docs/RUN-A-NODE.md, including the
delegated-key caveat below. README also carries a dated cutover for
`GITLAWB_REQUIRE_SIGNED_PEER_WRITES`, which stays `false` for now so live peers
can finish upgrading.
BREAKING CHANGE: `git-receive-pack` now rejects a push whose authenticated DID
is not the repo owner, returning 403 before any ref update is applied.
Delegated and CI keys count as non-owners: a UCAN `git/push` capability is
verified but not yet honored for authorization, so an agent pushing under its
own DID cannot push while this is on. Set `GITLAWB_ENFORCE_OWNER_PUSH=false`
during a rolling upgrade, or have automation push as the repo owner, until
scoped collaborator / UCAN-delegated push rights land.
Two tests build a fake git as a `/bin/sh` script, mark it executable through
`std::os::unix::fs::PermissionsExt`, and reap the hung `rev-list` with
`libc::kill(SIGKILL)`. None of that exists on Windows, and neither test was
cfg-gated, so `cargo test -p gitlawb-node` failed to compile there with six
errors before running anything. A Windows checkout could not run a single test
in the crate, including the ones that are platform-independent.
Gate both with `#[cfg(unix)]`. The attribute is a no-op on Linux, so CI keeps
running them exactly as before; only the Windows build changes, from "does not
compile" to "runs the platform-independent tests".
Refs #228
The dependency allowlist caught this: sharing the predicate pulled url, and
behind it idna and the icu crates, into the closure of the one crate that is
supposed to stay embeddable. Both clients that need the predicate already parse
URLs, so they opt in and nothing else pays for it.
The test dependency is not optional, so the origin matrix still runs under a
bare cargo test -p gitlawb-core. Gating the module on the feature alone would
have left those tests silently unbuilt, which is the failure the module is
there to prevent.
The previous commit stopped the CLI carrying request signatures to a host the
node redirected it to, and left the other signing client alone. That one is the
git remote helper, which is what actually runs on clone, fetch and push, and it
had no redirect policy at all. A push signs from its first request, so a node
answering 302 handed out a working credential, and on a 307 or 308 the pack
body went with it. The rule now lives in core and both clients use it, so the
two cannot drift apart again.
Three guards were not holding what they claimed. Bounding the retry wait by the
remaining deadline could be deleted with every test still green, which the
timing shows was four real seconds of overshoot past the bound the docs
advertise; the assertion was loose enough to pass either way and is now tight
enough to fail. The refusing-gateway warning was asserted through a test-only
mirror rather than the write the user sees, so deleting the write kept the test
green. And both no-hint exceptions, the definitive not-found and the wrapped
scan, were only ever reached without a token in hand, which is the one state
where they do nothing.
A response whose body failed partway through the read still replayed the held
token for every remaining attempt, spending the whole ladder on one position and
then reporting that the object might lie further on. A body cut by the read cap
was already terminal for that reason; a body cut by a broken connection is the
same unreadable state and is terminal now too.
A successful fetch no longer buffers the whole object before writing a byte of
it, so a node cannot make the client hold an arbitrary amount of memory inside a
time limit that was never a memory limit. The remaining silent skips in the
recovery loops now say what went wrong, including the manifest fetch and the
body reads, so a failure there is reported rather than looking like an absent
blob.
Also: the redirect chain bound is off by one against reqwest's own, has a test,
and its comment no longer claims to prevent something the request timeout
already prevents.
The node client followed redirects with no policy, and reqwest strips only the
classic credential headers when a redirect crosses origins. RFC 9421 signature
headers are not on that list, so a node answering 302 could hand a caller's
signature to another host, which then served whatever bytes it liked for a
content address that never hashed to them. Following continuations made it
worse by turning one signed request per invocation into as many as nine. The
signature covers the method, the path and the body digest but not the
authority, and the node accepts any signature inside a five minute skew with no
nonce ledger, so one harvested header reads a caller's private paths at any
node in the fleet.
Redirects are now scoped to the origin that issued them. Same-origin ones still
follow, so an http to https upgrade or a trailing-slash normalization keeps
working, and a cross-origin one stops rather than errors, so the caller sees
the 3xx through the status path it already has. A downgrade back to http on the
same host is refused too: the origin matches, but the signature would leave in
cleartext, which is the same leak by a slower route. A custom policy replaces
reqwest's built-in chain limit, so the limit is restated rather than lost.
The recovery path in clone had the same silence the CLI just lost: it skipped a
blob on any non-success without saying why, so a truncated scan, a permission
denial and a dead gateway were one indistinguishable outcome. It now names the
status or the transport error, capped and stripped of control characters since
the text comes from a caller-chosen gateway, and continues as before.
Reading a response body no longer conflates three endings. The cap cutting the
body, the body finishing, and the read failing partway were all one silent
break, so a failed read surfaced as an error message that stopped after the
colon. Each is now reported, and a body that could not be read says so.
Coverage for three paths that were previously argued rather than run: a
transport failure mid-ladder still hands back the resume token, a scan that
ends before any ceiling still answers a definitive not-found, and a response
whose body stalls after its headers is still ended by the client's own
deadline.
Review of the scan-continuation work found the ladder surfacing its token on
three terminals and dropping it on three others: a mid-ladder 5xx that is not
an overload, a truncation whose offered continuation fails validation while a
good token is still held, and a transport error. In each case the user is told
to start over, and a bare re-run restarts at the first row and re-spends the
node's per-IP budget, which is the failure the token exists to prevent. It is
surfaced on those paths now, still never on a definitive 404 or on the node's
deliberate scan-wrapped signal, and a rejected token is still never echoed
back.
A body cut at the read cap used to change the answer. Truncated mid-JSON it
failed to parse, lost its error code, and fell to the transient-overload arm,
which retried the old position for every remaining attempt and then reported
that the object might lie beyond the rows scanned when nothing past the first
page had been. The cap now reports that it cut, and a cut body is terminal.
Three guards were doing nothing. Mutation testing showed the Retry-After clamp
could be deleted with every test still green, so a node advertising a day-long
wait would have parked the client inside a single sleep that neither the
attempt cap nor the deadline can interrupt. The rejection of a malformed
caller-supplied token was equally untested, though the same property is proven
on the node-offered side. The withheld-blob ladder asserted only that each rung
was not a success, which any early exit satisfies without the rule ever being
evaluated; it now witnesses that the scan reached the end of the table.
The inter-attempt wait is bounded by the time left on the deadline, so a run
can no longer overshoot the bound it advertises. That drops the wait out of the
worst case entirely rather than shrinking it: the last attempt starts before
the deadline and the client's total timeout covers that request through the end
of its body, so a run is over by the deadline plus that timeout. The docs said
about 95 seconds and now say about 90.
They also said a large download already streaming is never cut off part way.
That was wrong. The scan deadline does not cover the download, but the client's
timeout is a total request deadline running until the body finishes, so a slow
transfer is cut off. Both prose sites now say what the code does.
Also: stdout is flushed before the command returns, so a closed pipe fails
loudly instead of yielding a truncated object with a success status.
Document the behaviour rather than the mechanism: that an object pinned before
the node recorded its repo is found by scanning the inventory, that the scan
stops at the node's ceilings, and that the command follows the resume token the
node hands back instead of reporting a false not-found.
State the bounds in values, not constant names, and state honestly what each
one covers. The deadline bounds the search: every attempt gets the time left on
it to produce response headers, and it deliberately does not cover downloading
an object once found, so a large blob already streaming is never cut off. The
error-body read runs outside it under the client's own timeout, which is why
the longest a single run can take is about 95 seconds rather than the flat
deadline, and the in-code comment now derives that instead of asserting it.
Say what is not promised, too. A rate limit ends the ladder because its window
is an hour and its Retry-After cannot be honored inside one invocation, and the
node's per-IP fanout brake can end a ladder well short of the cap, so automatic
resumption is not a guarantee that the object will be reached.
Where a bound stops the ladder, the printed invocation is the only thing that
makes progress: a bare re-run restarts at the first row, reproduces the same
truncation and spends the per-IP budget again.
The bounded terminals printed "gl ipfs get <cid> --scan <token>", but the
command took no --scan argument, so following that advice failed on an
unexpected argument and the resume input existed only behind a private seam.
The flag now exists and forwards to the same validation a node-supplied token
gets, which is what makes surfacing the token worth anything: without it a
stopped ladder restarts from the front on every rerun.
Three test fixes ride along, all found by running the mutation matrix rather
than by reading.
The deadline scenario asserted its message "must name the deadline" while its
fixture CID was bafkreideadline, so any message echoing the CID satisfied it.
Under a mutant that removed the deadline the ladder ran to the cap and that
assertion still passed. The CID is renamed so the assertion can only pass when
the message genuinely names the deadline, and the mutant now reddens on it
instead of on the call count.
The default classification arm was certified only for the token-less case: no
fixture reached it with a token already held, so degrading it to retry was an
equivalent mutation against the whole suite. A scenario now drives an
unrecognized code mid-ladder with a token in hand and asserts it is terminal.
The wired path from the argument to the resume input had no coverage; only the
seam was tested. A scenario now goes through the function clap dispatches to,
so dropping the argument turns it red.
A truncated legacy CID scan answers 503 with a sealed continuation token, and
the command turned that into a hard error, so content past a scan ceiling was
unreachable from the first-party client. It now follows the continuation
automatically.
Three bounds, each protecting a different resource. The attempt cap bounds the
work asked of the node. A wall-clock deadline bounds the caller's own time and
is passed into each request as that request's timeout, so a slow-but-alive node
cannot outlast it. The per-wait clamp bounds a single Retry-After, so a hostile
value cannot stall the client. There is deliberately no non-advancing-token
check: tokens are nonce-randomized ciphertext, so two seals of one position are
never byte-equal and a client cannot detect a wedged ladder by comparing them.
Classification happens at one site with no wildcard to retry. Status is checked
before the body is read. A rate limit is terminal on status alone, since its
window is an hour and its Retry-After cannot be honored inside one invocation.
An overload mid-ladder is retried on the still-held token, because that shed
releases its permit when the request ends and the node advertises a retry. Any
other status or code is terminal through the default arm, including an overload
on the first request, where no token is held and there is nothing to resume.
Every terminal message is capped and sanitized before it reaches the terminal,
and a rejected token is never echoed back; the bound is named instead. Tokens
are accepted only as non-empty base64url within a generous ceiling, so a future
change to the sealed layout cannot silently disable resume.
Where a bound stops the ladder, the still-held token is printed with the
invocation that resumes from it, rather than wording that implies a bare re-run
makes progress.
gl suite 332 green, fmt and clippy clean.
The node answers a truncated legacy CID scan with a 503 carrying a sealed
continuation token, and gl ipfs get turns every non-success into a hard error,
so content reachable only past a scan ceiling cannot be fetched with the
first-party client at all. The old test asserted that terminal 503 as the
desired behaviour, and its fixture body predates the token, so it is replaced
rather than adjusted.
Thirteen scenarios pin the contract: a resume that completes, the attempt cap,
a wedged server that keeps minting fresh tokens for the same position, a
missing or malformed token, a first-request overload, the signed resume, the
wall-clock deadline, a mid-ladder rate limit that is terminal, a mid-ladder
overload that is retried, the token length boundary pair, the token surfaced at
every bounded terminal, and a caller-supplied continuation that starts from the
token rather than the front.
Hostile-input scenarios assert the surfaced message is capped and stripped of
control and bidi characters and that a rejected token is never echoed back.
Ladder fixtures answer Retry-After: 0 so the clamped waits are zero and the
whole gl suite still runs in under a second in real time.
The seam signatures land here so the tests compile and fail on their
assertions: cmd_get delegates to cmd_get_inner, which threads a deadline, an
attempt cap, and an optional starting continuation. Its body is unchanged and
still bails on any non-success, so all fourteen new tests fail on the property
each one names.
GITLAWB_IPFS_MAX_LEGACY_SCAN_ROWS accepts any value from one upward, but the
pager always asked the database for a full page and the ceiling was only
checked afterwards, in the scan loop. With the ceiling below the page size an
anonymous request still selected and rule-loaded a whole page while holding
walk admission, and any ceiling that is not a page multiple overshot by up to
a page. Compute the remaining budget first and ask for min(page, remaining),
so the page size stays a batching detail rather than an implicit floor under an
operator ceiling.
The exhaustion inference moves with it. A short page used to prove the table
was exhausted; once the limit can be shortened by the budget it proves nothing,
and marking the pager exhausted breaks at the top-of-loop arm that sits ahead
of every ceiling arm, taints nothing, and mints no continuation. That path ends
in a definitive 404 for content that exists, so the predicate now compares the
returned rows against the limit actually sent.
The budget computation sits above the timeout opener so the deadline-wrapping
guard's lookback window is unaffected, and the limit is read from the AppState
ceiling the row-ceiling arm reads, not the config field, so the arm and the
query cannot disagree.
Full suite 1059 green, fmt, clippy, and --locked clean.
The row ceiling is enforced in the scan loop, after a full page has already
been fetched and rule-loaded, so a cap below the page size is ignored for the
first fetch and any cap that is not a page multiple overshoots by up to a page.
The existing coverage could not catch that. Both row-ceiling tests asserted
rows <= ceiling + one page, which passes whether or not the query is bounded,
and rows-returned cannot distinguish a bounded query from a full fetch that is
trimmed afterwards. Add a scan_limit seam recording the limit actually sent to
the database, tighten both tolerant assertions to exact counts, and add two
tests that pin the limit: a ceiling below the page size and a ceiling that is
not a page multiple.
Also add the fail-open fixture. A repo evaluated with a partial rule set is
served as if it had no rules, so the boundary repo is seeded public at root
with a path-scoped rule and a real blob on disk, pinned with NULL provenance so
the scan path's rule map is the one consulted, and asserted never served across
its own ladder. Its positive control serves the same blob with the rule absent,
so a failure there is attributable to the rules rather than to a probe or a
budget skip.
Both new tests fail against current code on the scan_limit assertion.
Both write-lock tests hand-copied advisory_lock_key's derivation. The copy
still used DefaultHasher, so once the key moved to SHA-256 (#210) the probe
checked a key nobody held. The disconnect test then failed asserting the
lock was held, and the success test passed asserting a release it never
actually observed.
Import the production function, as the acquire-deadline test already does,
so the two cannot diverge again.
An external review found three costs that outlive the request or the boot
that paid for them.
The scan token was sealed with a key minted per boot, so a restart or a
rolling deploy made every outstanding continuation undecryptable and the
handler reads an unopenable token as absent. A caller walking a large
inventory could be sent back to the front indefinitely. The key is now
derived from the node's Ed25519 seed with HKDF, so it is the same after a
reload, under a distinct salt and info string so it is not the signing
key and cannot be turned against signatures. Rotation is a version bump in
the info string. An earlier comment rejected derivation on the grounds
that surviving restarts bought nothing; that is exactly what it buys, and
the comment now says so.
The rule byte ceiling was summed after the page had already been fetched,
so it truncated the request without bounding the work: the allocation and
the transfer had happened. The budget now bounds the query itself. The cut
lands on a repo boundary, which is not cosmetic: a mid-repo cut leaves a
partial rule set, and a partial rule set at the gate is indistinguishable
from a repo with no rules, so it would fail open and serve an object those
rules withheld. Only fully loaded repos are gated. The query always admits
the first rule-carrying repo of a page, so the bound is the budget plus
one repo rather than the budget; without that a single oversized repo
would put the cut at the cursor and every retry would reproduce the same
page.
The old check carried a not-exhausted condition so a short final page
could not turn a complete scan into a permanent shed. That condition is
gone because the taint no longer keys on page length; it keys on the query
having left repos unloaded. A short final page that fits produces no cut
and still 404s, and one that is cut is honestly incomplete and resumable.
Discovery paged the whole repo table and stat'd every warm repo to pick a
sixteen candidate window. Its comment called that acceptable because the
cost was paid once, which was true when the sweep ran once per boot. This
round made the sweep periodic, so an upgraded node with one unrepairable
row paid a full table scan on every run, forever. The load now stops once
the window is full. It collects one candidate past the cap so that running
out still distinguishes a warm set that fits from one that does not, which
is what the reset arm needs now that the total count is no longer known.
Ordering is unchanged, so an attacker who mints repos still cannot move
the boundary.
Verified by mutation: a per-boot key strands a continuation across a
restart, using the signing seed directly defeats the domain separation,
summing bytes after the fetch materializes the whole page, paging to
exhaustion buys pages the window cannot use, and reading the fit under cap
from the window length loses the reset.
Code review found two ways the scan ladder still dead-ends, both the same
shape the round has been fixing: a truncation that sheds without a token,
which this design reads as "the ladder is over", so the caller restarts at
the front forever and everything past that point is unreachable.
The continuation was wired into the row and rules ceilings but not the
probe and visit ceilings, and those two bind first on any inventory with
root-readable repos, long before the row ceiling. Every fixture seeded
root-denying repos, so no probe was ever charged and no test could reach
the arms that were missing it. All four breaks stop at the same cursor, so
all four now carry it.
The token's field width was 128 bytes against a repo id the write path
admits at 356 (255 owner, a slash, 100 name); a plain full DID with a long
name already passes 128. A repo over the width at a truncation boundary
failed the seal and shed tokenless, and repo names are peer-controlled, so
the boundary was steerable rather than unlucky. Fields are now sized
separately, 64 for the timestamp key and 384 for the id, and the version
byte moves so older tokens fail open to absent instead of mis-parsing.
Tokens stay constant length at 668 characters.
The sweep no longer exits for good on a failed pass. Making it periodic
was the point of the round, and returning on the first error meant one
deadlock disabled repair until the next boot, silently. It re-arms on a
longer interval, escalates the log once failures are consecutive, and
backs off again when a run repairs nothing so a table that will never
repair stops costing 64 object reads every five minutes.
The rule ceiling counts retained bytes instead of rule rows and is checked
where the rules are actually fetched, since a page can carry unboundedly
many rules and rule count is the wrong unit for a memory bound. It skips a
short final page: without that, a complete scan of an absent object turns
into a permanent 503.
Two docs contradicted the code and are corrected: the walk cap is charged
per phase now, so a request can spend twice the knob, and the README's
retry bound holds when the row ceiling binds.
Verified by mutation: reverting either ceiling's continuation strands the
holder, narrowing the token width fails the seal at the boundary,
restoring the terminal return stops the sweep, counting rules instead of
bytes lets a page overshoot, and removing the idle backoff keeps a
fruitless table under the base interval.
get_by_cid takes the global walk permit and the per-source permit before
resolving the CID, and that lookup was clamped only by the 600s request
budget. A well-formed CID with no pinned row does no probe and no walk,
but against a stalled pool it held a scarce walk slot for the full
window, so requests from enough distinct sources could reject every real
retrieval at admission while nothing was walking.
The lookup now runs under its own budget, defaulting to ten seconds and
clamped by the request budget, so a larger value degrades rather than
extends. Admission still comes first. Resolving before admission was the
other way to fix this and it trades one amplification for another: taking
the read out from behind the gate lets unadmitted anonymous callers stack
concurrent queries, and shedding a flood cheaply before it reaches the
database is what admission-first is for.
Only that one await moves. Everything after it stays on the request
budget because it runs interleaved with work the caller has already been
admitted for: the per-oid source lookup runs after real probing from the
second candidate on, the marker pair only on a provenance miss, and the
pager fetches between pages that cost probes. A short clock anchored at
admission would be long spent by then, so widening it would shed
requests that are making progress. The region comment now carries that
assignment per await.
The shed names the knob an operator can actually turn, separately from
the request-budget shed, so the two are distinguishable in the body and
the log.
Verified by mutation: reverting the clamp overruns the short budget, and
anchoring one short clock across the whole permit-held region sheds a
slow but progressing walk. The second leaves the first test green, which
is what makes it a boundary case rather than a second removal.
MAX_HISTORY_WALKS_PER_REQUEST is MAX_PIN_SOURCES + 1, which is exactly
enough for the first pinner plus every recorded additional source. Those
seventeen sources each spend a walk reaching their deny when they are
root-readable but path-scoped, so the walk budget is gone by the time the
at-cap fallback reaches a public source that record_pin_source dropped at
its sixteen-row cap. If that source needs a walk too it is skipped at the
ceiling and the request returns 503 search_incomplete on every retry: a
public object nobody can ever fetch.
The counter is now per phase. gate_and_serve already knows which phase it
is in, so provenance and fallback each get their own budget against the
same cap, and the fallback's capacity no longer depends on what the
provenance phase spent. Total walk work per request stays bounded at
twice the cap, a constant, so the amplification story holds.
Raising the shared ceiling was the obvious alternative and does not work:
the adversary picks how many provenance slots exist, since they are
grindable repo ids filling pin_repo_sources, so cap + N denials recreate
the exhaustion for any constant N.
The test uses real repos with real path-scoped rules that each spend a
real walk, proven by the walk log. The existing buried-source test seeds
repos that do not exist, which never enter the walk block at all, and
that is why it could not see this.
The must-not assertion needed changing to mean anything. A CID is
content-addressed, so every holder of it has byte-identical content and
comparing the served bytes against the public repo's object cannot show
which repo served. The gate is pinned instead by a second object held
only by a path-denying repo, with the fallback armed for it, asserted to
404 for an anonymous caller.
Verified by mutation: re-merging the counters restores the deterministic
503; capping the fallback at what the provenance phase left over is a
reserve that still couples them and fails the same way; and dropping the
fallback cap entirely unbounds the total. Each reddens exactly one of the
two tests, so neither is standing in for the other.
A denial-only inventory paged through the whole repo table. gate_and_serve
returns Skip on quarantine and on the root visibility deny before either
walk.probes or walk.visits increments, and those two counters were the
pager's only stop conditions, so an anonymous caller holding a CID from
the public pins index could read every row and its rules while the global
IPFS walk admission stayed held for the request budget.
Three parts, and the shape of each is set by a way an earlier attempt was
wrong.
A row ceiling, with a rules companion. Rows scanned bounds the DB reads
and the pager's row retention; it does not bound rules, since the rules
query has no LIMIT and an owner picks how many rules their repos carry, so
that gets its own ceiling. Both taint into the existing retryable 503 and
never a 404.
A ceiling alone would stop there and strand content. The scan starts from
a NULL cursor over (created_at, id) ASC, so a bare ceiling cuts the same
oldest prefix on every request and a holder past it is unservable for
good, which is the defect this PR already carries a finding about. So the
503 hands back the position. Server-side alternatives were tried and
dropped: a node-global cursor lets concurrent callers skip each other's
windows, and a per-caller map lets a key farm evict an honest caller
mid-ladder. State the caller carries has neither problem.
The token is sealed, not encoded. It names a row the caller may not be
allowed to see, and id derives from the owner DID, so plaintext would leak
a withheld repo's existence, creation time and owner. XChaCha20-Poly1305
with a fresh nonce per seal, the canonical CID as associated data, an
expiry inside the sealed bytes, and a fixed-width plaintext so token
length cannot vary with the row it names. Undecryptable, expired,
malformed and foreign-CID tokens are one indistinguishable case: absent,
start at the front.
The wrap taint keys on the pager reaching exhaustion while resumed, not
on any break site, so a token pointing at the last row still sheds a
retryable 503 instead of falling through to the 404 tail.
Pages are charged to the per-IP work bucket. Without that the ceiling
bounds one request and nothing across requests, and the denial path pays
nothing, which is the per-request-free-allowance shape already rejected
once on this branch. The derived work-budget floor grows by the same page
term so a deep scan cannot self-throttle mid-ladder.
A toll shed renders 429 rather than 503, which reverses the tail's
documented precedence for this one case. The tail comment names the
exception: the caller's own bucket stopped the scan, not the node's
search, and a 503 would invite an immediate retry into an empty bucket.
Four existing tests hand-size the work limiter to their exact probe count
and now also pay for a page; each gained one token and a comment showing
the arithmetic. Production is unaffected because the floor grew by the
same term.
Verified by mutation: removing the ceiling unbounds the scan; truncating
without tainting turns it into a 404; emitting a token but never resuming
never reaches the holder; dropping the toll makes pages free; encoding
the token as base64 plaintext leaks the withheld fields; dropping the CID
from the associated data honors a cross-CID resume; a variable-width
encoding makes token length vary with the row; and a constant nonce makes
two seals identical.
satisfies_threshold counted raw signature entries against the maintainer
list, so a certificate carrying the same maintainer's signature duplicated
across two entries satisfied a 2-of-N threshold with only one real
signer. Dedupe by collecting valid signers into a set before comparing
against the threshold, so each maintainer counts once regardless of how
many times their signature appears in the certificate.
Discovery recorded the source and armed the fallback in two independent
best-effort writes. If the insert landed and the marker did not, the row
was left raw CIDv1 with a nonempty, below-cap, unmarked source set. Later
sweeps skip such a row, and get_by_cid reads that set as complete and
suppresses the fallback scan, so once the transient error cleared, a
public duplicate of an object whose recorded holder is private stayed
unreachable for good.
record_discovered_pin_source does both writes in one transaction: the
capped source insert, the gated failure-row delete, and the sentinel that
arms the fallback. The sentinel is unconditional because one holder found
by a bounded warm-only probe never proves the set complete.
The method ends in an explicit commit, so an elapsed deadline is a
definite non-write rather than an unknown. A failed record still leaves
an empty source set, which the resolver's own empty-set signal routes to
the fallback, so the degradation posture is unchanged and now has a test.
The fault injection is a BEFORE INSERT trigger on pin_source_failures,
not a dropped table. Dropping it would fail the delete inside
record_pin_source's own transaction, roll that back, and produce the same
empty set as the fix, so the test would pass without ever reaching the
half-state.
Verified by mutation: splitting the transaction back into two writes
404s the public copy, and dropping the sentinel leaves the fallback
unarmed.
Discovery probed the first sixteen warm candidates of a stable
(created_at, id) ordering on every pass, so an object held only by the
seventeenth repo was never repaired and every boot repeated the same
work. The probe cap bounds expensive reads, which is right; it was also
silently bounding which candidates are ever considered.
A persisted continuation now rotates the window. Four choices carry it,
each forced by a way an earlier draft was wrong:
Keyset, not offset. An offset indexes into a list whose length anyone can
change, so candidates entering below the boundary shift the window off the
holder. A key names a position that survives insertions. created_at is
server-stamped, so minted repos only ever append.
Per traversal, not per pass. The row cursor also advances per pass, so
per-pass rotation gives row R the window index (t*m + j) mod W and a row
sees only W/gcd(m, W) of the W windows. Sharing one window per traversal
makes coverage a per-row property.
Advanced only over probes that started with a live deadline. A probe
against a spent deadline is charged a read but never opens the repo, so
counting it would step the window over candidates nobody examined.
Persisted from inside sweep_pass, next to the row cursor. The run-end site
never executes: spawn_legacy_cid_sweep drops the future on SIGTERM.
Traversal state is owned by the wrapper because the dead-read cap can pause
a run mid-traversal and a later run ends it.
The sweep re-arms on a timer instead of running once per boot, so coverage
is wall-clock rather than a reboot count, and returns only on a failing
pass query. The unconditional sleep is what keeps a repaired table from
spinning.
Verified by mutation: no rotation strands the holder; an offset
continuation shifts the window; advancing to the window end both skips an
unread holder and burns an unprobed window; per-pass advancement drifts
one row's window from another's; no re-arm leaves the holder unreached; a
run-scoped accumulator applies the wrong arm after a cap pause; and
deleting the v26 entry fails the upgrade path.
DiscoveryCtx held one deadline for a whole sweep pass, so a single hung
candidate on the first source-less row spent it for every row behind that
one. Those rows then called repair_legacy_provider_cid with an expired
deadline, came back retryable without a meaningful probe, and were charged
reads for work they could not do. CID order is stable, so the same first
row consumed the budget on every boot and the rows behind it starved
permanently.
Each row now probes under min(now + git_timeout / 4, pass_deadline), so at
least four rows per pass get a live deadline and no row can take more than
a quarter. A row that starts with the pass deadline already spent returns
PassBudgetSpent: no probe, zero reads charged, retryable, and surfaced on
SweepStats so a run can say it happened. Charging those reads would burn
MAX_DEAD_ROW_READS_PER_RUN on probes that cannot succeed.
The field is renamed to pass_deadline because that is what it is:
load_discovery_ctx runs once per pass and a run loops passes. Naming it
for the run would have stated a bound the code does not hold.
The divisor is not a bound on a single probe. A row's slice is still
shared by up to MAX_LEGACY_DISCOVERY_PROBES candidates, and the doc says
so rather than implying a guarantee.
Verified by mutation: collapsing the row deadline back to the pass
deadline starves the second row, and folding the spent-budget arm into
ordinary retryable accounting charges reads that must stay free.
The sweep's source-less discovery had no fixture built from a real
pre-migration row. Every existing test seeded its pin through the modern
path, which cannot reproduce the shape an upgraded node actually carries:
a row written when pinned_cids.repo_id and pin_repo_sources did not exist
yet, so its source set is empty with no other signal.
The fixture un-applies migrations 19 and 20, inserts the provider-keyed
pin naming only (sha256_hex, cid, pinned_at), then migrates forward. That
ordering is what makes it real; seed_legacy_pin binds repo_id, the column
being proven absent, so the insert is inline.
It brackets the repair rather than asserting an end state: the anonymous
GET is driven before the sweep and asserted not to serve, then after the
sweep asserted to return the object's bytes. The marker assertion queries
pin_source_failures directly for the "" sentinel instead of reading the
pin_sources_incomplete bool, which cannot distinguish the sentinel from a
repo id that record_pin_source would later clear.
Verified by mutation: short-circuiting the sources.is_empty() discovery
arm restores the pre-fold fall-through and the test goes red on the
repair assertion.
The domain separation owner_slug + ':' + repo_name has no length prefix, so
it is injective only while owner_slug cannot contain a colon. State that on
the pub(crate) function and guard it with a debug_assert, since a raw DID
would collide (('did:key:abc','x') and ('did','key:abc:x') hash the same).
#173 removed Db::list_all_repos in favour of a keyset-paged
list_repos_page_for_scan, so this branch stopped compiling on rebase. One call
site, load_discovery_ctx, ported with its reviewed semantics intact.
The quarantine drop gets simpler rather than different: each page row carries its
own quarantined flag, so the separate list_quarantined_repos query is gone and
the filter runs over rows the pass already read. Private non-quarantined repos
still stay in the list. Warm-only via validated_repo_disk_path inside
spawn_blocking is untouched, so nothing here reaches repo_store.acquire and a
repair pass still cannot become a bulk restore, and an unsafe path is still
dropped with the same warn and stays terminal.
Oldest-first by (created_at, id) now comes from the query's ORDER BY (index
backed by migration v25) instead of a client-side sort, and pages concatenate in
that order, so the explicit sort_by is redundant and removed. The anti-grinding
reason for that order is unchanged and the doc comment keeps it: repo_id derives
from a grindable owner DID, so an id sort would let someone bury the true holder
past the probe cap. The probe cap still bounds expensive reads rather than
candidates considered, and no cap on candidates was added.
This pager runs to exhaustion, unlike the resolver's, which drives the same query
and stops when its probe or visit budget is spent. The comment says why: the
resolver is an anonymously reachable route holding scarce walk admission, while
this is background maintenance on a timer with no caller and no permit
contention that needs the whole warm candidate set to settle a row. Paging is per
pass, matching load_discovery_ctx's existing once-per-pass lifetime.
Refs Gitlawb/node#321.
The two halves of round 12 left a hole between them. The cursor rewind became
unconditional on reaching the end of the table, so every completed run re-walks
every row, and MAX_DEAD_ROW_READS_PER_RUN is what stops that from costing
O(rows) object reads on every boot. But row_read_attempted was only ever set in
the provenance loop, so a source-less row, the one shape discovery exists for,
spent up to MAX_LEGACY_DISCOVERY_PROBES reads and contributed nothing. The
per-row cap bounded one row; nothing bounded the run.
discover_legacy_row now reports the probes it spent and the caller charges them
whenever the row was not repaired.
Per probe rather than per row, because at 16 probes a row the budget would
otherwise admit 16 times the reads it names.
Including the retryable arm, which is where discovery differs from the
provenance loop. There a retryable read is against a repo the row names as a
holder and is expected to succeed once that repo warms. Discovery probes repos
the row does not name and re-derives its candidate list from scratch each run,
so an unrepaired row costs the same reads again on the next boot whatever its
outcome was. The cap-reached arm is retryable by design, to stop a grindable
repo id burying the true holder, and leaving it uncharged would leave exactly
the steerable path uncharged.
Rebasing onto the #173 head that made the marker per (object, repo) broke
three assumptions this branch was written against.
Discovery re-marked the set incomplete after recording a discovered source,
because a successful record_pin_source used to clear the whole per-object
boolean. It no longer does, so that call went from compensating for a clear to
being the only thing arming the resolver's fallback. It now marks against the
unknown-repo sentinel rather than the repo it just recorded, which would claim
that repo failed when it is the one that succeeded. The sentinel is what the
v24 migration carries pre-upgrade markers under and means the same thing here:
the bounded warm-only probe may have missed a holder and nobody knows which.
Two tests asserted the superseded behavior and are inverted rather than
deleted, which is what their own comments asked for. GAP 2 documented the
per-object marker being cleared by any later record; that residual is closed,
so the test now proves discovery's marker survives one. The cold-candidate test
proved the cursor was not rewound; the rewind is unconditional on completion
now, so it asserts the cost that still matters, that re-walking the row reads
no object bytes and restores nothing.
record_pin_source's doc already names this residual and accepts it: the
pin_sources_incomplete marker is one boolean per object rather than per
(object, repo), so a genuine record from a third repo clears a marker that a
different repo's failed record set. Its two neighbouring cases are tested,
the same repo re-recording and a no-op insert that must not clear, and this
one sat between them described in prose and guarded by nothing.
The test drives the real path rather than writing rows. Repo A first-pins the
object. Repo B genuinely holds it and pushes, its skip-branch record fails on
the definite-error arm, and the code marks the set incomplete. An anonymous
read is then SERVED from B, which is the assertion that matters: it proves the
unrecorded holder is reachable while the marker stands. Repo C pushes, its
record succeeds, and the clear fires inside C's transaction. The identical
read now 404s, with the preload counter proving the fallback scan never ran.
Nothing here changes behavior. It converts an accepted tradeoff from a comment
into a guard, so widening it takes a deliberate edit rather than going
unnoticed. The doc comment says the residual fails closed, and that holds:
the provenance loop and the fallback scan both serve through the same
gate_and_serve, so clearing the marker only ever removes a gated search and
can never serve an object the caller could not already read. The window is
exactly one to fifteen sources, because at cap the count-guarded insert is a
no-op and the marker survives.
If someone implements the per-(oid, repo) marker the doc proposes, this test
is expected to go red. It should be updated to the new behavior, not deleted.
Proven load-bearing: removing the clear from record_pin_source leaves the
second read at 200 and turns the test red on the marker assertion.
The boot sweep resolved each legacy row's bytes only through
pin_sources_for_oid. A row predating provenance has repo_id NULL and no
pin_repo_sources entry, so the source set is empty, the `for repo_id in
sources` body never runs, and the cursor advances past it. list_pinned_cids
also filters those rows out on is_raw_cidv1, so they were both unadvertised
and unrepairable: nothing on the node would ever fix them. The sweep exists
precisely for pre-provenance rows, and that was the shape it could not
handle.
For a source-less row, discovery now probes a bounded set of warm local repos
for the object and, on a hit, repairs the key through the existing
repair_legacy_provider_cid.
What it records is the part worth reading closely. It calls record_pin_source
additively and then mark_pin_sources_incomplete. It does NOT call
backfill_pin_provenance, and repo_id stays NULL. Reading identical bytes out
of a repo proves that repo HOLDS the object, not that it is the source: forks,
a vendored file, a shared LICENSE blob and the empty tree all collide across
repos. Since backfill_pin_provenance guards on AND repo_id IS NULL, the first
claim written is permanent, so an exclusive claim would let probe order decide
which repo's visibility rules gate a public read surface, with no correction
path. It would also disable the resolver's own fallback: needs_scan is
sources.is_empty() || at_cap || incomplete, so a non-empty below-cap unmarked
set reads as complete and stops the scan that would have found a servable
copy. The additive record plus the incomplete marker keeps that fallback
alive and makes nothing permanent.
Candidates are quarantined-filtered. list_all_repos is a bare SELECT over
repos with no quarantine or visibility filter, while the resolver's own legacy
scan loads list_quarantined_repos and gates on it, so discovery loads the same
set and drops those ids. A quarantined repo is hidden from every reader, so it
must not become a recorded source. Private non-quarantined repos stay in the
list: an additive source record does not widen what the resolver will serve,
since every source is gated independently.
The candidate list and the warm filter load once per pass, lazily, on the
first source-less row, and the is_dir sweep runs under spawn_blocking rather
than inline on a worker. Nothing goes through repo_store.acquire, so a cold
repo is never pulled back from remote storage; a repair pass must not become
a bulk restore.
Two classification rules matter more than they look. A cold or quarantined
candidate does NOT mark the row retryable. The candidate list is every repo on
the node rather than the row's holders, and sweep_legacy_provider_cids rewinds
the cursor whenever any retryable skip occurred, so counting a cold candidate
would make every source-less row retryable on every run on a cold-storage
node: the cursor would rewind forever, the sweep would never drain, and each
pass would pay full discovery cost. Only a read error on a warm candidate is
evidence about the row. Conversely, cap exhaustion IS retryable rather than
terminal, and candidates are ordered oldest-first by (created_at, id) rather
than by id: repo_id derives from the owner DID, which anyone can grind, so a
first-N-wins cap over an id-sorted list would let someone bury the true holder
permanently behind cheap registrations.
MAX_LEGACY_DISCOVERY_PROBES bounds the expensive unit, a bounded object read
from a warm repo. A candidate dropped cold, quarantined, or unsafe costs no
probe. Its doc states the contract it encodes rather than its resemblance to
the resolver's own cap, which happens to share the value under a different
contract.
The source record is best-effort and says so at the site rather than claiming
a later pass heals it. Once the key is rewritten the row is raw CIDv1, so the
cost gate skips it free and nothing revisits it; the healing path is the
resolver's needs_scan fallback, which the incomplete marker keeps available.
The no-remote-fetch property is asserted from both sides. On the effect side,
a cold candidate's disk path still does not exist after two full runs, with a
control proving the sweep declined an available copy rather than finding
nothing to take. On the call side, a source scan of this module (comments and
string literals stripped, four anti-vacuity assertions) rejects the
fetch-capable call shapes, and a mutation planting a real acquire in the probe
loop turns it red. What that pair does not cover, and the test says so: a
fetch reached transitively through git::store or db, and git's own promisor
lazy-fetch on a partial clone. Neither shape exists in the tree today.
Concurrency around the marker is executed rather than assumed. Discovery
records then marks, in that order, because a successful record_pin_source
clears the marker. Under a proven lock-wait interleaving, a concurrent record
committing before discovery's mark leaves the row marked, and two concurrent
passes over one row converge on a single source with the marker set. Both the
swap and the drop of that pair are red-checked.
One residual is pinned by a test rather than papered over, and it is not
introduced here. pin_sources_incomplete is one boolean per object, not per
(object, repo), so a genuine pusher whose record_pin_source commits strictly
AFTER discovery's mark clears it, leaving a non-empty below-cap unmarked set
that tells needs_scan to stop scanning. No ordering inside this sweep can
prevent that, and record_pin_source's own doc already states the same residual
for its existing call sites; closing it needs a per-(oid, repo) marker, which
is a change to the marker's shape. It is bounded in practice: the clearing
writer really holds the bytes, so every source in the set is servable, and for
these rows it cannot be a regression, since before discovery ran they were
keyed on a provider CID and were neither served nor advertised.
Tests: fifteen regressions driving the sweep entry points. The repair, cheap
second pass, probe cap and multi-holder cases were observed RED first. The
must-not cases carry mutations instead, since a guard against something
happening has no honest pre-fix red: quarantined holder never serves, a cold
candidate is neither fetched nor materialized and does not rewind the cursor,
an unsafe candidate name is dropped without aborting the pass while a later
safe candidate still repairs, a raw-CIDv1 row never enters discovery, and a
provenanced row still resolves through the source loop alone. Every guard
carries a red-check mutation with an attributed message, including three on
the design itself: one adds backfill_pin_provenance beside the additive record
and must redden the no-exclusive-claim assertion, one neutralizes the
incomplete marker, and one swaps the record and mark order.
GITLAWB_IPFS_MAX_LEGACY_PROBES bounded the probe fan-out, but nothing bounded
the load that ran before the first probe. On a legacy (NULL-provenance) or
capped/incomplete provenance set, get_by_cid called list_all_repos, cloned every
id, fetched visibility rules for the whole set, and fetched the node's entire
quarantine list, then retained all three for the request. /ipfs/{cid} is
anon-reachable and CIDs are enumerable from the unsigned pins index, and an
object pinned from MAX_PIN_SOURCES repos is permanently at cap, so every later
anonymous GET for it paid that whole preload with the scarce walk permits held,
even at a probe budget of 1.
The selection is now keyset-paged on (created_at, id) ASC, with the quarantine
flag returned per row so the separate whole-node quarantine query is gone. Both
key columns are immutable, so paging is exact: a repo touched mid-scan cannot
cross a page boundary and go unvisited, and updated_at is attacker-bumpable,
which would have let a caller sort their own repos ahead of the true holder.
The pager lives outside the oid loop, so its cursor and its accounting are per
request rather than per candidate. Before fetching another page the scan checks
that a row on it could still reach a verdict; a spent probe or visit budget
stops the read and TAINTS, so a truncated search still sheds the retryable 503
and never a false 404. A page of pure denials is deliberately not a stop
condition, since a cheap denial costs no probe and a public object buried behind
private repos must still be reached. Both page queries stay clamped to the
remaining request budget, and the rules query still fails closed.
A new v25 migration adds idx_repos_created_at_id. Bounding what the handler
materializes is only half the property: repos carried no index in the new keyset
order, so Postgres seq-scanned the table and top-N sorted it to return every
page while the walk admission was held, which moves the same O(rows) cost onto
the database rather than removing it. Measured on 50k rows: 951 shared buffers
and ~44ms per page without the index, versus an Index Only Scan at 7 buffers and
~0.09ms with it, and the keyset predicate pushed down as an Index Cond instead
of filtering after a scan. Nothing names the index in any query text, so its
entry says why it exists and a test asserts its presence structurally.
Two order-dependent tests were reseeded rather than left green: the buried-row
and acquire-taint cases encoded their ordering through updated_at, which no
longer determines iteration. Both were re-confirmed RED under their documented
mutations after the reseed. Three regressions land with the fix. One drives a
one-probe, one-row-per-page request against three candidate repos and asserts it
materializes one row and runs one query, which a query counter alone could not
have caught. One seeds two distinct source-less oids under a single CID and pins
the per-request property directly: the second candidate re-reads the pages the
first paid for, so the whole request costs one pass rather than one per
candidate. The third covers the index, including the INV-7 upgrade path.
Refs Gitlawb/node#173.
Spelling the escape character as `ESCAPE '\'` puts a backslash inside an
ordinary SQL string literal. Under the legacy `standard_conforming_strings=off`
setting that backslash escapes the closing quote, so the literal runs on and the
statement fails to parse. The failure happens before the bound parameters are
considered, so every prefix lookup breaks, including prefixes that contain no
metacharacter at all, and short certificate IDs stop resolving entirely.
Connection settings come from outside the process, so that mode can arrive from
a database- or role-level setting.
Use `!` as the LIKE escape character instead, escaping `%`, `_`, and `!` in the
bound pattern. The statement then holds no backslash and parses the same either
way, without needing a second layer of backslash escaping. A backslash in the
prefix becomes an ordinary character, which is the literal match the search
wants anyway.
Cover it with a test that runs the production query through a second pool that
requests the legacy mode at connection startup, so a future return to a
parser-mode-dependent literal is caught rather than only wildcard handling under
the default configuration.
Refs #319
The comment beside ipfs_max_legacy_probes told operators to tune
GITLAWB_IPFS_MAX_REPOS_WALKED, but AppState::ipfs_legacy_probe_budget reads
GITLAWB_IPFS_MAX_LEGACY_PROBES. The knob it named is the separate cap on
expensive visibility walks, so anyone following the guidance next to this line
adjusted the walk cap and left the legacy scan fan-out at its default.
pinned_cids.pin_sources_incomplete was one boolean per object, so any
successful source record cleared it, including a genuine record from a repo
unrelated to the failure. The resolver reads a cleared marker as "every source
is recorded", drops the scan fallback, and 404s an anonymous caller whose only
servable copy is the one whose record failed. The missing source is a property
of an (object, repo) pair, so v24 stores it as one and only a later record from
that same repo clears it.
Pre-upgrade markers carry over with an empty sentinel repo id that no real
record matches, so they keep their fallback instead of being cleared by the
next unrelated record. pinned_cids.pin_sources_incomplete is left in place,
unread, so a rollback to the previous release still finds its markers.
Also adds the handler-level coverage the size-probe commit could not: a test
seam drops the object between the type probe and the size probe, and the
request tails to a retryable 503 rather than a definitive 404. The seam is
keyed on (repo path, oid) because fixture oids are content-derived and shared
across tests, and an oid-only key deleted objects out from under two unrelated
tests running in parallel.
object_size_bounded mapped every non-timeout failure to Ok(None), which
gate_and_serve reads as a verified absence and does not taint the search for.
A corrupt object, an unreadable pack, or a failed spawn therefore ended the
search cleanly and handed an authorized caller a definitive 404 instead of the
retryable 503 tail, even though the type probe had already reported the object
present. Confirmed by execution against the old code: a corrupt loose object
returned Ok(None) where the healthy read returned Ok(Some(n)).
The probe now returns Result<u64, ProbeError>, the vocabulary the type and
content stages either side of it already use: a reaped child is Transient, and
everything else is classified by store readability, Transient when the store
cannot be read and Deterministic when it can. There is no absence value left,
so the collapse is unrepresentable rather than fixed at one call site, and the
gc-between-stages case is classified like any other fault.
ServedRead::Gone lost its only producer and is removed with it.
Covered by two store tests, each asserting a healthy read first so the fault
case cannot pass on a broken fixture. Not covered: the handler path that turns
the fault into a 503, because that call site hardcodes "git" rather than
state.git_bin, so no shim can be injected there.
Rewinding on every completed walk means a later run re-attempts the object
read for every row whose bytes are permanently gone, so a node that collected
dead pins from a deleted repo or a force-pushed history paid O(dead rows) git
invocations on every boot with no decay. A run now stops after 64 such reads
and keeps its cursor, so the next boot resumes past what it already walked and
the table is still covered across boots.
Also corrects two pieces of prose that described the superseded round 11
behavior: the SweepStats::retryable_skips field doc still claimed that field
drives the rewind, and the new below-cursor test's comments still said a clean
run parks its cursor.
The pass-failure test waited on a fixed 100ms sleep for the first pass to
finish before renaming the table out from under the walk. It now waits on the
cursor that pass writes, so a runner slow enough to push the first pass past
the sleep no longer fails the assertion for the wrong reason.
The deadline test hand-copied the advisory-lock key derivation, and that
copy drifted once the key moved to SHA-256, silently breaking the lock
collision the test is built around. Import the production function
instead so the key cannot diverge again.
A clean sweep run parked its cursor at the table's maximum sha256_hex and
later runs read only rows above it, so a provider-CID row written below that
point by another node during a rolling upgrade was never swept. The resolver
withholds such a row's advertised key, leaving the object unretrievable with
nothing left to repair it.
The round 11 rewind could not cover this: it fired only when a run skipped a
retryable row, and the run that parks the cursor is by definition clean. The
rewind is now unconditional on reaching the end of the table. A run that stops
on a pass error keeps its position instead, so a node whose DB fails part-way
through does not restart the walk from the beginning on every boot.
The cost is one ordered scan per run, plus one repair attempt per run for a
row that is unrepairable in principle. Rows already carrying the canonical raw
key cost a codec decode and no object read, so a node that has finished
repairing pays the scan and nothing more. The terminal-skip test now asserts
that bounded cost rather than the row going unread.
#254 landed after this branch was last pushed and it moves the same surface.
It converts get_by_cid's sqlx failure arms from AppError::Internal to a bare
Into<AppError> conversion, so a connection-class failure downcasts to
AppError::Db and answers 503 db_unavailable instead of a 500 (#251).
Both textual conflicts were spurious alignments rather than real overlaps.
Git matched this branch's provenance path against main's legacy-scan preload
because both regions end the same way, and oids_for_cid does not exist on main
at all. Resolved to this branch's structure, which already contains main's
preload in its own place with all three of its queries clamped.
The semantic change is the part that mattered and it was applied by hand
afterwards, not inherited from the resolution. All ten clamped DB awaits in
get_by_cid now use the bare conversion. Taking this branch's side wholesale
would have silently reverted #251 across every one of them, including the four
this branch added, since the arm shape they were written to mirror predated
#254.
Suite green on the merged tree with fmt, clippy, and metadata --locked clean.
Short-ID resolution passes a caller-provided prefix into a LIKE pattern. Any '%', '_' or '\' in that prefix is interpreted as a LIKE wildcard rather than a literal character, so lookups can match certificates outside the requested prefix.
Escape the three metacharacters in the prefix and add an explicit ESCAPE '\' clause so prefix matching is always literal.
Add a regression test covering literal '%', '_' and '\' prefixes.
Co-Authored-By: cairn-code <[email protected]>
api/repos.rs holds the global pin_semaphore across ipfs_pin::pin_new_objects
and across pinata::pin_new_objects. Inside both, batch_budget_gate only gates
between objects and the git read was already clamped, but the DB calls were
bare awaits. One stalled query parked the permit past every budget, and once
all pin permits were so held, post-push IPFS and Pinata replication stopped
for every repository on the node. The pool defers rather than sheds, so the
queue behind it only grows.
The bound set is derived callee-transitively from each budgeted region's call
graph, not from the shape of the fix. A grep for db./retry_db_record over the
loop bodies cannot see a DB await behind a call site, and that is not
hypothetical: it missed repair_legacy_provider_cid, whose own cid_for_oid and
repair_legacy_provider_cid awaits run under the permit from both lanes while
its deadline argument bounds only the spawn_blocking git read. Kubo: six
direct sites plus those two transitive ones. Pinata: eight direct sites, two
more than Kubo because its post-upload path has its own record_pin_source and
incomplete marker. Completeness here is the enumeration, scoped to the call
graph at this head, not a compile-enforced guarantee.
db_bounded takes the ABSOLUTE deadline, so a slow predecessor cannot hand a
later call a fresh budget. What its Elapsed arm MEANS is a property of the
operation, not of the timeout. A multi-statement transaction whose tx.commit()
is never reached definitively did not land: record_pin_source is
pool.begin(), INSERT, conditional UPDATE, tx.commit(), so a future cancelled
while the INSERT waits on a lock cannot have committed. A single autocommit
statement may well have landed server-side after the client future was
dropped, since tokio cancels the future but not the statement Postgres is
running; mark_pin_sources_incomplete and record_pinata_cid are that shape.
Each site maps its elapsed arm on that basis. Reads skip the object.
Provenance and backfill warn and continue, since the backfill is idempotent
under its repo_id IS NULL guard. A timed-out record_pin_source writes the
incomplete marker, exactly as its definite-error arm does, because the source
definitively was not recorded and leaving the set incomplete AND unmarked is
the state this change exists to prevent: /ipfs would read a non-empty
below-cap source set as complete and 404 a copy the repo would serve. The
arms stay separate so the warn still distinguishes a stalled batch from
scattered per-object failures.
The durability writes take a floored remainder. batch_budget_gate only
guarantees PIN_READ_FLOOR before an object starts and the add gets the whole
remainder, so a successful add can finish with ~0 budget left; an unfloored
bound would fail a write that today completes in milliseconds. That leaves
bytes in Kubo with no pinned_cids row, and on the Pinata side it is worse,
because pinned.push is unconditional there, so repos.rs would build cid_map
and gossip publish_ref_update for a CID that /ipfs will 404. The floor is
re-taken from now at each call, so graces chain within one iteration: the
worst case is deadline+4s on the Kubo skip branch and deadline+6s on the
Pinata add path, about 5% of a 120s PIN_BATCH_BUDGET. It does not stack per
object, because batch_budget_gate breaks at the next iteration's first
statement.
pin_new_objects_gated now takes the batch budget instead of hardcoding
PIN_BATCH_BUDGET. The permit-release regression asserts a bound that IS the
budget, so with the constant inlined it cost 120s of wall clock; passing it
drops the test to 3.4s. Production passes the same constant.
Both twin doc comments are updated together. Each previously stated that the
DB round-trips were untimed inside the budgeted region, which is now false.
The Pinata permit's total hold across pinata_object_list_for_refs stays
unbounded and is called out as such: repos.rs acquires the permit before that
pre-loop walk, so this change closes the DB-stall trigger, not the git-walk
one.
Tests: thirteen regressions, the stall cases observed RED first with the bare
await sitting on the lock until the outer wrap fired. Every guard is proven
load-bearing by a mutation carrying an attributed message, so a RED is pinned
to the named property rather than to seam removal.
Three properties needed a fixture built specifically to bind them, because
the obvious test passed either way. The record floor: a zero remainder still
landed the write, since tokio polls the inner future before the timer and a
local UPDATE round-trips inside one tick, so the marker's table is now locked
and released inside the grace window, which fails without the floor and lands
with it. The absolute deadline: every single-object stall test passes under a
per-call duration too, so a helper unit test now runs two calls against one
shared deadline and asserts the second is cut off by the remainder its
predecessor left, with the mutation granting a fresh duration equal to the
budget rather than larger than it. The Pinata post-upload elapsed arm: it had
been changed in lockstep with the two covered sites but nothing executed it,
so a fixture now lets the upload succeed and stalls only the record that
follows.
get_by_cid builds its walk admission (the global permit plus the
per-caller permit) and then made several DB awaits with no deadline while
those scarce permits were held. A stalled pool or a blocked query pinned
every walk slot for the whole stall, so later /ipfs requests shed capacity
503s long after GITLAWB_IPFS_REQUEST_BUDGET_SECS had elapsed. Any
unauthenticated caller can reach it.
The clamped set is derived from the admission-held region itself, not from
the two sites the review named. Walking the handler body found four bare
awaits:
oids_for_cid, pin_sources_for_oid, pin_sources_at_cap,
pin_sources_incomplete
The last two were not named in review. Already clamped and left alone: the
three per-source lookups and the legacy-scan preload trio. Two negatives
worth recording: gate_and_serve is the only callee reached from inside the
region and holds no DB awaits of its own, so the region does not extend
transitively, and ipfs_work_rate_limiter.is_throttled is in-memory, not a
round trip. Ten DB awaits in the body, zero bare after this change.
Each clamp takes the arm shape the per-source lookups already use, so a
timeout returns the budget-tainted retryable 503 and the early return
RAII-drops the admission. budget_shed and remaining are hoisted above the
oids_for_cid await so the provenance-path sites share one definition; the
legacy-scan preload keeps its own in its nested scope.
Tests: four stall regressions, one per clamped site. The first two lock the
table their query reads and were observed RED first, with the bare await
sitting on the lock until the 10s outer wrap fired.
The marker pair needed a test-only seam rather than a lock. Both queries read
tables that earlier queries in the same request touch first, so a LOCK TABLE
aimed at them stalls an earlier clamp instead, and budget exhaustion has the
same problem because an earlier query elapses before the pair is reached. The
remaining window between pin_sources_for_oid returning and the pair running
is a single get_repo_by_id round trip, measured at 0.14 to 0.17 ms, which is
too small to time a lock into without flaking. So there is a cfg(test) stall
armed per target, placed INSIDE the timeout wrapper so the clamp is what
fires. Every seam item is cfg(test), including the function itself, so a
non-test caller would not compile, and a full release build is clean.
Each of the four asserts the same contract: a 503 inside the budget, the
budget taint in the body, the walk permit returned, and a follow-up request
admitted rather than capacity-shed. All four clamps are proven load-bearing
by reverting the clamp alone with an attributed message, and the mutations
cross-isolate: reverting one turns only its own test red, so each test
stalls at its own query rather than a sibling's.
Cross-model review round, findings corroborated by the in-process reliability
reviewer.
Both channel waits were unbounded. If the reader thread never reached its read,
or never returned after the peer close, the test wedged the binary instead of
failing, and cargo test has no per-test timeout to rescue it. Both are now
recv_timeout with bounds far above any real scheduling delay, so hitting one is
a genuine defect report rather than a flake, and the failure names which wait
stalled.
The handshake paragraph also overstated what the handshake buys. It listed "the
reader thread not yet scheduled to reach the read at all" among the states the
handshake removes, which reads as though the whole scheduling window is closed.
It is not: the handshake removes thread spawn from the window, and a reader
descheduled between its own Started send and its read is still a false-green
path. That path is now stated plainly instead of being left to the following
paragraph to correct.
Re-verified after the change: probe RED 50/50 under a deleted
set_nonblocking(false) at full-suite parallelism, RED on the named message under
a deleted set_read_timeout, 50/50 green unmutated, fmt and clippy clean.
Two findings from review of the probe test.
The helper has two postconditions and only one was guarded. Deleting
set_read_timeout left every test green while a stalled peer would park the
shim's only accept-loop thread forever, since handle_conn runs inline in that
loop. std exposes read_timeout() as a getter, so this half is checkable exactly
with no timing component. Confirmed load-bearing: deleting the line reds the
probe on the new message.
The probe also normalized one socket and then read from a try_clone of it, so
it only proved anything if blocking mode is shared across the two. On Linux it
is, because try_clone is dup(2) and O_NONBLOCK lives on the shared open file
description. On Windows try_clone goes through WSADuplicateSocketW and WSASocketW
(std os/windows/io/socket.rs), which mints a new socket descriptor, and whether
FIONBIO follows it is a question this test would have depended on without being
able to answer. It would have failed silently and green, on the one platform
where the underlying defect is real. The clone was unnecessary, so the reader
now owns the socket the helper normalized and the question does not arise.
Also correct two overstatements in the probe's own doc comment. It called
itself deterministic while its verdict is a 250ms window, and it described the
residual false-green path as "the few instructions between the signal and the
call" when the window is a full cross-thread round trip: signal, read, return,
channel send, and this thread waking. Both now say what the code does.
Re-verified after the change: probe RED 50/50 under a deleted
set_nonblocking(false) at full-suite parallelism, RED on the named message under
a deleted set_read_timeout, and 50/50 green unmutated. The end-to-end test came
in at 39/50 on this run against the first mutation, a third sample alongside 43
and 47, so its doc comment now reports the spread instead of two numbers.
The regression test added alongside the shim fix could pass with the fix
removed. It synchronized on a channel send that happens after accept() but
before handle_conn runs, so the request bytes could already be buffered by
the time the shim read them, and the read then succeeded on a non-blocking
socket. Measured at 43 and 47 of 50 full-suite runs catching a deleted
set_nonblocking(false), so it went false-green roughly one run in eight.
Extract the normalization into normalize_accepted_stream and assert its
postcondition directly, with no client and no request-arrival race. The new
probe puts a connected socket into non-blocking mode, calls the helper, and
checks that a reader parks instead of returning WouldBlock. Blocking mode has
no getter to query: std exposes none and Winsock has no FIONBIO getter at all,
so the property has to be observed through behavior.
The reader signals Started immediately before its read and the main thread
waits for that signal unbounded before opening the 250ms window. Without the
handshake the window would also be satisfied by a thread that has not been
scheduled yet, which would reintroduce the same silent hole one level down.
Peer close comes after the assertion, never before; closing first unblocks the
read and fails the test even when the helper is correct.
Also correct a platform claim this PR had in five places. The bug is not
Windows-only. An accepted socket inherits the listener's flag on BSD-derived
stacks, so macOS and the BSDs are exposed too; Linux is the exception because
std uses accept4 there, which does not inherit file status flags
(rust-lang/rust#67027). A macOS contributor could hit this today while the
comments said it was impossible. The macOS half is documented rather than
executed, and the comments now say so.
Verification, full-suite runs at default parallelism:
helper body mutated (set_nonblocking(false) deleted): probe RED 50/50
call site mutated (helper call deleted): probe GREEN 0/50,
integration test 45
and 46 of 50
unmutated: 50/50 green
The call-site row is the probe's blind spot, measured rather than assumed: it
calls the helper directly, so only the integration test notices a deleted
call, and it does so at a sampled rate rather than reliably. That residual is
recorded in the integration test's doc comment.
The test reproduces acquire_write's advisory-lock key derivation locally so
a held session lock can stall the handler. repo_store's advisory_lock_key
now uses SHA-256, so the DefaultHasher copy no longer collided and the
stall never happened. Update the local copy to the SHA-256 derivation.
Resolve conflicts: keep the SHA-256 advisory-lock key change alongside
upstream's connection-affine RepoWriteGuard rework and its F1/F3b/F4/U8
test suite; RUN-A-NODE.md keeps both the updated Public URL row and the
rolling-upgrade advisory-lock row.
start_shim sets the listener non-blocking so its accept loop can poll the
stop flag. On unix accept() hands back a blocking socket, so that never
showed; on Windows the accepted socket inherits the listener's FIONBIO
state and std does not reset it, so every connection arrived non-blocking.
handle_conn reads with read_line(..).unwrap_or(0), which turns the
resulting WouldBlock into "peer closed" and returns without writing a
response, dropping the connection and aborting the client mid-request. The
shim accepts as soon as the handshake completes, strictly before the
request bytes arrive, so the window was open on every connection and
closed only because the 5ms accept poll usually lagged the client. That is
what made real_git_withheld_shaped_first_post fail on windows-latest with
os error 10053 and zero POSTs recorded.
Normalizing at handle_conn entry also restores the 30s read timeout on the
next line, which a non-blocking socket silently made meaningless.
The regression test forces the Windows socket shape explicitly, so it is
load-bearing on every platform rather than only the one that exhibits the
bug, and it establishes the accept-before-request interleaving with a
handshake rather than a sleep: a timing-based version cannot flake red but
can flake green under load, which would make the guard vacuous.
Rebased onto current main. Keep budget-timeout wrappers on get_by_cid;
convert sqlx failures with Into<AppError> (not AppError::Internal) so
pool closed/timeouts become 503 db_unavailable. Same for list_pins and
arweave anchors, with closed-pool regressions.
repair_legacy_provider_cid built its own deadline from git_timeout and never
consulted the batch deadline, but both pin loops call it with a pin_semaphore
permit held. At shipped defaults that is a 600s git_service_timeout_secs against
a 120s PIN_BATCH_BUDGET, so one legacy row whose cat-file wedges could hold a
global pin slot for five times the budget the batch is supposed to cost and
starve every other repo's pin work. The loop's budget gate cannot help: it runs
at the top of the next iteration and cannot preempt a call already in flight.
The helper now takes the deadline rather than deriving it, since its two kinds
of caller can afford different holds. Both pin loops clamp to
min(batch deadline, now + git_timeout). The boot sweep holds no permit and has
no batch to overrun, so it keeps the plain git_timeout.
The Kubo side has had this since the repair landed and the Pinata side inherited
it from the lockstep work in the previous commit, so both get a test: a wedged
cat-file with a 60s git_timeout and a 2s batch budget must return on the batch
order. Both were RED at 62s before the clamp and are load-bearing by mutation.
Also stops the read-bound comments implying both arms bind in a default
deployment. At shipped defaults the batch deadline is always the tighter one;
the git_timeout arm is what an operator who lowers that knob gets.
Add a regression test for verify_strict: a signature forged under a weak (small-order) public key satisfies the verification equation but must be rejected. The non-strict verify path accepts it.
Co-Authored-By: cairn-code <[email protected]>
Rebased onto current main. Keep Overloaded/UnresolvableDid arms; make
Internal and non-unavailable Db responses opaque with server-side logs,
and pin exact JSON in unit + anonymous route tests.
Two gaps remained after the main merge closed the unbounded-read one. The Kubo
skip branch repairs a legacy provider-CID row to the raw-content resolver key on
re-push and the Pinata skip branch did not, so a Pinata-first node left those
rows unresolvable until the deferred sweep caught them. And the Pinata object
read was bounded only by the batch deadline, so one wedged cat-file could hold
the pin permit for the whole budget.
Both are fixed by threading git_timeout into pinata::pin_new_objects, which it
needed anyway: repair_legacy_provider_cid takes a Duration, and the read bound
is now min(batch deadline, now + git_timeout), the same shape the Kubo side
uses. repair_legacy_provider_cid becomes pub(crate).
The returned pairs still diverge on purpose and the twin docs say why: Kubo omits
a record-failed pin because its return is log-only, Pinata keeps it because its
return feeds the announcement cid_map. Pinata's return behavior is unchanged.
Four tests drive pinata::pin_new_objects rather than the Kubo twin, because a
repair call merely present in pinata.rs proves nothing about the Pinata lane:
the repair runs on the skip branch, the codec cost gate reads no bytes for a
row that is already canonical, a failed repair does not break the batch, and the
upload path never runs the repair. The two must-not guards have no obtainable
pre-fix RED, so they are proven by mutation instead: removing the cost gate and
moving the repair onto the upload path each turn one red.
On record_pinned_cid_with_source exhausting its retries, pin_new_objects warned
and then pushed the object onto the returned vector anyway, so the post-push log
told operators an object was pinned when the durable index had no row for it and
the resolver could not serve that CID. The push is now conditional on the record
succeeding.
The twins deliberately diverge here and the old comment said the opposite, that
the push mirrored pinata to keep them structurally identical. The Kubo return is
log-only, so dropping a record-failed pin costs nothing. The pinata return is a
real input: api/repos.rs builds the sha-to-cid map from it, which drives
upsert_branch_cid and the p2p ref-update gossip, so omitting rows there would
silently drop branch-to-CID records. Both comments now say so, since the next
reader's instinct will be to re-align them.
Test asserts the record-failed object is absent from the return while the add
mock still fired, and that a second object is still attempted rather than the
batch breaking. Verified RED against the unconditional push before the fix, and
load-bearing by re-injecting it.
Two doc drifts jatmn flagged on #173, both docs-only.
The /ipfs walk cap is min(internal ceiling, GITLAWB_IPFS_MAX_REPOS_WALKED),
where the internal ceiling is MAX_PIN_SOURCES + 1 = 17. config.rs, README, and
.env.example all described a max() floor instead, so an operator reading them
would think the default of 64 was the effective ceiling and that raising the
knob bought headroom. Neither is true. The runtime is not changed: two tests set
the knob to 1 and depend on the cap actually being 1, so wiring the documented
floor would make the knob inert and break them.
db_max_connections has defaulted to 48 for a while and four comments still said
20. Two of them did arithmetic on it. main.rs and repo_store.rs justified the
separate advisory-lock pool by "20 is below 32", which is now false and made the
separation look like a consequence of the defaults; it is structural, since a
push holds its connection for the whole receive-pack at any pool size. Also adds
a RUN-A-NODE troubleshooting entry for the boot failure an in-place upgrade hits
when it is still configured with the old pool size of 20.
#174 landed on main at 5981257, so the integration merge this branch carried is
now redundant with main's own history. Five files conflicted, and two more
auto-merged into states that did not compile.
store.rs: main migrated the bounded-read family from a per-call timeout:
Duration to a shared deadline: Instant. The auto-merge took main's
batch_check_probe alongside this branch's object_type_bounded, which compose,
and produced two read_object_bounded definitions with incompatible signatures
and no conflict marker. Took main's convention throughout and re-applied this
branch's additions on top: object_size_bounded, the wedged-child reaper test,
and the doc enrichments.
ipfs_pin.rs: the pin read is bounded by the earlier of the batch deadline
(#174) and the object's own git_timeout (#173). Neither implies the other and
both are load-bearing: with only the batch deadline, one wedged cat-file holds
the pin permit for the whole 120s budget; with only git_timeout, a batch of
slow reads runs past the budget. Each direction was verified by injecting it
and observing the RED.
api/repos.rs: kept main's choice of the literal "git" over state.git_bin for
the pin path's cat-file, since that knob is the walk binary and tests point it
at a fake. The parameter stays so the seam is still drivable from a test, and
the reason now sits at the call site.
main.rs: dropped main's free sweep_rate_limiters, superseded by
AppState::sweep_rate_limiters, which sweeps the same limiters plus
ipfs_work_rate_limiter. Both call sites already used the method.
db/mod.rs: dispositioned main's two new peers-table test writers in the
completeness ledger.
Migration numbering is unchanged. v17 is byte-identical on both sides, and
main's own reservation comment already allots 18-23 to this work.
Resolves jatmn's #192 P2. run_bounded's clean-exit teardown only terminated
the job when one had been established; when assign_job returned None it did
nothing, so a git-remote-gitlawb helper descendant that outlived the `git
fetch` leader while holding the inherited stdout/stderr pipe write-ends stalled
the reader joins on the helper's ~300s HTTP timeout, past the bound run_bounded
promises. The timeout path's reap_tree (taskkill /T) fallback cannot rescue the
clean path: once the leader has cleanly exited it no longer roots the
descendant in the parent-PID tree, so taskkill /T cannot reach the orphan.
The Job Object is the only reliable Windows teardown, so make it mandatory:
assign_job now returns JobHandle and panics per-step if the job cannot be
established, and both teardown paths unconditionally terminate it. Nested
unnamed jobs with KILL_ON_JOB_CLOSE are standard on the shipped
x86_64-pc-windows-msvc / CI runners, so this costs no coverage; it converts a
silent ~300s hang into a loud, actionable failure and strengthens the bound
(KILL_ON_JOB_CLOSE kills the descendant at latest when JobHandle drops).
reap_tree, whose only Windows use was the removed None arm, is narrowed to
cfg(all(not(unix), not(windows))).
The change is entirely cfg(windows). Verified by cross-compiling the tests for
x86_64-pc-windows-gnu (compiles clean, clippy -D warnings clean, no dead_code
from the reap_tree narrowing) and by the unchanged Linux git-remote-gitlawb
suite (green). The Windows runtime teardown is reasoned-not-run here; CI's
Windows job is the runtime gate.
run_bounded's clean-exit descendant reap was Unix-only: on Windows a
git fetch leader that exits cleanly leaves the git-remote-gitlawb helper
blocked in its ~300s HTTP request holding the inherited pipes, and
taskkill /T can't reach a descendant once the leader is gone, so the
reader joins blew past the promised bound. The child is now assigned to
a Job Object at spawn (the analog of the Unix process_group(0)), and the
job is terminated on both the timeout and clean-exit paths, which kills
the whole tree regardless of leader liveness. KILL_ON_JOB_CLOSE is the
safety net. The clean-exit regression is now cross-platform so it
exercises this on a future Windows CI lane (#228).
Unix tests unchanged and green; the Windows path is compile-verified for
x86_64-pc-windows-msvc (runtime coverage lands with #228's CI lane).
The test accepted any nonzero git exit (with posts>=1) as the recorded #191
withheld rejection, so a truncated response or a later HTTP failure after the
first POST could pass green. Assert the specific `expected ACK/NAK` signature so
only that break passes, and pin LC_ALL=C so the diagnostic stays the untranslated
English string on a git build with l10n installed. (#192 F6)
The gitlawb_fetches_total gate keyed on whether the request carried `done`,
which both undercounts a no-done completion (a client can finish a flush-
terminated have round with `ACK <oid> ready` + pack and no `done` -> counted 0)
and, on the filtered path, counts a rejected response per POST (the pre-#191
two-POST negotiation the client refuses was counted twice).
Derive completion from the response instead. upload_pack / upload_pack_excluding
now return whether they served a pack; response_served_pack detects the PACK
magic in both side-band-64k and raw framing. The plain path counts when the
response delivered a pack (catches no-done, skips negotiation-only rounds so an
N-round fetch counts once); the filtered path stays gated on the finalizing
`done` round until #191 makes that negotiation valid. Also tighten
request_finalizes_fetch to reject 0003 framing and trailing bytes after the
`done` pkt-line. (#192 F1/F2 + CodeRabbit repos.rs:761)
init() did an unsynchronized REGISTRY.get() check then OnceLock::set(...).
expect(...) for each metric, so two concurrent callers both passed the check
and the loser panicked at 'set INFO once'. Two tests call init(), so the
parallel test runner could hit this. Guard the body with std::sync::Once so it
runs exactly once. Verified: a 32-thread barrier stress-test panics before the
fix and passes after (run in isolation; not committed since the process-global
registry makes the race non-deterministic in the shared-binary suite). (#192)
A non-UTF-8 or non-hex 4-byte length prefix was collapsed to 0000 via
unwrap_or, so a garbled header was handled exactly like a flush -- masking
corrupted/desynchronized protocol input and letting accumulated have lines
go out as a synthesized flush round. Propagate the parse error instead;
a real 0000 still parses as a valid flush. (#192)
run_bounded only tore down the process group on the deadline path. When the git
leader exits on its own but a descendant (the git-remote-gitlawb helper mid-HTTP
request) outlives it holding the inherited stdout/stderr write-ends, the reader
joins blocked on the helper's ~300s timeout, past the deadline run_bounded
promises. Close the group on the clean-exit path too, guarded by a kill(-pgid, 0)
liveness probe so a recycled pid is never signalled (the process-group ID stays
reserved while any member is alive). Adds a leader-exits-first fixture and a
regression test.
Verified by execution: with the clean-path reap removed the new test blocks ~10s
on the descendant (RED); with the reap it returns promptly.
git_upload_pack recorded a fetch and observed a pack size on every successful
POST. In HTTP stateless-RPC the plain serve path streams a pack only on the
finalizing `done` round, so an N-round incremental fetch was reported as N
completed fetches. The filtered path (upload_pack_excluding) instead serves a
self-contained full pack on the single POST that reaches it, with no `done`, so
gating purely on `done` would drop those private (path-scoped) fetches to zero.
Count once on the POST that actually completes a fetch: the `done` round on the
plain path (request_finalizes_fetch parses pkt-lines and fails closed on a
malformed body), or the one filtered-serve POST (served_filtered_pack).
should_count_fetch captures the decision; a completed fetch is exactly one such
POST, so a multi-round negotiation is never double-counted.
The observe_pack_size call still measures the request body, not the served pack;
that mislabel predates this change and is left as a follow-up.
Verified by execution: a three-round plain fetch and a no-`done` filtered serve
each record exactly one completion; reverting the gate to count per POST makes
the plain fetch record 3, and breaking the decision to AND drops both to 0.
The large-output and descendants tests re-invoke the test binary as their
fixtures instead of spawning sh/seq, so they run on x86_64-pc-windows-msvc;
the non-unix timeout path tears down the whole fetch tree with taskkill /T
before joining the drainers instead of killing only the leader. Also folds
the receive-pack POST block into post_pack_round (review nitpick), removing
the duplicated status-check/forward logic.
Address jatmn's two P2 review findings on real_git_fetch.rs.
F1 (INV-22): run_bounded killed only the top-level `git fetch` on the
timeout path, so the git-remote-gitlawb helper descendant kept the
inherited stdout/stderr pipe write-ends open and the reader joins blocked
on the helper's ~300s HTTP timeout instead of the 30s deadline. Spawn the
child with process_group(0) and, on timeout, tear the whole group down
(SIGTERM, ESRCH poll, SIGKILL escalation, bounded cap, then reap) before
joining, mirroring gitlawb-node/src/git/smart_http.rs. libc is added as a
cfg(unix) dev-dep; non-unix keeps the child.kill() fallback so the
always-on test still compiles. New regression run_bounded_reaps_descendants_holding_the_pipe:
a backgrounded sleep inherits the pipe and outlives a 1s-deadline leader;
reverting to a leader-only kill turns it RED (elapsed ~10s).
F2 (INV-21): the non-success branch of real_git_withheld_shaped_first_post
accepted any `git fetch` failure and only printed the POST count, so a
failure before the first POST (broken advertisement, helper lookup,
connection) passed as the expected withheld rejection. Assert posts >= 1
before accepting the nonzero exit. Forcing a pre-POST failure (0 POSTs)
turns the assertion RED.
F1: a nonempty have-batch terminated by EOF means git aborted mid-negotiation, but
the fetch loop folded RoundEnd::Eof into the Done arm, appended a synthetic
0009done, and POSTed it — so a cancelled private fetch still issued an authorized,
signed upload-pack request and made the node generate and stream a full pack to a
client that had gone away. Split the arm: only a real done pkt finalizes; a
nonempty EOF terminates without a POST. The nonempty_eof_batch_posts_as_final_round
test that locked in the wrong behavior is flipped to assert no POST.
F2: fetch_with_helper piped stdout+stderr but only polled try_wait, so a fetch
emitting more than an OS pipe buffer (~64 KiB) blocked git before it exited and the
harness reported a false 30s deadlock. Drain both streams on reader threads while
enforcing the deadline, so the deadline measures a real protocol hang only.
F3: the integration repos were removed only on the success path (predictable
pid/counter names), so a timeout/panic leaked real git repos and could poison a
later run. Root them in a RAII tempfile::TempDir that drops on unwind.
RED->GREEN: nonempty_eof_batch_aborts_without_posting (pre-fix POSTs a synthetic
done → the aborted-negotiation POST fails against the single-response server; fixed
makes one request, no POST). repos_are_cleaned_up_on_unwind (temp dir gone after a
panic unwinds past the guard). Full git-remote-gitlawb suite 42 unit + 3 integration
green, fmt + clippy clean.
Adds executed coverage for the negotiation branches the first pass left
unrun: an invalid pkt-line length (rejected, not underflowed), a non-EOF
read error (propagated, not swallowed as end-of-stream), an immediately
empty upload-pack request (skips the POST), a nonempty have-batch
terminated by EOF (POSTed as a final done round), and an empty
receive-pack body (skips the POST). Each was mutation-checked to fail
without its guard, so the coverage is load-bearing rather than vacuous.
A code-review pass found the negotiation loop would POST once per bare `0000`
flush that carried no haves, so a malformed `wants + N*0000 + EOF` stream
amplified into N signed POSTs. Real git never sends a content-free mid-negotiation
flush and the peer is upstream of the local git process, so this was not
reachable in practice, but it left the loop unbounded per input. Skip a round
that carries no new haves (an empty flush) and only POST rounds with have content
or a terminator, which also makes the loop provably bounded by git's finite have
set. The fresh-clone done-with-no-haves round still POSTs.
A committed integration test drives a real `git fetch` through the built helper
against a real `git upload-pack --stateless-rpc`, so the stateful-to-stateless
ACK bridging is proven by execution rather than reasoned. The main.rs mock tests
cannot falsify it: a pre-scripted Cursor never reacts to the server's ACKs. An
in-test shim replicates the node's v0 serving; the fetch is forced to at least
two negotiation rounds (a fixture that resolved in one round fails the test) and
the resulting object graph is verified.
The second scenario drives the withheld-blob shape (a full pack on the first
POST, as upload_pack_excluding does) through real git and records the outcome:
real git rejects a pack where it expected an ACK continuation ("expected
ACK/NAK, got ..."), so a multi-round fetch of a withheld repo does not complete.
The helper forwards and terminates cleanly rather than hanging, so this is a
node-side concern, not a helper defect, left as a follow-up per the plan's
Withheld-Path Decision. The test guards the helper's non-hang behavior and
surfaces the break if the assumption ever changes.
git-remote-gitlawb advertised `connect`, so git spoke the stateful native
protocol, but handle_connect collapsed the exchange into one GET plus one POST.
A multi-round fetch (more than ~32 overlapping commits) deadlocked: git sent a
flush-terminated have-batch and blocked for an ACK/NAK the helper never sent,
while the helper blocked reading for a `done` git never sent.
Turn Phase 2 into a per-round stateless-RPC client loop. read_upload_pack_round
returns one round at a time (at a flush, done, or EOF) instead of buffering past
flushes waiting for `done`; negotiate_upload_pack captures the wants once and
POSTs a self-contained request per flush-terminated batch: wants, one flush,
every have accumulated so far, and exactly one terminator. Each ACK/NAK is
streamed back to git so it advances, until the done round returns the pack. The
node's `git upload-pack --stateless-rpc` keeps no state between POSTs, so wants
and all prior haves are re-sent every round and no intermediate flush survives
into a body (which would truncate the negotiation server-side).
Signing is preserved per round: every POST carries the Phase-1 decision (signed
after the 404 escalation for a private repo, anonymous for a public one), and a
mid-negotiation denial surfaces through the sanitized error path rather than
reading as an empty or successful fetch. The receive-pack path is unchanged.
Covers the deadlock repro, multi-round accumulation on the wire, per-round
signing for private and public fetches, mid-negotiation denial surfacing, the
withheld-shaped forwarding path, and single-round non-regression.
The git_read_semaphore doc claimed it bounds both info/refs advertisements,
but the receive-pack advertisement draws from git_push_advert_semaphore
(api/repos.rs), so an operator sizing the read pool for both would undercount
their read capacity.
README's settings table also omitted the served-git concurrency env vars that
.env.example already documents: MAX_CONCURRENT_GIT_OPS, MAX_CONCURRENT_GIT_PUSHES,
MAX_CONCURRENT_READS_PER_CALLER, and MAX_CONCURRENT_PIN_TASKS.
GITLAWB_MAX_CONCURRENT_PIN_TASKS caps how many pin loops run at once, not how
much object-id list memory is retained. On the local IPFS path the caller
materializes the list before pin_new_objects_gated acquires, so a parked loop
still holds its full list and the parked count is capped only per repo by
EncryptInflight. config.rs and the pin_new_objects_gated docstring already say
this; the env template still said the knob bounds the memory.
The drain barrier sampled target = count + 1 before connecting the flush and
waited for any next increment. A leak accepted after that sample and before
the flush was counted satisfied the wait; the test then subtracted the flush
as if it were that leak, so leak_count could be 0 with a real leak present
(false green).
Tag the flush connection instead: the accept loop records each accepted
connection's source port, and drain waits until the flush's own source port
is recorded (the flush is never counted). Because accept() is FIFO, when the
flush is recorded every connection that completed its handshake before it is
recorded too, so the post-drain count still covers all egress in flight.
Verified with a probe that reproduces the reported scenario (leak accepted
before the flush is now counted) before removing it.
DefaultHasher is not frozen by the Rust standard and has shifted across
toolchain versions, so the same (owner_slug, repo_name) produced a
different i64 advisory-lock key on different builds and broke
cross-machine write-exclusion for the same repo.
Hash with SHA-256 instead and pin the result with a golden-value test,
plus a one-axis-at-a-time differential test so a regression that drops
either parameter is caught. Document the SHA-256 re-keying caveat on
acquire_write and in the RUN-A-NODE.md operator checklist (drain writes
or cut over single-node during a rolling upgrade).
The tripwire bound a hard-coded 127.0.0.2 alias, which macOS does not route
by default (only 127.0.0.1), so the test could not run on apple-darwin
targets. Switch the fixture to [::1] (IPv6 loopback), which is present by
default on Linux, macOS, and Windows and is NOT covered by NO_PROXY
(127.0.0.1, localhost).
The negative control still holds: hyper-util's proxy matcher has no implicit
loopback bypass — a host is only proxied-around when literally listed in
NO_PROXY (verified in the pinned hyper-util source). Both the positive
control (direct reach of [::1]) and negative control (proxy intercept) still
pass.
- Tripwire: fail loudly (expect) when 127.0.0.2 cannot be bound, instead of
printing SKIP and returning green with zero coverage of the negative
control. Linux routes 127/8 as loopback; macOS needs an explicit alias,
and the message says so.
- Observer: restore a bounded drain/ack barrier (Mutex + Condvar). The
detached accept thread signals after each accept; the test opens a flush
connection and blocks until it is acknowledged. accept() is FIFO, so every
leak connection that completed its handshake before the flush is counted
before the flush. Leaks that start their handshake after the drain remain
the irreducible window, now documented as such.
- support: document that arm_blackhole / disarm_proxy_env mutate process-wide
env vars and are safe only because each consuming binary has exactly one
test; a future second test must be serialized against them.
The stabilise() polling loop cannot prove no more connections are coming,
only that none arrived in the last 100ms. A carefully timed background
leak could schedule its connection after stabilise stabilises and the test
would pass despite a live egress. Replace with a simple counter read
before and after obtain_proof. Connections accepted during the synchronous
call are counted; a leak whose TCP handshake completes after the counter
is read is outside the assertion window — this is now documented as a
known limitation of the single-threaded test harness.
Also narrow the fire-and-forget claim: the observer catches leaks that a
closed-port blackhole would miss (because it counts connections regardless
of error propagation), but the timing race inherent in a detached accept
thread means the guarantee is best-effort for post-return handshakes.
Replace the closed-port proxy with a ProxyObserver (TCP listener with
acknowledged accept-loop flushing) so fire-and-forget leaks whose error
is discarded are caught and the count is synchronized before assertion.
Add a blackhole tripwire asserting obtain_proof fails against 127.0.0.2
with the proxy armed, narrowed to a connect-level error to prevent
vacuous passes. Share arm_blackhole across both test files via a
tests/support module.
Document the observer's coverage scope: it only witnesses leaks on the
single exercised happy path; error/retry branches and non-proxy
transports are outside its guarantee.
Two review findings from the last round.
The object-store readability probe swallowed EACCES on objects/pack exactly
like the benign NotFound of a loose-only store, and never checked the loose
fan-out at all. A present object behind an unreadable directory therefore
looked absent: git cat-file printed a clean missing, the probe said the store
was fine, and /ipfs/{cid} answered 404 for content that exists. The same bool
drives classify_store_fault, so an unreadable store also came back as a
terminal 500 rather than a retryable 503. Only NotFound stays benign now, and
the probe additionally opens the object's own fan-out path. That check is
scoped to the oid (validated as 40 or 64 hex before any path component is
derived) rather than sweeping all 256 directories, so an unrelated broken
directory cannot veto every lookup in the repo, and the caller pays one open
instead of a directory drain it does not need. An unreadable directory now
fails closed: absence cannot be certified through something we cannot read, so
a missing object there sheds a retryable 503 instead of a 404.
Both pin loops held the shared pin permit across a synchronous, unbounded
git read, so a wedged filesystem pinned a slot in a pool that defers rather
than sheds, and blocked a runtime worker while it did. The read now runs
through the bounded, process-group-reaped helper under spawn_blocking, against
the batch deadline, and the Pinata loop gets the batch budget it never had.
The budget gate is shared between the two loops so they cannot drift, and it
declines to start a read it cannot pay for rather than spawning a child only
to reap it.
A fault stops the batch only when the object store is genuinely unreadable,
where every remaining object would fail the same way. An object-scoped fault
skips that object and continues, because one corrupt object should not cost a
healthy batch, and the recovery path re-offers the same list and would break at
the same place forever.
Jatmn's #209 round: known_ungated is empty on main and the three listed
handlers are gated, so the guide must not teach them as exceptions. Restore
the signature/certificate section with a trusted positive case and a forged
rejection case instead of blessing a forged "valid" drive.
Scope the denial-test guidance to each route's actual authorization
rule instead of a blanket non-owner assertion, since several
documented routes (star_repo, replica actions, bounty actions,
author-based closing) intentionally authorize non-owners.
Name the known_ungated exceptions (list_webhooks, list_replicas,
list_protected_branches) instead of implying full repo-scoped read
coverage.
Describe the listed jobs as workflow checks that run on every PR
rather than merge-blocking CI, matching the live main branch ruleset
(one required review, no required status checks).
Encode the contributor-facing gate rules (the three owner-gate forms in
use and the by-design non-owner exceptions, path-scoped reads and the
listings contract, denial semantics, deny-test expectations, migration
versioning, client denial surfacing, MSRV and the exact CI gate set) in
a root AGENTS.md that coding agents load automatically, and point
CONTRIBUTING.md at it. Enforcement labels state only what is on main
today.
Whitelist eight safe AppError variants; opaque Git and anything else.
Add a source-scrape guard for map_err sites, a real sqlx context chain
test, and a tasks(limit) ceiling pin at 200.
Drop the head-file fetch and Rust lexer. Match test attributes on
added lines only, with horizontal-whitespace anchoring, covering bare
#[test], #[cfg(test)], namespaced forms (#[tokio::test(...)]), optional
leading ::, and raw-identifier spellings. needs-tests is advisory, so
the simpler matcher is enough.
Addresses jatmn's review: an exhausted 15-file scan or an oversized/
unreadable blob no longer asserts "no test found" (needs-tests was
being applied on unverified evidence), blobs over 5MB are skipped
before decoding, the "\ No newline at end of file" patch marker no
longer miscounts as an added context line, and #[r#test] /
#[tokio::r#test] raw-identifier attributes are now recognized.
Use function-pointer map_err to clear redundant_closure. Opaque only
when a sqlx::Error is in the anyhow chain so claim/finish business
messages survive. Clamp GraphQL tasks(limit); add mutation coverage.
Addresses jatmn's second round of review on pr-triage's needs-tests check:
hitting the per-PR fetch cap now logs and stops scanning instead of
throwing (which was skipping label reconciliation entirely), file content
comes from the Git blob API instead of the Contents API to avoid the
encoding: none gap for 1-100MB files, the added-line check now requires a
non-whitespace attribute character (not just added whitespace) on the
added line, stripped literals leave a placeholder so they can't glue
adjacent tokens into a false match, and the namespaced-test regex now
accepts a leading :: for absolute-path attributes.
Route query and mutation DB failures through a shared helper that
logs the cause chain server-side and returns a generic client
message, so unauthenticated POST /graphql cannot leak sqlx text.
A diff hunk carries no lexical context from lines outside it, so scanning
patch text alone could misclassify an added line that sits inside a comment
or raw string opened earlier in the file (or outside the hunk's context
window). Fetch the head-version file content via the contents API and lex it
in file order, checking only the lines the diff actually added.
The needs-tests detection regex only matched a literal #[test] or #[cfg(test)
immediately after #[, so it missed namespaced test attributes like
#[tokio::test] or #[async_std::test]. Any PR whose only new tests were async
(the common pattern in this codebase, e.g. crates/gl/src/agent.rs) got falsely
labeled needs-tests even when tests were added in the same commit.
Reproduced against PR #198: it added two #[tokio::test]-only tests in the same
commit as the source change, and the triage bot still commented needs-tests.
Widens the regex to also match any attribute ending in ::test], verified
against representative patches including PR #198's actual diff.
Closes#201
The `test (beta)` lane failed on 17f9eda reading `leader.pid`, NotFound, while
`test (stable)` passed the same commit. The test armed a 100ms watchdog deadline
and the fake git records its pid as its first statement, so a runner slow enough
to delay `/bin/sh` past 100ms gets the whole process group torn down before the
pid is ever written.
Reproduced by delaying the fake git's first statement by 300ms, which yields the
identical NotFound panic at the same line. With the deadline at 2s the same
injection passes at 0.3s, 1.0s and 1.8s of start delay, so the race window moves
from 100ms to 2s.
The property under test is unchanged: a pre-fix build stands the watchdog down on
drain and blocks on `child.wait()` for the child's full `sleep 30`, which still
trips the 10s recv budget. Confirmed by injecting that exact defect, which goes
red on the budget assertion rather than on some other failure.
local_add_refusal had six unit tests and the wiring had none, so
replacing the whole match at the call site with an unconditional "Added
to local peer list." left every test green. That is the shape the store
already names: a helper test plus a status-mapping test does not prove
the call site.
Extract the selection into local_add_report, following the same
print-selection-helper precedent used for the certificate DID hint, and
assert both directions: a refusal never yields the success line, and a
200 never yields the warning. Collapsing the match to always return the
success line now turns that red. The call site is left as a two-line
dispatch carrying no logic.
The scan matched a verb-plus-table needle per line, so it was whitespace
and case sensitive. The identical statement written across lines walked
straight past it, and that form is how the longer queries in this file
are already written, so it is the shape a new writer most likely takes.
Normalize the whole file before matching, mapping offsets back to line
numbers so each statement is still attributed to the function issuing
it. The scan is split into a helper so the property can be asserted
against a synthetic source: the tree contains no multi-line peers write
today, so a test that only ran against the real tree would prove nothing.
Fixtures derive from the needles at runtime for the reason the needles
themselves do, which is that a literal here is found by the scan of this
very file.
The check ran before anything looked at whether the row existed, so it
refused a pure liveness refresh of a row already in the table. Nothing is
protected by that: the row is already there, and an identical-URL refresh
changes no authority. What it did instead was freeze every peer a
deployed database already holds from before this gate, since upsert_peer
validated nothing then and the handler accepted any parseable DID. Those
rows take a 400 on every unsigned announce and cannot be corrected by a
signed one either, because no key resolves.
Now the DID is judged only when no row exists, which is the only place
refusing it prevents anything. The read-then-write race is benign: if a
row appears in between, the write becomes an update and the existing
repoint guard still refuses any http_url change.
The capped read means a peer answering with a body past the cap leaves
truncated JSON behind, and parsing that with ? before the status check
turned a 503 into "invalid JSON response", losing the status the user
needs. A non-JSON error page from a caller-chosen peer URL does the same.
The failure check now runs first and falls back to the body text when
there is no message field. Bound by a test that drives the command
against a mock peer, since reordering the two statements compiles and
passes every unit test in the file.
The peer-add refusal warning added earlier in this branch copied the
node's message straight into stderr, and both announce bodies were read
unbounded, so a hostile --node endpoint, a compromised local node, or a
MITM on plain http could put ANSI or OSC escapes into the user's
terminal. The accepted-announce block had the same hole on the success
path, where the DID and URL come from a peer whose URL the caller chose.
sanitize_node_msg and read_body_capped move from sync.rs to http.rs,
which already owns the wire boundary, and both paths in peer.rs now use
them. Only the display is sanitized: the values forwarded to the local
node stay verbatim, since that is data rather than terminal output.
The split reported the cause in the message but left the code as
bad_request, so a client telling this apart from the other validation
failures on the announce route had to substring-match prose. The auth
middleware already answers unresolvable_did for the same failure on a
request's keyid; the peer-write path now answers the same way. Status is
unchanged at 400, and an unsupported DID method keeps bad_request.
The unproven insert path folded every to_verifying_key failure into
UnsupportedDidMethod, so an oversize, malformed, or non-ed25519 did:key
was told its DID method is unsupported, which is false and leaves the
caller nothing to act on. Split the arm by cause: a parse or resolution
failure now returns UnresolvableDid carrying the underlying reason, in
the same wording the signed path already uses, while a foreign method
keeps methodNotSupported. Every input stays refused in the 400 class.
The refusal test fed local_add_refusal an invented message, so it could
never go red when the node's wording changed. Drive it with the Display
of PeerWriteDenied::UnprovenRepoint and assert the same substring the
node-side pin asserts against a real router response.
The announce route now answers 403 when an unproven caller tries to
repoint an existing peer. gl peer add discarded the local node's reply
with let _ and printed the success line unconditionally, so a refused add
reported as a completed one. The refusal is split into a small function so
the path is assertable, and it is tested for a 403 with a message, a
refusal with no body, and the accepted case. The local add stays best
effort: a transport failure warns rather than failing the command, which
is what the surrounding code already assumed.
The README described the staged rollout as accepting unsigned peers on
these routes, which is no longer the whole rule. With signed peer writes
off, an unsigned announce may still register an unseen peer and refresh a
row whose URL is unchanged, but changing an existing peer's http_url needs
a signature from that peer's own DID, and only a did:key can be registered
unsigned.
The gate restricting unproven inserts to did:key checked the method label
only. Did::from_str validates that the method is one of key, web, or
gitlawb and never inspects the key material, so did:key:notarealkey passed
and created exactly the row nobody can ever correct that the gate exists
to prevent. It now resolves the key the way the signature path does.
Six test fixtures using invented did:key strings stopped inserting once
the gate became real. That failure is the evidence it now reads key
material rather than a prefix, and it means those fixtures were exercising
a path no real DID could take. They carry derivable values now.
Resolving the key put a quadratic base58 decode on a route that accepts
anonymous callers by default. Measured through the router, a 64,000
character did:key cost 8.05s against 4.6ms for the previous label check,
on a tokio worker with no spawn_blocking, and a 1.5 MB body reaches the
handler. The bound sits in to_verifying_key so it also covers the
HTTP-signature keyid, which reads an untrusted string through the same
function. An ed25519 method-id is a fixed 48 characters, so 64 is slack.
Its test asserts the error names the length rather than merely being an
error, because an oversized id fails either way and a check for is_err()
would pass with the guard removed, having already paid the cost.
Three last_ping_ok assertions were vacuous: the seed helper never granted
reachability, so they compared false against false and would have held
against code that never cleared the flag. The helper grants it now and
asserts the seed took. Nothing was hiding behind them.
Drives the must-not set through the production router with real RFC 9421
signatures rather than injected extensions, so the signed path is exercised
as a caller would reach it, in both require_signed_peer_writes modes.
Covers the allowed cases and every rejection: unsigned insert of an unseen
DID, unsigned repoint refused with the row byte-identical after, a
signature from the row's own DID accepted, a signature from a different
DID refused, an identical-URL re-announce refreshing only last_seen, and a
did:web announce refused with no row created.
The signature-from-a-different-DID case is the RUSTSEC-2022-0009 shape,
where libp2p-core accepted a valid signature without checking it derived
the claimed identity. It asserts the key is bound to the row rather than
to anything the request carries.
The announce handler's keyid-mismatch branch had no test at all and
returned 400. It now returns 403 and is tested: a caller denied for want
of proof of control over the targeted row gets the same response class
whether or not it presented a signature.
The completeness ledger derives its set from every writer of the peers
table rather than from upsert_peer's callers, because a caller scan cannot
see a future writer that issues its own SQL. A second table records the
call-site authority for main.rs's bootstrap announce-back, which issues no
SQL and so can never appear in the writer ledger; its authority choice has
no runtime test in this change and is recorded as such.
The type system carries the guard: the authority parameter has no default,
so a new caller cannot omit it. The scan is the backstop for a raw
statement that bypasses upsert_peer entirely, and stays minimal rather
than copying the handler-directory apparatus, which does not apply to a
scan over SQL literals.
upsert_peer ended in an unconditional ON CONFLICT DO UPDATE SET http_url,
so any unauthenticated caller could repoint any peer. A repointed row
steers the sync worker's git remote, the post-receive notify fan-out,
trigger_sync, and the public GET /api/v1/resolve/{did}, none of which
consult the reachability flag that #270's stopgap resets.
An unsigned write may now insert an unseen DID but may never change an
existing row's http_url; changing one requires a verified signature from
that row's own DID. The gate is at the db boundary rather than in the
announce handler because there is a second writer: the bootstrap
announce-back in main.rs reads a peer's own JSON response body with no
proof of anything, and now declares itself unproven, which makes it
insert-only by construction.
The proven variant carries the DID it proves and upsert_peer compares it
against the row being written. A bare proven/unproven flag would say that
something was proven without saying which DID, leaving the real check in
the handler and reproducing RUSTSEC-2022-0009, where a valid signature was
accepted without checking it derived the claimed identity.
An unproven insert is restricted to did:key. Only did:key can ever
authenticate here, so a did:web or did:gitlawb row created unsigned would
become unwritable by anyone once this rule lands.
The unproven path derives its refusal from rows_affected, because an
ON CONFLICT DO UPDATE with a guarding WHERE reports zero rows rather than
raising, and a silent no-op is the fail-open this gate exists to prevent.
The refusal is a typed error the handler maps to 403, downcast out of the
anyhow chain the way sqlx errors already are, so it does not render as a
500.
Observed at c83cbc5 before the fix: an unsigned announce naming an
existing peer returned 200 OK and left the attacker's URL stored. That
test is the RED this change turns green.
Two of #270's reachability tests now drive the proven path. Under this
rule an unproven repoint is refused, so they can no longer express a URL
change through it; they assert the same behavior as before.
The coverage gate matched `# x-release-please-version` anywhere in a manifest,
so an annotated `version` under any other table satisfied it. A crate could
leave the version cargo actually reads unannotated, pass the gate, and still
freeze at release time, with release-please rewriting the decoy line instead:
the exact failure this gate exists to catch.
Scope the match to the [package] table.
IcaptchaCfg::new normalizes an empty GITLAWB_ICAPTCHA_API_KEY to None
via .filter(|s| !s.is_empty()) before storing api_key. No test drove
the empty-string value through new(), so a regression would attach an
empty bearer token to the challenge and answer requests and ship green.
The new test drives an empty key through new() on a trusted operator
origin and asserts it normalizes to None, with a contrast arm showing
a real key still attaches. Mutation check: removing the filter fails
this test and no other.
The merge kept the knob and its clap range but carried over only the
default-value assertion, so nothing asserted that 0 and 1048577 are
rejected. Restored verbatim from the #174 side, beside its
ipfs_max_repo_visits sibling.
Load-bearing: widening the range to 0..=2_097_152 turns it red.
Three gates scanned the "production half" of api/repos.rs by splitting at the
first cfg(test) attribute. Adding test-only items above the code they check
moved that split point above every line being scanned, so all three failed
looking for production code that was still right there.
They now split at the test module, which is what they meant. This is the same
defect already fixed in the ipfs gate during the merge, and no split-on-the-
attribute site remains.
Both behavior-bearing gates were re-proven load-bearing: F2 goes red when the
Pinata object set is re-derived before the pin permit is taken, and the tail
gate goes red when release stops consuming the same success flag. F3 is a
presence check whose marker cannot be removed by a single compiling edit, so it
is not mutation-proven here.
The drain read the repo row and its visibility rules once each. Either read
failing dropped that coalesced push's pins and recovery copy for good: the repo
read returned None on error, and the rules read collapsed through ok(), which
makes the repo read as not announceable. Both fail closed, so nothing leaked,
but nothing sweeps the lost work up either. A single database blip was enough.
resolve_drain_object_list now refreshes through a bounded retry: three
attempts, 50ms doubling, and either read failing is transient. A missing repo
row is the one terminal answer, so it releases at once and spends no retry
budget. Exhaustion logs at ERROR with the repo id and attempt count, so giving
up is observable rather than silent.
The coalescing loop is untouched. It already has the shape the retry needs: a
refresh that gives up skips the pin call, and the next finish_or_take_pending
picks up anything that coalesced while the retry was running.
Eight restored tests cover it, written and observed red against the seam-only
tree first. The five the plan predicted red came up red for the reasons it
predicted, and the three controls stayed green. All eight are mutation-proven:
no-retry, an off-by-one bound, gone burning budget, a rules error read as
empty, an unfiltered candidate set, a duplicated refresh, break-on-giveup, and
a pending slot that never drains each turn their own test red.
A real Postgres pool will not fail on demand, so the drain's error arms have no
way to be driven from a test. This ports the injection table the removed
requeue loop used, keyed on the task's repo id, which is a fresh uuid per test
so parallel cases cannot cross-inject.
The two reads now go through drain_get_repo and drain_list_rules, whose bodies
consult the seam under cfg(test) and then make the same call as before. Error
handling is untouched: the repo read keeps its arms, the rules read keeps its
ok() collapse, and no retry exists yet. The suite is unchanged at 856, which is
the point.
inject and counters carry a dead_code allow until the tests that call them
land.
These covered post-push work being requeued rather than dropped, and were
removed in the merge because they drove the replaced requeue loop. The
behaviour they check is still there, so they re-seam onto the coalescer without
any production change.
The seam moved in three ways. The inflight key is the repo identity key, not
the row id. A coalescing push now merges its (old, new) tip pairs into the
pending slot, so a test that passes an empty vec merges nothing and the drain
never laps: every coalesce carries real commit oids. And the full-scan arm is
reached through the pending-slot overflow, so the leak test coalesces 1025
pairs and asserts the slot degraded to FullScan before the run.
run_encrypt_pin_task_for_test builds the task context the way the production
spawn site does and takes a real owner and name, because the drain re-fetches
by owner and name; a blank name would resolve Gone on every lap.
All four are mutation-proven: drain-does-nothing, stale-empty-rules,
full-scan-skips-the-fail-closed-filter, and a leaked inflight key each turn
their own test red.
The merge deleted coverage that no longer compiled against the reconciled
designs, rather than re-seaming it. Three of those tests have counterparts in
the merged tree after all:
unlock_error_connection_close_is_bounded and
write_guard_dropped_off_runtime_disposes_the_connection came back with
close_conn_bounded and the Drop backstop, which the lock-pool design needed
after all (the after_release hook cannot free a lock when the unlock errors on
a live session). Both now observe the store's derived lock pool rather than the
pool handed to for_testing, which is where the guard's connection actually
lives.
get_by_cid_caps_repos_walked_knob_bounds_the_walks was orphaned when its body
was spliced into a sibling test. It drives GITLAWB_IPFS_MAX_REPOS_WALKED, which
is the binding cap only now that the gate takes the tighter of the two walk
ceilings.
The one test not restored is write_guard_release_when_not_locked: the guard has
a single construction site and it sits below the lock-or-bail, so a guard that
never took the lock cannot exist.
Reconciles the served-git concurrency cap with the /ipfs CID tree gate. The two
branches independently reworked the same three seams, so most of this is picking
one mechanism per seam rather than taking a side per file.
/ipfs walk admission: keeps this branch's provenance-first resolver and its lazy
legacy scan, but drops the detached serve task in favour of #174's shared
Arc<WalkAdmission>, cloned into each spawn_blocking. Both close the permit-release
bypass; the Arc does it without leaving an abandoned request's full legacy scan
running against a held slot. Ported in with it: the request-budget gate at all four
stages with each child deadline clamped to the remainder, the walk timeout computed
inside the closure so queue delay is charged, taint sources in place of a single
truncated flag, and the deterministic-fault 500 gated on nothing else having tainted.
The two per-request ceilings are now composed rather than one shadowing the other.
The visit ceiling was unwired by the merge and the walk cap answered to only one of
the two knobs; both are honoured, the tighter one winning.
Object probe: object_type_bounded takes a Duration and returns ProbeError, so this
branch's call shape keeps #174's absence-vs-unreadable-store discriminator. A clean
`missing` from cat-file is byte-identical whether the object is absent or the pack
is unreadable, and the discriminator is store readability, not git's wording.
Write lock: the advisory-lock pool's after_release hook does not cover an unlock that
errors on a live session. Measured: a poisoned connection returned to the pool still
held the lock 15s later, because the hook's own unlock_all fails the same way. So the
guard now disposes of that connection, in release and in the Drop backstop.
Post-push replication goes to #174's coalescer. This branch's requeue_faults
integration suite has no equivalent seam on that design and is not carried over; it
is recoverable from the pre-merge head.
Also clamps the provenance path's three per-source lookups to the request budget.
They run while the walk permits are held, which is the stall hazard the preload
clamp already covered, and the structural guard found them because it now requires
every occurrence to be wrapped instead of assuming exactly one.
1346 tests pass across the workspace; fmt and clippy are clean under --locked. The
four re-anchored structural guards were mutation-tested and all four go red on the
defect they name.
Both doc comments describing GIT_SERVICE_TIMEOUT_SECS_MAX said the 100-year
ceiling sits "an order of magnitude" under the ~584-year limit of a
u64-nanosecond Instant. The real margin is 3,153,600,000 seconds against
18,446,744,074, a factor of about 5.85. An order of magnitude means ten.
Worth noting how it spread, since the fix is trivial and the pattern is not.
The phrase was written once on the constant, then copied into the
ipfs_request_budget_secs doc by the commit that was at that moment correcting
a different false claim in the same file. Nobody re-ran the division at either
site, because quoting an existing sentence feels like citation rather than
assertion.
The direction of the error is the part that matters: a vague "well under"
would have been true and would have invited a reader to check. The
precise-sounding "order of magnitude" was false and discouraged checking.
Comments only; the diff carries no executable line, checked by filtering it
for non-comment changes. fmt and clippy clean under the CI form.
Review round on the three preceding commits.
The knob's doc said a value past the ceiling "aborts every /ipfs/{cid}
request in a release build". That is wrong, and three reviewers plus an
outside model landed on it independently: the ceiling is 100 years, the
Instant addition does not actually overflow until roughly 584 years on a
u64-nanosecond Instant, so values in between are representable and are
rejected as a deliberate safety margin, not because they panic. Reworded
to say that, since a confidently wrong comment is worse than the vague
one it replaced.
Also on the knob:
- GIT_SERVICE_TIMEOUT_SECS_MAX has two consumers now, so its own doc says
so, and notes this knob derives only the Instant addition and not the
lease-steal multiply.
- The "clears every disable setting" phrasing overclaimed: u32::MAX sits
above the ceiling. Now names the documented sentinels instead.
Operator-facing docs: the sibling knob's ceiling commit updated both
README.md and .env.example in the same change, and this one had not. Both
now carry the accepted range, matching that precedent.
House style: five comment lines added by the preceding commits used em
dashes, which this project does not allow. Four were carried over from the
text being edited and one was copied from the sibling test. All replaced.
Checked mechanically over the added lines rather than by eye.
Two comments restated an argument in full immediately after pointing at
the canonical site for it. Trimmed to the pointer plus the local delta.
Scoping correction worth recording: the two preceding commits claimed "no
behavior change". That holds for repos.rs, state.rs, main.rs and
test_support.rs, but a config.rs field doc IS clap --help output, so the
pin-knob doc edit changes operator-visible help text. No runtime behavior
changes either way.
Verification: full suite 733 passed, fmt clean, clippy clean under the CI
form. The range guard was separately proven load-bearing in both
directions by mutation: reverting to range(1..) goes RED on the rejection
assertions, and tightening the ceiling to 500_000_000 goes RED on the
disable-value assertions.
Known residual, not fixed here: the bound is parse-side only, so a Config
built by direct construction rather than through clap can still hold an
unrepresentable value.
The three handler peeks said they shed "before any DB/disk work", which
reads as an admission bound on the DB window. It is not one. The peek
holds no permit, and permits are only held from the authoritative acquire
below, after the visibility and rate gates, so a burst arriving while
permits are free all proceeds into the DB and none of it sheds. What the
peek actually gives is a cheap 503 for one request once the pool has
ALREADY filled, sparing that request's DB work.
The gate ordering that makes it a peek is deliberate and stays: a denied
or rate-limited request must consume no slot, and one source must not
hold global slots through the DB/visibility window. Reordering to bound
the window would undo that. So this corrects the claims and changes no
behavior; bounding the DB window would need an admission mechanism the
peek is not.
test_support.rs's receive-pack docstring additionally called the peek
"the permit" and "the first statement". Both were wrong at this head: it
is a non-holding peek, and the authoritative write-pool acquire sits
after the per-repo lease. Corrected along with the shed claim. The
"remove the block and this goes red" sentences stay, since those tests
drive an already-saturated pool, which is the case the peek does handle.
Sweep over "before any DB", "before DB", "pre-DB". Left alone with
reason: api/ipfs.rs:142 describes a HELD walk-admission acquire, so its
claim is true; graphql/mutation.rs:201 and test_support.rs:528 are about
auth ordering, unrelated; the repos.rs test docstrings at 4194/4213/4556/
4618/4682 describe test mechanics against a saturated pool, which is the
case the peek delivers.
No behavior change: the diff is comments and docstrings only, checked by
filtering the diff for non-comment lines.
The pin pool's docs claimed it bounded "how many MB-scale lists are held
at once across all repos". It does not. On the local IPFS path the caller
materializes the object list before pin_new_objects_gated acquires, so a
task parked waiting for a permit is still holding its full list, and how
many tasks park is capped only per repo by EncryptInflight. Across
distinct repos that retained memory is unbounded, which is the gap a
reviewer raised against this claim.
Correct the claim at each site rather than implying the pool closes an
exposure it cannot see. The pool does bound concurrent pin loops, and
therefore lists held while being pinned; that part was always true and is
kept. The residual is now named where each claim lives, including on the
config knob, so an operator does not read the knob as a memory cap it
cannot deliver.
Bounding the retention is a real change to the capture shape (the Pinata
twin already avoids it by acquiring before it derives) and is not
attempted here.
Sites corrected: pin_new_objects_gated's docstring and its F2b empty-list
comment, the empty-list test's docstring, AppState::pin_semaphore and
AppState::encrypt_inflight field docs, the max_concurrent_pin_tasks knob
doc, and the construction comment in main.
Sweep over pin_semaphore, F6, and "held at once". Left alone with reason:
ipfs_pin.rs 16/31/174-200/518 describe the loop that genuinely holds the
permit, so their claims are about hold duration and are true; repos.rs
1029-1036 and 2276-2294 describe the Pinata path, which acquires before
materializing, so its "at most pin_semaphore permits' worth exist at once"
holds; repos.rs 5898 claims concurrent loops are bounded, which is the
true concurrency claim; the F6 hits in api/ipfs.rs are that file's own
review numbering for walk-permit ordering, unrelated to this pool;
constructions and test fixtures carry no claim.
No behavior change: the diff is comments and docstrings only, checked by
filtering the diff for non-comment lines.
ipfs_request_budget_secs parsed range(1..), so every positive u64 was
accepted, but get_by_cid derives the request deadline as
Instant::now() + Duration::from_secs(that). That addition is an explicit
overflow check rather than a debug-only one, so an oversized value aborts
every /ipfs/{cid} request in a release build instead of setting a very
long budget. The route is anon-reachable, so an operator typo is felt
publicly.
Reuse GIT_SERVICE_TIMEOUT_SECS_MAX, the ceiling the sibling knob already
carries: the bound is representability, not a policy view of a sane
budget, so the large "set it very large to disable" values operators were
told to use still parse. Reject rather than clamp, matching the other
bounds in this config.
Execution evidence:
- RED before the change: the new test failed with "3153600001 is past the
representable ceiling and must be rejected at parse time" while its
disable-value assertions passed, so the test is not vacuous in either
direction.
- GREEN after: config::tests 13 passed, including the sibling
git_service_timeout regression test.
Sweep for the same defect class, re-run with a widened shape because the
original grep missed a site. Passes: Instant::now() sites; from_secs()
fed by config fields; config fields reaching a Duration through an
arithmetic expression; config fields reaching a tokio timer constructor.
The arithmetic and timer passes each return exactly one hit,
heartbeat_interval_hours (config.rs 150, main.rs 916, operator.rs 161),
which has no range at all and multiplies by 3600 before the Duration.
That knob is a separate defect on a path this branch does not touch, so
it is recorded for a follow-up rather than widened into here. Every other
Instant addition derives from the already-bounded git_service_timeout_secs
or from a constant.
A second-model pass on the previous two commits caught an upgrade footgun in the
first and stale prose in the second.
The 365-day ceiling was tighter than the defect required. The help text has
always told operators to set this very large to disable the bound, and clap
accepted every positive u64, so values like 999999999 (~31 years) are working
production "off" settings that neither overflow `* 2 + 60` nor exceed an Instant
deadline. A node running one of those would have failed to start after this
upgrade, over a value that was never the defect. The ceiling moves to 100 years,
which clears every such setting and stays an order of magnitude under the
~584-year limit of a u64-nanosecond Instant, the tightest representation on any
platform we build for. Every value that worked before still parses; only the
ones that would have panicked are rejected. The test now pins the pre-existing
disable values alongside the rejections so the distinction cannot quietly
regress.
Clamp-and-warn was the suggested remedy and is not what landed: it would keep
accepting a value the node cannot honor and silently substitute a different one,
where the range says what is representable and fails fast, matching how every
other bound in this file behaves.
The lease-bound comment and its test still described the pre-ceiling world. Both
now say what is true, and the test deliberately keeps u64::MAX rather than moving
to the ceiling as suggested: Config is reachable by direct construction, and at
the ceiling the arithmetic does not overflow, so a test pinned there would pass
with the saturation removed and prove nothing. Re-confirmed load-bearing after
the rewording.
Tracing the lease-bound overflow to its other consumers turned up a worse
instance of the same defect. This PR routed git_service_timeout_secs into
build_filtered_pack and blob_paths, and both build a deadline as
`Instant::now() + Duration::from_secs(..)`. That addition panics on overflow in
RELEASE as well as debug, so a large-but-accepted value crashes the serve path
rather than merely disabling a timeout.
Hardening each arithmetic site as it is found would leave the next one open, so
the bound goes where the value enters: clap now rejects anything above 365 days
at parse time, which keeps every derived duration in range at once and fails
fast at boot instead of at the first path-scoped clone. A year is many orders of
magnitude above any real clone or push, so it remains the practical way to
disable the bound.
The test pins both derivations at the maximum and rejects the value one past it
and u64::MAX. RED when the upper bound is dropped from the value_parser.
GITLAWB_GIT_SERVICE_TIMEOUT_SECS accepts every positive u64, so deriving the
per-repo write-lease steal bound as `* 2 + 60` panics every push in a debug
build across the top of the accepted range, and in release wraps to a short
Duration that would let a waiter steal a live push's lease.
Saturating keeps the knob's documented meaning: a timeout that large means the
service bound is effectively off, so the steal bound should be effectively
infinite rather than tiny.
The regression drives git_receive_pack rather than the arithmetic, so the
call-site wiring is what is under test. RED before at repos.rs:1733 with
"attempt to multiply with overflow", GREEN after; re-confirmed load-bearing by
reinstating the unchecked expression.
Review finding from jatmn, round nine. RepoWriteGuard::release closes the
connection when pg_advisory_unlock returns Err (F3b): the await resolved, so
the session is alive and still holds the lock, and returning that connection
to the pool hands the next caller a lock nobody tracks. The Drop backstop
carried the same hazard and only logged.
Both arms are fixed.
The spawned detached unlock now closes the connection through
close_conn_bounded on the error path, instead of letting the async block end
and return it to the pool.
The off-runtime arm was worse than the finding described. It dropped an
already-taken connection with no unlock attempted at all, and this turned out
to panic rather than leak: PoolConnection::drop calls rt::spawn for its
return-to-pool task, whose no-runtime fallback panics with "this
functionality requires a Tokio context". So any off-runtime guard drop
panicked in a destructor today. It now detaches, which gives up the pool slot
and yields a plain PgConnection whose drop closes the socket, ending the
session that holds the lock. Drop cannot await, so detach is the whole
disposal available here. Confirmed against the vendored sqlx 0.8.6 source
rather than assumed, and min_connections is 0 everywhere in this repo.
Three tests, modeled on the release F3b cases, on a pool with the idle reaper
disabled so a reaped session cannot pass the assertion for us. Observed RED
first:
write_guard_dropped_off_runtime_disposes_the_connection ... FAILED
panicked at sqlx-core-0.8.6/src/pool/connection.rs:208:
this functionality requires a Tokio context
then: dropping a write guard off a Tokio runtime must not panic
write_guard_drop_with_failing_unlock_does_not_return_the_connection ... FAILED
timed out waiting for the connection whose detached unlock errored to be
closed rather than returned to the pool still holding the session lock
write_guard_drop_with_successful_unlock_keeps_the_connection ... ok
The success case is green on both sides on purpose: it is the must-not case,
so it only ever catches a widening of disposal onto the healthy path.
GREEN after, and all 12 guard tests pass, so the release coverage did not
regress. Both new guards mutation-proved load-bearing, each red on its own
message, 2/2.
Full suite 729 passed, 0 failed. fmt and clippy clean.
Both new comments said saturating the pool takes ~8 distinct source IPs,
each also rate-limited. Neither half holds for an IPv6 caller with a routed
/64: the sub-cap and the rate limiter both key on rate_limit::client_key,
which with TrustedProxy::None falls through to the full peer address with no
prefix folding, so one caller has 2^64 keys. max_keys does not compensate,
since keys are freed on permit drop and live keys never approach the 100,000
default.
The comments now say what the cap actually bounds, concurrent slots per
resolved client key, and name the residual.
The keying fix stays deferred as a design call, and that deferral is
defensible: full-IP keying predates this PR, so these caps help IPv4 and
single-address callers without regressing anything. Landing a comment that
overstates the guarantee is not defensible, in a round that corrects a
weaker overclaim one screen away.
The per_source_push_cap docstring added earlier in this round repeated the
same claim, so it is corrected here rather than shipping fresh.
Comments only, no behaviour change. check, fmt clean.
The pinata_object_list_for_refs docstring said a visibility tightening that
lands after tail-start is covered by the reconciliation sweep. No such sweep
exists; state.rs says so directly, that a dropped job would be lost forever
precisely because there is none.
The claim is dropped and nothing replaces it. An earlier draft wanted to
substitute a next-push claim, which no site establishes either, and swapping
one unbacked claim for another is the same defect.
Four pre-existing sites make the same claim (repos.rs:30, push_delta.rs at 8,
189 and 281 with warns at 349 and 358). They predate this PR and are left for
a follow-up rather than widening a nine-round diff.
advert_per_caller_cap_sizing_is_never_zero opened with a local closure
re-implementing (pushes / 8).max(1), so it bound nothing. Production derives
that expression twice, and the second site was added after the test was
written. per_source_push_cap now lives in rate_limit.rs next to
PerCallerConcurrency, the type the value feeds, and both main.rs sites and
the test call it.
The finding was confirmed by execution before the fix. Deleting .max(1) from
both production sites with the closure still in place:
injected (2 sites), re-running ... still green
VACUOUS the guard's own defect was injected and nothing failed
0/1 load-bearing
And after, deleting .max(1) from the helper:
injected (1 site), re-running ... red, on the expected message
LOAD-BEARING matched 'advert cap must be >= 1 for pushes=1'
1/1 load-bearing
The first attempt at that second proof came back RED-WRONG-REASON against
'minimum write pool must derive cap 1, not 0': the loop's assertion for
pushes=1 fires first and aborts before that line is reached. Same property,
earlier assertion.
No mutation residue. Test green, fmt and clippy clean.
separator_prevents_the_owner_name_boundary_collision asserted that
("a","b_c") and ("a_b","c") produce different keys. Those differ under any
join at all, bare included (ab_c against a_bc), so deleting the separator
from repo_identity_key left the test green. It could not fail for the reason
it named.
The inputs are now ("a","bc") against ("ab","c"), both abc under a bare
join. The test docstring said the separator was \0; the code joins with /.
Corrected. The docstring on repo_identity_key itself carried the same wrong
collision example, so it is corrected too.
Proven by mutation rather than argued, joining with no separator:
old inputs, injected, re-running ... still green
VACUOUS the guard's own defect was injected and nothing failed
0/1 load-bearing
new inputs, injected, re-running ... red, on the expected message
LOAD-BEARING matched 'owner/name boundary must not be ambiguous'
1/1 load-bearing
Separator restored, no mutation residue. All 6 repo_identity_key_tests pass.
The walk's deadline clamp was computed on the async side, before
spawn_blocking was called, then used inside the closure. Time spent queued
for a blocking thread went uncharged, so the walk ran with a budget larger
than the request's true remainder. The clamp is now derived inside the
closure at task start.
This is the sibling of the upload-pack site fixed in 28a6ca4, and unlike
that one it sits on the anonymously reachable route.
A queue delay that eats the whole remainder saturates the clamp to zero and
the bounded walk fails closed through the existing taint arm, no verdict,
which is the safe direction.
Testing gap, recorded at the site rather than papered over: the queue-delay
path is reasoned, not executed. Observing it needs a runtime with the
blocking pool pinned and parked, and the sqlx::test harness gives no seam
for that. 28a6ca4 shipped with the same gap for the same reason, verified
against its diff and message rather than assumed. The tests in this file
saturate the admission semaphores, never the blocking pool, so there was
nothing to reuse. An assertion on the arithmetic would only restate the
implementation.
cargo test ipfs::: 24 passed, 0 failed.
pin_new_objects runs under a pin_semaphore permit and that pool defers rather
than sheds. Both sinks in ipfs_pin.rs built a bare reqwest::Client::new(),
which carries no timeout, so a silent endpoint parked the pool indefinitely.
Both now use the shared no-redirect client from build_http_client() through a
module-local OnceLock, rather than the hand-rolled twin its docstring forbids.
The client alone does not bound the hold: without a ceiling the loop is O(N)
with N chosen by the pusher. A 120s batch budget is taken once at loop start
and gated at the top of every iteration, so no object's work begins with zero
budget left. The remainder is also passed to each add as a per-request
timeout, which is what lets one large healthy upload run past the client's 10s
default without letting the batch run forever. RequestBuilder::timeout
replaces the client value for that request only and leaks nothing to other
callers, verified by execution against the pinned reqwest.
An earlier version of this commit classified a dead endpoint by
reqwest::Error::is_timeout() || is_connect() and broke the batch. That was
wrong in both directions and is gone. It missed the ordinary dead-peer shapes,
since a peer closing mid-request yields IncompleteMessage and a peer RST
mid-upload yields ConnectionReset, both observed with is_timeout and is_connect
false, so a load-balancer idle timeout evaded it entirely. And it fired on a
slow but alive endpoint, because .timeout() is a total deadline covering body
upload, so a healthy multi-megabyte add was read as a dead endpoint and
abandoned the rest of the push. A wall-clock bound needs neither judgement.
The docstring now states what is bounded and what is not, rather than implying
the hold is closed. Not bounded: the git read, since store::read_object shells
two bare Command::output() calls with no timeout or process-group reaping and
this loop does not run it under spawn_blocking; the DB round-trips; and the
pool itself, since the Pinata replication task holds the same semaphore across
a full git re-derivation with no deadline. Truncation is stated plainly: a
batch stopped at the deadline leaves the rest unpinned and nothing sweeps them
up, so recovery is opportunistic through a later full-scan push.
Tests written first, both RED by assertion rather than by a harness timeout:
pin_new_objects_stops_the_batch_at_its_deadline ... FAILED
the batch must stop partway, not pin all five and not stall on the
first: pinned 5
pin_new_objects_does_not_abandon_the_batch_on_a_slow_but_alive_endpoint ... FAILED
a slow but progressing endpoint must pin both objects: an upload past
the client's 10s default is not a dead endpoint
left: 1, right: 2
Both guards mutation-proved load-bearing against present-but-wrong mutants,
each red on its own message: the loop-top break widened to continue, and the
per-request timeout replaced with None.
Also bounds encrypted_pin.rs's pin_git_object call, where a hang wedged a
repo's EncryptInflight key. Its Drop backstop covers unwind, not hang.
Full suite 730 passed, 0 failed. fmt and clippy clean.
icaptcha-client became a workspace member in #138, after #130 last wrote
release-please-config.json, and nothing updated extra-files. release-please
rewrites a version line only when the line carries the annotation AND the
manifest is listed, so the crate was skipped every release and sat at 0.4.0
while the other five walked to 0.7.1.
Nothing broke: every intra-workspace path dep is bare, so no version
requirement could go unsatisfied; no crate is published to a registry; and
the crate reads no CARGO_PKG_VERSION of its own. The cost was the workspace
Cargo.toml comment asserting a lockstep rule the config did not keep, which
is what the next person adding a crate would read and copy.
Add the missing entry, and gate the rule so it cannot silently lapse again.
The gate derives its required set from `cargo metadata`, so a new crate reds
CI whether or not its author knows the gate exists, and it covers all three
release-silent shapes: a member absent from extra-files, a member whose
version line lost the annotation, and an extra-files path that is no longer
a member.
Next release moves icaptcha-client 0.4.0 to the current workspace version in
one step. Nothing depends on that version, so the jump is cosmetic.
The absent-CID path disambiguates a clean `missing` by re-running the probe,
but the re-probe took the same deadline with no check that any budget was left.
A first probe that nearly exhausted the budget still spawned a second child
that could only be reaped, and the watchdog's SIGTERM grace plus SIGKILL settle
carried the whole call to roughly twice the budget. Measured on
object_type_bounded with a 1s budget and a probe that burns 0.9s of it: two
spawns and 2021ms elapsed.
That is the same ~2x occupancy shape this PR exists to remove, on a different
handler, and it is worse here than on the served-git path: /ipfs/{cid} carries
only optional_signature, and an absent CID drives this branch once per repo, so
an unauthenticated caller spraying well-formed absent CIDs pays the overshoot
per repo. It also degraded an honest absence into a retryable 503.
Re-probe only when the remaining budget covers what the first probe actually
took, since the re-probe runs the same command. Below that, taint to Transient
rather than spawn: an inconclusive disambiguation is not an absence verdict, so
it must not become a false 404 either. The same 1s case is now one spawn and
finishes inside the budget.
Both introduced by this PR (d39fc8d, the F5 disambiguation); neither
object_type_bounded nor the re-probe exists at the merge base, so this is not a
pre-existing condition. The companion test pins that an ample budget still
re-probes, so the check cannot silently disable the disambiguation it guards.
The shared deadline computed the walk's budget on the async side, before the
task was queued on the blocking pool. withheld_blob_oids_bounded starts its own
clock when it actually runs, so a budget measured at queue time handed the walk
a full budget from whenever the pool got to it: the read permit was held for
queue delay PLUS the budget, and only the serve phase saw the outer deadline
already spent.
Computing the remainder inside the closure charges that delay against the shared
deadline, which is what makes the ~1x bound in the docs true rather than
approximate. Not a regression from the previous commit (the old code passed a
constant git_timeout, so the delay went uncharged there too, and the combined
hold was queue delay plus two full budgets), but the claim this PR now prints in
README and .env.example should be exact.
The two shared-deadline tests still pass and still fail on revert; they cover
the budget arithmetic, not the queue-saturation path. Reproducing that needs a
runtime with the blocking pool pinned and parked, which the sqlx::test harness
does not give a seam for, so the queue-delay path is reasoned, not executed.
Two table rows needed correcting for the changes in this PR: the
GITLAWB_GIT_SERVICE_TIMEOUT_SECS row now says the classification walk and the
pack serve share one deadline on the path-scoped upload-pack path, so the value
bounds them combined; and the GITLAWB_IPFS_REQUEST_BUDGET_SECS row no longer
claims the probe and content-read subprocesses are unbounded, while keeping the
object-store-readability sweep named as a live residual.
The other thirteen changed lines are punctuation only, no wording changes: em
dashes replaced with colons, commas, or sentence breaks, and one pair of curly
quotes straightened. The repo's prose-style hook validates a whole file rather
than a diff, so these pre-existing characters were rejecting every edit to
README.md, including the two rows above.
Four prose corrections on this PR's surfaces, no behavior change.
The receive-pack advertisement no longer draws from the write pool, it draws
from a dedicated advert pool, but five comments still described the old
shape: the info/refs per-source block, the cap construction in main.rs, the
State field doc, and a test doc plus its assertion message. The state.rs and
main.rs ones mattered most because they contradicted the operator text being
written in this same commit.
The /ipfs probe and content-read subprocesses each run under their own
deadline now (the lesser of git_service_timeout_secs and the remaining
budget, reaped by process-group teardown), but four surfaces still claimed
they had no duration bound past the pre-start budget check. The correction
stays scoped rather than declaring the route safe: the probe's
object_store_readable check is a synchronous filesystem sweep with nothing to
reap, so a wedged filesystem can still hold the walk slot past the deadline,
and that residual is now stated where the old warning used to be. A grep for
the full phrase misses two of these surfaces, since one wraps across doc-comment
lines and one says "duration clamp" instead, so the sweep uses the fragment.
Both push-side per-source concurrency caps are derived as
max_concurrent_git_pushes / 8 with a floor of 1 and have no environment
variable, so neither was documented anywhere an operator reads. The write-side
one is load-bearing for push availability: it is acquired before the global
write permit, and owner enforcement defaults off, so without it one host
minting disposable did:key identities could monopolize the write pool. They
are described in .env.example beside the read cap rather than in README's
table, which is keyed by variable name.
The hung-walk assertion in visibility_pack printed {err}, showing only the top
anyhow context and dropping the underlying io error. A beta-lane failure
surfaced as "failed to spawn git for-each-ref" with no errno, which made it
undiagnosable. {err:#} prints the chain. The underlying flake is still
undiagnosed: it did not reproduce in 123 local runs and the stable lane passed
on the same commit, so this only improves the next occurrence.
The path-scoped git-upload-pack branch spent git_service_timeout_secs twice:
once on the withheld-blob classification walk, then a second full budget on
the serve. One clone could hold a read-pool permit for ~2x the configured
budget, which is the occupancy hole this PR otherwise exists to close.
Thread one Instant deadline through both phases, mirroring
fail_closed_full_scan_objects and build_filtered_pack: the walk runs against
the remainder and the serve gets what the walk left, computed after the
blocking task's await so it cannot pick up a fresh budget. A walk that
consumes the whole budget saturates the serve to zero, which surfaces as
GitServiceTimeout -> 504 rather than running unbounded. Under-serving one
slow clone is the safe direction versus over-holding the pool.
Both serve arms are covered because a fix that reached only the plain arm
would leave the ~2x hold reachable by any clone of a repo that actually
withholds a blob. Each test was observed RED before the change (status 200,
the serve completing on a fresh budget) and GREEN after, and reverting the
filtered arm alone flips only its own test, so neither guard is a proxy for
the other.
The knob's meaning changed for this path, so .env.example now says the value
bounds the two phases combined rather than each separately.
Spawning the replication tail above guard.release() also moves it above the
Tigris upload, so a ref can be announced while the shared durable copy is still
the old one, and a disconnect there cancels the upload while the detached tail
still pins and announces.
Accepted rather than fixed. Upload-then-announce was never guaranteed: release
tolerates a failed upload by design, warning and continuing to the unlock, so an
announce over a stale Tigris copy was already reachable before this reorder, and
it self-heals through acquire_fresh's local fallback plus the next push. The
alternative, detaching release and the tail together, would return 200 before
the durable copy lands, which changes the client contract more than the window
it closes.
A handler-level "the next acquire_write succeeds" check does not discriminate:
it still passes with the RepoWriteGuard::Drop backstop disabled. Measured with
the backstop off, the lock is held at the drop and reads free about two seconds
later from a session outside the pool. #[sqlx::test] builds its pool with
idle_timeout(1s), so the guard's connection is reaped once the handler future
drops it, the session ends, and postgres frees the lock with no help from the
code under test. acquire_write retries for far longer than that window, so it
waits the release out and reports success either way.
write_guard_release_cancelled_mid_unlock_frees_the_lock is the real proof: it
probes from a connection held out of the pool, 400ms after the drop, inside that
window rather than past it.
The post-receive tail was spawned after `guard.release(...)`, which on a
successful push awaits the Tigris upload and then the advisory unlock while
the handler future is still tied to the client connection. The pack has
already landed on disk by then, so a disconnect in that window left a durable
push whose pins, recovery copy and announce were never spawned: the dropped
tail F2 closed, one step earlier in the handler.
Spawn it above `release`, gated on an explicit `receive_result.is_ok()`. The
`?` below the spawn can no longer be what keeps a rejected push from pinning
and announcing a half-applied repo, so the success check is now a named flag
that `release` consumes too. The tail is read-only on the repo directory and
holds neither the lease nor the advisory lock, so overlapping it with the
upload waits on nothing the handler still owns.
Tests: a handler-level disconnect driven through a parked `release` (new
store-level test seam), plus the must-not direction, a failed receive-pack
that spawns no tail. The U5 source gate now binds the new boundary: spawn
inside `if push_succeeded`, above `guard.release(push_succeeded)`.
The key is logged as the `repo` field on the lease waiter-cap shed and the
steal-bound warning (state.rs), and it was joined with a NUL. Those are the two
messages an operator reads to find a contended repo, and a NUL-hostile log sink
truncates the field at the separator, rendering two different repos' warnings
identically. Before this PR that field was a UUID or a mirror id, both printable,
so the NUL was a regression this change introduced.
Join with '/' instead. It cannot occur in the owner slug by construction - the
slug is owner_did.replace([':','/'], "_") - and it mirrors the shape of the disk
path the key exists to reproduce. A test asserts the key carries no control
characters so this cannot regress silently.
Rotating the U2 lease test's row to a UUID id is not incidental: the mirror path
that seeds it uses 'owner_short/name' as the row id, which now coincides with the
identity key and would leave the test unable to tell the two keys apart. The
rotated id is also the real shape of the bug. Re-verified RED against the reverted
call site after the change.
The post-receive replication tail was spawned at the end of git_receive_pack, after
touch_repo, the trust-score update, per-ref certificate issuance and webhook fan-out.
A client or proxy disconnect anywhere in that window dropped the tail, and with it
this push's pins, recovery copy, and announcements - even though the push was already
durable on disk.
Spawn it immediately after the receive_result ? instead. That is the first point where
the push is known to have landed: guard.release() a few lines up runs on failure too,
so anchoring there would spawn a tail for a push git rejected, pinning and announcing
a half-applied repo - the state release() deliberately refuses to upload to Tigris,
and one a hostile pusher can produce on demand by aborting a pack mid-transfer.
RefUpdate gains Clone so the tail takes the parsed updates while the certificate and
webhook loops keep iterating their own copy.
This makes the tail run concurrently with certificate issuance rather than after it,
so a ref can be announced before its signed certificate exists. Accepted, and stated
here rather than left for a reviewer to find: cert issuance already fails open (errors
are logged and skipped) and the gossip event carries cert_id: None regardless, so no
announce consumer reads a certificate out of it.
Evidence is an ordering gate in inv22_gates.rs rather than a cancellation-race test,
matching the F3 gate's instrument: forcing a disconnect to land inside the cert/webhook
window is timing-dependent and would be flaky, while the ordering is precisely what
changed. The gate is mutation-verified in both directions - moving the spawn back below
the webhooks fails the upper bound, hoisting it above the ? fails the lower one.
Two single-site wiring fixes from the same review round.
withheld_recipients_gated held its git_encrypt_semaphore permit in the async frame
and then awaited spawn_blocking, so dropping the future released the permit while the
uncancellable walk kept running. Every other post-receive scan helper moves the permit
into the closure - replication_withheld_set says so in as many words - so this aligns
the last site with the F4 contract. The regression drops the future mid-walk and
asserts the pool stays at zero while the git child is still alive.
git_info_refs passed a literal "git" to smart_http::info_refs while upload-pack and
receive-pack both pass state.git_bin, so no fake-git harness could drive the
advertisement path. The regression points git_bin at a fake emitting a distinctive
advertisement and asserts it reaches the response body; before the fix the body came
back from real system git (agent=git/2.43.0).
The global and per-source walk permits were handler locals, so a client disconnect
released them the moment the future dropped while the uncancellable spawn_blocking
probe/walk/read kept running against a slot the pool now considered free. A disconnect
flood could admit replacements while prior blocking git children still occupied
threads and PIDs, weakening GITLAWB_MAX_CONCURRENT_IPFS_WALKS and
GITLAWB_IPFS_WALK_PER_SOURCE on a surface any unauthenticated caller can reach.
Hold the pair in an Arc and clone it into each of the three blocking closures. The
permits release when the last clone drops, which collapses both failure directions
into one case: a disconnect leaves the abandoned closure holding the slot, and a
panicking closure leaves the handler holding it. No loop arm changes and the
JoinError arms keep today's taint-and-continue.
A move-and-return shuttle was implemented and rejected. It is correct only if every
arm that continues the loop re-binds the returned value, which the compiler cannot
enforce - the absent-object arm Ok(Ok(None)) => continue, taken on nearly every
iteration of a random-CID scan, drops it - and it forces a second decision about the
panic arms whose least-bad answer would let one broken repo shed every other owner's
/ipfs lookups.
The regression strengthens the existing admission test: it now asserts the slot is
still held after the future is dropped and before the child is killed, with a
child-still-alive precondition so it cannot pass vacuously. Only the walk site runs
under state.git_bin and can be pinned that way; the probe and read deliberately shell
to real git, so a source-scan gate in inv22_gates.rs binds all three clone sites
instead. Removing any one clone turns that gate red.
repo_write_leases and encrypt_inflight keyed on record.id, but RepoStore::local_path
and the pg advisory lock key on the sanitized owner slug plus repo name, and the bare
repo on disk is reused when a repo is deleted and recreated under the same slug. The
row id rotates across that recreate, so both in-process serializers stopped serializing
exactly where the shared objects/ directory still needed them: a disconnected push's
reaper could still be tearing down while a new push took a different key and proceeded.
Add repo_identity_key(owner_did, repo_name) and use it at both sites. Three details are
load-bearing and carry tests: the sanitization is repo_disk_path's replace([':','/'],"_")
rather than normalize_owner_key, which folds a did🔑 prefix and would disagree with
the disk path; the separator is a NUL, without which owner 'a' + name 'b_c' and owner
'a_b' + name 'c' collide; and both call sites pass record.owner_did / record.name rather
than the request path segments, since get_repo normalizes DID aliases and the raw
segments would mint two keys for one directory.
The sanitization is deliberately not injective. Two did:web spellings can fold to one
slug, which would misroute encrypt coalescing if it were reachable, but repos.disk_path
is NOT NULL UNIQUE and holds exactly this derivation, so two such rows cannot coexist.
A test pins that fold in place: making the key injective is what would let two owners
sharing one objects/ directory push concurrently.
resolve_drain_object_list already re-fetches the repo row by owner/name and
deliberately reads visibility rules under record.id, because a delete+recreate
under the same slug gives the row a new id while the bare repo on disk is
reused. The encrypt write a few lines later still took ctx.repo_id, frozen when
the task was spawned, so recovery-pin metadata for a coalesced drain landed
against the deleted row and was unreachable from the live one.
Return the re-fetched id from the resolver and pass it explicitly to
pin_and_encrypt_objects. The parameter is explicit rather than read from ctx so
the two callers stay honest about which id they mean: the snapshot iteration
passes its own spawn-time id, the drain passes the fresh one.
The Arweave anchor is unaffected - EncryptedManifest carries no repo id and
builds its slug from owner and name, both stable across the rotation.
The 300s cleanup task swept the DID, create-IP, push, sync-trigger and
peer-write limiters but not ipfs_rate_limiter, so its expired keys sat
until the map hit max_keys and the inline capacity sweep reclaimed them.
Drive the sweep off AppState in a named sweep_rate_limiters so a limiter
added to the state and forgotten here fails a test instead of leaking
keys, and cover all six with a regression that goes red when any one is
dropped from the sweep.
GITLAWB_REPO_LEASE_MAX_WAITERS was also missing from .env.example and the
README settings table, leaving operators without a way to discover or
tune the cap behind the "repo is busy with another push" 503.
Three findings from the review round.
Moving the Pinata gate to the rules-only predicate was necessary, because a
coalesced push has no walk of its own to consult. But it also meant a push whose
OWN walk failed now acquired a global pin permit and ran a second full
re-derivation that failed the same way. Measured on a repo whose rev-list always
fails: twelve git children and three walk attempts, against four and one before.
With eight pin slots that defer rather than shed, eight such pushes stall pins for
every repo on the node. The gate now excludes a non-coalesced push whose own walk
failed, leaving the coalesced path on the rules-only predicate it needs, and the
pin helper returns before taking a permit for an empty object list.
The connection close on the failing-unlock path had no deadline, and release is
awaited inline while the write lease and both admission permits are still held.
The condition that produces that branch is a connection whose last statement
errored, and a blackholed socket to Postgres is one way to get there, so the await
could pin three admission resources for a TCP timeout. It now runs under a five
second bound; on expiry the future is dropped, which closes the socket and ends
the session anyway, which is what frees the lock. What is tested is the deadline.
That sqlx's close is the thing that stalls in production is reasoned from its
source, not reproduced, and the test says so.
The two tests for that path waited a fixed four hundred milliseconds before
asserting. Replacing the sleep with polling was not enough on its own: the test
pool sets a one second idle timeout, so a connection returned to the pool is
reaped shortly after and the session ends without any help from the code under
test. Both tests passed under mutation once the sleep was long enough to reach
that reaper, which means a straightforward flake fix would have quietly gutted
them. They now run against a pool with no idle reaper, so release is the only
thing that can end the session, and both mutations go red again.
Adds tokio's test-util as a dev-dependency for paused-time assertions. Production
dependencies are unchanged and the build stays --locked clean.
The previous attempt moved the per-source write permit above the lease park to
bound buffered pack bodies. Two reviewers demonstrated that it denies far more
than the source doing the parking. A source parked on one repo's lease was shed
on repos with no contention at all, and because the trusted-proxy setting
defaults to trusting nothing and keying on the socket peer, every pusher behind a
proxy, NAT, or CI runner pool resolves to one key: four parked pushes shed every
push from every pusher to every repo for up to twenty-one minutes. That is the
node-wide shed the permit was moved below the lease to prevent in the first
place, reintroduced at an eighth of the threshold.
The permit goes back below the lease. What is actually unbounded is the waiter
set, so that is what is bounded now, per repo, with a shed past the cap.
The cap counts live waiters, never the entry refcount. The refcount includes the
holder, so capping it would let a lease whose owner died without unwinding pin a
slot forever, which is the permanent wedge the steal timeout exists to survive.
The count is claimed in the same critical section as the refcount and held by a
guard scoped to the cancellable wait, so a cancelled or shed waiter gives it back.
An uncontended acquire takes the fast path and spends no waiter budget; that path
does not barge a queued waiter, which was checked over two thousand rounds rather
than assumed.
Two tests now guard the defect that was just removed: a source parked on one repo
is still served on another, and a parked push does not shed an unrelated pusher
sharing its resolved key. Restoring the old ordering turns both red. Capping on
the refcount instead of live waiters turns the steal-backstop test red. Holding
the waiter slot past the wait turns the fast-path test red.
What this bounds is the parked count, not the parked bytes. At a two gigabyte
pack limit the byte figure stays large, and the total across repos, whose
dominant term is the buffering the signature middleware does before any admission
point at all, is tracked in #263. Past the cap, same-repo concurrency sheds
instead of queueing; that is the trade the knob buys.
The two F1 tests asserted "the capped source shed rather than parked" by racing a
ten second timeout, and drove the holding push in ten millisecond slices. Under
`cargo test --workspace` that is a race against the scheduler rather than a test:
a reviewer saw two of five runs go red, and under load the pair stretched from
1.8s to 19.2s while the fake git holding the lease only lived for 10s, past which
every assertion in both tests is meaningless.
Shed versus park is a question of which path ran, so both are now read as state.
A shed is `Err(Overloaded)`. A park is a second reference on the lease entry,
exposed by a test-only `refs_for`; `acquire` takes that reference synchronously
before the cancellable wait, so a waiter is visible the instant it reaches the
lease, and it stays visible because the steal bound is 1260s. The tests spawn
their pushes and poll until one of the two mutually exclusive states appears, so
a starved machine iterates more rather than failing.
Every wall-clock assertion is gone. The worst of them was a sleep followed by
`!is_finished()`: a runtime that had not yet polled that task satisfied it
vacuously, which means the constant-key mutation could have gone green under
load, and the discriminator is the whole reason that test exists.
All three mutations still go red, now in seconds rather than by timeout expiry:
the per-caller acquire below the lease, the global permit above it, and the
per-caller key replaced with a constant.
Also fixes a third flake found on the way: the acquire-deadline test's follow-up
push inherited the deliberately short two second acquire timeout, which has to
cover a real advisory-lock round trip, so its `Overloaded` was indistinguishable
from the drained-pool `Overloaded` the assertion was actually testing for.
The sqlx `databases_pkey` collisions seen alongside this are machine residue, not
a leak from these tests: sqlx derives the test database name from a hash of the
test path and does cleanup-then-insert without a lock, so any two concurrent test
binaries running the same path collide. The orphan count here held at 53 across
every run.
The post-push tail resolved the withheld set and materialized the object list
before it ever consulted the per-repo coalescer, so rapid pushes to one repo each
parked on the encrypt semaphore and repeated the scan before discovering they
should have coalesced. The gate now runs first, keyed on the cheap
listable-at-root predicate that `replication_withheld_set` already computes from
rules alone before it touches the semaphore.
Three things made this less mechanical than it looks, all of them ways a naive
move would have shipped a regression.
The withheld set had a second consumer. It also gated the Pinata pin and the
branch-to-CID upsert, so a coalesced push that skipped the walk would have pinned
nothing and recorded no CID, while a test asserting merely that the announce
spawn ran would have stayed green. That gate moves to the announce predicate too.
Coalescing must not return from the tail. The announce spawn sits below the
coalescer in the same task, and returning early would drop gossip, the GraphQL
broadcast, Arweave anchoring and peer notify, which are per-push and
non-idempotent. A flag skips the encrypt section and falls through.
The publish paths are gated on announce and iterate the ref updates, not the
object list, so handing the tail the cheap predicate would have made a repo whose
walk is failing start gossiping and anchoring, losing the fail-closed property.
`pinata_object_list_for_refs` already recomputes the real set inside the pin
permit, so it now returns that verdict alongside the list and the tail uses it.
No additional walk is introduced.
Moving the gate earlier also opens a window where a push can coalesce while the
admitted one is still walking. If that walk then fails, the admitted task now
still spawns the drain with an empty snapshot so those tips are consumed, rather
than dropping the guard and discarding them. Note a released guard and a dropped
guard are indistinguishable by the in-flight map, since Drop clears the key
either way, so that case is asserted on drained work rather than on the map being
empty.
Each of the four is pinned by a mutation: walking anyway, returning early,
leaving the Pinata gate on the withheld set, and dropping instead of draining
each turn exactly one test red.
`release` discarded the unlock's result and set `released` regardless, so an
unlock that errored while its session stayed alive (a statement timeout, an admin
cancel) returned a connection to the pool still holding the repo's advisory lock,
with the Drop backstop already disarmed by that flag.
The error is now captured. On failure the connection is taken and closed, which
ends the session and makes Postgres drop the lock server-side, and only then is
the guard marked released, because the lock went with the session rather than
with the unlock. `close()` rather than `detach()` since we are already in an async
fn and can wait for the teardown deterministically.
Disposal is the only mechanism on this path. It cannot coexist with an armed Drop
backstop: disposal needs the connection taken, and Drop returns early when there
is none, while leaving the connection in place so Drop can use it just puts the
locked connection back in the pool when that spawned task ends. The Drop doc now
says so rather than claiming to cover a case it cannot.
The cancellation path is untouched. It was already correct, since `released` is
set after the await, and its test still passes unmodified.
The new tests assert against `pg_locks` from a connection held out of the pool
before the acquire, never by calling `acquire_write` a second time. That shortcut
would pass whether or not the fix is present, because the pool can hand back the
same connection and session advisory locks are re-entrant. The unlock error is
injected by leaving the pinned connection in an aborted transaction, which errors
the next statement while keeping the session and its lock alive, and the test
asserts that precondition before releasing.
Not covered: that the failure path still skips the Tigris upload. The test store
builds with no Tigris client, so the upload site is unobservable here without a
new seam, and that block is untouched by this change.
A push buffers its whole pack before the handler runs, then parks on the per-repo
write lease for up to git_service_timeout_secs * 2 + 60, which is 1260s at
defaults. Both admission permits were taken after that park, so nothing accounted
for waiters: any number of requests could sit on one contended repo, each holding
a fully buffered body, invisible to both semaphores.
The per-source sub-cap now runs before the park. It already exists, is sized at a
quarter of the push pool, and is keyed on the resolved source IP, so parked bodies
are bounded at four per source without a new knob or a new cap to tune.
The global permit deliberately stays below the lease. Moving it up is what
f02c5e1 fixed: a lease-blocked waiter holding a global slot sheds pushes to every
other repo node-wide. A per-source key does not have that shape, since exhausting
it denies only the source doing the exhausting, so the two permits belong on
opposite sides of the park and the comment there now says why.
The load-bearing test is not that a capped source sheds, which any cap satisfies,
including the global one this rejects. It is that a different source still parks
and is served while the first is at its cap. Keying the permit on a constant
leaves the shed test green and turns only that one red. Moving the global permit
above the lease turns the f02c5e1 guard red.
Bounded per source, not globally. A distributed set of sources can still each hold
their four, and the unbounded-total term is the pre-auth buffering in #263 rather
than the lease.
Two claims from earlier commits were argued rather than executed. Both are now
observed.
The legacy-CID sweep's spawn sat inline in main() where no test could reach it,
so the batch size, the inter-batch delay, the detachment and the shutdown select
were all reasoned. The block moves verbatim into `spawn_legacy_cid_sweep`, which
a test calls directly. It seeds rows whose keys are already canonical, so the
cost gate skips each and the persisted cursor reads out walk progress cleanly,
then asserts the cursor stops at the configured batch boundary, the call returns
without blocking, the task is still alive inside its delay, and shutdown ends it.
Never spawning, reading the batch or delay from the wrong knob, or removing the
shutdown branch each turns it red. What remains reasoned is the single call line
inside main(), down from a twenty-six line block.
The claim that a disconnect-interrupted push never uploads to Tigris was
structural: the tests build the store with no Tigris client, so a counter inside
the client arm could never move and would have proved nothing. The counter now
sits at the upload decision, before the client is consulted, and the disconnect
test asserts zero while the success test asserts exactly one. That positive
control is what makes the zero meaningful, and it is itself load-bearing:
releasing the success path with failure semantics drops it to zero, and adding a
Drop impl that uploads raises the disconnect count to one.
The counter observes reaching the upload site, not an S3 request. TigrisClient
wraps a concrete SDK client with no injectable endpoint, and plumbing one through
would be out of proportion to the risk here.
The lock-pool work gave `acquire_write` a distinct exhaustion error and mapped it
to 503 with Retry-After, but only on the push handler. The three other callers,
both issue writes and the PR merge, still turned it into a generic git error and
returned 500. Those are exactly the callers that hold no concurrency permit, so
they are the likeliest to meet an exhausted pool in the first place.
All three now go through the push handler's helper rather than repeating the
mapping, so the log level and the message cannot drift between the four sites.
Each has a test that occupies the single connection of a one-slot pool with a
guard on a different repo, so the failure is pool capacity rather than
advisory-lock contention, and asserts status 503 plus retry-after through
IntoResponse. Reverting the mapping at one site turns only that site's test red.
Worth knowing for review, and left alone deliberately: close_issue takes the
write guard before its author check, so this shed is reachable by any
authenticated caller whether or not they could close the issue. That ordering
predates this change and reordering the handler is not this commit's business,
but it does mean the 503 discloses that the repo's write pool is busy.
Four follow-ups from an adversarial pass over the previous commits.
The marker clear ran unconditionally inside `record_pin_source`'s transaction,
but its INSERT is `ON CONFLICT DO NOTHING` guarded by the cap, so it records
nothing when the repo is already a source. That is the common case, not a rare
one: the pin path calls `record_pin_source` for every already-pinned object, and
on a requeue pass that is the whole-repo enumeration. So a re-push from the repo
that was already recorded cleared a marker set by a different repo's failure and
the public copy went back to 404ing. The clear is now gated on the insert
actually adding a row. A genuine record from a third repo still clears the
marker, which needs a per-(oid, repo) table to close and is documented rather
than papered over.
The sweep advanced its cursor before every skip, which stops a wedge but made a
transient skip permanent: once the walk reached the end of the table the cursor
parked there and no later boot read another row. On a Tigris-backed node the
common skip is "repo is not on local disk", so the sweep would repair whatever
happened to be warm at boot and then never run again, and the new advertise
filter turned those rows from "advertised but 404" into "never advertised, never
repaired". Repairs now report whether a skip was retryable or terminal, and a run
that skipped anything retryable rewinds the cursor once, after the walk, so the
next run re-walks. A permanently unrepairable row is terminal and does not
trigger a rewind, so this cannot become a hot loop.
The sweep also reached its repo paths through the raw join helper, skipping the
three-layer traversal barrier `RepoStore::local_path` applies. A test with a repo
row named `../../escapee` showed it reading and repairing bytes from outside
`repos_dir`. That barrier is now extracted and shared, so the two callers cannot
drift. And the repair's synchronous `git cat-file` moved under `spawn_blocking`,
matching the existing treatment of the withheld walk, so a wedged read no longer
parks a worker thread for the full service timeout.
Finally the resolver's two source-set queries ran ahead of the work-budget peek
that exists to shed a spent caller before doing work. The peek now runs first. A
throttled caller whose set turns out complete gets 429 where it used to get 404,
which is the honest answer since its search never ran, and it removes an oracle
that let a throttled caller tell a complete source set from an incomplete one.
Three regressions from the previous four commits, each demonstrated before being
fixed.
The advisory-lock retry loop checked its connection out before spinning, so a
caller that lost the lock held a pool connection through all sixty one-second
attempts. The old pool-per-query shape returned it between iterations, so a
spinner occupied nothing. That matters more than the pool sizing suggested:
`acquire_write` has three non-push callers (two in issues, one in pulls) that
hold no concurrency permit, and the issue routes carry no rate limit, so any
self-minted DID could fire enough concurrent closes at one repo to strand the
pool and fail authenticated pushes on every repo. The checkout now happens per
attempt and a losing attempt returns the connection before sleeping, so only the
winner retains one. Lock-pool exhaustion also carries its own error now and sheds
503 with Retry-After instead of surfacing as a 500.
Holding the lock on one connection then meant a client disconnect released it
early: the handler future dies mid-receive-pack, the guard drops without
`release`, and `after_release` frees the lock while the detached reaper is still
giving the process group its SIGTERM grace. A second writer was admitted 90ms
after the drop, well inside that window, which is the invariant smart_http
already documents. The write guard now rides the AdmissionGuard into the reaper,
the same seam the admission permits already use, so the lock outlives the group.
The success path still reclaims the guard and releases it synchronously, and a
dropped guard performs no Tigris upload, so an interrupted push cannot publish a
half-applied repo.
The requeue loop's exit on an exhausted re-read discarded a push that coalesced
during the retry window: `requeue_or_release` had cleared the dirty bit without
marking the guard released, so `Drop` removed the key and no pass was ever
attempted. It now falls through with nothing to replicate and lets the tail's
atomic check-and-clear decide. A lap only happens when a push actually coalesced
and each pays a full bounded re-read, so a sustained outage still terminates,
which is tested with ten thousand injected faults under a watchdog rather than
argued.
Also clamps the lock pool's derived size, which previously followed
max_concurrent_git_pushes up to its million-connection ceiling, and documents the
node's total connection budget on that knob.
`repair_legacy_provider_cid` had exactly one trigger: the already-pinned skip
branch inside `pin_new_objects`, which a push only reaches when it re-carries the
object. Normal git negotiation omits objects the node already has, so on an
upgraded node that push generally never arrives. Meanwhile `list_pinned_cids`
kept advertising the stored provider CID that this branch's `/ipfs/{cid}`
deliberately withholds, so the node handed out CIDs it would then refuse to
serve, and previously pinned data stayed unresolvable.
Two halves, both needed. A detached background sweep walks `pinned_cids` in
bounded batches with an inter-batch delay, resolves each row's repo from its
recorded provenance, and reuses the existing repair (and its cost gate, so a row
already keyed on a raw CIDv1 reads no bytes). Migration v16 adds a single-row
cursor table so a restart mid-walk resumes rather than rewinding; the cursor
advances on every row read, before any skip, so an unrepairable row cannot wedge
the walk. A row whose bytes are gone, whose provenance is NULL, or whose repo is
not on local disk is left exactly as it is. And `list_pinned_cids` now omits rows
whose key is not a raw CIDv1, so the window before the sweep catches up
advertises nothing the resolver withholds.
The sweep resolves paths directly rather than through `repo_store.acquire`, which
would pull cold repos back from Tigris and turn a repair pass into a bulk
restore. Defaults are deliberately quiet: 64 rows per batch, 60s between batches.
Ten tests cover the repair, the bytes-gone and unrepairable arms, the cost gate,
the batch bound, cursor resumption, both states of one row through the advertise
filter, and the degenerate empty and no-legacy-rows tables. Removing the filter,
the batch bound, the cursor advance, the bytes-gone early return, or the
inter-batch sleep each turns a test red.
One pre-existing test used placeholder resolver keys that the new filter drops;
its values are now real raw CIDv1s with every assertion unchanged.
The resolver treated a non-empty, below-cap pin-source set as proof that every
source had been recorded, and skipped the bounded scan on that basis. But
`record_pin_source` is best effort at every call site: the ipfs_pin sites were
warn-only once their retries exhausted, and both pinata sites had no retry at
all. An object first pinned from a private repo and later pushed from a public
one whose source write failed keeps a set naming only the private source, so
`GET /ipfs/{cid}` returned 404 for an object the public repo would have served.
Migration v15 adds `pinned_cids.pin_sources_incomplete`, NOT NULL DEFAULT FALSE
so every existing row reads as complete. A record that fails outright sets it; a
later successful record clears it, inside the same transaction as the insert so
the two paths cannot drift. `needs_scan` now ORs the marker in, short-circuiting
after the existing empty and at-cap checks so it costs nothing on the serve path.
Pinata's two bare calls now share `retry_db_record` with ipfs_pin. The ipfs_pin
first-pin path additionally records the CID and its source in one transaction;
Pinata's remains two retried calls.
INV-10 is preserved: a complete set still tail-404s with no preload, asserted on
the existing preload counter. The fallback is not an authorization bypass, and
the test for that asserts the scan actually ran before checking the denial, so
the denial cannot pass vacuously. Dropping the marker term from the gate, forcing
it true, or removing the clear each turns a test red.
The marker is per-object rather than per-(object, repo): a successful record from
one repo clears a marker set by another's failure. That is the conservative
direction, since the marker only ever adds the fallback back.
`requeue_or_release` clears the dirty flag atomically with the decision to loop,
which is correct and stays as it is. The problem was what the pass did next: a
failed `get_repo_by_id` fell into an Err arm that yielded no rules, and
`list_visibility_rules(...).ok()` collapsed its own error the same way. With no
rules, `replication_withheld_set` returns `(false, None)`, the pass skipped its
work entirely, and the loop exited having consumed the dirty bit that stood for
the coalesced push. There is no reconciliation sweep to re-derive it, so that
push's objects were never pinned and its withheld blobs never sealed.
`requeue_refresh_state` now separates the three cases the old code collapsed.
`Ok(Some)` proceeds. `Ok(None)` means the repo really is gone and releases
without touching the retry budget. `Err` from either read is transient and is
retried with doubling backoff up to three attempts. A rules-read error is no
longer indistinguishable from a repo that has no rules.
Exhausting the retries still drops the pass, which is the pre-existing residual,
but it now logs at error level with the repo id and attempt count instead of
disappearing behind a warning. Six tests cover the retried-then-lands path, the
bounded-exhaustion path, repo-gone, the rules-read arm, freshness of the applied
rules, and the coalescing property that had to survive. Reverting the Err arm,
restoring the `.ok()`, raising the bound, or downgrading the error log each turns
a test red.
`acquire_write` took `pg_try_advisory_lock` through the shared pool, then awaited
the Tigris exists/download path before constructing `RepoWriteGuard`. Two bugs
fell out of that. The `tokio::time::timeout` this branch added around the call
can drop the future inside that window, after the lock is held and before the
only unlock (`RepoWriteGuard::release`) exists, wedging every later push to the
repo. And because a session advisory lock belongs to the connection that took
it, running the lock and the unlock through the pool lets them land on different
connections: the unlock returns false and leaks, while a competing acquire that
draws the holding connection re-enters the lock and two writers run against the
same repo concurrently.
`RepoStore` now owns a dedicated lock pool built with an `after_release` hook
that runs `pg_advisory_unlock_all()`. `acquire_write` checks out one connection,
takes the lock on it, and moves it into the guard, which unlocks on that same
connection. sqlx's `PoolConnection::drop` spawns `return_to_pool`, which runs the
hook, so a connection dropped by cancellation still clears its locks; the unlock
is asynchronous with respect to the drop, which the tests account for. The pool
is separate from the main query pool because a push holds its connection for the
whole receive-pack and `db_max_connections` (20) sits below
`max_concurrent_git_pushes` (32).
Seven tests cover the cancellation window, the same-session release, and the
serialization the pool-mismatch bug was breaking. Both the cancellation
regression and the concurrent-writer case were observed red before the fix, and
removing the `after_release` hook or moving the unlock back to the pool turns
them red again.
A cross-model adversarial pass found that run_post_push_replication holds the
per-repo coalescing key until requeue_or_release, but pin_new_objects reached it
only after two unbounded git reads: the U7 legacy-repair read and the
pre-existing pin read, both plain Command::output with no teardown. A wedged
cat-file on a stuck backend hung the task forever, so the key was held until
process death and later pushes only marked it dirty without spawning a
replacement (the same class this PR closed on the /ipfs serve path).
Add read_object_bounded (the bounded twins under one shared deadline) and use it
at both sites; a timeout skips that object and lets the task proceed to
requeue_or_release. Adds a wedge-reap regression proven load-bearing two ways.
Code review found each candidate's size and content reads were each granted a
full git_service_timeout, so a single served candidate could hold the /ipfs
walk permit for 2x the timeout. Share one deadline across both stages (mirrors
build_filtered_pack) so the read holds admission for one timeout total, and
correct the admission-bound comment: the worst case is O(candidates x timeout)
bounded by the walk concurrency cap, not a single deadline.
Before this PR the pin path stored the provider CID (Kubo dag-pb / Pinata) in
pinned_cids.cid, which the new resolver recomputes-and-404s, while
/api/v1/ipfs/pins still advertises the stale value. Repair a legacy row in the
already-pinned skip branch: gate on a cheap stored-CID codec check (only a
non-raw-codec CID reads bytes, so raw rows keep the DB-only skip cost),
recompute the raw CID, and rewrite it while stashing the old value in a new v14
legacy_provider_cid column (distinct from pinata_cid, which gates the Pinata
pin-skip). Rows whose bytes are gone stay withheld. Documents that the deferred
one-shot sweep, not this opportunistic path, fully retires the window.
The three warn-only record sites in the detached post-push task dropped a
transient DB failure silently, leaving a permanently incomplete pin-source set
that makes the resolver 404 a valid public copy. Wrap them in a bounded retry
(inserts are idempotent via ON CONFLICT DO NOTHING); on exhaustion the warn
still fires so behavior degrades to today's. Documents the residual crash/outage
window that only a reconciliation sweep retires.
The route middleware charged ipfs_rate_limiter once per request while the
legacy scan and provenance walk charged the same bucket per probe/walk, so a
one-probe request cost two tokens and GITLAWB_IPFS_RATE_LIMIT=1 admitted the
request at the route then 429'd it at the pre-scan peek.
Add a bounded ipfs_work_rate_limiter, move the in-scan, per-walk, and pre-scan
charges onto it, and leave the route limiter as the pure once-per-request
brake. The work-budget capacity derives from the route limit with a floor of
one full legacy-probe budget per window (no new operator knob), so a
default-config deep search never self-throttles. Adds it to the periodic sweep,
updates all three AppState construction sites, and rewrites the contradictory
limiter comments.
The knob was parsed and documented as the per-request /ipfs fan-out cap but
never read in production: AppState seeded the legacy-probe budget from a fixed
256 constant, so setting the knob did nothing and an operator could still incur
256 acquire/probe operations per request.
Seed ipfs_max_legacy_probes from the knob (default raised to 256 to preserve
shipped behavior); leave the history-walk ceiling on its MAX_PIN_SOURCES+1
constant so a provenanced request is never truncated into a false 503. Updates
the README and .env.example to the shipped behavior and adds cap-honored and
plumbing tests.
Per-repo coalescing dropped every push that arrived while a repo's encryption
task was in flight, and no reconciliation ever processed it, so a withheld
blob added by the coalesced push had its recovery copy permanently absent. The
drop also silently skipped the coalesced push's local-IPFS pin work.
EncryptInflight becomes a dirty-flag map; the detached task loops, and the
atomic check-and-clear sits at the task tail, unconditional on the
has_path_scoped_rule gate and walk success (a public or rules-free repo would
otherwise exit before it ran). Each requeue re-reads repo state fresh and
re-enumerates the pin half through the fail-closed full scan, never bare
list_all_objects, so a coalesced rule change is honored and no withheld or
dangling object leaks. At-most-one-task-per-repo is preserved.
get_by_cid held the global walk permit and the per-source permit as handler
locals, so dropping the request future released both while the spawn_blocking
probe, visibility walk, and object read kept running; a cancel-spam or timeout
client could exceed MAX_CONCURRENT_IPFS_WALKS and the per-source cap, and the
bare cat-file probe/read children had no teardown at all.
Move the gated serve pipeline into a detached task that owns an AdmissionGuard,
awaited by the handler, so admission releases only after the bounded work is
gone; back the /ipfs probe and read with run_bounded_git twins for
process-group teardown. Adds cancel-mid-walk, timeout-reap, and cancel-spam
regressions plus two load-bearing INV-22 rows. Also reflows one U1 test line to
satisfy cargo fmt.
The path-scoped upload-pack branch kept the read and per-caller permits as
handler locals and called upload_pack_excluding with no AdmissionGuard, so a
cancelled filtered clone released admission while KillGroupOnDrop was still
reaping the git group, bypassing the read and per-caller concurrency caps.
drive_git_child now returns the disarmed guard on success so one guard rides
both build_filtered_pack stages (rev-list then pack-objects); the handler builds
the guard from the permits as the plain path does. Adds a disconnect regression
and an INV-22 gate row, both proven load-bearing.
- steal_after: the reclaim is NOT a guarantee that only a leaked lease is
reclaimed. A waiter's timeout starts at acquire(), not at the FIFO head, so a
same-repo backlog whose cumulative wait exceeds steal_after can steal while an
earlier waiter still writes; correctness rests on the retained pg advisory
lock (which serializes the stealer at acquire_write), not on the bound.
- pinata_object_list_for_refs: recomputes from the tail-start rules snapshot,
not a fresh read at pin-worker time, so a rule tightened AFTER tail-start is
not reflected (matches the old retained-list behavior; reconciliation sweep is
the backstop). Removed the overstated 'rule tightened since the push is honored'.
- RepoWriteGuard::Drop: note that on runtime shutdown the detached unlock task
may be dropped before it polls, bounded by pool teardown / connection close.
The deterministic-fault terminal 500 was checked before the transient
truncated_by 503, unconditionally. When one repo is corrupt (deterministic)
while a DIFFERENT repo is transiently skipped in the same scan, the requested
CID may live in the transient repo -- a retryable 503 is correct, but the
handler returned a terminal 500 and a conformant client won't retry, hiding
retrievable content until the unrelated corrupt repo is repaired.
Gate the 500 on truncated_by.is_empty(): it fires only when a deterministic
fault is the SOLE reason nothing served; a co-occurring transient taint falls
through to the retryable 503. A pure deterministic fault still terminally 500s
an absent lookup by design. Adds a co-occurrence regression test.
Code review found a DoS the F3 lease introduced: git_receive_pack acquired the
global and per-source write permits before block-waiting on the per-repo lease,
so a second same-repo push that parked on the lease pinned a scarce global
write-pool slot (1 of 32) for up to steal_after while sending zero bytes. A few
hostile sources stacking same-repo pushes could hold every slot on idle
lease-waiters and shed 503 on every push to every other repo node-wide.
Move both permit acquisitions to after the lease acquire so a lease-blocked
waiter pins no write-pool slot, and keep a cheap non-holding availability peek
before the DB lookup so a push flood on a saturated pool still sheds 503 without
touching Postgres (the authoritative held permit is taken after the lease).
Adds a blocked-waiter-holds-no-permit regression test.
On client disconnect, RepoWriteGuard::Drop released the per-repo advisory
lock immediately (it holds no child/pgid), while KillGroupOnDrop's detached
reaper was still tearing down the git process group for ~4s. A second same-node
push could then acquire the repo and race the first's still-writing objects/
dir and Tigris upload.
Add an in-process per-repo write lease (block-and-wait, keyed like the advisory
lock) that SUPPLEMENTS the retained cluster-wide PG lock. The lease rides into
the disconnect reaper via a clone on the write-path AdmissionGuard (None on all
read paths) and a second clone held across guard.release() for the clean-path
Tigris upload, so it frees only after the group is reaped. A bounded-wait steal
reclaims a leaked lease (block-and-wait has no coalesce degradation). The lease
is always taken before the PG lock and released after, so the two serializers
cannot invert. Deterministic fake-git race test + inv22 gate row.
The detached Pinata task moved each push's full object-id list into the
closure and parked it on pin_semaphore.acquire_owned(); a tokio semaphore
bounds active holders, not the waiter queue, so a slow Pinata backend let N
queued pushes each retain an MB-scale OID list (unbounded memory/tasks).
Capture only the small ref-update tuples and re-derive the object set via the
existing reaped/deadline-bounded git helpers AFTER the pin permit is acquired,
so retained memory is O(ref tuples), not O(object lists). Per-ref effects are
unchanged (still one spawn per push, no coalescing or shedding, so no dropped
announcements). Adds an inv22 gate row forbidding a retained object list across
the acquire, plus re-derivation-equivalence and reaping tests.
list_all_repos and list_visibility_rules_for_repos ran as bare awaits while
holding the scarce IPFS walk permits, so a query blocked in Postgres pinned the
slots past GITLAWB_IPFS_REQUEST_BUDGET_SECS and shed later requests as 503s.
Wrap both in the remaining request-deadline budget; on timeout the RAII permits
drop and the handler sheds a retryable budget 503. Fails closed: a timeout on
the visibility-rules query denies the request rather than serving an unfiltered
listing.
Appends an inv22 gate row asserting both queries stay deadline-wrapped.
Probe existence via `git cat-file --batch-check` (structured `<oid> missing`
on exit 0) instead of matching English `cat-file -t` prose, so a git wording
change cannot flip a miss into an error or vice-versa. A bad-config/corrupt
repo on a readable store is now a terminal, non-retryable 500 (not a false 404,
and not a retryable 503 that would fan out cat-file retries); a transient
unreadable store stays 503. Error bodies are opaque (no raw git stderr).
Layers onto the existing object_store_readable + re-probe disambiguation.
Await pg_advisory_unlock while the pooled connection is still owned by
self; take it and set released only after the await resolves. A cancel
mid-unlock now leaves the Drop backstop armed (conn still Some), so the
session-level advisory lock is no longer leaked onto a pooled connection.
Adds a cancel-mid-unlock regression test (via a test-only pre-unlock gate)
and an append-only inv22 gate row asserting the take/released orderings.
EncryptInflight bounds the outstanding post-push encryption-task set to one per
repo, but its state.rs doc implied a global cap it does not deliver, and each pin
loop (ipfs_pin + pinata pin_new_objects) holds a full per-push object-id list
while walking it — so N distinct authenticated first-pushes could hold N MB-scale
lists at once, unbounded.
Add a global pin_semaphore (GITLAWB_MAX_CONCURRENT_PIN_TASKS, default 8) and gate
BOTH pin loops through it: they DEFER when the pool is full, never drop a pin.
Tighten the state.rs comment to state the bound is PER-REPO, with the cross-repo
residual (throttled by auth + rate limits, memory bounded by the pin permit)
called out honestly.
Also correct .env.example GITLAWB_DB_MAX_CONNECTIONS (20 -> 48): with the #174 F1
boot validation, 20 is below max_concurrent_git_pushes(32)+headroom(8) and would
fail startup — closing the INV-24 doc gap from the F1 change.
Tests: pin_new_objects_gated_defers_when_pin_pool_exhausted (RED without the
permit acquire); max_concurrent_pin_tasks range/default.
object_type_bounded treated any bare `fatal:` cat-file exit as absence. A packed
object whose pack/idx is transiently unreadable (permissions, or a mid-repack
race) emits `could not get object info` — byte-identical to a genuine miss — so a
present object could return Ok(None) and the /ipfs scan a definitive 404 instead
of the intended retryable 503. The stderr cannot separate the two (the CID is
pre-validated, so that string is also the only legitimate absence path), so the
disambiguation must be out of band.
On the collided fatal, probe object_store_readable (objects/ listable, every
pack/idx openable). Unreadable -> Err (taint -> 503). Readable -> re-probe once;
still absent -> Ok(None), now present -> the type. This narrows but does not close
the concurrent-repack TOCTOU (the check samples a different instant), stated
honestly rather than claimed deterministic.
Test: object_type_bounded_unreadable_pack_is_error_not_absence packs a real repo,
chmod 000 the pack, and asserts Err (root-guarded). RED with the readability check
disabled: the present blob returned Ok(None).
fail_closed_full_scan_objects ran phase 1 (replicable_blob_set_bounded) then
phase 2 (all_blob_oids) with a FRESH Instant::now() + timeout for phase 2, under
one held scan permit. A large-but-successful phase 1 plus a full phase 2 held the
permit ~2x the configured budget instead of the one whole-scan budget.
Compute one deadline before phase 1 and share it: phase 2 takes the shared
Instant directly, phase 1 takes the remaining-until-deadline. Total occupancy is
now ~1x. The completeness cost is honest: a repo whose phase 1 nears the budget
under-pins this push (fail closed) rather than over-holding, so size the budget so
both phases normally fit.
Test: full_scan_shares_one_deadline_across_both_phases drives a slow-but-
successful phase 1 that eats the budget; the shared deadline reaps phase 2 and the
scan fails closed (empty). RED before: phase 2's fresh budget completed and kept
the non-blob candidate.
The GET /ipfs/{cid} object-type probe and content read ran bare synchronous
Command::output() on the async worker, budget-checked only before they started.
A hung or corrupt object store therefore pinned a Tokio runtime worker and both
held IPFS admission permits indefinitely, so enough requests exhausted the route
despite GITLAWB_IPFS_REQUEST_BUDGET_SECS. The walk beside them already used
spawn_blocking + the reaped bounded runner; these two stages were left bare.
Extract run_bounded_git_raw (returns ExitStatus/stdout/stderr so callers can
classify exit codes; run_bounded_git stays a thin bail-on-nonzero wrapper for the
walk callers) and add store::object_type_bounded / read_object_content_bounded on
it, preserving object_type's Ok(None)-on-absence vs Err classification so the
serve path's 404-vs-503 semantics are unchanged. Run both /ipfs stages in
spawn_blocking under a deadline clamped to min(git timeout, remaining budget),
with the IPFS permits held across the awaited spawn_blocking.
Test: get_by_cid_hung_probe_is_reaped_and_sheds_503 hangs git on a FIFO alternates
with no feeder; the reaped probe sheds a 503 in bounded time (GREEN), while a
neutralized deadline blocks the handler forever (RED). The existing probe-error /
corrupt-repo tests still assert the absence-vs-error classification at the handler.
The post-receive replication tail (withheld/candidate/full-scan resolution, then
the encrypt+recovery spawn and the announce spawn) ran inline in the request
future and parked on git_encrypt_semaphore before the durable try_begin gate. A
client/proxy disconnect while parked dropped the future and silently lost this
push's pins, recovery copy, and announcements, with no reconciliation sweep to
recover them (state.rs documented this exact residual).
Move the whole tail into an independently owned tokio task and return the git
response immediately, so a disconnect can no longer drop the parked work. Each
push owns its own tail, including its own always-spawned announce, so the
announce is never brought under the per-repo encrypt coalescing (which would
drop a coalesced push's per-ref announcements). Update state.rs to retire the
disconnect-loses-work residual.
Test: receive_pack_landed_push_returns_without_parking_on_scan_pool asserts the
handler returns 200 while the scan pool is held (RED with the tail inline: the
request future parks and the timeout fires; GREEN detached). The burst
serialization test is retimed to poll for scan completion since the withheld
walk is now detached too.
acquire_write took the pg advisory lock, then awaited Tigris, then built the
guard. An outer tokio::time::timeout firing during the Tigris await dropped the
future with the lock held and no guard, so no unlock ran and later pushes to the
repo spun 60s. release() also unlocked on an arbitrary pooled connection, not the
one that locked (session advisory locks are connection-affine), so even the
normal release was unreliable.
Pin a PoolConnection in RepoWriteGuard, build the guard before the lock query,
run the lock loop through it, unlock on that same connection in release(), and
add a Drop backstop that spawns a detached affine unlock (Handle::try_current so
an off-runtime drop logs instead of panicking). Two distinct-session tests
(drop-without-release, affine release) go RED without the fix, GREEN with it.
Pinning a connection per active write is a DB-pool DoS unless the pool clears
the concurrent-write cap: the shipped default (20) was below
max_concurrent_git_pushes (32). Raise the db_max_connections default to 48 and
add Config::validate() to reject db_max_connections < max_concurrent_git_pushes
+ 8 headroom at boot; wire it into startup.
The budget-knob docs claimed the overshoot was bounded by one unclamped
probe; an unclamped subprocess is not a bound, so the README,
.env.example, config help, and module comments now say plainly that a
hung git probe holds the walk slot for the hang's duration. New
regression: an acquire-tainting ghost row ahead of a healthy public copy
must not stop the scan; the later found short-circuit still serves 200.
object_type now distinguishes repo-level git failures (not-a-repository,
error:-prefixed corruption) from genuine object absence, so a corrupt
repo taints the /ipfs scan into the retryable 503 instead of vouching
for a definitive 404; the drain reads visibility rules by the freshly
fetched record id, closing a latent fail-open on id divergence. The four
budget gates collapse into one stage-labelled helper, the three scan
sites share state::acquire_scan_permit (inv22 tripwire repointed and
sever-verified), and the Tigris stall tests use a local accept-and-park
listener instead of a non-routable address.
try_begin now merges a losing push's tip pairs into the in-flight key's
pending slot in the same critical section as the presence check, bounded
at 1024 pairs with a full-scan marker on overflow (force_full_scan
threads to the resolver; empty tips never encode the marker). The task
loop-drains: finish_or_take_pending either hands back the batch with the
key retained, or removes the key and disarms the guard atomically, so a
successor task's key survives the guard drop. Drains re-fetch rules
fresh and replay the tail pipeline under the U4 helper gates only; the
loop holds no task-level pool permit, so a hot repo cannot deadlock the
pool at size 1.
replication_withheld_set's walk arm, resolve_candidates_for_push's git
stages, and fail_closed_full_scan_objects each take a
git_encrypt_semaphore permit before their spawn_blocking and carry it
inside the closure, so a push burst holds at most pool-many concurrent
scans and a started scan finishes with its permit. Deletion-only pushes
and repos without path-scoped rules never park. Contention defers the
landed push's tail (logged with queue_wait_ms), never errors it; the
acquire sites document the accepted disconnect-during-park residual.
GITLAWB_IPFS_REQUEST_BUDGET_SECS (default 600) is captured as one
deadline at handler entry: every acquire and expensive walk is clamped
to the remaining budget, every stage checks remaining before starting,
and exhaustion taints the scan into the retryable 503. A walk in flight
is never aborted at the tokio layer (that would free the walk permit
while the blocking thread still runs); its clamped git deadline plus
group teardown ends it. Docs state the residual overshoot: the in-flight
stage's kill/reap slack plus the unclamped probe subprocesses.
A request-scoped taint list replaces the silent folds: any repo skipped
without a verdict (acquire timeout, probe or read failure, walk failure,
cap or ceiling exhaustion) turns the terminal into a retryable 503 with
Retry-After, and 404 is reserved for a scan in which every candidate
reached a verdict. The walk cap now counts only expensive allowed-set
walks and skip-continues on exhaustion so a plain public copy past the
cap still serves; a new GITLAWB_IPFS_MAX_REPO_VISITS knob (default 1024)
bounds the acquire+probe cost class and stops the scan, documented with
its worst-case object-store fetch count.
drive_git_child hands the AdmissionGuard back on success so
upload_pack_excluding can thread it across the rev-list and pack-objects
stages; on timeout or error it is dropped post-reap, and on future-drop
it rides the detached reaper as before. The upload-pack handler builds
the guard once ahead of the withheld/plain branch, so a client
disconnect mid-filtered-serve no longer frees the read-pool and
per-source permits while the git process group is still being reaped.
An attacker who pins a public object from MAX_PIN_SOURCES repos before the
legitimate public source registers fills pin_repo_sources (record_pin_source is
first-N-wins and drops later sources silently); the buried public source is then
never reached because a non-empty provenance set suppressed the legacy scan, so
anon GET /ipfs/{cid} 404s forever for a public object (re-breaks F1).
U1: db::pin_sources_at_cap reports whether pin_repo_sources is at MAX_PIN_SOURCES
for an oid (the only observable signal that a servable source may have been
dropped, since the write cap never overshoots).
U2: get_by_cid falls back to the bounded legacy scan on a provenance miss when the
set is empty OR at_cap. The scan gates every repo through the real per-caller gate,
so it finds the buried public copy. A complete (non-full) set still fast-404s, so
ordinary denials never fan out to O(repos) (INV-10/F3); the fallback honors the
is_throttled peek.
Regression ipfs_cid_buried_public_source_still_serves_via_scan_fallback: 404 before,
200 after; pin_sources_at_cap_flips_at_max covers the boundary.
- ipfs_pin::pin_new_objects returns the provider Hash (not the raw resolver key),
matching the pinata twin's contract; the DB cid stays the raw content CID. The
return is logging-only here, so this is drift-avoidance, not a functional fix.
- AppError::SearchIncomplete now carries Retry-After: 1 like Overloaded, so both
retryable 503s from the /ipfs handler advertise the retry hint consistently.
A provenanced object has a bounded source set (first-pinner + up to
MAX_PIN_SOURCES additional). With the walk ceiling at 16 == MAX_PIN_SOURCES, an
authorizing public source sorting after 16 path-scoped denials was never reached:
the ceiling truncated the search and returned a false 503 for a readable object.
Raise the ceiling to MAX_PIN_SOURCES + 1 so the whole bounded provenance set is
always tried; the legacy scan stays bounded by MAX_LEGACY_PROBES_PER_REQUEST.
P2-E: the object-type probe and the F6 size+read ran bare git cat-file inside
spawn_blocking while the /ipfs walk permit was held, so a wedged cat-file could
pin the global walk slot for the request's life. Bound both under
git_service_timeout_secs; on timeout free the slot (truncated -> retryable 503)
and skip the repo.
P2-D: the pinata already-pinned branch now backfills NULL first-pinner
provenance in lockstep with the ipfs_pin skip branch (consistency; the object
was already resolvable via the pin_repo_sources union).
P1-A: gate_and_serve held the /ipfs walk permit across a bare repo_store.acquire;
wrap it in git_acquire_timeout_secs (mirroring #174's own handler) so a cold/hung
Tigris acquire cannot pin the global walk slot. On expiry skip the repo and mark
the search truncated (retryable 503, not a false 404).
P1-B: the local-IPFS pin path recorded Kubo's provider Hash as pinned_cids.cid;
for objects above the block size that is a dag-pb root that does not hash the raw
content, so GET /ipfs/{cid} listed then 404'd them under the F2 integrity check.
Record the locally-computed raw-content CID, mirroring the pinata twin.
Rebased onto #174 (fix/served-git-concurrency-cap). #173's incremental history
cannot replay commit-by-commit because #174 rewrote the same handler files in
parallel; this single commit carries the fully-integrated tree (identical to the
verified merge result), with the follow-up fixes as separate commits on top.
Brings in #173: GET /ipfs/{cid} CID->oid resolution via pinned_cids, per-caller
path-scoped blob/tree/commit-tag gating (#135/#173 F1-F6), and the pin-source
provenance table. Adapts #173's tree/commit-tag walks to #174's run_bounded_git
so the held /ipfs walk-concurrency permit is duration-safe (F5).
P2-1: git_info_refs treated every service other than git-receive-pack as a read op
and did the DB/visibility/Tigris work under a read permit, validating the service
string only downstream in smart_http. So an unauthenticated `?service=anything` to
a public repo consumed a read slot + storage work before rejection. Validate the
service is exactly git-upload-pack or git-receive-pack immediately after parsing,
returning 400 before the pre-DB shed and the permit. Handler-layer regression:
with the read pool exhausted, `?service=git-explode` returns 400 (validated first),
not 503 — removing the validation makes it 503 (RED).
U5/INV-24 docs: document the new knobs from U2/U3 in .env.example and the README
env-var table — GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS, GITLAWB_MAX_CONCURRENT_IPFS_WALKS,
GITLAWB_IPFS_WALK_PER_SOURCE, GITLAWB_IPFS_MAX_REPOS_WALKED, GITLAWB_IPFS_RATE_LIMIT.
Each is already enforced on the path it names (U2/U3 tests), so the docs match the
implemented routing.
After a successful path-scoped push the handler spawns a DETACHED task that runs
pin_new_objects then withheld_recipients_gated, which acquire_owned().await's on
git_encrypt_semaphore — it DEFERS (blocks) when the pool is full. The semaphore
caps active walks but nothing capped how many detached tasks were spawned and
PARKED on that await: N rapid path-scoped pushes spawned N tasks, each holding
cloned object lists/rules/paths/keys — an unbounded parked-waiter set (P2-2).
2a54c15 deliberately chose defer-not-shed because dropping the walk loses the
withheld-blob recovery copy and there is no reconciliation sweep to rebuild it.
So bound the OUTSTANDING-TASK set by per-repo coalescing rather than shedding:
EncryptInflight (a bounded Arc<Mutex<HashSet<repo_id>>>) + try_begin — before the
spawn, if a task for the repo is already in-flight skip the duplicate (the pending
walk covers the newer objects); otherwise spawn and move an RAII
EncryptInflightGuard into the task that removes the key on drop (completion, error,
or panic-unwind). This bounds the set to <=1 pending task per repo and never drops
work; withheld_recipients_gated's defer is unchanged.
The second detached post-push spawn (Pinata replication, also runs pin_new_objects)
is scoped out with rationale: it parks on no semaphore (no unbounded waiter set)
and does per-push per-ref work (branch->CID, gossip, subscription, Arweave,
peer-notify) keyed to this push's ref_updates — coalescing it would DROP a later
push's announcements, a correctness regression.
Regressions (mutation-verified RED->GREEN): the outstanding set is bounded to 1
per repo under saturation (never-coalesce -> 32 tasks RED); a coalesced repo's key
is released when its task ends so it is reprocessed, never permanently skipped
(guard-drop no-op -> repo locked out, recovery copy lost forever, RED — the exact
durability regression the fix prevents); distinct repos each admit; cold-set first
push admits; plus an inv22_gates structural tripwire that fails if the handler gate
is removed.
GET /ipfs/{cid} (get_by_cid, publicly reachable via optional_signature) ran
allowed_blob_set_for_caller_bounded in spawn_blocking inside a per-repo loop with
NO global/per-source admission and NO route rate limit — the surface had zero
concurrency or rate control. A permissionless caller could fan out concurrent
full-history git walks (each up to the walk timeout) exhausting blocking-pool
threads and PIDs outside every served-git pool (P1-3).
Acquire a global git_ipfs_walk permit (try_acquire_owned, shed 503) plus a
per-source sub-cap (with_default_max_keys, reject-before-insert, keyed via
client_key/TrustedProxy so XFF can't spoof it) ONCE after CID validation, held as
handler locals across the whole request — including every spawn_blocking walk, so
the slot reflects real blocking-thread occupancy (a tokio timeout cannot cancel
spawn_blocking). None key -> global pool only. Cap repos walked per request so one
request can't serialize N full-history walks. Attach rate_limit_by_ip +
IpRateLimiter to the /ipfs route (mirrors the git/create/peer routers; the
extension is required or the middleware is a silent no-op). Visibility/authz
semantics unchanged.
New knobs: GITLAWB_MAX_CONCURRENT_IPFS_WALKS (32), GITLAWB_IPFS_WALK_PER_SOURCE
(4), GITLAWB_IPFS_MAX_REPOS_WALKED (64), GITLAWB_IPFS_RATE_LIMIT (600/hr, 0
disables) — all range-validated and enforced on-path.
Handler-layer regressions (all mutation-verified RED->GREEN): shed-at-capacity
503 (delete acquire -> falls through), per-source cap sheds same source / admits
another, None-key arm sheds on the global pool, map self-bounds reject-before-insert,
the walk permit is held while the spawn_blocking walk runs (available_permits()==0;
dropping it before the loop -> RED), repos-walked cap (remove break -> 2 walks),
and the route IP rate limit actually fires 429 (drop the Extension -> RED no-op).
The served-git admission permit is acquired, then RepoStore::acquire*/advisory-lock
runs before the bounded git runner starts — with no local deadline. acquire/acquire_fresh
await Tigris HEAD/GET, and acquire_write's advisory-lock loop awaits a per-iteration
pg_try_advisory_lock that can block indefinitely on a hung Postgres pool. Since
GITLAWB_GIT_SERVICE_TIMEOUT_SECS only starts once git spawns, a stalled backend pins
the permit and drains the pool until every later request 503s (P1-2).
Wrap each git-path acquire (git_info_refs read/advert, git_upload_pack, git_receive_pack)
and the /ipfs per-repo acquire loop in tokio::time::timeout(git_acquire_timeout_secs, ..).
On expiry the git paths return AppError::Overloaded (503 + Retry-After); the permit is a
handler-local at that point (moved into the AdmissionGuard only after acquire) so the early
return frees the slot. The /ipfs per-repo acquire keeps its fail-closed `continue` on
timeout (never serves an un-acquired repo). No repo_store.rs signature change: the outer
timeout cancels a mid-sleep/mid-fetch_one future, so no in-loop deadline is needed.
New knob GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS (default 30, range 1..), modeled on
git_service_timeout_secs, kept separate because acquisition and git execution are distinct
cost centers.
Handler-layer regression receive_pack_acquire_deadline_sheds_and_releases_permit: holds the
same pg advisory lock acquire_write derives on a second connection so the real loop must
retry, drives git-receive-pack with the deadline at 2s and write pool size 1, and asserts
503 at ~2s + the permit recovers + a follow-up admits once the lock frees. Removing the
timeout wrapper turns it RED (blocks ~31s to the test ceiling on the ~59s advisory loop).
On the plain (non-path-scoped) info_refs / upload-pack / receive-pack paths the
global + per-source admission permits were handler-locals that dropped the
instant the handler future dropped on a client disconnect, while KillGroupOnDrop
reaps the git process group in a DETACHED task (SIGTERM -> grace -> SIGKILL ->
reap). So a disconnect-spammer admitted replacements while prior groups were
still alive, defeating the process cap and the per-source cap during teardown.
be0cdd6 already fixed this for the path-scoped upload-pack walk; this closes the
residual plain paths (P1-a).
Introduce AdmissionGuard, a move-only holder of the permits, threaded through
info_refs/upload_pack/receive_pack/run_git_service into drive_git_child's
KillGroupOnDrop. On disconnect the guard moves into the detached reaper and drops
only after the group is ESRCH-confirmed reaped; on success/timeout disarm()
returns it for the earliest provably-free drop. The rev-list/pack-objects
visibility-walk callers hold no admission and pass None.
Thread git_bin through upload_pack/receive_pack (they hardcoded "git") so the
plain POST path is drivable by an injected fake git, matching the existing
injectable seam on info_refs/run_git_service/upload_pack_excluding.
Handler-layer regression upload_pack_plain_permit_held_through_group_reap_after_disconnect:
isolates the global read pool (size 1, per-source + rate limiter permissive), a
SIGTERM-ignoring descendant keeps the group alive across the reap window, and
asserts available_permits()==0 held on disconnect + a cross-source replacement
sheds 503, then frees after ESRCH. Reverting the guard-into-reaper move (release
immediately) turns it RED at the held-through-reap assertion. Plus a None-key arm.
A source-scan tripwire that fails when a new site reintroduces the concurrency-cap
class PR #174 fixed, or when one of the five gates (U1-U5) is removed. INV-22: a
permit held per op recovers only if every path is duration-bounded and reaps the
group before releasing admission, and every detached git task carries admission.
Lives in tests/ (a separate crate) on purpose — a guard scanning the file it lives
in would match its own identifier literals and pass vacuously; scanning src/ from
here keeps every check load-bearing. Checks: run_bounded_git confirms child exit via
child_terminated_without_reaping (U1); KillGroupOnDrop launches the reaper via
Handle::try_current (U2); git_receive_pack acquires state.git_write_per_caller (U4);
the encryption walk runs through withheld_recipients_gated over git_encrypt_semaphore
(U5); and the bounded recipients walk is invoked exactly once — only inside the gated
helper — so a new spawn_blocking bypass (count > 1) fails (P1-e non-bypass).
Proven load-bearing: a synthetic second call site turns the count tripwire RED, and
each named gate maps to a fix whose own regression already goes RED on revert.
After a successful path-scoped push the handler released its write permit and
then ran a DETACHED tokio task whose spawn_blocking(withheld_blob_recipients_bounded)
performed another full-history git walk under no admission. N fast completed
pushes spawned N concurrent full-history walks past GITLAWB_MAX_CONCURRENT_GIT_PUSHES
(which bounds only the in-handler phase), and the per-child bounding from 71389e6
caps each walk's duration but not their count (P1-e).
Add git_encrypt_semaphore, a pool of its own (sized from max_concurrent_git_pushes,
no new knob — Q1) so a long background walk never holds a foreground write slot and
a handler holding a write permit can't self-deadlock. Route the walk through a new
withheld_recipients_gated helper that acquires the pool before the walk and DEFERS
(blocks) when full rather than shedding — dropping the walk would lose the
withheld-blob recovery copy, so durability stays fail-closed.
Regression: encrypt_walk_defers_when_pool_exhausted drives the gated helper with the
pool exhausted and asserts the walk blocks and does not run its rev-list, then runs
once a permit frees. RED before (the walk runs regardless of the pool), GREEN after.
The detached push task calls this exact helper.
The authenticated git-receive-pack POST acquired only the global write semaphore
— no per-source sub-cap, unlike the read and anon-advert pools. Owner enforcement
defaults off, so one host minting disposable did:key identities could open
max_concurrent_git_pushes slow POSTs from a single source IP and 503 every other
source's push; the 600/hour push limiter bounds arrival rate, not in-flight
concurrency (P1-d).
Add git_write_per_caller (a PerCallerConcurrency sized like the advert cap,
max_concurrent_git_pushes/8), and acquire it before the global write permit,
keyed on the resolved source IP via read_caller_key — never the signed DID (a DID
farm defeats a DID key). This required adding the PeerAddr + HeaderMap extractors
to git_receive_pack, which it lacked (without them the key is None and the cap is
inert). Neutralized the shared acquire helper's log wording now that it serves
both read and write paths.
Regression: receive_pack_per_source_write_cap_sheds_capped_source_not_others — a
source at its write sub-cap sheds Overloaded/503 (proving the extractors resolve a
key), while a different source is not shed. RED before (the capped source proceeds
past admission to a git error instead of shedding), GREEN after; the existing
receive-pack advert cap test still passes.
On the path-scoped upload-pack path the per-source and global read permits were
handler locals, so a client disconnect dropped the handler future and released
both permits while the awaited spawn_blocking withheld-blob walk kept running
(spawn_blocking can't be cancelled). A permitless caller could disconnect-spam to
exceed the read cap while real git work continued (P1-b).
Move both permits into the blocking task and return them out: on success they
flow back so the serve phase keeps them; on a dropped future the returned tuple
(with the permits) is discarded only when the blocking task completes, so
admission tracks the walk's real duration.
Regression: upload_pack_permit_held_through_walk_after_disconnect drives the
handler to mid-walk (fake git hangs on rev-list), disconnects, and asserts the
global read slot stays held (available_permits == 0). RED before (the slot frees
to 1 the instant the future drops), GREEN after. The test roots the repo store at
its own TempDir so it is isolated per run.
On a client disconnect KillGroupOnDrop sent a lone SIGTERM with no escalation or
reap, so a group member that ignores SIGTERM (a wedged pack-objects, a malicious
helper) survived while the handler released its concurrency permit — letting
disconnect-spam accumulate live git processes and defeat the load shed (P1-c).
The timeout path already ran the full teardown.
Make the disconnect teardown as strong as the timeout path: the guard now owns
the child and, on drop, launches a detached reaper that runs the same
TERM -> grace -> SIGKILL -> reap-the-group sequence (reap_group_on_timeout).
Owning the tokio Child in the reaper keeps it the sole reaper of the leader, so
it never races tokio's own orphan reaper (a raw waitpid(-pgid) would).
Regression: run_git_service_sigkills_a_sigterm_ignoring_child_on_disconnect, RED
before (the descendant survives the lone SIGTERM), GREEN after; all existing
teardown/timeout/disarm tests still pass (the three hand-built-guard tests were
updated to the owned-child shape).
Residual: capacity (the handler's semaphore permit) still releases at disconnect
rather than strictly after the reaper confirms the group dead, so accumulation is
now bounded to ~one teardown window (<=4s) rather than eliminated; gating the
permit on the reap needs threading it through the serve handlers and is left for
a follow-up (it also touches the receive-pack permit lifetime U4 changes).
A group member (or the leader itself) that closes stdout before the deadline made
the stdout drain return EOF early, which stood the watchdog down via done_tx
before it could ever fire; child.wait() then blocked on the still-live child,
pinning the walk thread and its read/write permit past
GITLAWB_GIT_SERVICE_TIMEOUT_SECS. This is distinct from the descendant case
already covered: there the leader sleeps until the deadline so the watchdog does
time out; here the drain-EOF races ahead of it.
Stand the watchdog down only once the child has actually terminated, detected
without reaping (waitid + WNOWAIT) so the leader's pid stays unreaped and its
pgid un-recycled until the watchdog finishes its teardown and is joined. Past the
deadline the watchdog still owns the full SIGTERM -> grace -> SIGKILL sequence.
Regression: run_bounded_git_reaps_a_leader_that_closes_stdout_then_hangs, RED
before (blocks on child.wait past the recv budget), GREEN after; the three
existing teardown tests still pass (no recycle-hazard regression).
build_filtered_pack_times_out_a_hung_rev_list and _a_hung_pack_objects
spawn a freshly-written fake git directly, bypassing the fake_git_run_with_pids
retry the rest of the module uses. Under cargo test fork-storm load a concurrent
worker forks while the fake's write fd is still open, so execve returns ETXTBSY
and the raw io::Error surfaces where the test asserts GitServiceTimeout, failing
test (stable) intermittently.
Route both tests through build_filtered_pack_or_exec_race_retry, which retries
ONLY errno 26 (ETXTBSY) and keeps the outer 10s watchdog per attempt, so a
genuinely-missing async timeout bound still trips the watchdog and a wrong error
type still fails at the assertion. Production paths unchanged.
The finding-2 wiring test covered only list_all_objects (the full-scan fallback).
Add delta_path_exec_fns_time_out_on_a_hung_git covering the common push path:
peeled_object_type (cat-file), rev_list_delta (rev-list), and list_all_objects_with_type
(cat-file, which all_blob_oids delegates to). Each must return within the watchdog
budget on a hung git; reverting any to a bare Command::output() blocks past the recv
budget (RED-verified).
jatmn P2 (INV-24): the config help, .env.example, and README described routing the
handler no longer does.
- The read pool serves upload-pack and the UPLOAD-PACK info/refs advertisement only;
the anon receive-pack info/refs advertisement draws from its own dedicated pool
(sized like the write pool but disjoint), not the read pool and not the write pool.
- The write pool is the authenticated push POST only.
- The per-caller read cap is keyed on the resolved source IP, never the DID: a
signature does not move a caller off it (the docs claimed a per-DID budget, which
would tell a NATed authenticated client it escapes the shared cap when it does not).
- git_service_timeout_secs now also bounds the push-side pin-candidate discovery
(rev-list / cat-file), reaped via process-group teardown, alongside the info/refs
advertisements and the withheld-blob walk.
Route push_delta's candidate-discovery git children (cat-file / rev-list) through the
bounded run_bounded_git (deadline + process-group SIGKILL teardown) instead of bare
Command::output(). A hung scan no longer pins the git-receive-pack write permit past
the deadline or a client disconnect. Threads git_bin + a shared per-scan deadline
(git_service_timeout_secs) through resolve_candidates_for_push and all_blob_oids.
jatmn/CodeRabbit P1: the walk watchdog stood down as soon as the group LEADER was
reaped. A leader that exits cleanly on SIGTERM while a background member ignores
SIGTERM and closes its inherited pipes let the main drain unblock, the leader be
reaped, and the watchdog skip its SIGKILL escalation — so the handler returned while
the descendant kept running. jatmn's suggested "continue until ESRCH" could not work:
the main thread reaping the leader is what frees the pgid, so continuing to kill(-pgid)
after that races a recycled group.
Restructure the coordination instead: the main thread now defers reaping the leader
until the watchdog thread returns. The watchdog always escalates through an
unconditional group SIGKILL (no leader-reap short-circuit), and because the leader is
still unreaped while every kill(-pgid) fires, the pgid cannot have been recycled. The
grace before SIGKILL is a fixed budget (only paid on the exceptional timeout path).
Regression run_bounded_git_sigkills_a_sigterm_ignoring_descendant_after_leader_exits:
a leader that exits 0 on SIGTERM spawns an sh -c descendant (its own $$, not a ( )
subshell's parent pid) that ignores SIGTERM and closes its pipes; the descendant must
be dead when the runner returns. RED on the old teardown (survives), GREEN now. The
existing leader-ignores-SIGTERM and hung-walk teardown tests still pass.
jatmn P1: run_bounded_git was gated #[cfg(unix)], but its callers
(assert_all_refs_are_commits, blob_paths) and the module are compiled on all
targets, so a non-Unix build (the continue-on-error Windows release target) could
not resolve the function.
Add a #[cfg(not(unix))] twin with the same signature and result semantics. It bounds
a single child (no process-group teardown, which is Unix-only): threads feed stdin
and drain stderr, the main thread drains stdout, and a watchdog thread kills the
child at the deadline. The child is shared with the watchdog behind a mutex the main
thread does not hold while draining, so the watchdog can always acquire it to kill,
and killing closes stdout to unblock the drain. Best-effort (reaps only the direct
child), which is why the hardened group-aware path stays Unix-only.
Verified the twin type-checks by compiling its (portable-std) body on the native
target; the full non-Unix link is CI's Windows job.
jatmn P1: the anon-reachable GET info/refs?service=git-receive-pack drew from the
same git_write_semaphore as the authenticated git-receive-pack POST. The per-source
advert cap reserves no global slots, so ~8 sources (each within the per-IP push rate
limit) could hold all 32 write permits across acquire_fresh + the advertisement, and
the next authenticated push then 503s at admission. That is an anonymous caller
shedding an authenticated push across the auth boundary (INV-10).
Add a dedicated git_push_advert_semaphore, disjoint from the write pool. Both the
pre-DB early-shed peek and the held acquire for the receive-pack advertisement now
draw from it; the write pool is left exclusively for the POST. An advert flood can at
worst exhaust the advert pool (each source still capped by git_push_advert_per_caller
and the per-IP push rate limiter); the authenticated push pool is untouched.
Test receive_pack_advertisement_draws_from_dedicated_advert_pool proves at the handler
layer that the advert sheds when its own pool is saturated AND survives when the write
pool is saturated (the reservation). Both flip under revert, verified independently for
the pre-DB peek and the held acquire.
CodeRabbit review: the held git_permit was acquired at the top of git_info_refs
and git_upload_pack, before the per-source cap, so one source could occupy the
global pool during the DB/visibility window before its sub-cap rejected the
excess. Move the held git_permit to after acquire_read_caller_permit (still
before acquire_fresh/git, so INV-10's bound on the fresh Tigris acquire holds),
and add a cheap pre-DB early shed (available_permits() == 0 -> 503, holds no
permit) so the #62 shed-before-DB property is preserved without holding a permit
across the DB work.
The receive-pack POST is unchanged (owner-only; its top-of-handler git_permit is
deliberate for acquire_write bounding).
Tests: the two #62 shed-before-DB tests now exercise the explicit early check
(load-bearing: disable it and they fall through to the DB); three new
*_per_source_cap_sheds_with_global_capacity tests prove the per-source sub-cap
sheds a capped source even with global capacity free (load-bearing on the caller
cap). Full workspace green.
Follow-up to threading the configured timeout into the walk: the walk now honors
GITLAWB_GIT_SERVICE_TIMEOUT_SECS on both the serve and replication paths, so the
README no longer says a fixed internal deadline.
Close the reasoned-not-run gaps from the code review by making the walk's git
binary and timeout injectable, then driving the missing branches with a real
handler and a fake git instead of reasoning about them.
- Add state.git_bin and *_bounded variants of the walk entry points taking
(git_bin, timeout); the served handlers (upload-pack serve, receive-pack
replication and full-scan and encrypt-pin, and the ipfs gate) now pass the
operator-configured GITLAWB_GIT_SERVICE_TIMEOUT_SECS, so the whole walk is bounded
by the same budget as the other served-git ops rather than a fixed constant. The
git_bin-less wrappers stay for the real-git classification tests.
Newly vetted by execution (not reasoning):
- receive-pack replication path is bounded: replication_withheld_set with an injected
hung git returns within the budget and fails closed, so it cannot pin the write
permit git_receive_pack holds across it.
- a hung withheld-blob walk on the upload-pack POST returns 504 (real handler, real
repo on disk, injected hung git), proving the GitServiceTimeout -> git_service_app_error
wiring end to end.
- the watchdog status-gate: a child that exits successfully is not reported as a
timeout even when the watchdog fired (mutation-checked: drop the guard -> RED).
- SIGKILL escalation: a SIGTERM-ignoring child is still reaped via SIGKILL and the
group is gone; a truly uninterruptible D-state child (unreapable by any signal) is
the documented residual, matching the async teardown.
- the advertisement per-source cap sizing never derives 0.
Full gitlawb-node suite 515 green.
U2 moved the receive-pack info/refs advertisement onto git_write_semaphore to keep
an anonymous read flood from starving the push handshake. But the advertisement is
anon-reachable on public repos and holds its write permit across the slow
acquire_fresh Tigris download, and the only per-source brake on it was the push
RATE limiter, not a concurrency cap. So a multi-source flood of receive-pack
advertisements could hold the write pool's slots across those downloads and shed
authenticated pushes (both the advertisement and the owner-gated git-receive-pack
POST draw from the same pool). U2 thus introduced the first anonymous consumer of
the write pool the state doc promised anon could never reach; the plan's residual
note (no worse than the POST) was wrong, because the POST is owner-gated and the
advertisement is not.
Add git_push_advert_per_caller, a per-source concurrency sub-cap on the receive-pack
advertisement keyed on the resolved source IP (the same PerCallerConcurrency
mechanism U1 uses for reads), sized to an eighth of the write pool so a single
source holds at most that share and saturating the pool takes many distinct source
IPs, each also braked by the per-IP push rate limiter. The upload-pack advertisement
keeps its read-pool per-caller cap; the owner-gated POST is unchanged. Correct the
state doc for git_write_semaphore accordingly.
Handler-layer proof: a source at its receive-pack advertisement cap sheds 503 (RED
before the acquisition, 500-not-503), while a different source and the upload-pack
advertisement are unaffected. Full suite 510 green.
Code review found run_bounded_git's watchdog could return a spurious 504 and
signal a recycled process group. The watchdog runs off a wall clock on its own
thread; done_tx.send() only fires after child.wait() reaps the leader, so a walk
that finished within microseconds of the deadline took the watchdog's Timeout
branch, discarded a fully-captured successful result, and returned GitServiceTimeout
(a 504 for a walk that actually completed). Worse, the Timeout branch SIGTERMed
-pgid unconditionally after the leader was reaped, so a recycled pgid could be
signalled, the exact hazard smart_http guards via disarm-after-wait.
Set a reaped AtomicBool the instant the main thread reaps the child; the watchdog
checks it before every kill and stands down if the leader is already reaped. Gate
the timeout verdict on !status.success(), so a child that exited on its own is never
reported as a timeout even if the watchdog fired late. Add the survived-SIGKILL warn
smart_http's reap already emits, for operator visibility on a wedged (D-state) git.
The hung-walk test stays green (a killed child exits by signal, not success, so it
still surfaces GitServiceTimeout and reaps the group) and the 32 real-git
classification tests stay green (a fast walk is never spuriously killed).
GITLAWB_GIT_SERVICE_TIMEOUT_SECS bounds the info/refs advertisement too:
smart_http::info_refs drives it through drive_git_child under this timeout, with a
passing test proving the 504. The old note claimed it does not. It also claimed the
withheld-blob path is unbounded; after the blob_paths seam bound (this PR) the walk
is bounded and reaped, by a fixed internal deadline rather than this env var, so the
line now states that precisely instead of overclaiming this setting covers it.
The withheld-blob classification walk (blob_paths) fanned out blocking git children
with no deadline and no process-group teardown: git for-each-ref, git cat-file, git
rev-list, a git ls-tree per commit, and an uncounted git rev-parse (via
store::head_commit). A hung or pathologically slow child pinned the caller's
served-git permit for the whole hang, and on client disconnect the spawn_blocking
task and its git children ran on, orphaned. blob_paths is the shared core of five
callers: the upload-pack serve path (holds a read permit) AND, inside
git_receive_pack, the post-push replication and encrypt-then-pin walks (hold the
write permit U2 reserves for pushes). So the same unbounded walk could pin either
pool, and leaving the write-side twin unbounded would have made U2's reservation a
claim that does not match behavior.
Bound every git child at the blob_paths spawn seam on the blocking side: each child
runs in its own process group with a watchdog thread that SIGTERMs (then SIGKILLs)
the group on one shared deadline spanning the whole walk, and retains admission
until the group is reaped. This is the blocking-side counterpart of
smart_http::drive_git_child (spawn_blocking cannot be cancelled by an async
timeout). blob_paths stays sync, so all five callers keep their signatures and the
32 classification tests are unchanged; because every caller funnels through
blob_paths, one seam bounds both the serve and replication paths. The previously
unbounded store::head_commit child becomes a bounded git rev-parse inside the walk.
A walk that hits its deadline carries GitServiceTimeout, which the serve handler now
maps to 504 rather than a generic 500.
Proof: a fake git that hangs on rev-list makes blob_paths return GitServiceTimeout
within the watchdog budget (not block on the child) and the recorded process-group
leader is reaped, not orphaned; neutralizing the watchdog kill makes it hang past
the budget (RED). The 32 real-git classification tests stay green through the
refactor, including detached-HEAD, non-standard-ref, and deleted-in-history cases.
git_info_refs acquired git_read_semaphore for BOTH services, so the push handshake
(GET /info/refs?service=git-receive-pack) competed in the global read pool. An
anonymous clone flood could exhaust that pool and shed a legitimate push with 503
during its required advertisement phase, before it ever reached git_write_semaphore
on the POST. The write pool exists precisely so anonymous reads cannot shed an
authenticated push, but only the POST drew from it.
Select the pool by service: the receive-pack advertisement (phase one of a push)
now draws from git_write_semaphore, like the git-receive-pack POST, so a saturated
read pool cannot starve it. The per-IP push_rate_limiter that already brakes the
advertisement stays as the anti-flood control, and the advertisement stays
reader-visible with no new auth requirement. Because the receive-pack branch is now
a write-path op, it no longer consumes a read per-caller slot.
Handler-layer proofs: with the read pool at zero the receive-pack advertisement
survives while the upload-pack advertisement sheds; with the write pool at zero the
receive-pack advertisement sheds while upload-pack is unaffected; and a receive-pack
advertisement from an IP whose read per-caller budget is full still gets through
(mutation-checked, RED when the skip is neutralized).
read_caller_key returned the authenticated DID when a caller signed, dropping the
source-IP key. Public read routes accept any valid did:key via optional_signature
with no admission step, so one host could mint N disposable DIDs and hold
max_concurrent_reads_per_caller slots under each, multiplying its budget N-fold and
filling the global read pool, while the same host unsigned was capped on its IP.
Key the read sub-cap on the resolved source IP for every caller, signed or not,
mirroring the push path's IpRateLimiter which already throttles on source IP for
this exact DID-farm reason. Drops the now-unused caller_did parameter at both call
sites (git_info_refs, git_upload_pack).
Inverts info_refs_per_caller_cap_keys_on_did_not_ip into
info_refs_per_caller_cap_keys_on_ip_not_did: fill one source IP's slot, then two
requests signed under different DIDs from that same IP both shed 503 (farm
defeated), while a signed request from a different IP keeps its own budget. RED on
the DID-keyed tree, GREEN after.
The read-pool knob was referenced by the push and per-caller entries'
comments but had no example line of its own, so operators couldn't
discover it from the template. Add it with the config default (128).
Review of the served-git concurrency cap found no P0/P1; these are the
verified P2 follow-ups.
config: the max_concurrent_git_ops doc overclaimed that "every capped path
is duration-bounded." The rev-list object enumeration in the withheld-blob
path still runs in an uncancellable spawn_blocking, so a stuck rev-list can
hold its slot until git exits. Scope the guarantee to the streaming stages
and name the residual. Also tighten the fairness claim: the receive-pack
advertisement shares the read pool (a shed advertisement is a cheap retryable
GET); only the push POST is on the isolated write pool.
api/repos: add info_refs_per_caller_cap_keys_on_did_not_ip, the missing
handler proof that a signed caller is keyed by its DID, not its source IP.
Filling the DID slot sheds a request from a free IP; collapsing read_caller_key
to its IP arm turns the assertion green-not-503 (mutation-verified RED).
api/repos: extract acquire_read_caller_permit so both read handlers share one
shed path instead of a duplicated match block.
rate_limit: recover from a poisoned PerCallerConcurrency mutex instead of
panicking. The critical section is pure counter arithmetic and cannot poison
the lock, but a panic there would brick the limiter for every caller.
505 tests pass; clippy -D warnings and fmt clean.
Part of #174.
The max_concurrent_git_ops and git_service_timeout_secs doc-comments (and .env.example) described the info/refs and withheld-blob paths as unbounded follow-up gaps. Both are now closed (#174): the comments reflect the read/write pool split, the per-caller sub-cap, and that every capped path is duration-bounded with process-group teardown. Verified the pattern-doc pre-ship checklist: every git_permit / write-permit / per-caller site holds only a timeout+teardown git path, and all three size knobs are range(1..=1_048_576).
Closes the #174 work. No behavior change.
The filtered-pack path ran the whole rev-list + pack-objects build inside a spawn_blocking, so an outer tokio timeout could not cancel the blocking thread and a client disconnect orphaned the git child while the permit freed (jatmn P1b, the second gap path). Split it: rev-list enumeration stays blocking off the runtime (rev_list_keep), but the streaming pack-objects stage now runs under the shared drive_git_child on the async side, so it is duration-bounded (GitServiceTimeout -> 504) and its process group is reaped on disconnect. build_filtered_pack becomes async and takes a git_bin seam + timeout; upload_pack_excluding threads the git_service_timeout through.
A watchdog-bounded test proves a hung pack-objects times out (rev-list fast, pack-objects hangs). The refactor's happy path is covered by the existing filtered-pack correctness and real-git partial-clone/fetch tests, all still green; disconnect/group-teardown is the shared drive_git_child code proven by the run_git_service tests. 504 tests pass.
Part of #174.
info_refs ran a bare Command::output() with no timeout and no process-group teardown, so a hung git pinned its concurrency slot indefinitely and a client disconnect orphaned the child (jatmn P1b). Extract the timeout + process_group(0) + KillGroupOnDrop core from run_git_service into a shared drive_git_child, and route info_refs through it with an injectable git_bin. A hung advertisement now aborts with GitServiceTimeout (mapped to 504); disconnect reaps the group.
run_git_service's teardown tests all pass through the shared core (proving the group teardown info_refs inherits), the real-git filter tests cover the advertisement happy path, and a new watchdog-bounded test proves a hung advertisement times out. 503 tests pass.
Part of #174.
Adds PerCallerConcurrency, a bounded-keyed in-flight limiter (distinct from the request-rate RateLimiter) so no single caller monopolizes the served-git read pool. Each caller (per-DID when signed via optional_signature, else per-source-IP via client_key) may hold at most max_concurrent_reads_per_caller concurrent reads; over that it sheds 503. The key map is self-bounding (a key is dropped when its in-flight count hits zero) with a reject-before-insert max_keys backstop so a key farm can't grow it (INV-15). Applied in git_upload_pack and both info/refs advertisements, acquired after the visibility gate so a denied request never consumes a slot (KTD7).
Primitive unit-tested (cap + self-bounding + reject-before-insert) and mutation-checked. Handler-layer SC2: same-caller sheds while a different caller passes, proven on BOTH git_info_refs and git_upload_pack with independent mutation probes; plus a None-key bypass test. Per-source-IP keying is trust-config dependent, documented on the config knob. 502 tests pass.
Part of #174.
git-receive-pack now draws from a separate git_write_semaphore (max_concurrent_git_pushes) instead of the shared pool, so a flood of anonymous reads can no longer shed an authenticated push at admission (jatmn P1a). The shared field is renamed git_read_semaphore and continues to gate upload-pack and both info/refs advertisements. The write permit stays above acquire_write so it precedes the Tigris fresh-acquire (INV-10).
Handler-layer tests: write-pool shed (503), and a cross-boundary proof that an exhausted read pool does NOT shed a push; both mutation-checked (routing receive-pack back to the read pool flips each RED). 497 tests pass.
Part of #174.
Add max_concurrent_git_pushes (default 32) and max_concurrent_reads_per_caller (default 16), both clap range(1..=1_048_576) so an oversized value is a clean CLI error, not a Semaphore::new boot panic. The per-caller knob documents that per-source-IP keying is only as granular as GITLAWB_TRUSTED_PROXY. Wiring lands in the following commits; these are the config surface for the #174 concurrency-fairness fix.
Resolves jatmn P1a/P1b groundwork on #174.
PR3 of the #62 served-git hardening stack (timeout #165 and teardown
wiring #150 are merged). A bounded semaphore caps how many upload-pack /
receive-pack / info-refs operations run at once; past the cap a request
is shed with a clean 503 + Retry-After before spawning another git
subprocess, instead of exhausting the PID/thread table. A permit is
acquired at the top of each of the three handlers and held for the whole
op, releasing on return.
The cap is a portable backstop: the compose pids_limit is absent on Fly,
whose 500-connection cap is a different axis. Size --max-concurrent-git-ops
(GITLAWB_MAX_CONCURRENT_GIT_OPS, default 128) below the process budget.
Range 1..=1_048_576 so 0 (shed everything) and an oversized value that
would panic tokio's Semaphore at boot are clean CLI errors.
Known gap, tracked separately: info/refs and the withheld-blob
(upload_pack_excluding) path are not duration-bounded and do not reap
their git child on client disconnect, so a hung git on those two paths
holds its slot until it exits and live git can briefly exceed the cap.
The main pack path (run_git_service) tears its group down on drop.
Tests: Overloaded maps to 503 + Retry-After; the config knob defaults
and rejects out-of-range; git_permit sheds at capacity and releases; and
each of the three endpoints sheds with 503 when the semaphore is
exhausted (load-bearing: drop the permit line and the endpoint test goes
red).
2026-07-10 12:10:30 -05:00
168 changed files with 107754 additions and 3861 deletions
This file carries the rules a patch author cannot recover from the code alone, for humans and coding agents alike. For build, test, and PR basics, read `CONTRIBUTING.md` first.
## Authorization invariants (the rules PRs most often break)
Every repo-scoped endpoint must bind the caller to an authorization decision before serving or mutating anything. The `known_ungated` allowlist in `authz_guard::every_repo_scoped_handler_is_gated` is empty on main: `list_webhooks`, `list_replicas`, and `list_protected_branches` are gated (`list_webhooks` with read-visibility then owner; the other two with `authorize_repo_read`). Do not re-open those surfaces as exceptions.
- Owner-only mutations (visibility, webhooks, protected branches, merges) are gated against the repo owner. Three gate forms are in use and all are recognized: webhooks and merges call `require_repo_owner` (`crates/gitlawb-node/src/api/mod.rs`), visibility uses its module-local `require_owner`, and protected branches compare inline with `did_matches`. Use the form the surrounding module already uses.
- Not every write is owner-only, and that is by design: `star_repo`/`unstar_repo` bind to the signer (they still require repo read access), `register_replica`/`unregister_replica` bind to the replica's own DID, the bounty actions (claim, submit, approve, cancel, dispute) have their own multi-party rules, and closing a PR or issue is owner-or-author. Do not add owner gates to these.
- Content-serving reads (blobs, trees, raw files) gate through `authorize_repo_read` with the specific path being served, so a withheld subtree is denied even on an otherwise-public repo; repo-level reads (listings) pass `"/"` by the helper's own contract. The source scans below cannot check this argument's value, so review will.
- A route that reaches repo data through a global id, with no repo in its path, or through an extractor shape the scans do not recognize still needs the same gate; the scans cannot see those.
- Read-surface denials must not reveal existence: a caller who may not read a repo (or a withheld subtree) gets the same 404 as a missing repo, never a 403. (Owner-gated mutations currently return 403 after the repo lookup; do not widen that shape to read paths.)
- Two tests in module `authz_guard` (`crates/gitlawb-node/src/api/mod.rs`) scan handler sources and fail on a repo-scoped handler without a recognized gate: `every_in_scope_mutation_has_its_gate` and `every_repo_scoped_handler_is_gated`. If they fail on your handler, add the gate. The second test's `known_ungated` allowlist is empty; never add an entry to it without an open issue and review.
## Adding or changing routes
- A new gated handler needs tests proving its route-specific gate fires: exercise an unauthorized authenticated caller and an anonymous caller where applicable, assert the exact denial status each way, and assert the body leaks nothing. The non-owner tests in `crates/gitlawb-node/src/test_support.rs` show the shape for owner-gated routes.
- A new mutation handler also gets a row in `every_in_scope_mutation_has_its_gate` naming its gate type; the per-handler row is the point of that test.
- A table-driven deny suite covering every deny-bearing route is in review (#194, #195). If it is present in your checkout, add a probe row there for any new deny-bearing route.
## Verifying signatures and certificates
- Derive the verifying key from something outside the artifact you are checking: the node's own DID, a pinned or configured value, a peers lookup. A key read from a field of the object being verified (`Did::from_str(&cert.node_did)`) only proves the artifact is self-consistent, and identities here are permissionless, so anyone can mint a keypair, name it in the artifact, self-sign, and be told it is valid. A mismatch against that anchor has to fail the result, not log a warning.
- A handler whose correct answer is "valid" needs a positive case driven by a trusted, independently anchored artifact, and a rejection case for a well-formed, fully populated forged artifact. Invalid-only suites miss an implementation that rejects everything; a forged artifact that still returns valid is the regression that matters.
- The fields a signature covers are a versioned format. Adding one invalidates every artifact already signed, so give the payload a version, keep a path that verifies the older form, and add a test that verifies an artifact signed before your change.
## Removing a serving path
When a fix removes a path that used to serve content, the same PR must add a test asserting the removed path now denies. The deletion is what turns that test green; without it the path can quietly come back.
## Database migrations
Schema changes are code-defined in `MIGRATIONS` (`crates/gitlawb-node/src/db/mod.rs`). Always append a new versioned entry; never edit an entry that has merged (operators deploy from main). Test runs build the schema from scratch, so editing an applied version stays green in CI while breaking every existing deployment.
## Client behavior (`gl`, `git-remote-gitlawb`)
When a node denies a request, surface the denial to the user. Never render a denial as an empty list or a silent success; that turns an authorization boundary into invisible data loss.
## Toolchain and CI reality
- MSRV is Rust 1.91 (`rust-version` in the workspace `Cargo.toml`). The MSRV CI job runs `cargo check --workspace --all-targets` on exactly 1.91, so check that an API you rely on is stable there; the full test suite runs on stable.
- The PR workflow runs these jobs on every PR: `cargo fmt --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo test --workspace` (against Postgres), a release build, `cargo audit`, the MSRV check, and a Docker build. The `main` branch ruleset currently requires one approving review but does not require these checks to pass before merge; a failing job should block review approval in practice, but nothing enforces that mechanically today.
- Conventional commit titles (`feat:`, `fix:`, `docs:`, ...) are required by convention and drive releases, but no CI job checks them; a bad title fails review, not the pipeline.
└── git-remote-gitlawb/ git remote helper for gitlawb:// URLs
├── git-remote-gitlawb/ git remote helper for gitlawb:// URLs
├── gitlawb-attest/ provenance attestations for ref-update certificates
└── icaptcha-client/ client for the iCaptcha proof-of-intelligence service
docs/ Operator guides
scripts/ Build helpers
```
Smart contracts live in a separate repo: [github.com/Gitlawb/contracts](https://github.com/Gitlawb/contracts).
Smart contracts live in a separate repo: [github.com/Twigpine/contracts](https://github.com/Twigpine/contracts).
## Ways to contribute
@@ -54,6 +56,7 @@ Smart contracts live in a separate repo: [github.com/Gitlawb/contracts](https://
3.**No breaking changes without discussion.** Open an issue first for protocol-level changes.
4.**Conventional commits.** Use `feat:`, `fix:`, `docs:`, `refactor:`, `test:`, `chore:`. Releases are automated by [release-please](https://github.com/googleapis/release-please) — your commit prefixes drive the next version bump.
5.**Format and lint.** Run `cargo fmt --all` and `cargo clippy --workspace --all-targets -- -D warnings` before submitting. CI will reject anything that fails these.
6.**Security-surface rules.** [`AGENTS.md`](AGENTS.md) carries the authorization invariants PRs most often break; it is written for coding agents and humans alike, so read it before touching endpoints, visibility, or migrations.
## What gets merged, what gets closed
@@ -105,7 +108,7 @@ cargo test -p gitlawb-node
- **TypeScript SDK** (`@gitlawb/sdk`) — client library for the HTTP API
- **Python SDK** (`gitlawb`) — for ML/agent pipeline integration
- **UCAN chain validation** — complete the auth middleware
**Decentralized git infrastructure for developers, AI agents, and app delivery.**
Gitlawb Node is the open-source node software behind the Gitlawb network. It lets anyone run a self-hosted node, publish repositories under a DID, sign writes with Ed25519 HTTP signatures, replicate git activity across peers, and move toward a resilient app-delivery network where code and build assets can be served closer to users.
Twigpine Node is the open-source node software behind the Twigpine network. It lets anyone run a self-hosted node, publish repositories under a DID, sign writes with Ed25519 HTTP signatures, replicate git activity across peers, and move toward a resilient app-delivery network where code and build assets can be served closer to users.
Gitlawb is not trying to be only “another git host.” The long-term direction is:
Twigpine is not trying to be only "another git host." The long-term direction is:
```txt
Decentralized GitHub
@@ -19,33 +19,35 @@ The mission is simple: once code is pushed to the network, it should not disappe
| `icaptcha-client` | Client for the iCaptcha proof-of-intelligence flow used to protect spam-prone writes. |
---
## Why Gitlawb Nodes?
## Why Twigpine Nodes?
Most git hosting today depends on a small number of centralized platforms. Gitlawb Nodes are designed for a different model:
Most git hosting today depends on a small number of centralized platforms. Twigpine Nodes are designed for a different model:
- **Own your identity** — every user, agent, and node is an Ed25519 keypair represented as `did:key:z6Mk...`.
- **Signed writes by default** — write requests use RFC 9421 HTTP Signatures instead of passwords.
- **Git-native transport** — repositories are still real git repositories served over smart HTTP.
- **Agent-native workflows** — the `gl` CLI and MCP server expose repo, issue, task, PR, and UCAN flows to AI agents.
- **Peer-aware delivery** — nodes can announce, discover, gossip, and sync with each other.
- **App CDN direction** — the network can evolve from decentralized code storage into code + asset + app delivery.
- **Own your identity**: every user, agent, and node is an Ed25519 keypair represented as `did:key:z6Mk...`.
- **Signed writes by default**: write requests use RFC 9421 HTTP Signatures instead of passwords.
- **Git-native transport**: repositories are still real git repositories served over smart HTTP.
- **Agent-native workflows**: the `gl` CLI and MCP server expose repo, issue, task, PR, and UCAN flows to AI agents.
- **Peer-aware delivery**: nodes can announce, discover, gossip, and sync with each other.
- **App CDN direction**: the network can evolve from decentralized code storage into code + asset + app delivery.
---
## Current status
Gitlawb Node is live early infrastructure. It is useful today, but some security and reliability features are intentionally staged for compatibility with existing nodes.
Twigpine Node is live early infrastructure. It is useful today, but some security and reliability features are intentionally staged for compatibility with existing nodes.
Good today:
@@ -54,6 +56,7 @@ Good today:
- Bare git repository storage.
- Git smart-HTTP clone/fetch/push.
- RFC 9421-signed writes.
- Repository and path-scoped visibility enforcement for repository and Git content reads, with 404-shaped repository denials.
- DID identities.
-`gl` CLI workflows.
- libp2p peer discovery/gossip foundation.
@@ -63,11 +66,13 @@ Good today:
Known limitations:
-Private repository read enforcement is not wired yet. Treat public nodes as public infrastructure unless you restrict access at your proxy/firewall.
- UCAN chain validation and revocation are not complete.
-Repository write authorization is not capability-complete yet; HTTP signatures prove identity, not full authorization policy.
-Repository write authorization is secure by default: `GITLAWB_ENFORCE_OWNER_PUSH` defaults to `true`, so a valid HTTP Signature identifies a pusher and the node additionally requires that pusher to be the repository owner. Set `GITLAWB_ENFORCE_OWNER_PUSH=false` only for the duration of a rolling upgrade whose pushers are not yet the repo owner. Authentication is not authorization on its own (anyone can mint a `did:key` and sign), so owner enforcement is the only thing that prevents every signed caller from pushing to every repository, private ones included.
- UCAN proof chains are validated when supplied, but UCAN capabilities are not consulted by write authorization and the root issuer is not independently trust-anchored. UCANs therefore do not yet grant scoped collaborator access.
-Agent lifecycle revocation is not enforced by HTTP Signature authorization; do not rely on removing or revoking an agent record to block a compromised signer.
- Read visibility is not a blanket data-classification boundary: task, IPFS-pin, and Arweave-anchor listings are not repository-gated; withheld path names can be visible to a root reader; and later visibility changes cannot retract content already announced or externally anchored.
- Peer writes are signed by upgraded nodes, but strict signed-peer enforcement is opt-in during rolling upgrades.
- GraphQL mutations need mutation-aware auth before becoming a public write surface.
- GraphQL mutations require a signed identity (a verified signer bound to the acting DID), are limited to the agent-task queue, and are covered by a CI completeness fence that requires signer acquisition and actor binding on every registered mutation; repo writes are not exposed over GraphQL and remain REST-only.
- Pull-request review comments do not yet have threaded line-level anchors, and merges do not enforce approval requirements.
See:
@@ -82,7 +87,7 @@ See:
The fastest path is Docker Compose. It starts a node and Postgres.
git push origin2 main # origin2 = gitlawb://<your-did>/memlawb (signed)
git clone gitlawb://<your-did>/memlawb # public read, no proof needed
gl doctor # preflight: identity, node, version, iCaptcha
@@ -215,14 +220,61 @@ Notes:
- **Proofs are short-lived (~5 min TTL) and single-use.** If one expires between
solving and use, the client transparently solves a fresh one and retries.
- **What needs what:** create / fork / register are signed **and** iCaptcha-gated;
`git push` is **signed-only** (owner signature is the gate — no per-push challenge);
`git push` is **signed-only** (owner signature is the gate, no per-push challenge);
reads (clone / fetch / `repo info`) need no proof. A non-existent repo returns a
clear `404`, never a placeholder.
- **API-key iCaptcha deployments:** set `GITLAWB_ICAPTCHA_URL` to your iCaptcha
origin and `GITLAWB_ICAPTCHA_API_KEY` to its key. The client only talks to an
`https` origin whose host is allowlisted (that URL or the public default), and
sends the bearer token **only** to your configured origin — never to a URL a
node advertises — so a hostile node can't capture the key or redirect the solve.
sends the bearer token **only** to your configured origin, never to a URL a
node advertises, so a hostile node can't capture the key or redirect the solve.
### Fetching an object by CID
```bash
gl ipfs list # CIDs this node has pinned
gl ipfs get bafkrei... > object.bin # object bytes on stdout
```
Objects that were pinned before the node started recording which repo they came
from are found by scanning its repo inventory, and that scan stops at the
per-request ceilings in the [Configuration](#configuration) table. A stopped scan
answers 503 with a resume token instead of a false "not found", and `gl ipfs get`
follows the token automatically: up to 8 resumes after the first request, so at
most 9 calls to the node, waiting between attempts for as long as the node's
`Retry-After` asks and never longer than 5 seconds.
The whole ladder runs under a 60 second wall-clock deadline. The deadline bounds
the search, not the download: each attempt gets the time left on it to produce
response headers, and once an object is found its bytes stream outside that
deadline. They are not unbounded, though. The client's own 30 second HTTP timeout
is a total request timeout, running from the start of a request until its body
has finished, so a transfer still going 30 seconds after its request began is cut
off. Waits between attempts never run past the deadline either, so a single run
spends at most around 90 seconds on the network: the deadline plus the 30 second
timeout covering the last attempt. Writing the object out sits outside both
bounds, so piping into a reader that stops reading can hold the command open
longer than that.
Two node-side brakes end a ladder early and are reported rather than retried
around. A 429 is terminal, because the node's rate-limit window is an hour and
that wait cannot be honored inside one invocation; a transient overload (a 503
carrying no incomplete-scan code) is retried on the token already held, under the
same cap, clamp and deadline. The per-IP fanout brake can also stop a ladder well
short of the 9 calls, so automatic resumption is not a guarantee of reaching the
object.
When a bound stops the ladder with a usable token in hand, the command prints the
token and the invocation that continues from it before exiting nonzero:
```txt
resume from where this stopped: gl ipfs get bafkrei... --scan <token>
```
Run that to carry on from where the scan stopped. Re-running without `--scan`
restarts at the first row, reproduces the same truncation and spends the node's
per-IP budget again, so the token is the only thing that makes progress. Tokens
are valid for an hour.
---
@@ -309,7 +361,7 @@ POST /api/v1/peers/announce
POST /api/v1/sync/notify
```
When `GITLAWB_REQUIRE_SIGNED_PEER_WRITES=false`, unsigned legacy peers are accepted on those two routes, but signed requests are verified when signature headers are present. Once all live peers upgrade, operators can set:
When `GITLAWB_REQUIRE_SIGNED_PEER_WRITES=false`, unsigned legacy peers are accepted on those two routes, but signed requests are verified when signature headers are present. Staged rollout relaxes who may announce, not who owns a peer row. An unsigned announce may register a previously unseen peer, and it may refresh an existing row whose `http_url` is unchanged, but changing an existing peer's `http_url` requires an RFC 9421 signature from that peer's own DID and is refused with 403 otherwise. An unsigned announce can only register a `did:key`, since that is the only method whose verifying key can be resolved and therefore the only kind of row anyone could ever correct through the signed path; any other DID is refused with 400. Once all live peers upgrade, operators can set:
`POST /api/v1/sync/trigger` is not part of the staged rollout: it always requires a signature in both config modes and returns 401 without one, because each call drives an O(peers) outbound fan-out.
The flag is not HTTP-only. It also gates inbound gossip ref-update events on the libp2p mesh, which carry an Ed25519 payload signature rather than RFC 9421 headers because there are no HTTP headers to sign. Set to `true`, an unsigned gossip event is dropped. Set to `false`, an unsigned event is accepted with a warning during the rolling-upgrade window, and the write budget it consumes is charged to the forwarding peer rather than to the DID it claims, since an unsigned event's claimed DID is asserted and not proven. A present-but-invalid signature is refused in both modes; that is forgery, not an un-upgraded peer.
Because the flag now spans both transports, the rollout order matters in one direction: **upgrade every gossip-publishing peer to a build that signs events before you set this to `true`.** Enabling it while an old publisher is still live drops that publisher's ref-updates on arrival, and the publisher sees no error, because gossip has no response to carry one. Upgrading the HTTP peers alone is not sufficient.
There is a second precondition on gossip ingest that the flag does not control. A publisher's `node_did` must already exist in the receiving node's peers table, or the event is dropped as an unknown peer DID. This holds in both modes, signed and unsigned alike: a valid signature proves key possession, not membership. Rows reach that table over HTTP, either from `GITLAWB_BOOTSTRAP_PEERS` when this node contacts a bootstrap peer and records the DID it reports, or from a prior `POST /api/v1/peers/announce`. So a peer that only ever joined the libp2p mesh, with no HTTP announce and no bootstrap contact in either direction, will have its ref-updates dropped even with a good signature and the flag off. If gossip is silently not landing from a peer you can see on the mesh, check that its DID is in `GET /api/v1/peers` on the receiving side first.
---
## Configuration
@@ -339,22 +397,63 @@ Important node settings:
| `GITLAWB_P2P_PORT` | libp2p QUIC/UDP port. Use `0` to disable. |
| `GITLAWB_P2P_KEY` | Path to the node's persistent libp2p identity key file, which fixes the PeerId across restarts. Must name a key file inside a directory, such as `/data/keys/p2p.key` or `./keys/p2p.key`. Two kinds of problem are handled differently. A value that cannot name a securable key file at all (a bare filename, `..` traversal, a trailing `/`, the filesystem root, or a `~/` path that escapes home) is refused before the node binds, so the process exits. Anything wrong with the storage itself (a symlinked or non-directory parent, an unsafe ancestor or working directory, an unreadable or malformed key) leaves p2p off while HTTP keeps serving; the node logs it at error level as `p2p_identity_key_load_failed` and sets the `gitlawb_p2p_identity_key_load_failed` metric to 1. Note that `/health` still reports healthy in that state, so alert on the log event or the metric rather than on the health check. Correcting the storage and restarting restores the same PeerId. On Unix a new key is created `0600` inside a `0700` directory. A key directory that grants access beyond its owner is tightened to `0700` on start; one that is closed too far is refused with a `chmod 700` remedy rather than widened, since a directory locked down on purpose is an operator decision. Rotate the key only when the node logs that the directory was group- or world-writable (or the key file itself was group- or world-readable). An ordinary `0755` directory is tightened too, but others could not read a `0600` key or replace its directory entry. An existing key file is never modified: one with group or other bits set is rejected (`chmod 600`), and so is one this node cannot read. On other platforms no permissions are enforced. Default `~/.gitlawb/p2p.key`; point it at a persistent volume when running in a container. |
| `GITLAWB_REQUIRE_SIGNED_PEER_WRITES` | Require signed peer announce/sync writes. |
| `GITLAWB_REQUIRE_SIGNED_PEER_WRITES` | Require peer writes to prove their DID on every transport: RFC 9421 signatures on the announce/sync routes, and an Ed25519 payload signature on gossip ref-update events. Defaults to `false` during the staged rollout below. Upgrade every gossip publisher before enabling, or their updates are dropped with no error on their side. |
| `GITLAWB_ENFORCE_OWNER_PUSH` | Require the authenticated pusher to be the repo owner on `git-receive-pack`. **Defaults to `true`.** A `did:key` signature is authentication, not authorization (anyone can mint a key and sign), so with this off every signed caller may push to every repository, private ones included. Delegated and CI keys count as non-owners: a UCAN `git/push` capability is verified but not yet honored for authorization, so they cannot push while this is on. Set `false` only for a rolling upgrade; see [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md). |
| `GITLAWB_AUTO_SYNC` | Enable automatic sync from known peers. |
| `GITLAWB_MAX_PACK_BYTES` | Max git pack body size for smart-HTTP routes. |
| `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack/receive-pack may run before it is aborted (504). Default 600. Does not bound `info/refs` or the withheld-blob path. |
| `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack, receive-pack, or `info/refs` advertisement may run before it is aborted (504). Default 600. Also bounds the withheld-blob classification walk (on both the upload-pack serve and receive-pack replication paths) and the push-side pin-candidate discovery (`rev-list` / `cat-file`), each reaped via process-group teardown at the deadline. On the path-scoped upload-pack path the classification walk and the pack serve share ONE deadline, so this value bounds their combined duration rather than granting each stage a full budget: a walk that consumes it leaves the serve nothing and the clone gets a 504. Serving large path-scoped repos may therefore need a higher value than they did when each stage was budgeted separately. Accepted range is 1 to 3153600000 (100 years), since the node derives deadlines from this value and a larger one cannot be represented. |
| `GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS` | Max seconds the storage-acquisition phase (Tigris HEAD/GET, push advisory-lock) of a served git op may run before the request is shed with a 503, separate from the git-run timeout. The concurrency permit is released on expiry so a stalled backend cannot pin the pool. Default 30. |
| `GITLAWB_MAX_CONCURRENT_GIT_OPS` | Max concurrent served git READ ops (upload-pack and its `info/refs` advertisement) across all callers; over-cap sheds a 503 + Retry-After. Anonymous reads draw from this pool, so pair it with `GITLAWB_MAX_CONCURRENT_READS_PER_CALLER`. Pushes and the receive-pack advertisement have their own pools, so a read flood cannot shed an authenticated push. Default 128. |
| `GITLAWB_MAX_CONCURRENT_GIT_PUSHES` | Max concurrent `git-receive-pack` POST operations, in a pool separate from the read pool. The anon receive-pack `info/refs` advertisement runs in a third pool of the same size, disjoint from both, so an advertisement flood cannot shed a push either. Two per-source push caps are derived from this value (`/8`, floor 1) and have no env var of their own. Over-cap sheds a 503 + Retry-After. Default 32. |
| `GITLAWB_MAX_CONCURRENT_READS_PER_CALLER` | Max concurrent read ops a single caller may hold, so one caller cannot monopolize the read pool. Keyed on the resolved source IP, never the DID, and only as granular as `GITLAWB_TRUSTED_PROXY`: left unset, a node behind an edge or NAT keys every caller on the edge IP and this collapses to one global cap. Default 16. |
| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | Max post-push pin loops (IPFS + Pinata) running concurrently across all repos. This caps how many loops RUN at once, not how much object-id list memory the node retains: on the local IPFS path a loop parked waiting for a permit still holds its full list. Do not size memory from this knob alone. A loop over cap waits, never drops a pin. Default 8. |
| `GITLAWB_REPO_LEASE_MAX_WAITERS` | Max pushes parked at once waiting for the same repo's write lease. Each waiter pins its buffered pack body, so this bounds that memory for a hot repo; past the cap the newest push sheds a 503 + Retry-After instead of queueing. Pushes to other repos are unaffected, and the lease holder is not counted. Default 8. |
| `GITLAWB_MAX_CONCURRENT_IPFS_WALKS` | Max concurrent `GET /ipfs/{cid}` visibility walks across all callers (own pool, disjoint from the served-git pools); over-cap sheds 503. Default 32. |
| `GITLAWB_IPFS_WALK_PER_SOURCE` | Max concurrent `/ipfs` walks a single source IP may hold. Default 4. |
| `GITLAWB_IPFS_MAX_LEGACY_PROBES` | Max legacy (NULL-provenance) repos probed per `/ipfs/{cid}` request, bounding the scan-fallback fan-out. A truncated scan returns a retryable 503, not a false 404. Default 256. |
| `GITLAWB_IPFS_MAX_LEGACY_SCAN_ROWS` | Max repo rows one `/ipfs/{cid}` request's legacy scan may read from the database. The probe ceiling above only starts counting once a probe runs, and quarantined or private repos are denied before that, so this is what bounds a scan over an inventory that denies the caller everywhere. A truncated scan sheds a retryable 503 carrying an opaque `continuation` token; echoing it as `?scan=` resumes the scan where it stopped. Every per-request ceiling on this path (rows, probes, visits, retained rule bytes) mints one, so each request advances the ladder by at least the rows it read and a holder buried past a ceiling is reached in a bounded number of requests, `ceil(repos / ceiling) + 1` when this row ceiling is the one that binds. No ceiling ever produces a 404. Every page is charged to the caller's `/ipfs` work allowance, so raising this raises that allowance too. Lowering it sharpens an oracle: because a truncation emits a token and a completed wrap does not, laddering to the end reveals the node's total repo count (private and quarantined included) to within one ceiling. Default 2048. |
| `GITLAWB_IPFS_MAX_REPOS_WALKED` | Max expensive path-scope visibility walks a `/ipfs/{cid}` request may run per phase; over-cap repos are skipped and the scan continues, shedding a retryable 503 (not a false 404) if the object is then found nowhere. The effective cap is the tighter of this knob and the node's internal history-walk ceiling of 17 (`MAX_PIN_SOURCES + 1`), so a value above 17 has no effect while a value below it does tighten the cap. It is charged per phase: the provenance lookup and the legacy-scan fallback get separate equal budgets, so one request can run up to twice the cap in total. Default 64. |
| `GITLAWB_IPFS_MAX_REPO_VISITS` | Ceiling on repos one `/ipfs/{cid}` request may visit (acquire + probe) past the visibility gate. Also the worst-case per-request Tigris fetch count. On exhaustion the scan stops with a retryable 503. Default 1024. |
| `GITLAWB_IPFS_REQUEST_BUDGET_SECS` | Absolute wall-clock budget for one admitted `/ipfs/{cid}` request's acquire+walk lifetime. Per-stage clamps bound the acquire and walk stages to the remaining budget, and no stage starts once it is exhausted; the scan then stops with a retryable 503. The object-type probe and content-read `cat-file` subprocesses are budget-checked before starting and each also run under their own deadline (the lesser of `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` and the remaining budget), reaped via process-group teardown, so a hung `cat-file` cannot hold the request's walk slot past it. One hang path is still unbounded: the probe's object-store readability check is a plain filesystem sweep with nothing to reap, so a wedged filesystem can hold the slot past the deadline. Default 600. Accepted range is 1 to 3153600000 (100 years), since the node derives a deadline from this value and a larger one cannot be represented. |
| `GITLAWB_IPFS_RESOLVE_BUDGET_SECS` | Shorter budget for the pre-walk CID resolve inside an admitted `/ipfs/{cid}` request: the lookup that maps the requested CID to its git oid(s), which runs while the scarce walk admission is already held. A well-formed CID with no pin row does no probe and no walk work, so without this it could hold a walk slot for the whole request budget while nothing walked, and enough such requests shed every real retrieval at admission. The effective deadline is the lesser of this and the remaining request budget, so a value above `GITLAWB_IPFS_REQUEST_BUDGET_SECS` degrades to the request budget. Only the resolve is on this clock; walk and probe work stay on the request budget, so a slow but progressing scan is never shed by it. Default 10. Accepted range is 1 to 3153600000 (100 years). |
| `GITLAWB_IPFS_RATE_LIMIT` | Max `/ipfs/{cid}` requests per client IP per hour (route flood brake). 0 disables. Default 600. |
Production note: change the default Postgres password before exposing a node publicly.
Legacy-pin window: releases before the CID-resolver work stored the provider CID (Kubo dag-pb / Pinata) as a pinned object's resolver key. The `/ipfs/{cid}` resolver now recomputes the raw-content CID from the object bytes and refuses to serve a key that does not match, so `GET /api/v1/ipfs/pins` can still advertise an unrepaired legacy CID that 404s. Such a row is repaired opportunistically the next time a push carries the object again (its key is rewritten to the raw CID, the old value kept in `legacy_provider_cid`), but git negotiation omits objects the node already has, so most legacy rows never re-enter a push delta. A deferred one-shot startup sweep, not this opportunistic path, is what fully retires the advertise-then-404 window. Rows whose object bytes are gone stay withheld.
### Upgrading: the PeerId rotates once
This node's libp2p identity is now a keypair generated on first start and kept
at `GITLAWB_P2P_KEY`, rather than one derived from the node DID. Every node
therefore gets a new PeerId once, on the first start after upgrading, and keeps
it from then on as long as that key file survives (put it on a persistent volume
in a container).
Two things to check before upgrading:
- Any `GITLAWB_P2P_BOOTSTRAP` multiaddr that pins a peer's old PeerId with a
`/p2p/<PeerId>` suffix stops matching once that peer upgrades. Update the
suffix, or drop it and let identify supply the current one. Addresses without
the suffix keep working untouched.
-`GITLAWB_P2P_KEY` must name a key file inside a directory, such as
`/data/keys/p2p.key` or `./keys/p2p.key`. A bare filename is refused at
startup, since the node will not keep its identity key in the working
directory.
Peers found over `GITLAWB_BOOTSTRAP_PEERS` and the embedded seed list are
unaffected, since those are HTTP URLs and carry no PeerId.
---
## Optional node staking
Gitlawb Node includes optional Base L2 node-operator hooks. Operators can register a node DID, stake `$GITLAWB`, and post heartbeats.
Twigpine Node includes optional Base L2 node-operator hooks. Operators can register a node DID, stake `$GITLAWB`, and post heartbeats.
PoS is disabled unless these are configured:
@@ -424,8 +523,8 @@ Build:
Output:
```txt
dist/Gitlawb Node.app
dist/GitlawbNode.dmg
dist/Twigpine Node.app
dist/GitlawbNode-macOS.dmg
```
Features:
@@ -439,7 +538,7 @@ Features:
Unsigned local build:
```bash
xattr -cr "dist/Gitlawb Node.app"
xattr -cr "dist/Twigpine Node.app"
```
---
@@ -454,8 +553,8 @@ Short-term priorities:
2. Add Docker and installer smoke tests.
3. Improve operator docs and `gl doctor` checks.
4. Harden peer writes and publish the signed-peer rollout plan.
- Bootstrap UCAN tokens are issued at registration.
-A supplied token's signature, audience, expiry, and proof-chain attenuation are validated.
-Tokens use a signed JSON wire format with expiry.
- Capability grants are not yet consulted by repository write authorization; see the limitations below.
**Smart contracts (Base Sepolia testnet)**
-`GitlawbDIDRegistry` — on-chain DID → document registry
@@ -48,21 +49,20 @@ We will acknowledge receipt within 48 hours and aim to release a fix within 14 d
---
## Known Limitations (Planned for v0.2)
## Known Limitations
These are **documented, accepted limitations** for the current live release and should be prioritized without breaking existing nodes during rolling upgrades.
These are documented limitations of the current live release. They should be prioritized without breaking existing nodes during rolling upgrades.
### UCAN chain validation
-The auth middleware verifies HTTP Signatures and token structure, but does not yet walk the full UCAN delegation chain.
- **Impact:** A node cannot yet enforce fine-grained capability delegation. Currently, any registered agent with a valid HTTP Signature can push.
- **Mitigation:** Keep write endpoints signed, treat public nodes as public infrastructure, and treat trust scores as soft rate-limiting signals rather than authorization.
- **Fix target:** v0.2
### Repository write authorization defaults
-`git-receive-pack` verifies HTTP Signatures and, with the default configuration, also requires the pusher DID to be the repository owner: `GITLAWB_ENFORCE_OWNER_PUSH` defaults to `true`.
- **Impact:** With the default setting, a valid signature authenticates the pusher *and* the pusher must be the repository owner. Authentication is not authorization on its own — anyone can mint a `did:key` and sign — so owner enforcement is the only thing that prevents every signed caller from pushing to every repository, including private ones.
- **Mitigation:** `GITLAWB_ENFORCE_OWNER_PUSH=false` is the explicit rolling-upgrade compatibility override. Disable it only for the duration of an upgrade whose pushers are not yet the repo owner, then turn it back on. Delegated and CI keys count as non-owners: a UCAN `git/push` capability is verified but not yet honored for authorization, so they cannot push while this is on.
### UCAN revocation
-Issued UCAN tokens cannot be revoked before expiry.
-**Impact:** If a keypair is compromised, the attacker retains access until the UCAN expires (default: 30 days).
- **Mitigation:** Regenerate your identity (`gl identity new --force`) and re-register to issue a new UCAN. Until revocation/blocklisting is implemented, operators should remove compromised DIDs directly from their local database.
- **Fix target:** v0.2
### UCAN delegation and revocation
-The middleware validates a supplied UCAN's complete proof chain, but a root token is accepted without an independently trusted issuer anchor. `Ucan::can` is not yet used by write handlers, so a UCAN does not grant scoped repository access.
-Agent lifecycle revocation is not checked by HTTP Signature authorization. Removing or revoking an agent record does not itself block a compromised DID from authenticating.
- **Impact:** Do not use UCANs for collaborator permissions or rely on agent-record revocation as a key-compromise response.
- **Mitigation:** Keep sensitive deployments behind operational network controls and enable owner-only push enforcement where it fits the deployment until trusted delegation and authorization-aware revocation are implemented.
### git-receive-pack authentication
- The `git-receive-pack` endpoint enforces HTTP Signature auth. Plain Git smart-HTTP clients do not generate those headers, so the `git-remote-gitlawb` helper is required for pushes.
@@ -71,10 +71,20 @@ These are **documented, accepted limitations** for the current live release and
- **Fix target:** v0.2
### Private repository reads
- Repository records have an `is_public` field and the node exposes `GITLAWB_PUBLIC_READ`, but per-repository private-read enforcement is not wired in the current live release.
-**Impact:** Do not store private repositories or secrets on public nodes.
-**Mitigation:** Run isolated nodes for non-public data and restrict network access at the reverse proxy or firewall layer.
-**Fix target:** v0.2
- Repository and path-scoped visibility checks are enforced for repository API and Git content reads. A denied whole-repository or root read returns the same 404 shape as a missing repository, so the denial does not reveal private-repository existence.
-Sparse-clone support exposes withheld path globs to callers who may read the repository root. Do not put sensitive information in withheld path names.
-`GET /api/v1/tasks`, `/api/v1/ipfs/pins`, and `/api/v1/arweave/anchors` are not repository-gated. Task records include a UCAN token; pin and anchor listings expose object and ref metadata.
-Changing a repository's visibility controls future serving, but cannot retract ref metadata or configured external pins and anchors already announced while the repository was public. Do not push secrets to an announceable repository.
- **Impact:** Visibility policies protect the repository and Git content routes they gate, not every metadata endpoint or previously published content.
- **Remaining boundary:** This read control does not address the independent write-authorization and UCAN-delegation limitations described above.
### Pull-request review enforcement
- Pull-request review comments are not yet threaded or line-anchored, and merges do not enforce required approvals.
- **Impact:** Teams must use their own review policy or external controls for merge approval requirements.
### GraphQL mutation coverage
- Existing GraphQL mutations require an authenticated signer, but a mutation-specific source-level guardrail has not yet been added for future mutations.
- **Impact:** A new mutation could accidentally omit its signer check without an explicit test fence.
### Peer route hardening rollout
- Peer announce and sync notification routes accept signed requests and verify DID matches when a signature is present.
@@ -101,20 +111,20 @@ These are **documented, accepted limitations** for the current live release and
// Prove the mocked route was actually requested; a non-matching request (mockito's 501, also non-2xx) would otherwise satisfy the error assertion vacuously.
// Prove the mocked route was actually requested; a non-matching request (mockito's 501, also non-2xx) would otherwise satisfy the error assertion vacuously.
// Prove the mocked route was actually requested; a non-matching request (mockito's 501, also non-2xx) would otherwise satisfy the error assertion vacuously.
// Prove the mocked route was actually requested; a non-matching request (mockito's 501, also non-2xx) would otherwise satisfy the error assertion vacuously.
_m.assert_async().await;
}
#[tokio::test]
@@ -544,4 +548,86 @@ mod tests {
cmd_stats(server.url()).await.unwrap();
}
/// #186 (F1): a node denial on the stats endpoint must surface as an error,
/// not print a fake `Bounty Stats` with every counter zeroed. RED before the
/// fix (unwrap_or_default + no status check -> Ok with zeros), GREEN after
// Prove the mocked route was actually requested; a non-matching request (mockito's 501, also non-2xx) would otherwise satisfy the error assertion vacuously.
assert!(err.to_string().contains("issue not found"),"got: {err}");
// Prove the mocked route was actually requested; a non-matching request (mockito's 501, also non-2xx) would otherwise satisfy the error assertion vacuously.
// Prove the mocked route was actually requested; a non-matching request (mockito's 501, also non-2xx) would otherwise satisfy the error assertion vacuously.
_m.assert_async().await;
}
#[tokio::test]
@@ -670,6 +701,7 @@ mod tests {
.with_status(404)
.with_header("content-type","application/json")
.with_body(r#"{"message":"issue not found"}"#)
.expect(1)
.create_async()
.await;
@@ -683,6 +715,8 @@ mod tests {
.await
.unwrap_err();
assert!(err.to_string().contains("issue not found"),"got: {err}");
// Prove the mocked route was actually requested; a non-matching request (mockito's 501, also non-2xx) would otherwise satisfy the error assertion vacuously.
_m.assert_async().await;
}
#[tokio::test]
@@ -740,4 +774,48 @@ mod tests {
.await
.unwrap();
}
#[tokio::test]
asyncfncmd_list_surfaces_denial_not_empty(){
// A gated 404 on the issue-list read must Err, not print "No issues".
File diff suppressed because it is too large.
Load diff
Loaded 100 of 168 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.