Under-lock GET failures now always shed as RepoUnavailable, even when a local
copy exists, so a transient download error cannot publish over a newer stored
generation. PublishDurabilitySlot only synthesizes UploadUnknowable after
release starts, and receive-pack marks that boundary before spawning the tail.
Adds regression tests for both paths; inv22 F4 gate accepts the release wrapper.
GITLAWB_IPFS_MAX_LEGACY_PROBES bounded the probe fan-out, but nothing bounded
the load that ran before the first probe. On a legacy (NULL-provenance) or
capped/incomplete provenance set, get_by_cid called list_all_repos, cloned every
id, fetched visibility rules for the whole set, and fetched the node's entire
quarantine list, then retained all three for the request. /ipfs/{cid} is
anon-reachable and CIDs are enumerable from the unsigned pins index, and an
object pinned from MAX_PIN_SOURCES repos is permanently at cap, so every later
anonymous GET for it paid that whole preload with the scarce walk permits held,
even at a probe budget of 1.
The selection is now keyset-paged on (created_at, id) ASC, with the quarantine
flag returned per row so the separate whole-node quarantine query is gone. Both
key columns are immutable, so paging is exact: a repo touched mid-scan cannot
cross a page boundary and go unvisited, and updated_at is attacker-bumpable,
which would have let a caller sort their own repos ahead of the true holder.
The pager lives outside the oid loop, so its cursor and its accounting are per
request rather than per candidate. Before fetching another page the scan checks
that a row on it could still reach a verdict; a spent probe or visit budget
stops the read and TAINTS, so a truncated search still sheds the retryable 503
and never a false 404. A page of pure denials is deliberately not a stop
condition, since a cheap denial costs no probe and a public object buried behind
private repos must still be reached. Both page queries stay clamped to the
remaining request budget, and the rules query still fails closed.
A new v25 migration adds idx_repos_created_at_id. Bounding what the handler
materializes is only half the property: repos carried no index in the new keyset
order, so Postgres seq-scanned the table and top-N sorted it to return every
page while the walk admission was held, which moves the same O(rows) cost onto
the database rather than removing it. Measured on 50k rows: 951 shared buffers
and ~44ms per page without the index, versus an Index Only Scan at 7 buffers and
~0.09ms with it, and the keyset predicate pushed down as an Index Cond instead
of filtering after a scan. Nothing names the index in any query text, so its
entry says why it exists and a test asserts its presence structurally.
Two order-dependent tests were reseeded rather than left green: the buried-row
and acquire-taint cases encoded their ordering through updated_at, which no
longer determines iteration. Both were re-confirmed RED under their documented
mutations after the reseed. Three regressions land with the fix. One drives a
one-probe, one-row-per-page request against three candidate repos and asserts it
materializes one row and runs one query, which a query counter alone could not
have caught. One seeds two distinct source-less oids under a single CID and pins
the per-request property directly: the second candidate re-reads the pages the
first paid for, so the whole request costs one pass rather than one per
candidate. The third covers the index, including the INV-7 upgrade path.
Refs Gitlawb/node#173.
Three gates scanned the "production half" of api/repos.rs by splitting at the
first cfg(test) attribute. Adding test-only items above the code they check
moved that split point above every line being scanned, so all three failed
looking for production code that was still right there.
They now split at the test module, which is what they meant. This is the same
defect already fixed in the ipfs gate during the merge, and no split-on-the-
attribute site remains.
Both behavior-bearing gates were re-proven load-bearing: F2 goes red when the
Pinata object set is re-derived before the pin permit is taken, and the tail
gate goes red when release stops consuming the same success flag. F3 is a
presence check whose marker cannot be removed by a single compiling edit, so it
is not mutation-proven here.
Reconciles the served-git concurrency cap with the /ipfs CID tree gate. The two
branches independently reworked the same three seams, so most of this is picking
one mechanism per seam rather than taking a side per file.
/ipfs walk admission: keeps this branch's provenance-first resolver and its lazy
legacy scan, but drops the detached serve task in favour of #174's shared
Arc<WalkAdmission>, cloned into each spawn_blocking. Both close the permit-release
bypass; the Arc does it without leaving an abandoned request's full legacy scan
running against a held slot. Ported in with it: the request-budget gate at all four
stages with each child deadline clamped to the remainder, the walk timeout computed
inside the closure so queue delay is charged, taint sources in place of a single
truncated flag, and the deterministic-fault 500 gated on nothing else having tainted.
The two per-request ceilings are now composed rather than one shadowing the other.
The visit ceiling was unwired by the merge and the walk cap answered to only one of
the two knobs; both are honoured, the tighter one winning.
Object probe: object_type_bounded takes a Duration and returns ProbeError, so this
branch's call shape keeps #174's absence-vs-unreadable-store discriminator. A clean
`missing` from cat-file is byte-identical whether the object is absent or the pack
is unreadable, and the discriminator is store readability, not git's wording.
Write lock: the advisory-lock pool's after_release hook does not cover an unlock that
errors on a live session. Measured: a poisoned connection returned to the pool still
held the lock 15s later, because the hook's own unlock_all fails the same way. So the
guard now disposes of that connection, in release and in the Drop backstop.
Post-push replication goes to #174's coalescer. This branch's requeue_faults
integration suite has no equivalent seam on that design and is not carried over; it
is recoverable from the pre-merge head.
Also clamps the provenance path's three per-source lookups to the request budget.
They run while the walk permits are held, which is the stall hazard the preload
clamp already covered, and the structural guard found them because it now requires
every occurrence to be wrapped instead of assuming exactly one.
1346 tests pass across the workspace; fmt and clippy are clean under --locked. The
four re-anchored structural guards were mutation-tested and all four go red on the
defect they name.
The post-receive tail was spawned after `guard.release(...)`, which on a
successful push awaits the Tigris upload and then the advisory unlock while
the handler future is still tied to the client connection. The pack has
already landed on disk by then, so a disconnect in that window left a durable
push whose pins, recovery copy and announce were never spawned: the dropped
tail F2 closed, one step earlier in the handler.
Spawn it above `release`, gated on an explicit `receive_result.is_ok()`. The
`?` below the spawn can no longer be what keeps a rejected push from pinning
and announcing a half-applied repo, so the success check is now a named flag
that `release` consumes too. The tail is read-only on the repo directory and
holds neither the lease nor the advisory lock, so overlapping it with the
upload waits on nothing the handler still owns.
Tests: a handler-level disconnect driven through a parked `release` (new
store-level test seam), plus the must-not direction, a failed receive-pack
that spawns no tail. The U5 source gate now binds the new boundary: spawn
inside `if push_succeeded`, above `guard.release(push_succeeded)`.
The post-receive replication tail was spawned at the end of git_receive_pack, after
touch_repo, the trust-score update, per-ref certificate issuance and webhook fan-out.
A client or proxy disconnect anywhere in that window dropped the tail, and with it
this push's pins, recovery copy, and announcements - even though the push was already
durable on disk.
Spawn it immediately after the receive_result ? instead. That is the first point where
the push is known to have landed: guard.release() a few lines up runs on failure too,
so anchoring there would spawn a tail for a push git rejected, pinning and announcing
a half-applied repo - the state release() deliberately refuses to upload to Tigris,
and one a hostile pusher can produce on demand by aborting a pack mid-transfer.
RefUpdate gains Clone so the tail takes the parsed updates while the certificate and
webhook loops keep iterating their own copy.
This makes the tail run concurrently with certificate issuance rather than after it,
so a ref can be announced before its signed certificate exists. Accepted, and stated
here rather than left for a reviewer to find: cert issuance already fails open (errors
are logged and skipped) and the gossip event carries cert_id: None regardless, so no
announce consumer reads a certificate out of it.
Evidence is an ordering gate in inv22_gates.rs rather than a cancellation-race test,
matching the F3 gate's instrument: forcing a disconnect to land inside the cert/webhook
window is timing-dependent and would be flaky, while the ordering is precisely what
changed. The gate is mutation-verified in both directions - moving the spawn back below
the webhooks fails the upper bound, hoisting it above the ? fails the lower one.
The global and per-source walk permits were handler locals, so a client disconnect
released them the moment the future dropped while the uncancellable spawn_blocking
probe/walk/read kept running against a slot the pool now considered free. A disconnect
flood could admit replacements while prior blocking git children still occupied
threads and PIDs, weakening GITLAWB_MAX_CONCURRENT_IPFS_WALKS and
GITLAWB_IPFS_WALK_PER_SOURCE on a surface any unauthenticated caller can reach.
Hold the pair in an Arc and clone it into each of the three blocking closures. The
permits release when the last clone drops, which collapses both failure directions
into one case: a disconnect leaves the abandoned closure holding the slot, and a
panicking closure leaves the handler holding it. No loop arm changes and the
JoinError arms keep today's taint-and-continue.
A move-and-return shuttle was implemented and rejected. It is correct only if every
arm that continues the loop re-binds the returned value, which the compiler cannot
enforce - the absent-object arm Ok(Ok(None)) => continue, taken on nearly every
iteration of a random-CID scan, drops it - and it forces a second decision about the
panic arms whose least-bad answer would let one broken repo shed every other owner's
/ipfs lookups.
The regression strengthens the existing admission test: it now asserts the slot is
still held after the future is dropped and before the child is killed, with a
child-still-alive precondition so it cannot pass vacuously. Only the walk site runs
under state.git_bin and can be pinned that way; the probe and read deliberately shell
to real git, so a source-scan gate in inv22_gates.rs binds all three clone sites
instead. Removing any one clone turns that gate red.
get_by_cid held the global walk permit and the per-source permit as handler
locals, so dropping the request future released both while the spawn_blocking
probe, visibility walk, and object read kept running; a cancel-spam or timeout
client could exceed MAX_CONCURRENT_IPFS_WALKS and the per-source cap, and the
bare cat-file probe/read children had no teardown at all.
Move the gated serve pipeline into a detached task that owns an AdmissionGuard,
awaited by the handler, so admission releases only after the bounded work is
gone; back the /ipfs probe and read with run_bounded_git twins for
process-group teardown. Adds cancel-mid-walk, timeout-reap, and cancel-spam
regressions plus two load-bearing INV-22 rows. Also reflows one U1 test line to
satisfy cargo fmt.
The path-scoped upload-pack branch kept the read and per-caller permits as
handler locals and called upload_pack_excluding with no AdmissionGuard, so a
cancelled filtered clone released admission while KillGroupOnDrop was still
reaping the git group, bypassing the read and per-caller concurrency caps.
drive_git_child now returns the disarmed guard on success so one guard rides
both build_filtered_pack stages (rev-list then pack-objects); the handler builds
the guard from the permits as the plain path does. Adds a disconnect regression
and an INV-22 gate row, both proven load-bearing.
On client disconnect, RepoWriteGuard::Drop released the per-repo advisory
lock immediately (it holds no child/pgid), while KillGroupOnDrop's detached
reaper was still tearing down the git process group for ~4s. A second same-node
push could then acquire the repo and race the first's still-writing objects/
dir and Tigris upload.
Add an in-process per-repo write lease (block-and-wait, keyed like the advisory
lock) that SUPPLEMENTS the retained cluster-wide PG lock. The lease rides into
the disconnect reaper via a clone on the write-path AdmissionGuard (None on all
read paths) and a second clone held across guard.release() for the clean-path
Tigris upload, so it frees only after the group is reaped. A bounded-wait steal
reclaims a leaked lease (block-and-wait has no coalesce degradation). The lease
is always taken before the PG lock and released after, so the two serializers
cannot invert. Deterministic fake-git race test + inv22 gate row.
The detached Pinata task moved each push's full object-id list into the
closure and parked it on pin_semaphore.acquire_owned(); a tokio semaphore
bounds active holders, not the waiter queue, so a slow Pinata backend let N
queued pushes each retain an MB-scale OID list (unbounded memory/tasks).
Capture only the small ref-update tuples and re-derive the object set via the
existing reaped/deadline-bounded git helpers AFTER the pin permit is acquired,
so retained memory is O(ref tuples), not O(object lists). Per-ref effects are
unchanged (still one spawn per push, no coalescing or shedding, so no dropped
announcements). Adds an inv22 gate row forbidding a retained object list across
the acquire, plus re-derivation-equivalence and reaping tests.
list_all_repos and list_visibility_rules_for_repos ran as bare awaits while
holding the scarce IPFS walk permits, so a query blocked in Postgres pinned the
slots past GITLAWB_IPFS_REQUEST_BUDGET_SECS and shed later requests as 503s.
Wrap both in the remaining request-deadline budget; on timeout the RAII permits
drop and the handler sheds a retryable budget 503. Fails closed: a timeout on
the visibility-rules query denies the request rather than serving an unfiltered
listing.
Appends an inv22 gate row asserting both queries stay deadline-wrapped.
Await pg_advisory_unlock while the pooled connection is still owned by
self; take it and set released only after the await resolves. A cancel
mid-unlock now leaves the Drop backstop armed (conn still Some), so the
session-level advisory lock is no longer leaked onto a pooled connection.
Adds a cancel-mid-unlock regression test (via a test-only pre-unlock gate)
and an append-only inv22 gate row asserting the take/released orderings.
object_type now distinguishes repo-level git failures (not-a-repository,
error:-prefixed corruption) from genuine object absence, so a corrupt
repo taints the /ipfs scan into the retryable 503 instead of vouching
for a definitive 404; the drain reads visibility rules by the freshly
fetched record id, closing a latent fail-open on id divergence. The four
budget gates collapse into one stage-labelled helper, the three scan
sites share state::acquire_scan_permit (inv22 tripwire repointed and
sever-verified), and the Tigris stall tests use a local accept-and-park
listener instead of a non-routable address.
try_begin now merges a losing push's tip pairs into the in-flight key's
pending slot in the same critical section as the presence check, bounded
at 1024 pairs with a full-scan marker on overflow (force_full_scan
threads to the resolver; empty tips never encode the marker). The task
loop-drains: finish_or_take_pending either hands back the batch with the
key retained, or removes the key and disarms the guard atomically, so a
successor task's key survives the guard drop. Drains re-fetch rules
fresh and replay the tail pipeline under the U4 helper gates only; the
loop holds no task-level pool permit, so a hot repo cannot deadlock the
pool at size 1.
replication_withheld_set's walk arm, resolve_candidates_for_push's git
stages, and fail_closed_full_scan_objects each take a
git_encrypt_semaphore permit before their spawn_blocking and carry it
inside the closure, so a push burst holds at most pool-many concurrent
scans and a started scan finishes with its permit. Deletion-only pushes
and repos without path-scoped rules never park. Contention defers the
landed push's tail (logged with queue_wait_ms), never errors it; the
acquire sites document the accepted disconnect-during-park residual.
After a successful path-scoped push the handler spawns a DETACHED task that runs
pin_new_objects then withheld_recipients_gated, which acquire_owned().await's on
git_encrypt_semaphore — it DEFERS (blocks) when the pool is full. The semaphore
caps active walks but nothing capped how many detached tasks were spawned and
PARKED on that await: N rapid path-scoped pushes spawned N tasks, each holding
cloned object lists/rules/paths/keys — an unbounded parked-waiter set (P2-2).
2a54c15 deliberately chose defer-not-shed because dropping the walk loses the
withheld-blob recovery copy and there is no reconciliation sweep to rebuild it.
So bound the OUTSTANDING-TASK set by per-repo coalescing rather than shedding:
EncryptInflight (a bounded Arc<Mutex<HashSet<repo_id>>>) + try_begin — before the
spawn, if a task for the repo is already in-flight skip the duplicate (the pending
walk covers the newer objects); otherwise spawn and move an RAII
EncryptInflightGuard into the task that removes the key on drop (completion, error,
or panic-unwind). This bounds the set to <=1 pending task per repo and never drops
work; withheld_recipients_gated's defer is unchanged.
The second detached post-push spawn (Pinata replication, also runs pin_new_objects)
is scoped out with rationale: it parks on no semaphore (no unbounded waiter set)
and does per-push per-ref work (branch->CID, gossip, subscription, Arweave,
peer-notify) keyed to this push's ref_updates — coalescing it would DROP a later
push's announcements, a correctness regression.
Regressions (mutation-verified RED->GREEN): the outstanding set is bounded to 1
per repo under saturation (never-coalesce -> 32 tasks RED); a coalesced repo's key
is released when its task ends so it is reprocessed, never permanently skipped
(guard-drop no-op -> repo locked out, recovery copy lost forever, RED — the exact
durability regression the fix prevents); distinct repos each admit; cold-set first
push admits; plus an inv22_gates structural tripwire that fails if the handler gate
is removed.
A source-scan tripwire that fails when a new site reintroduces the concurrency-cap
class PR #174 fixed, or when one of the five gates (U1-U5) is removed. INV-22: a
permit held per op recovers only if every path is duration-bounded and reaps the
group before releasing admission, and every detached git task carries admission.
Lives in tests/ (a separate crate) on purpose — a guard scanning the file it lives
in would match its own identifier literals and pass vacuously; scanning src/ from
here keeps every check load-bearing. Checks: run_bounded_git confirms child exit via
child_terminated_without_reaping (U1); KillGroupOnDrop launches the reaper via
Handle::try_current (U2); git_receive_pack acquires state.git_write_per_caller (U4);
the encryption walk runs through withheld_recipients_gated over git_encrypt_semaphore
(U5); and the bounded recipients walk is invoked exactly once — only inside the gated
helper — so a new spawn_blocking bypass (count > 1) fails (P1-e non-bypass).
Proven load-bearing: a synthetic second call site turns the count tripwire RED, and
each named gate maps to a fix whose own regression already goes RED on revert.
beardthelion