Compare commits

..
Author SHA1 Message Date
beardthelion 2219c5d23f test(node): realign the inv22 tail-spawn gate and metrics help text (#580)
- inv22_replication_tail_spawns_at_the_durability_boundary matched the
  literal tokio::spawn(post_receive_replication_tail( shape, which went
  blind when production wrapped the tail in tokio::spawn(async move {
  ... }). The gate now matches the tail call itself (filtering the fn
  definition) while keeping the disposition.spawn_tail gate, brace
  depth, no-second-spawn, landed-set, and ordering assertions.
- metrics help for gitlawb_push_duration_seconds described receive-pack
  subprocess time but observes the whole successful handler interval;
  gitlawb_sync_lag_seconds described the origin ref-update timestamp but
  samples lag_origin (the land instant). Both strings and the module
  bullet now name what is measured.
2026-11-11 23:59:46 -06:00
beardthelion c564f8422b fix(node): align notify land surfaces and resolve land instant in-task
- the multi-ref notify regression now matches lag_origin for the
  supplied land instant instead of timestamp, which is a live send
  instant after the publish/land split; the tail comment now names the
  notify lag_origin field rather than timestamp
- both live replication-tail spawns resolve push_landed_at inside the
  spawned task so the handler no longer awaits the lookup while
  holding the reclaimed write lease or leaving the tail un-owned on
  disconnect
2026-11-11 21:04:46 -06:00
beardthelion 66d01f8a6f fix(node): use the deferred land-instant rule on live replication tails
Both live post_receive_replication_tail spawns now resolve
push_landed_at through db.request_push_landed_at: the latest
applied_at across this request's applied transitions, falling back to
the request row's created_at, the same rule the deferred drain uses.
The pre-receive-pack acceptance stamp no longer flows into gossip
lag_origin, the HTTP notify land field, or sync_queue.origin_timestamp
on the live path, so live and deferred pushes report one lag origin.
Regression: request_push_landed_at_returns_max_applied_or_none covers
the max-applied arm and both fallback cases.
2026-11-11 18:23:12 -06:00
beardthelion ea0b0571cc fix(node): split notify publish/land clocks and unify live push-land instant
- HTTP sync/notify now stamps timestamp with send time (publish
  semantics, matching what gossip ingest stores on the feed row) and
  carries the land instant in a new optional lag_origin field; the
  receiver plumbs lag_origin to sync_queue.origin_timestamp with a
  timestamp fallback for older senders
- both live replication-tail spawns pass the request row's acceptance
  instant (the same clock the deferred drain's created_at fallback
  uses) instead of minting a fresh clock at spawn time
2026-11-11 16:55:56 -06:00
beardthelion a42aab77e7 fix(node): tighten the two new lag-observation edges
- deferred tails source push_landed_at from the accepted transitions'
  applied_at (the instant refs actually moved), falling back to the
  request row's created_at rather than drain time
- the visibility gate confirms the mirror's ref tip matches the queued
  outcome before sampling, so a row superseded by a newer origin
  update (or a deletion the mirror absorbed) never records lag
2026-11-11 16:20:55 -06:00
beardthelion bfb491e0d8 fix(node): version-gate lag_origin, read persisted quarantine, keep push-land instant
- lag_origin ships under v=1: CURRENT_REF_UPDATE_VERSION bumped, ingest
  rejects a v0 event carrying the v1 field, goldens re-pinned for the
  v1 publish shape, and the legacy-parse test now pins the fleet rule
  (v0 still parses, v1 refuses loudly on the v key)
- the sync-lag observation gate reads is_repo_quarantined instead of
  the attempt-local flag, so a re-sync of a quarantined mirror no
  longer samples visibility readers still deny; fail-closed on lookup
- post_receive_replication_tail takes push_landed_at so a deferred
  drain reports the request row's landing instant rather than its own
  drain time; live callers pass the acceptance instant, tests pass None
2026-11-11 15:30:44 -06:00
beardthelion d5d9bbb5b3 fix(node): split sync-lag origin from gossip publish freshness
The single origin_timestamp captured after the durability wait was reused
as the signed RefUpdateEvent timestamp, so a tail that published more
than 600s later minted an already-stale event that verified ingest drops
in check_freshness, silently disabling gossip auto-sync on mirrors.

- Capture the lag origin at push landing, before the durability wait, and
  carry it on a new optional signed field RefUpdateEvent::lag_origin
  (serde default, skipped when absent, so events from older peers still
  parse and the null-optional signing shape is byte-identical).
- Sign the publish instant in event.timestamp again, matching merge-base,
  so check_freshness sees the time the event actually went on the wire.
- Ingest now feeds enqueue_sync lag_origin with a timestamp fallback, so
  sync-lag still measures push-to-visible instead of notify-to-visible.
- Re-pin the all-Some GOLDEN_SIGNING_BYTES for the new field; the
  all-None production golden is unchanged. Round-trip test asserts the
  field survives serde.

Also restore the v18 pinned-cid upgrade oracle: the test's rollback
delete targeted schema_migrations version 47 instead of 18, so v18 never
re-applied and the index_exists assertion failed on stable and beta.
2026-11-11 12:12:54 -06:00
beardthelion 966b2b8e0a fix(node): sample sync lag at mirror visibility, clamp untrusted origin ages
- the lag observation moves to the upsert point, before the
  encrypted-blob tail and mark_sync_done, so the metric reads
  push-to-visible rather than push-to-fully-replicated
- a quarantined mirror is admitted but withheld from readers and now
  reports no visibility lag
- MAX_SYNC_LAG_OBSERVATION (24h) discards absurd sender-supplied origin
  ages that would park fabricated values in the tail buckets; the
  timestamp is unauthenticated peer input
- the migration lands at v47 after main's slot moves; the test
  initializer gains origin_timestamp
2026-11-10 22:52:34 -06:00
beardthelion 2f99ad8555 Merge remote-tracking branch 'origin/main' into pr-362 2026-11-10 22:44:01 -06:00
beardthelion 3840b7392a Merge remote-tracking branch 'origin/main' into pr-362 2026-11-10 22:42:50 -06:00
anandh8x 9f4f509619 feat: add push fetch and sync latency metrics 2026-08-16 20:21:57 +05:30
29 changed files with 4693 additions and 10864 deletions

No files matched your search

-4
View File
@@ -1,4 +0,0 @@
name: "Gitlawb CodeQL config"
paths-ignore:
- "**/tests/**"
- "**/test_support/**"
-10
View File
@@ -195,16 +195,6 @@ jobs:
# shellcheck disable=SC2086 # PACKAGES is a word list by design
cargo test --locked $PACKAGES
# The real-node deny harness is behind the `test-harness` feature (so its
# spawn surface never compiles into the production binary), which means
# `--workspace` above skips it. Run it explicitly so the trust-boundary
# regression cases (INV-1/INV-2/INV-8) execute on every PR. Uses the same
# Postgres service; `#[sqlx::test]` provisions an isolated DB per test.
- name: cargo test (real-node deny harness)
env:
DATABASE_URL: postgres://postgres:postgres@localhost:5432/gitlawb_test
run: cargo test --locked -p gitlawb-node --features test-harness --lib --test deny_harness
build-release:
name: build --release
needs: changes
-4
View File
@@ -26,15 +26,11 @@ COPY crates/icaptcha-client/Cargo.toml crates/icaptcha-client/
# is what let it sit dead from the moment gl gained a path dependency on
# icaptcha-client. A failure here should stop the build, not silently turn every
# image build into a cold one.
# gitlawb-node declares BOTH a [[bin]] and a [lib], so it needs both stubs:
# cargo resolves every declared target before it fetches anything, and a
# missing src/lib.rs aborts the layer with a target resolution error.
RUN mkdir -p crates/gitlawb-core/src crates/gitlawb-node/src crates/gl/src \
crates/git-remote-gitlawb/src crates/gitlawb-attest/src crates/icaptcha-client/src && \
echo 'fn main() {}' > crates/gitlawb-node/src/main.rs && \
echo 'fn main() {}' > crates/gl/src/main.rs && \
echo 'fn main() {}' > crates/git-remote-gitlawb/src/main.rs && \
echo '' > crates/gitlawb-node/src/lib.rs && \
echo '' > crates/gitlawb-core/src/lib.rs && \
echo '' > crates/gitlawb-attest/src/lib.rs && \
echo '' > crates/icaptcha-client/src/lib.rs && \
-121
View File
@@ -1,121 +0,0 @@
# Gitlawb Wire Protocol Specification (v1, alpha)
This document describes the wire behavior a second implementation can interop
with **today**. Anything the node does not ship yet is explicitly marked
*planned*; nothing below is aspirational unless it says so.
## 1. Identity & DIDs
- **Key type:** Ed25519.
- **Method (alpha):** `did:key` only. Resolution is deterministic public-key
extraction from the multicodec prefix (`z6Mk…`, multicodec `0xed01`).
- Signed requests presenting any other DID method are rejected; the node's
auth layer answers with a hint that only `did:key` is supported in alpha.
- *Planned:* `did:web` and `did:gitlawb` resolution. Clients MUST NOT assume
either is accepted anywhere yet.
## 2. Authentication (RFC 9421)
- **Transport:** production deployments should run the node behind TLS
(typically a reverse proxy). The node itself has no scheme check: it verifies
RFC 9421 headers on whatever transport the request arrives over, so nothing
at the signature layer refuses a cleartext request.
- **HTTP signatures:** `Signature-Input` covers `@method`, `@path`, and
`content-digest`:
`sig1=("@method" "@path" "content-digest");keyid="did:key:…";alg="ed25519";created=<unix>`
All three components are mandatory: a `Signature-Input` missing one is
rejected as `incomplete_signature`.
- **`@path` value:** the node binds `@path` to the path *and* query string of
the request, not to the path-only value RFC 9421 defines for `@path`. A
second implementation must sign the request exactly as it goes on the
wire, e.g. `GET /zOwner/myrepo.git/info/refs?service=git-upload-pack`
including the `?service=…` part (see §5 for the owner and `.git`
normalization).
- **Signer identity:** `keyid` is the actor DID (`did:key` only, see §1).
- **Integrity:** `Content-Digest` is SHA-256 over the request body and is
verified against it. Every signed request MUST send the header: a signed
request without it is refused `400 missing_content_digest` before routing.
On a body-less request the header still carries the SHA-256 digest of the
empty body (`sha-256=:47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=:`),
it is never omitted.
## 3. Proof of Intelligence (iCaptcha)
- **Mode:** controlled by `ICAPTCHA_MODE` = `off` | `shadow` | `enforce`,
**default `off`**. A node with the gate off never demands proof.
- **Scope:** the gate applies to a small set of write endpoints (agent
registration and repo create/fork), not to all writes.
- **Wire shape:** the client-supplied header is `x-icaptcha-proof` — that is
the only iCaptcha header a node reads from a request. When the gate is
`enforce` and a verifier public key is loaded, a missing or invalid proof
answers `403 icaptcha_proof_required` and sets `x-icaptcha-url` and
`x-icaptcha-level` **on the response** (also mirrored as JSON fields) so
the client can go solve a challenge and retry. Fail-safe: `enforce` with
no public key loaded (unset and unfetchable) stays inert and admits the
request rather than rejecting every request.
## 4. Ref-Update Certificate
*Planned:* the `gitlawb/ref-update/v1` schema below is defined in code, but no
shipped endpoint serves or verifies these certificates today; nothing in
sections 2-3 produces one on the wire. It is documented here so a second
implementation shares the type vocabulary ahead of the wire surface.
The hash fields reserve 64 hex characters for the SHA-256 object format; the
repositories the node serves are sha1 with 40-hex OIDs (see §5), so the field
width is a forward reservation, not what a current push produces.
Two distinct certificate-shaped payloads exist; this section documents the
**core signing schema**, which is what `gitlawb/ref-update/v1` names:
- **Type string:** `gitlawb/ref-update/v1`.
- **Body (canonical JSON, the signed bytes):** `type`, `repo` (the
repository DID), `ref_name`, `from` (previous hash, 64 hex chars,
all-zeros for a new ref), `to` (target hash, 64 hex chars), `seq`
(monotonically increasing, replay prevention), `timestamp` (RFC 3339),
`nonce`.
- **Signatures:** a list of `{signer: did:key, sig: base64url-unpadded
Ed25519}` entries appended outside the signed body; thresholds are counted
over distinct signers that verify.
The node's REST API separately serves **push receipts** — a different,
node-issued JSON payload attesting that a push was processed. Receipts are
not `gitlawb/ref-update/v1` documents and are not interchangeable with them;
they are documented with the node's HTTP API, not here.
## 5. Storage & Git Transport
- **Object format:** bare repositories are created with
`--object-format=sha1`; production OIDs are 40 hex chars. *Planned:*
SHA-256 object format.
- **Path normalization:** `{repo}` may carry one trailing `.git` suffix,
stripped before lookup (so `myrepo.git` resolves to `myrepo`; a repo
literally named `foo.git` addresses as `foo.git.git`). `{owner}` may be the
full `did:key:…` string or the bare multibase segment the `did:key:` prefix
implies; both resolve to the same owner. None of this changes `@path`:
sign the request path and query exactly as sent.
- **Smart-HTTP endpoints:**
- `GET /{owner}/{repo}/info/refs?service=git-upload-pack` — fetch
advertisement.
- `GET /{owner}/{repo}/info/refs?service=git-receive-pack` — push
advertisement.
The `?service=` parameter is mandatory and takes exactly those two
values; a missing parameter is a `400`, not an omitted-service default.
- `POST /{owner}/{repo}/git-upload-pack` — clone/fetch. Request and
response bodies are pkt-line framed; the result content type is
`application/x-git-upload-pack-result`.
- `POST /{owner}/{repo}/git-receive-pack` — push. **Authentication is
mandatory**: an unsigned push is refused with `401`. Same pkt-line
framing; the result content type is
`application/x-git-receive-pack-result`.
Signed requests on this path follow the RFC 9421 header rules in §2.
§2's transport note applies here too: the verifier has no scheme check,
so nothing at the signature layer requires this push to arrive over TLS.
- **Fetch vs push auth split:** fetch-side requests against a withheld or
private repo answer `404` (existence is not confirmed to unauthorized
callers), while the push path answers `401` for missing/invalid
signatures. A second implementation must not treat the two symmetrically.
- **Content addressing:** the node records IPFS CID metadata per branch and
serves it over REST (the repo `refs` endpoint maps branch heads to CIDs).
*Planned:* IPNS-backed branch pointers; IPNS is not part of the shipped
wire contract.
+2 -29
View File
@@ -6,28 +6,10 @@ edition.workspace = true
rust-version.workspace = true
license.workspace = true
[lib]
name = "gitlawb_node"
path = "src/lib.rs"
[[bin]]
name = "gitlawb-node"
path = "src/main.rs"
# Exposes the `test_harness` spawn surface (src/test_harness.rs) to the
# real-node deny-harness integration crate. Off by default so the production
# binary never compiles test-only boot code. Enable with
# `cargo test -p gitlawb-node --features test-harness`.
[features]
test-harness = ["dep:tempfile"]
# Only builds when `test-harness` is enabled; otherwise skipped (not an error),
# so `cargo test --workspace` without the feature stays green.
[[test]]
name = "deny_harness"
path = "tests/deny_harness.rs"
required-features = ["test-harness"]
[dependencies]
gitlawb-core = { path = "../gitlawb-core" }
ed25519-dalek = { workspace = true }
@@ -96,19 +78,10 @@ alloy = { version = "1", default-features = false, features = [
] }
rand = { workspace = true }
libp2p-dns = { version = "0.45.0", features = ["tokio"] }
# Optional: only needed by the test-harness feature's lib modules
# (test_git_shim, test_harness); dev-deps are invisible to lib code.
tempfile = { version = "3", optional = true }
[dev-dependencies]
mockito = "1"
tempfile = "3"
# test-util enables tokio::test(start_paused): the delayed claim re-sweep
# test drives the real CLAIM_SWEEP_MIN_AGE-plus-margin sleep on paused time.
# `test-util` for `#[tokio::test(start_paused = true)]`: lets the deadline tests assert
# on which timer fired instead of on wall clock.
tokio = { workspace = true, features = ["test-util"] }
# Used by the deny-harness integration crate (tests/deny_harness.rs): the
# #[sqlx::test] macro for an ephemeral per-test pool, and reqwest as the real
# HTTP client that drives deny paths over the socket.
sqlx = { version = "0.8", features = ["postgres", "runtime-tokio-rustls", "macros", "migrate"] }
reqwest = { workspace = true }
hex = "0.4"
+62 -1
View File
@@ -439,6 +439,7 @@ pub async fn trigger_sync(State(state): State<AppState>) -> Result<Json<serde_js
"refs/heads/main",
"0000000000000000000000000000000000000000",
None,
None,
)
.await;
enqueued += 1;
@@ -474,6 +475,13 @@ pub struct NotifyRequest {
pub old_sha: Option<String>,
#[serde(default)]
pub timestamp: Option<String>,
/// The instant the origin push landed, for sync-lag measurement only —
/// mirrors RefUpdateEvent::lag_origin on the gossip transport. `timestamp`
/// stays publish semantics to match what gossip ingest stores on the feed
/// row; when an older sender omits this field the enqueue falls back to
/// `timestamp`, which was the pre-split approximation of land time.
#[serde(default)]
pub lag_origin: Option<String>,
#[serde(default)]
pub cert_id: Option<String>,
/// Full owner DID — added in #144 for DID-aware feed gating.
@@ -520,7 +528,14 @@ pub async fn notify_sync(
state
.db
.enqueue_sync(&req.repo, &req.node_did, &req.ref_name, &req.new_sha, None)
.enqueue_sync(
&req.repo,
&req.node_did,
&req.ref_name,
&req.new_sha,
None,
req.lag_origin.as_deref().or(req.timestamp.as_deref()),
)
.await?;
// Mirror the gossipsub-receive handler: insert the same record we'd
@@ -1282,6 +1297,52 @@ mod tests {
);
}
#[sqlx::test]
async fn sync_notify_preserves_origin_timestamp_for_sync_lag(pool: PgPool) {
let state = test_state(pool).await;
let db = state.db.clone();
let peer_did = seed_peer(&state).await;
// Distinct publish and land instants: the feed row must carry publish
// semantics while only the queue row sees lag_origin.
let timestamp = "2026-08-16T00:00:00Z";
let lag_origin = "2026-08-15T23:59:00Z";
let body = serde_json::json!({
"repo": "z6Mkfoo/timestamped",
"ref_name": "refs/heads/main",
"new_sha": "0000000000000000000000000000000000000000",
"node_did": peer_did,
"timestamp": timestamp,
"lag_origin": lag_origin,
})
.to_string();
let router = crate::server::build_router(state);
let resp = router
.oneshot(unsigned_post(
"/api/v1/sync/notify",
&body,
"198.51.100.23:5000",
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let queued = db.dequeue_pending_syncs(100).await.unwrap();
assert_eq!(queued.len(), 1);
assert_eq!(queued[0].origin_timestamp.as_deref(), Some(lag_origin));
let feed = db
.list_ref_updates_keyset(Some("z6Mkfoo/timestamped"), 10, None)
.await
.unwrap();
assert_eq!(
feed.iter()
.find(|u| u.repo == "z6Mkfoo/timestamped")
.map(|u| u.timestamp.as_str()),
Some(timestamp),
"the feed row must keep publish semantics, matching gossip ingest"
);
}
#[sqlx::test]
async fn sync_notify_unknown_peer_wins_over_malformed_slug(pool: PgPool) {
// Ordering: the slug check sits after the peer gate, so an unknown peer
+133 -18
View File
@@ -1777,6 +1777,7 @@ pub async fn git_upload_pack(
headers: axum::http::HeaderMap,
body: Bytes,
) -> Result<Response> {
let fetch_started = std::time::Instant::now();
// #62 cheap load shed. Permit-less snapshot, not admission; see git_info_refs for
// what it does and does not bound. The authoritative hold is `git_permit` below,
// after the per-source cap.
@@ -1979,6 +1980,7 @@ pub async fn git_upload_pack(
// that mislabel predates this change and is tracked as a follow-up.
if should_count_fetch(finalizes_fetch, served_filtered_pack, served_pack) {
crate::metrics::record_fetch(&format!("{owner}/{name}"));
crate::metrics::observe_fetch_duration(fetch_started.elapsed());
crate::metrics::observe_pack_size(body_len as f64);
}
Ok(resp)
@@ -2190,6 +2192,7 @@ async fn notify_peer_of_ref(
node_did: &str,
pusher_did: &str,
owner_did: &str,
origin_timestamp: &str,
) {
let body = serde_json::json!({
"repo": repo_slug,
@@ -2198,7 +2201,13 @@ async fn notify_peer_of_ref(
"node_did": node_did,
"pusher_did": pusher_did,
"old_sha": old_sha,
"timestamp": chrono::Utc::now().to_rfc3339(),
// Publish semantics, matching what gossip ingest stores on the feed
// row: the send instant, freshness-window meaningful. Land time travels
// separately in lag_origin (also an Option<String> on the receiver);
// stamping land here would put a different clock on HTTP rows than on
// gossip rows for the same logical update.
"timestamp": Utc::now().to_rfc3339(),
"lag_origin": origin_timestamp,
"owner_did": owner_did,
});
let body_bytes = match serde_json::to_vec(&body) {
@@ -2248,6 +2257,7 @@ async fn notify_peer_of_refs(
node_did: &str,
pusher_did: &str,
owner_did: &str,
origin_timestamp: &str,
) {
for (ref_name, old_sha, new_sha) in ref_updates {
notify_peer_of_ref(
@@ -2262,6 +2272,7 @@ async fn notify_peer_of_refs(
node_did,
pusher_did,
owner_did,
origin_timestamp,
)
.await;
}
@@ -2280,6 +2291,7 @@ pub async fn git_receive_pack(
headers: axum::http::HeaderMap,
body: Bytes,
) -> Result<Response> {
let push_started = std::time::Instant::now();
let name = smart_http_repo_name(&repo)?;
// Fast-path shed before the DB lookup when the write pool is ALREADY saturated, so a
// push flood against a full pool does not hit Postgres per request. Best-effort
@@ -3309,14 +3321,37 @@ pub async fn git_receive_pack(
Vec::new()
};
if disposition.spawn_tail && !push_succeeded && !landed_refs.is_empty() {
tokio::spawn(post_receive_replication_tail(
state.clone(),
record.clone(),
landed_refs.clone(),
disk_path.clone(),
auth.0.to_string(),
None,
));
let state = state.clone();
let record = record.clone();
let landed = landed_refs.clone();
let disk_path = disk_path.clone();
let did = auth.0.to_string();
let db = state.db.clone();
let request_id = request_id.clone();
let fallback = now.clone();
tokio::spawn(async move {
// Resolve inside the owned task: awaiting the lookup in the handler
// would hold the reclaimed write lease during a DB wait and leave
// the tail un-owned on a disconnect. Same land rule as the deferred
// drain — max applied_at across this request's landed transitions,
// else the request row's created_at.
let push_landed_at = db
.request_push_landed_at(&request_id)
.await
.ok()
.flatten()
.unwrap_or(fallback);
post_receive_replication_tail(
state,
record,
landed,
disk_path,
did,
Some(push_landed_at),
None,
)
.await;
});
}
// Always release the advisory lock — even on error — to prevent stale locks
@@ -3337,14 +3372,35 @@ pub async fn git_receive_pack(
// than by the mere fact that release was entered.
publish_durability.arm_for_release(reclaimed.publish_stage());
if disposition.spawn_tail && !landed_refs.is_empty() {
tokio::spawn(post_receive_replication_tail(
state.clone(),
record.clone(),
landed_refs,
disk_path.clone(),
auth.0.to_string(),
Some(publish_durability.arc()),
));
let state = state.clone();
let record = record.clone();
let disk_path = disk_path.clone();
let did = auth.0.to_string();
let db = state.db.clone();
let request_id = request_id.clone();
let fallback = now.clone();
let durability_slot = publish_durability.arc();
tokio::spawn(async move {
// Same in-task resolution and land rule as the carve-out spawn
// above — the handler must not await the lookup while it still
// holds the reclaimed write lease.
let push_landed_at = db
.request_push_landed_at(&request_id)
.await
.ok()
.flatten()
.unwrap_or(fallback);
post_receive_replication_tail(
state,
record,
landed_refs,
disk_path,
did,
Some(push_landed_at),
Some(durability_slot),
)
.await;
});
}
}
// Short-circuit on a refused publish BEFORE anything downstream observes
@@ -3399,6 +3455,7 @@ pub async fn git_receive_pack(
// that case via `run_effects`, so this is defensive).
let _ = state.db.touch_repo(&record.id).await;
crate::metrics::record_push(&record.id);
crate::metrics::observe_push_duration(push_started.elapsed());
crate::metrics::observe_pack_size(body_len as f64);
// Claim ownership before inline effects: a background worker may
@@ -3674,12 +3731,24 @@ async fn publish_durability_confirmed(
/// complete without ever having a live tail (capability-free pushes whose
/// landing reconcile proved). `pub(crate)` so the drain can call it; the
/// live handler is the other caller.
/// The lag origin the tail reports: the caller-supplied push-land instant when
/// one exists (a deferred drain passes the request row's created_at), else the
/// tail's own entry time for callers that never knew the landing instant.
fn lag_origin_timestamp(push_landed_at: Option<String>) -> String {
push_landed_at.unwrap_or_else(|| Utc::now().to_rfc3339())
}
pub(crate) async fn post_receive_replication_tail(
state: AppState,
record: RepoRecord,
ref_updates: Vec<RefUpdate>,
disk_path: std::path::PathBuf,
did: String,
// The instant the push landed, captured by the caller at acceptance so a
// deferred tail reports push-land lag rather than drain-time lag. Callers
// that cannot recover the landing instant pass None and the tail falls
// back to its own entry time, the pre-param behavior.
push_landed_at: Option<String>,
publish_durability: Option<
Arc<std::sync::Mutex<Option<crate::git::repo_store::ReleaseOutcome>>>,
>,
@@ -3690,6 +3759,14 @@ pub(crate) async fn post_receive_replication_tail(
.lock_held_transfer_timeout_secs
.saturating_add(5),
);
// The sync-lag origin: the instant this push landed, captured BEFORE the
// durability wait so a slow publish does not shrink the lag a mirror
// measures. Travels to receiving peers as RefUpdateEvent::lag_origin
// (gossip) and the notify lag_origin field (HTTP) — never as the signed
// event's own `timestamp` or the notify `timestamp`, which both stay at
// publish time for check_freshness and feed-row parity.
let origin_timestamp = lag_origin_timestamp(push_landed_at);
if !publish_durability_confirmed(&publish_durability, durability_wait).await {
tracing::warn!(
repo = %record.id,
@@ -3912,6 +3989,7 @@ pub(crate) async fn post_receive_replication_tail(
let owner_did_for_arweave = record.owner_did.clone();
let self_public_url = state.config.public_url.clone();
let node_keypair = Arc::clone(&state.node_keypair);
let origin_timestamp_for_announce = origin_timestamp.clone();
// #174 F2a: gated on the cheap announce predicate, not on `withheld`.
// `withheld` is None for a push that coalesced (it never walked), and
// this task's work is per-push and non-idempotent, so keying it on the
@@ -4037,6 +4115,7 @@ pub(crate) async fn post_receive_replication_tail(
old_sha: old_sha.clone(),
new_sha: new_sha.clone(),
timestamp: chrono::Utc::now().to_rfc3339(),
lag_origin: Some(origin_timestamp_for_announce.clone()),
cert_id: None,
cid: cid.map(|s| s.to_string()),
sig: None,
@@ -4139,6 +4218,7 @@ pub(crate) async fn post_receive_replication_tail(
&node_did_str,
&pusher_did_clone,
&record.owner_did,
&origin_timestamp_for_announce,
)
.await;
}
@@ -5352,7 +5432,7 @@ const MAX_REF_UPDATES: usize = 8192;
/// replication tail at the durability boundary while the certificate and webhook
/// loops below still iterate their own copy (#174 U5).
#[derive(Clone, Debug)]
pub struct RefUpdate {
pub(crate) struct RefUpdate {
pub(crate) old_sha: String,
pub(crate) new_sha: String,
pub(crate) ref_name: String,
@@ -8062,6 +8142,11 @@ mod tests {
mockito::Matcher::PartialJsonString(
r#"{"owner_did":"did:key:zOwner"}"#.to_string(),
),
// Land time rides lag_origin after the publish/land split;
// timestamp is a live send instant and cannot be pinned.
mockito::Matcher::PartialJsonString(
r#"{"lag_origin":"2026-08-16T00:00:00Z"}"#.to_string(),
),
]))
.with_status(200)
.expect(1)
@@ -8098,6 +8183,7 @@ mod tests {
"did:key:zNode",
"did:key:zPusher",
"did:key:zOwner",
"2026-08-16T00:00:00Z",
)
.await;
@@ -8146,6 +8232,7 @@ mod tests {
"did:key:zNode",
"did:key:zPusher",
"did:key:zOwner",
"2026-08-16T00:00:00Z",
)
.await;
@@ -13962,6 +14049,24 @@ mod tests {
}
#[cfg(unix)]
#[test]
fn lag_origin_timestamp_prefers_the_caller_supplied_push_land() {
let landed = "2019-01-01T00:00:00Z".to_string();
assert_eq!(
lag_origin_timestamp(Some(landed.clone())),
landed,
"a deferred tail must report the request row's landing instant, not drain time"
);
let fallback = lag_origin_timestamp(None);
let parsed = chrono::DateTime::parse_from_rfc3339(&fallback)
.expect("the fallback must still be a valid RFC-3339 instant");
let skew = (chrono::Utc::now() - parsed.with_timezone(&chrono::Utc)).num_seconds();
assert!(
skew.abs() < 60,
"a caller that never knew the landing instant falls back to tail entry time"
);
}
fn f2a_update(ref_name: &str, new_sha: &str) -> Vec<RefUpdate> {
vec![RefUpdate {
old_sha: ZERO_SHA.to_string(),
@@ -13994,6 +14099,7 @@ mod tests {
f2a_update("refs/heads/main", &c1),
repo.path().to_path_buf(),
F2A_PUSHER.to_string(),
None,
Some(slot),
)
.await;
@@ -14040,6 +14146,7 @@ mod tests {
repo.path().to_path_buf(),
F2A_PUSHER.to_string(),
None,
None,
)
.await;
let after_first = f2a_walks(&log);
@@ -14061,6 +14168,7 @@ mod tests {
repo.path().to_path_buf(),
F2A_PUSHER.to_string(),
None,
None,
)
.await;
@@ -14137,6 +14245,7 @@ mod tests {
repo.path().to_path_buf(),
F2A_PUSHER.to_string(),
None,
None,
)
.await;
@@ -14219,6 +14328,7 @@ mod tests {
repo.path().to_path_buf(),
F2A_PUSHER.to_string(),
None,
None,
));
f2a_wait_for(|| started.exists(), "the admitted push's walk to start").await;
@@ -14336,6 +14446,7 @@ mod tests {
repo.path().to_path_buf(),
F2A_PUSHER.to_string(),
None,
None,
)
.await;
@@ -15487,6 +15598,7 @@ exit 0
repo.path().to_path_buf(),
F2A_PUSHER.to_string(),
None,
None,
)
.await;
@@ -15543,6 +15655,7 @@ exit 0
repo.path().to_path_buf(),
F2A_PUSHER.to_string(),
None,
None,
)
.await;
tokio::time::sleep(std::time::Duration::from_millis(300)).await;
@@ -15585,6 +15698,7 @@ exit 0
repo.path().to_path_buf(),
F2A_PUSHER.to_string(),
None,
None,
)
.await;
@@ -15684,6 +15798,7 @@ exit 0
repo.path().to_path_buf(),
F2A_PUSHER.to_string(),
None,
None,
)
.await;
+3 -3
View File
@@ -278,9 +278,9 @@ pub struct Config {
#[arg(long, env = "GITLAWB_METRICS_ADDR", default_value = "")]
pub metrics_addr: String,
/// Shutdown grace setting, in seconds. Currently not enforced: the value
/// is logged at startup and axum drains in-flight requests on the shutdown
/// signal with no upper bound. Default: 30s.
/// Maximum time to wait for in-flight requests to drain on shutdown, in
/// seconds. After this elapses, the server returns 503 to anything still
/// in flight and exits. Default: 30s.
#[arg(long, env = "GITLAWB_SHUTDOWN_GRACE_SECS", default_value_t = 30)]
pub shutdown_grace_secs: u64,
+176 -14
View File
@@ -786,27 +786,27 @@ pub struct Db {
}
impl Db {
#[cfg(test)]
pub fn for_testing(pool: PgPool) -> Self {
Self { pool }
}
/// Test-only: access the underlying Postgres connection pool.
///
/// The production accessor was removed on main (ed76adf) once every
/// production caller had migrated to typed `Db` methods; this restores
/// it for tests only, for the raw-SQL fixtures and closed-pool cases in
/// this change that have no typed equivalent.
#[cfg(any(test, feature = "test-harness"))]
#[cfg(test)]
pub fn pool(&self) -> &PgPool {
&self.pool
}
#[cfg(any(test, feature = "test-harness"))]
pub fn for_testing(pool: PgPool) -> Self {
Self { pool }
}
/// Test-only: apply the full schema to a fresh pool. `#[sqlx::test]`
/// provisions an empty per-test database, so DB-backed tests must run this
/// before seeding. Reuses the production `migrate()` path (the advisory lock
/// is harmless on an isolated test DB and migrations are idempotent).
#[cfg(any(test, feature = "test-harness"))]
#[cfg(test)]
pub(crate) async fn run_migrations(&self) -> Result<()> {
self.migrate().await
}
@@ -2372,6 +2372,16 @@ const MIGRATIONS: &[Migration] = &[
"CREATE INDEX IF NOT EXISTS idx_webhook_deliveries_pending ON webhook_deliveries (sent_at, created_at)",
],
},
Migration {
version: 47,
name: "sync_queue_origin_timestamp",
stmts: &[
// The source timestamp accompanies a ref update across either
// transport. It remains nullable for rows from older peers and
// manually-triggered syncs, which must not report a fabricated lag.
"ALTER TABLE sync_queue ADD COLUMN IF NOT EXISTS origin_timestamp TEXT",
],
},
];
/// Max distinct source repos recorded per pinned object (F1, #173 jatmn round 8).
@@ -3532,6 +3542,7 @@ pub struct SyncQueueItem {
pub ref_name: String,
pub new_sha: String,
pub cid: Option<String>,
pub origin_timestamp: Option<String>,
pub status: String,
pub enqueued_at: String,
}
@@ -3544,10 +3555,11 @@ impl Db {
ref_name: &str,
new_sha: &str,
cid: Option<&str>,
origin_timestamp: Option<&str>,
) -> Result<()> {
sqlx::query(
"INSERT INTO sync_queue (id, repo, node_did, ref_name, new_sha, cid, status, enqueued_at)
VALUES ($1, $2, $3, $4, $5, $6, 'pending', $7)
"INSERT INTO sync_queue (id, repo, node_did, ref_name, new_sha, cid, origin_timestamp, status, enqueued_at)
VALUES ($1, $2, $3, $4, $5, $6, $7, 'pending', $8)
ON CONFLICT DO NOTHING",
)
.bind(Uuid::new_v4().to_string())
@@ -3556,6 +3568,7 @@ impl Db {
.bind(ref_name)
.bind(new_sha)
.bind(cid)
.bind(origin_timestamp)
.bind(Utc::now().to_rfc3339())
.execute(&self.pool)
.await?;
@@ -3598,7 +3611,7 @@ impl Db {
SELECT id FROM sync_queue WHERE status = 'pending'
ORDER BY COALESCE(attempted_at, enqueued_at) ASC LIMIT $1
)
RETURNING id, repo, node_did, ref_name, new_sha, cid, status, enqueued_at",
RETURNING id, repo, node_did, ref_name, new_sha, cid, origin_timestamp, status, enqueued_at",
)
.bind(limit)
.bind(Utc::now().to_rfc3339())
@@ -3613,6 +3626,7 @@ impl Db {
ref_name: r.get("ref_name"),
new_sha: r.get("new_sha"),
cid: r.get("cid"),
origin_timestamp: r.get("origin_timestamp"),
status: r.get("status"),
enqueued_at: r.get("enqueued_at"),
})
@@ -5392,6 +5406,25 @@ impl Db {
/// this once and processes each row by re-deriving the push event,
/// the per-ref cert, and the anchor handoff.
#[allow(dead_code)] // wired by the handler refactor in the next slice
/// The instant this request's refs actually moved: the latest `applied_at`
/// stamp across its applied transitions, or `None` when none are applied
/// yet. Shared definition of "push landed" for the live replication tail,
/// which must match the deferred drain's `max(applied_at) ?? created_at`
/// rule so live and deferred pushes report one lag origin.
pub async fn request_push_landed_at(&self, request_id: &str) -> Result<Option<String>> {
let row = sqlx::query_scalar::<_, Option<String>>(
"SELECT MAX(applied_at) FROM pending_ref_transitions WHERE request_id = $1 AND state = 'applied'",
)
.bind(request_id)
.fetch_one(&self.pool)
.await?;
Ok(row)
}
/// Test-only: the drain callers that used this moved to the request-scoped
/// `request_push_landed_at` and count helpers; kept for the transition-state
/// tests below.
#[cfg(test)]
pub async fn list_pending_ref_transitions_applied(
&self,
limit: i64,
@@ -10454,7 +10487,7 @@ mod migration_tests {
db.migrate().await.unwrap();
}
// ── sync_queue scheduling (attempted_at, v17) ────────────────────────────
// ── sync_queue scheduling and origin timestamps (v17/v18) ───────────────
async fn enqueue_one(db: &super::Db, repo: &str) {
db.enqueue_sync(
@@ -10463,6 +10496,7 @@ mod migration_tests {
"refs/heads/main",
&"0".repeat(40),
None,
None,
)
.await
.unwrap();
@@ -10532,6 +10566,72 @@ mod migration_tests {
db.migrate().await.unwrap();
}
/// Upgrade-path test for the nullable source timestamp. The worker must
/// preserve it when it exists, while rows written before v18 remain valid.
#[sqlx::test]
async fn migration_v47_adds_sync_queue_origin_timestamp(pool: sqlx::PgPool) {
let db = super::Db::for_testing(pool);
db.migrate().await.unwrap();
sqlx::query("ALTER TABLE sync_queue DROP COLUMN origin_timestamp")
.execute(&db.pool)
.await
.unwrap();
sqlx::query("DELETE FROM schema_migrations WHERE version = 47")
.execute(&db.pool)
.await
.unwrap();
sqlx::query(
"INSERT INTO sync_queue (id, repo, node_did, ref_name, new_sha, status, enqueued_at)
VALUES ('legacy-sync', 'z6Mkfoo/legacy', 'did:key:zPEER', 'refs/heads/main', $1, 'pending', $2)",
)
.bind("0".repeat(40))
.bind("2026-08-16T00:00:00Z")
.execute(&db.pool)
.await
.unwrap();
db.migrate().await.unwrap();
let col: (String, String) = sqlx::query_as(
"SELECT data_type, is_nullable
FROM information_schema.columns
WHERE table_name = 'sync_queue' AND column_name = 'origin_timestamp'",
)
.fetch_one(&db.pool)
.await
.unwrap();
assert_eq!(col.0, "text");
assert_eq!(col.1, "YES", "origin_timestamp must be nullable");
let items = db.dequeue_pending_syncs(10).await.unwrap();
assert_eq!(items.len(), 1);
assert_eq!(items[0].origin_timestamp, None);
db.migrate().await.unwrap();
}
#[sqlx::test]
async fn dequeue_preserves_sync_origin_timestamp(pool: sqlx::PgPool) {
let db = super::Db::for_testing(pool);
db.migrate().await.unwrap();
let timestamp = "2026-08-16T00:00:00Z";
db.enqueue_sync(
"z6Mkfoo/timestamped",
"did:key:zPEER",
"refs/heads/main",
&"0".repeat(40),
None,
Some(timestamp),
)
.await
.unwrap();
let items = db.dequeue_pending_syncs(10).await.unwrap();
assert_eq!(items.len(), 1);
assert_eq!(items[0].origin_timestamp.as_deref(), Some(timestamp));
}
#[sqlx::test]
async fn migration_v27_drops_raw_assignee_index(pool: sqlx::PgPool) {
let db = super::Db::for_testing(pool);
@@ -13128,6 +13228,24 @@ mod icaptcha_quarantine_tests {
);
}
/// The sync-lag gate in `sync.rs` reads this flag rather than the
/// attempt-local one, so the preserved flag is the exact contract the
/// gate's "withheld mirrors never sample" rule depends on.
#[sqlx::test]
async fn resync_preserved_quarantine_is_visible_to_the_lag_gate(pool: PgPool) {
let db = db(pool).await;
db.upsert_mirror_repo("z6owner", "garbage", "/srv/g", None, true)
.await
.unwrap();
db.upsert_mirror_repo("z6owner", "garbage", "/srv/g", None, false)
.await
.unwrap();
assert!(
db.is_repo_quarantined("z6owner/garbage").await.unwrap(),
"the lag gate reads is_repo_quarantined: a re-sync that passes quarantined=false must still read withheld here"
);
}
/// Quarantined repos are withheld from the deduped listing surfaces.
#[sqlx::test]
async fn listings_exclude_quarantined(pool: PgPool) {
@@ -15398,8 +15516,8 @@ mod peer_authority_tests {
/// | `prune_non_public_peers` (db/mod.rs) | a delete keyed on a computed bad-DID array; cannot repoint; boot-only caller in main.rs |
/// | `seed_local_peer` (sync.rs) | excluded by test-module location: a deliberate `upsert_peer` bypass for `file://` fixtures, which the public-URL gate rejects |
/// | `a_legacy_row_can_still_refresh_its_liveness` (db/mod.rs) | test-only. Seeds a PRE-GATE row by raw SQL on purpose: `upsert_peer` cannot create one, since the gate it is testing refuses exactly that DID. The fixture models what a deployed table already holds |
/// | `gossip_ping_round_requires_two_failures_before_persisting_unreachable` (main.rs) | test-only fixture seed. Raw SQL because the test drives the readiness HYSTERESIS, which needs a row already at `last_ping_ok = TRUE` before the round runs; it never exercises the announce gate |
/// | `manual_ping_uses_readiness_without_mutating_federation_gate` (api/peers.rs) | test-only fixture seed, same shape and same reason: the row under test must pre-exist so the assertion is about what the ping does NOT rewrite |
/// | `gossip_ping_round_requires_two_failures_before_persisting_unreachable` (lib.rs) | test-only fixture seed for the gossip hysteresis round; the row must pre-exist so the assertion is about when `mark_peer_ping` fires |
/// | `seed_peer` (p2p/mod.rs) | test-only fixture seed. Raw SQL so the gossip ingest tests can drive the known-peer gate directly, including the did:web case `upsert_peer` would refuse |
/// | `federated_peer_query_is_bounded` (api/repos.rs) | test-only fixture seed for the bounded federation query; inserts reachable rows without exercising peer admission |
/// | `federated_route_annotates_peer_repos_on_real_200_body` (api/repos.rs) | test-only fixture seed; inserts the mock peer row the bounded federation query reads, so the response-envelope assertions run against a real HTTP 200 body |
@@ -15498,11 +15616,11 @@ mod peers_table_writer_guard {
1,
),
(
"manual_ping_uses_readiness_without_mutating_federation_gate",
"gossip_ping_round_requires_two_failures_before_persisting_unreachable",
1,
),
(
"gossip_ping_round_requires_two_failures_before_persisting_unreachable",
"manual_ping_uses_readiness_without_mutating_federation_gate",
1,
),
("mark_peer_ping", 1),
@@ -16609,4 +16727,48 @@ mod policy_fence_tests {
"the unfenced sentinel skips the epoch comparison"
);
}
/// The live replication tail reads its land instant through the same rule
/// the deferred drain uses: max applied_at across the request's applied
/// transitions, or None so the caller falls back to created_at.
#[sqlx::test]
async fn request_push_landed_at_returns_max_applied_or_none(pool: PgPool) {
let db = db(pool).await;
let repo_id = seed_repo(&db).await;
db.insert_pending_ref_transitions(
"req-live",
&repo_id,
"did:key:zNode",
"did:key:zPusher",
&[
crate::api::repos::RefUpdate {
ref_name: "refs/heads/main".into(),
old_sha: "0".repeat(40),
new_sha: "a".repeat(40),
},
crate::api::repos::RefUpdate {
ref_name: "refs/heads/dev".into(),
old_sha: "0".repeat(40),
new_sha: "b".repeat(40),
},
],
"sig",
"sigin",
"digest",
)
.await
.unwrap();
// Nothing applied yet: the caller must fall back to created_at.
assert_eq!(db.request_push_landed_at("req-live").await.unwrap(), None);
db.mark_pending_ref_transitions_applied("req-live")
.await
.unwrap();
let landed = db.request_push_landed_at("req-live").await.unwrap();
assert!(
landed.is_some(),
"an applied request must report a land instant"
);
// A different request's rows never leak into this request's instant.
assert_eq!(db.request_push_landed_at("req-other").await.unwrap(), None);
}
}
+10
View File
@@ -1711,6 +1711,15 @@ fn spawn_deferred_tail_after_proof(
}
let repo_id = req.repo_id.clone();
let pusher_did = req.pusher_did.clone();
// The truest landing instant available: the applied_at stamp on the
// accepted transitions (written when the refs actually moved), with the
// request row's created_at (push acceptance) as the fallback. Either is
// truer than this deferred drain's Utc::now().
let push_landed_at = accepted
.iter()
.filter_map(|t| t.applied_at.clone())
.max()
.unwrap_or_else(|| req.created_at.clone());
let landed: Vec<(String, String, String)> = accepted
.iter()
.map(|c| (c.ref_name.clone(), c.old_sha.clone(), c.new_sha.clone()))
@@ -1751,6 +1760,7 @@ fn spawn_deferred_tail_after_proof(
ref_updates,
std::path::PathBuf::from(repo.disk_path.clone()),
pusher_did,
Some(push_landed_at),
None,
)
.await;
+13 -4
View File
@@ -82,9 +82,18 @@ pub struct RepoStore {
}
impl RepoStore {
#[cfg(any(test, feature = "test-harness"))]
#[cfg(test)]
pub fn for_testing(repos_dir: PathBuf, lock_pool: PgPool) -> Self {
Self::new(repos_dir, None, lock_pool, Duration::from_secs(300))
Self {
repos_dir,
tigris: None,
lock_pool,
lock_held_transfer_timeout: Duration::from_secs(300),
lock_acquire_deadline: LOCK_ACQUIRE_DEADLINE,
migrated: Arc::new(tokio::sync::Mutex::new(std::collections::HashSet::new())),
pre_unlock_gate: None,
upload_site_reached: Arc::new(std::sync::atomic::AtomicUsize::new(0)),
}
}
/// Same as [`RepoStore::for_testing`] but with Tigris enabled, so the paths
@@ -1228,7 +1237,7 @@ async fn retry_fork_archive_delete(
/// `remove_dir_all` / `rename` can take it and static analysers can treat it as
/// sanitised input (CodeQL `rust/path-injection`).
#[derive(Debug, Clone)]
pub struct ValidatedRepoDiskPath(PathBuf);
pub(crate) struct ValidatedRepoDiskPath(PathBuf);
impl ValidatedRepoDiskPath {
pub(crate) fn as_path(&self) -> &Path {
@@ -2399,7 +2408,7 @@ impl RepoWriteGuard {
/// Only the UNLOCK has to wait for the reaper. This token carries everything
/// the settlement itself needs, so the request future can hold it and settle
/// the tree at the disconnect instant while the reaper still holds the lock.
pub struct TreeSettlement {
pub(crate) struct TreeSettlement {
local_path: ValidatedRepoDiskPath,
repo_name: String,
/// Shared with the guard, so the stage read here is the stage the publish
+1 -1
View File
@@ -663,7 +663,7 @@ pub(crate) fn publish_lock(local_path: &Path) -> Arc<Mutex<()>> {
/// Owns a snapshot temp dir from extraction through handoff to [`RepoSnapshot`].
/// Dropped when the `spawn_blocking` join result is abandoned, so cancellation
/// before the outer future resumes still removes the directory.
pub struct TempSnapshotDir {
pub(crate) struct TempSnapshotDir {
path: PathBuf,
}
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+96 -2
View File
@@ -5,6 +5,8 @@
//!
//! * is push traffic flowing? — `gitlawb_pushes_total`
//! * is fetch traffic flowing? — `gitlawb_fetches_total`
//! * how long do successful push and fetch operations take? —
//! `gitlawb_push_duration_seconds` / `gitlawb_fetch_duration_seconds`
//! * are signature checks passing or failing? —
//! `gitlawb_auth_successes_total` / `gitlawb_auth_failures_total`
//! * is the sync worker making progress? —
@@ -18,6 +20,8 @@
//! `gitlawb_gossip_replay_guard_saturated_total`
//! * how big are the packs we're sending and receiving? —
//! `gitlawb_pack_size_bytes`
//! * how long after an origin lands a push is it visible on this mirror? —
//! `gitlawb_sync_lag_seconds`
//! * a single `gitlawb_info{version, did}` gauge = 1, for joins/dashboards
//! * currently-connected peer count — `gitlawb_peers_connected`
//! * reconciliation sweep gaps found and filled —
@@ -34,11 +38,10 @@
//! free.
//!
//! Follow-ups (not in this module):
//! * per-route latency histograms (TraceLayer already gives us spans)
//! * per-peer p2p counters
//! * ipfs / pinata counters
use std::sync::OnceLock;
use std::{sync::OnceLock, time::Duration};
use prometheus::{
Encoder, Histogram, HistogramOpts, IntCounter, IntCounterVec, IntGauge, IntGaugeVec, Opts,
@@ -60,6 +63,9 @@ static WEBHOOK_DELIVERIES: OnceLock<IntCounterVec> = OnceLock::new();
static GOSSIP_INGEST: OnceLock<IntCounterVec> = OnceLock::new();
static GOSSIP_REPLAY_GUARD_SATURATED: OnceLock<IntCounter> = OnceLock::new();
static PACK_SIZE: OnceLock<Histogram> = OnceLock::new();
static PUSH_DURATION: OnceLock<Histogram> = OnceLock::new();
static FETCH_DURATION: OnceLock<Histogram> = OnceLock::new();
static SYNC_LAG: OnceLock<Histogram> = OnceLock::new();
static PEERS_CONNECTED: OnceLock<IntGauge> = OnceLock::new();
static RECONCILIATION_GAPS_FOUND: OnceLock<IntCounter> = OnceLock::new();
static RECONCILIATION_GAPS_FILLED: OnceLock<IntCounter> = OnceLock::new();
@@ -125,6 +131,51 @@ fn init_inner(version: &str, node_did: &str) {
.expect("register gitlawb_fetches_total");
FETCHES.set(fetches).expect("set FETCHES once");
let push_duration = Histogram::with_opts(
HistogramOpts::new(
"gitlawb_push_duration_seconds",
"Wall time of a successful git push (receive-pack) request in seconds, from handler entry to the point its effects are recorded",
)
.buckets(operation_duration_buckets()),
)
.expect("gitlawb_push_duration_seconds definition");
registry
.register(Box::new(push_duration.clone()))
.expect("register gitlawb_push_duration_seconds");
PUSH_DURATION
.set(push_duration)
.expect("set PUSH_DURATION once");
let fetch_duration = Histogram::with_opts(
HistogramOpts::new(
"gitlawb_fetch_duration_seconds",
"Duration of successful git fetch (upload-pack) completion requests in seconds",
)
.buckets(operation_duration_buckets()),
)
.expect("gitlawb_fetch_duration_seconds definition");
registry
.register(Box::new(fetch_duration.clone()))
.expect("register gitlawb_fetch_duration_seconds");
FETCH_DURATION
.set(fetch_duration)
.expect("set FETCH_DURATION once");
let sync_lag = Histogram::with_opts(
HistogramOpts::new(
"gitlawb_sync_lag_seconds",
"Lag from the origin push-land timestamp (lag_origin) to successful local mirror visibility in seconds",
)
.buckets(vec![
1.0, 5.0, 10.0, 30.0, 60.0, 120.0, 300.0, 600.0, 1_800.0, 3_600.0,
]),
)
.expect("gitlawb_sync_lag_seconds definition");
registry
.register(Box::new(sync_lag.clone()))
.expect("register gitlawb_sync_lag_seconds");
SYNC_LAG.set(sync_lag).expect("set SYNC_LAG once");
let auth_successes = IntCounterVec::new(
Opts::new(
"gitlawb_auth_successes_total",
@@ -302,6 +353,27 @@ pub fn record_fetch(repo: &str) {
}
}
/// Observe the elapsed time of a successful git push handler.
pub fn observe_push_duration(duration: Duration) {
observe_duration(&PUSH_DURATION, duration);
}
/// Observe the elapsed time of a successful git fetch completion request.
pub fn observe_fetch_duration(duration: Duration) {
observe_duration(&FETCH_DURATION, duration);
}
/// Observe end-to-end lag until an origin ref update is visible on this mirror.
pub fn observe_sync_lag(duration: Duration) {
observe_duration(&SYNC_LAG, duration);
}
fn observe_duration(histogram: &OnceLock<Histogram>, duration: Duration) {
if let Some(h) = histogram.get() {
h.observe(duration.as_secs_f64());
}
}
/// Test-only: current `gitlawb_fetches_total` value for `repo` (0 if the registry
/// is not initialized). Lets api-layer tests assert the completed-fetch count with
/// a unique label instead of scraping the encoded text.
@@ -420,6 +492,12 @@ pub fn observe_pack_size(bytes: f64) {
}
}
fn operation_duration_buckets() -> Vec<f64> {
vec![
0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0, 60.0, 120.0, 300.0,
]
}
/// Update the currently-connected peer count gauge.
pub fn set_peers_connected(count: i64) {
if let Some(g) = PEERS_CONNECTED.get() {
@@ -485,6 +563,9 @@ mod tests {
.expect("PUSHES set after init")
.with_label_values(&["alice/repo"])
.inc();
observe_push_duration(Duration::from_millis(42));
observe_fetch_duration(Duration::from_millis(24));
observe_sync_lag(Duration::from_secs(12));
record_reconciliation_gaps_found(7);
record_reconciliation_gaps_filled(3);
@@ -501,6 +582,16 @@ mod tests {
body.contains("gitlawb_pushes_total{repo=\"alice/repo\"} 1"),
"expected the incremented counter to be visible in: {body}"
);
for metric in [
"gitlawb_push_duration_seconds",
"gitlawb_fetch_duration_seconds",
"gitlawb_sync_lag_seconds",
] {
assert!(
body.contains(&format!("# TYPE {metric} histogram")),
"expected {metric} histogram in: {body}"
);
}
assert!(
body.contains("gitlawb_reconciliation_gaps_found_total 7"),
"expected the reconciliation gaps-found counter to be visible in: {body}"
@@ -546,6 +637,9 @@ mod tests {
// no diagnostic value.
record_gossip_ingest("accepted");
observe_pack_size(1024.0);
observe_push_duration(Duration::from_secs(1));
observe_fetch_duration(Duration::from_secs(1));
observe_sync_lag(Duration::from_secs(1));
set_peers_connected(0);
}
+87 -22
View File
@@ -306,8 +306,9 @@ impl IngestLimiters {
///
/// 0 is the versionless form: the field set this struct shipped with, before
/// `v` existed. It is not a placeholder for "unset", it is a real version whose
/// wire encoding happens to omit the key.
pub(crate) const CURRENT_REF_UPDATE_VERSION: u32 = 0;
/// wire encoding happens to omit the key. 1 adds `lag_origin`, the first
/// version-gated field: it is emitted and verified only under v1.
pub(crate) const CURRENT_REF_UPDATE_VERSION: u32 = 1;
/// True for the version whose key is omitted from the wire form. Free function
/// because `skip_serializing_if` takes a path, not a closure.
@@ -376,8 +377,15 @@ pub struct RefUpdateEvent {
pub old_sha: String,
/// SHA after the push
pub new_sha: String,
/// RFC-3339 timestamp
/// RFC-3339 timestamp of the publish, freshness-checked at ingest
pub timestamp: String,
/// RFC-3339 instant the origin push landed, for sync-lag measurement only.
/// Kept out of `timestamp` so `check_freshness` still sees the publish
/// instant: a push whose durability wait ran long would otherwise sign an
/// already-stale event. Optional for older peers; ingest falls back to
/// `timestamp` when it is absent.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub lag_origin: Option<String>,
/// Certificate ID (from the ref certificate, if issued)
pub cert_id: Option<String>,
/// IPFS CID of the latest commit object (set after pinning completes)
@@ -1192,6 +1200,14 @@ pub(crate) async fn ingest_ref_update(
event.v, CURRENT_REF_UPDATE_VERSION
));
}
// `lag_origin` is a v1 field: a v0 event carrying it is a protocol
// violation, not a newer peer's wire form, and refusing it keeps the field
// set inside each version's signing bytes exactly what that version means.
if event.v < 1 && event.lag_origin.is_some() {
return IngestOutcome::Rejected(
"v0 ref-update event must not carry the v1 field lag_origin".to_string(),
);
}
// The cheapest denial under enforcement, ahead of every gate that reads a
// field: an unsigned event is refused before its `node_did` is resolved or
@@ -1498,6 +1514,12 @@ pub(crate) async fn ingest_ref_update(
&event.ref_name,
&event.new_sha,
event.cid.as_deref(),
Some(
event
.lag_origin
.as_deref()
.unwrap_or(event.timestamp.as_str()),
),
)
.await
{
@@ -7193,7 +7215,7 @@ mod tests {
#[test]
fn ref_update_event_round_trip_with_owner_did() {
let event = RefUpdateEvent {
v: 0,
v: CURRENT_REF_UPDATE_VERSION,
node_did: "did:key:zNode".into(),
pusher_did: "did:key:zPusher".into(),
repo: "zOwner/myrepo".into(),
@@ -7202,6 +7224,7 @@ mod tests {
old_sha: "0000000000000000000000000000000000000000".into(),
new_sha: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa".into(),
timestamp: "2026-07-02T12:00:00Z".into(),
lag_origin: Some("2026-07-02T11:59:30Z".into()),
cert_id: None,
cid: None,
sig: None,
@@ -7213,6 +7236,7 @@ mod tests {
let deserialized: RefUpdateEvent = serde_json::from_value(json).unwrap();
assert_eq!(deserialized.owner_did, Some("did:key:zOwner".into()));
assert_eq!(deserialized.lag_origin, Some("2026-07-02T11:59:30Z".into()));
}
#[test]
@@ -7255,7 +7279,7 @@ mod tests {
/// field is Some so the serialized form exercises the widest field set.
fn populated_event() -> RefUpdateEvent {
RefUpdateEvent {
v: 0,
v: CURRENT_REF_UPDATE_VERSION,
node_did: "did:key:zNode".into(),
pusher_did: "did:key:zPusher".into(),
repo: "zOwner/myrepo".into(),
@@ -7264,6 +7288,7 @@ mod tests {
old_sha: "0000000000000000000000000000000000000000".into(),
new_sha: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa".into(),
timestamp: "2026-07-02T12:00:00Z".into(),
lag_origin: Some("2026-07-02T11:59:30Z".into()),
cert_id: Some("cert-1".into()),
cid: Some("bafycid".into()),
sig: None,
@@ -7318,30 +7343,53 @@ mod tests {
cid: Option<String>,
}
let mut event = populated_event();
event.sig = Some("c2lnbmF0dXJl".into());
let bytes = serde_json::to_vec(&event).unwrap();
// The fleet rule this version split is built around:
//
// * A v0 event is the pre-change wire form: no `v` key, no `lag_origin`,
// and it must still parse under the OLD struct verbatim.
// * A v1 event is for nodes that bumped with us. An un-upgraded peer
// rejects it on the `v` key itself, which is the LOUD version drop the
// ingest ceiling was designed to name, not a silent signature
// mismatch against bytes it could never compute.
let mut v0_event = populated_event();
v0_event.v = 0;
v0_event.lag_origin = None;
v0_event.sig = Some("c2lnbmF0dXJl".into());
let bytes = serde_json::to_vec(&v0_event).unwrap();
let legacy: LegacyRefUpdateEvent = serde_json::from_slice(&bytes)
.expect("new-code bytes must deserialize under the old field set");
.expect("v0 bytes must deserialize under the old field set");
assert_eq!(legacy.repo, "zOwner/myrepo");
assert_eq!(legacy.owner_did, Some("did:key:zOwner".into()));
// An UNSIGNED event carries no `sig` key at all, so it is byte-identical
// An UNSIGNED v0 event carries no `sig` key at all, so it is byte-identical
// in shape to the pre-change wire form and parses even under the strict
// reader. This is what `skip_serializing_if` buys; drop it and a
// `"sig": null` key appears here and this goes red.
let unsigned_bytes = serde_json::to_vec(&populated_event()).unwrap();
v0_event.sig = None;
let unsigned_bytes = serde_json::to_vec(&v0_event).unwrap();
let strict: StrictLegacyRefUpdateEvent = serde_json::from_slice(&unsigned_bytes)
.expect("an unsigned event must carry no field the pre-change reader did not know");
.expect("an unsigned v0 event must carry no field the pre-change reader did not know");
assert_eq!(strict.repo, "zOwner/myrepo");
// A SIGNED event does carry the new key, and that is intentional, not a
// compatibility bug: the permissive reader above is what makes it
// harmless. Pinning the refusal here documents `sig` as the one added
// field, so a SECOND addition cannot slip in unnoticed.
serde_json::from_slice::<StrictLegacyRefUpdateEvent>(&bytes)
.expect_err("a signed event must be visibly carrying the added `sig` key");
// A SIGNED v0 event does carry the added `sig` key, and that is
// intentional, not a compatibility bug: the permissive reader above is
// what makes it harmless. Pinning the refusal here documents `sig` as
// an added field, so a SECOND unsigned-field addition cannot slip in
// unnoticed.
v0_event.sig = Some("c2lnbmF0dXJl".into());
serde_json::from_slice::<StrictLegacyRefUpdateEvent>(
&serde_json::to_vec(&v0_event).unwrap(),
)
.expect_err("a signed v0 event must be visibly carrying the added `sig` key");
// A v1 event is refused by the pre-change reader on the `v` key:
// refusing to parse IS the version drop old nodes perform, the same
// failure the ingest ceiling produces in the other direction.
serde_json::from_slice::<StrictLegacyRefUpdateEvent>(
&serde_json::to_vec(&populated_event()).unwrap(),
)
.expect_err("a v1 event must fail loudly on the `v` key for a pre-change reader");
}
#[test]
@@ -7398,12 +7446,13 @@ mod tests {
/// this failure, and the emit-to-ingest round trip is structurally blind to
/// it because both sides re-serialize the same new declaration order.
const GOLDEN_SIGNING_BYTES: &str = concat!(
r#"{"node_did":"did:key:zNode","pusher_did":"did:key:zPusher","#,
r#"{"v":1,"node_did":"did:key:zNode","pusher_did":"did:key:zPusher","#,
r#""repo":"zOwner/myrepo","owner_did":"did:key:zOwner","#,
r#""ref_name":"refs/heads/main","#,
r#""old_sha":"0000000000000000000000000000000000000000","#,
r#""new_sha":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","#,
r#""timestamp":"2026-07-02T12:00:00Z","cert_id":"cert-1","cid":"bafycid"}"#,
r#""timestamp":"2026-07-02T12:00:00Z","lag_origin":"2026-07-02T11:59:30Z","#,
r#""cert_id":"cert-1","cid":"bafycid"}"#,
);
#[test]
@@ -7448,9 +7497,15 @@ mod tests {
/// The all-`None` optional shape: what the production publish site emits.
fn all_none_optionals_event() -> RefUpdateEvent {
RefUpdateEvent {
// v0 pins the versionless form: the key stays off the wire and the
// signing bytes are byte-identical to what a pre-version build emits.
v: 0,
owner_did: None,
cert_id: None,
cid: None,
// None skips the key entirely, so the production-shape golden stays
// byte-identical even though lag_origin exists.
lag_origin: None,
..populated_event()
}
}
@@ -7675,8 +7730,11 @@ mod tests {
let keypair = Keypair::generate();
// Signed at v0, where the key is absent from the signing bytes.
// Raising it on the received copy makes the key appear.
// Raising it on the received copy makes the key appear. The v0 shape
// drops lag_origin too, since v0 may not carry the v1 field.
let mut at_zero = event_for(&keypair);
at_zero.v = 0;
at_zero.lag_origin = None;
sign_ref_update(&keypair, &mut at_zero).unwrap();
verify_ref_update(&at_zero).expect("baseline must verify before tampering");
let mut raised = at_zero.clone();
@@ -7981,7 +8039,14 @@ mod tests {
&bytes_of(&event),
"v1 event",
Some((
"unsupported ref-update event version 1; this build understands version 0",
Box::leak(
format!(
"unsupported ref-update event version {}; this build understands version {}",
CURRENT_REF_UPDATE_VERSION + 1,
CURRENT_REF_UPDATE_VERSION
)
.into_boxed_str(),
),
"an unknown version must be named as one, never reported as a signature failure",
)),
)
+101 -3
View File
@@ -16,7 +16,9 @@ use std::collections::HashMap;
use std::path::Path;
use std::path::PathBuf;
use std::sync::Arc;
use std::time::Duration;
use chrono::{DateTime, Utc};
use gitlawb_core::identity::Keypair;
use tracing::{error, info, warn};
@@ -180,6 +182,27 @@ fn resolve_origin_url(peers: &[crate::db::PeerRecord], node_did: &str) -> Option
.map(|p| p.http_url.trim_end_matches('/').to_string())
}
/// Convert a sender-supplied RFC-3339 ref-update timestamp into an observable
/// lag. Missing, malformed, and future timestamps are deliberately ignored:
/// older peers and clock-skewed peers must not create fabricated observations.
const MAX_SYNC_LAG_OBSERVATION: Duration = Duration::from_secs(86_400);
fn sync_lag_from_origin_timestamp(
origin_timestamp: Option<&str>,
visible_at: DateTime<Utc>,
) -> Option<Duration> {
let origin = DateTime::parse_from_rfc3339(origin_timestamp?).ok()?;
let lag = visible_at
.signed_duration_since(origin.with_timezone(&Utc))
.to_std()
.ok()?;
// The origin timestamp is unauthenticated input from a known-but-not-proofed
// peer: an absurd age (a 1970 date, a stale clock) would park a fabricated
// observation in the histogram's tail buckets, so past a day the value is
// ignored rather than sampled.
(lag <= MAX_SYNC_LAG_OBSERVATION).then_some(lag)
}
async fn process_batch(
db: &Db,
config: &Config,
@@ -396,7 +419,7 @@ async fn process_batch(
false
};
// Register in DB so git smart HTTP can serve the mirrored repo
let _ = db
let mirror_visible = db
.upsert_mirror_repo(
owner_short,
repo_name,
@@ -404,7 +427,34 @@ async fn process_batch(
machine_id,
quarantined,
)
.await;
.await
.is_ok();
// The lag is push-to-visible: sample at registration, before
// the encrypted-blob tail and mark_sync_done. A quarantined
// mirror is admitted but withheld from readers, so it reports
// no visibility lag. The gate reads the PERSISTED flag, not the
// attempt-local one: on a re-sync `quarantined` is false while
// `upsert_mirror_repo` preserves an existing quarantine, and
// sampling then would record visibility the readers still deny.
let withheld = db.is_repo_quarantined(&item.repo).await.unwrap_or(true);
// Only sample when the queued outcome is what the mirror now
// shows: if the origin advanced the same ref past this row
// before the fetch ran, the mirror carries the newer tip and
// this row's visibility was never real. Deletions confirm when
// the ref is absent.
let tip = mirror_ref_tip(&local_path, &item.ref_name).await;
let outcome_visible = if item.new_sha.bytes().all(|b| b == b'0') {
tip.is_none()
} else {
tip.as_deref() == Some(item.new_sha.as_str())
};
if mirror_visible && !withheld && outcome_visible {
if let Some(lag) =
sync_lag_from_origin_timestamp(item.origin_timestamp.as_deref(), Utc::now())
{
crate::metrics::observe_sync_lag(lag);
}
}
// Option B2: carry the encrypted withheld-blob envelopes too, so an
// authorized reader can recover private content from this mirror if
// the origin dies. `item.repo` is the slug "{owner_short}/{name}",
@@ -649,6 +699,32 @@ async fn git_run(args: &[&str]) -> anyhow::Result<()> {
Ok(())
}
/// The sha `ref_name` resolves to inside `repo`, or `None` when the ref is
/// absent or the read fails. Used by the lag observation gate to confirm the
/// queued outcome is what the mirror actually carries before sampling.
async fn mirror_ref_tip(repo: &std::path::Path, ref_name: &str) -> Option<String> {
// allow-unbounded-git: bounded by the tokio::time::timeout wrapping this
// read below, plus kill_on_drop; rev-parse is a metadata lookup, not a
// transfer, so the 10s cap is generous.
let mut cmd = tokio::process::Command::new("git");
cmd.kill_on_drop(true);
let out = tokio::time::timeout(
std::time::Duration::from_secs(10),
cmd.args(["-C"])
.arg(repo)
.args(["rev-parse", "--verify", ref_name])
.output(),
)
.await
.ok()?
.ok()?;
if !out.status.success() {
return None;
}
let sha = String::from_utf8_lossy(&out.stdout).trim().to_string();
(!sha.is_empty()).then_some(sha)
}
/// Run a git subprocess, ignoring a non-zero exit. Used for idempotent
/// `config --unset`, which exits non-zero when the key is already absent.
async fn git_run_lenient(args: &[&str]) {
@@ -1582,11 +1658,32 @@ mod tests {
}
async fn enqueue(db: &Db, repo: &str, did: &str) {
db.enqueue_sync(repo, did, "refs/heads/main", &"0".repeat(40), None)
db.enqueue_sync(repo, did, "refs/heads/main", &"0".repeat(40), None, None)
.await
.unwrap();
}
#[test]
fn sync_lag_uses_origin_timestamp_only_when_it_is_in_the_past() {
let visible_at = chrono::DateTime::parse_from_rfc3339("2026-08-16T00:00:12Z")
.unwrap()
.with_timezone(&chrono::Utc);
assert_eq!(
sync_lag_from_origin_timestamp(Some("2026-08-16T00:00:00Z"), visible_at),
Some(Duration::from_secs(12))
);
assert_eq!(sync_lag_from_origin_timestamp(None, visible_at), None);
assert_eq!(
sync_lag_from_origin_timestamp(Some("not-a-timestamp"), visible_at),
None
);
assert_eq!(
sync_lag_from_origin_timestamp(Some("2026-08-16T00:00:13Z"), visible_at),
None
);
}
fn dir_entries(dir: &Path) -> Vec<String> {
let mut names: Vec<String> = std::fs::read_dir(dir)
.unwrap()
@@ -2808,6 +2905,7 @@ mod tests {
cid: None,
status: "pending".to_string(),
enqueued_at: chrono::Utc::now().to_rfc3339(),
origin_timestamp: None,
};
async fn status_of(pool: &sqlx::PgPool, id: &str) -> String {
let row: (String,) = sqlx::query_as("SELECT status FROM sync_queue WHERE id = $1")
-3
View File
@@ -6,7 +6,6 @@ use std::sync::OnceLock;
/// The fixture delays or hangs in its own process, so the production watchdog
/// can terminate it without depending on a shell or descendant process support.
#[allow(dead_code)]
pub(super) enum Behavior<'a> {
Delay(u64),
Hang,
@@ -15,7 +14,6 @@ pub(super) enum Behavior<'a> {
LogTypes(&'a Path),
}
#[allow(dead_code)]
pub(super) struct GitShim {
_directory: tempfile::TempDir,
executable: PathBuf,
@@ -29,7 +27,6 @@ impl Deref for GitShim {
}
}
#[allow(dead_code)]
pub(super) fn create(name: &str, behavior: Behavior<'_>) -> GitShim {
static COMPILED: OnceLock<tempfile::TempDir> = OnceLock::new();
let compiled = COMPILED.get_or_init(|| {
-601
View File
@@ -1,601 +0,0 @@
//! Test-only spawn surface for the real-node deny harness (see
//! `tests/deny_harness.rs`). Feature-gated behind `test-harness` so it never
//! compiles into the production binary. It exposes a single boot constructor so
//! the out-of-crate integration test can bring up a real node over a bound TCP
//! socket without the integration crate needing access to `graphql`,
//! `rate_limit`, `repo_store`, or the other internals `AppState` is built from.
//!
//! The node is built the same way `test_support::build_state` builds it (p2p
//! disabled, real migrated pool), but bound on `127.0.0.1:0` and served through
//! the real `axum::serve` stack with connect-info, so the middleware order,
//! body limits, and per-IP rate limiters all run exactly as in production.
use std::net::SocketAddr;
use std::path::PathBuf;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
use std::time::Duration;
use chrono::Utc;
use clap::Parser;
use sqlx::PgPool;
use tokio::net::TcpListener;
use tokio::sync::watch;
use uuid::Uuid;
use gitlawb_core::identity::Keypair;
use crate::config::Config;
use crate::db::{Db, PullRequest, RepoRecord, VisibilityMode};
use crate::rate_limit::{RateLimiter, TrustedProxy};
use crate::state::AppState;
/// A running node bound to an ephemeral port. End every test with
/// [`TestNode::shutdown`]: it joins the serve task and closes the pool, so
/// `#[sqlx::test]`'s `DROP DATABASE` cleanup (which runs the moment the test
/// future returns) never races the server's still-open connections. Dropping
/// without it signals shutdown, aborts any remaining serve task, and removes
/// the temp repository directory, a fallback for tests that return or panic
/// before reaching teardown.
pub struct TestNode {
/// Base URL of the running node, e.g. `http://127.0.0.1:54321`.
pub base_url: String,
/// The node's own DID (for building requests that reference it).
pub node_did: String,
shutdown_tx: watch::Sender<bool>,
repos_dir: PathBuf,
/// Seeding handle over the same pool the node serves from. Kept private so
/// the integration crate seeds through the methods below rather than
/// naming `Db`/`RepoRecord` directly.
db: Arc<Db>,
/// The detached `axum::serve` task, joined by [`Self::shutdown`]. `None`
/// once taken, so the `Drop` fallback never overlaps a completed teardown.
/// Carries `serve`'s own `io::Result<()>` so `shutdown()` can observe it
/// rather than discard it (see the coverage note there).
server: Option<tokio::task::JoinHandle<std::io::Result<()>>>,
/// A clone of the served pool, kept so [`Self::shutdown`] can close it
/// (sqlx pool clones share one inner, so closing this closes the clones
/// inside `AppState`/`Db` too).
pool: PgPool,
/// Test-only: when set, `Drop` skips the graceful watch signal so ONLY
/// `handle.abort()` tears the serve task down. Defaults false, so every
/// other test keeps both redundant teardown mechanisms. Set via
/// [`Self::force_abort_only_teardown`] to fence that the abort alone
/// releases the database when the graceful chain is broken.
suppress_graceful_shutdown_on_drop: bool,
}
impl Drop for TestNode {
fn drop(&mut self) {
// Fallback for tests that return or panic before reaching
// [`TestNode::shutdown`]: flip the shared shutdown signal so the serve
// task exits gracefully, abort whatever is left of it, and remove the
// temp repos dir. Drop cannot await, so it cannot join the server or
// close the pool; a test that already failed should not panic again in
// teardown. All three actions are idempotent or no-ops after
// `shutdown()` (which takes the handle), so running afterwards is
// harmless.
//
// The abort matters when the graceful chain does not run to completion
// before sqlx's `DROP DATABASE` cleanup: `JoinHandle::abort()` is a
// synchronous cancellation request callable from non-async Drop, and
// the aborted task's future (with the `PgPool` clones inside its
// router/state) is dropped on a subsequent scheduler tick, which sqlx
// cleanup's own awaits provide before the drop statement runs. The
// race is closed by those scheduling points, not by the abort alone.
//
// `suppress_graceful_shutdown_on_drop` (test-only) forces the abort to
// carry teardown by itself, so a test can fence that the abort line is
// load-bearing; it is false everywhere else.
if !self.suppress_graceful_shutdown_on_drop {
let _ = self.shutdown_tx.send(true);
}
if let Some(handle) = self.server.take() {
handle.abort();
}
let _ = std::fs::remove_dir_all(&self.repos_dir);
}
}
/// Outcome of joining the serve task during teardown, split so the wedged-task
/// path is observable to a test (which injects a short timeout) without waiting
/// the production teardown window.
enum ServeTeardown {
/// The task returned (carrying `serve`'s `io::Result<()>`) — either within
/// the timeout, or in the race between the timeout firing and the abort
/// landing. A task that actually finished is never a wedge.
Graceful(std::io::Result<()>),
/// The task did not finish within the timeout and was aborted (cancelled)
/// and reaped.
TimedOutAborted,
/// The serve task panicked. The payload is carried out so the caller can
/// resume unwinding only AFTER the pool is closed and the temp dir removed.
Panicked(Box<dyn std::any::Any + Send>),
}
/// Join `handle` within `timeout`; on expiry `abort()` it and await the aborted
/// handle so the serve task (and the `PgPool` clones inside its router/state) is
/// reaped BEFORE the caller returns and `#[sqlx::test]` runs `DROP DATABASE`.
/// Takes `&mut` (a `JoinHandle` is `Unpin`) so ownership is retained across the
/// timeout and the handle can be aborted on expiry — the previous code moved the
/// handle into `timeout` and, on elapse, dropped it, which DETACHES the still
/// running task rather than aborting it, leaking the test database (#194 F2).
async fn join_or_abort(
handle: &mut tokio::task::JoinHandle<std::io::Result<()>>,
timeout: Duration,
) -> ServeTeardown {
match tokio::time::timeout(timeout, &mut *handle).await {
// A panic in the serve task is carried, not unwound here: the caller
// must close the pool and remove the repos dir before resuming it.
Ok(join_result) => match join_result {
Ok(result) => ServeTeardown::Graceful(result),
Err(e) => ServeTeardown::Panicked(e.into_panic()),
},
Err(_elapsed) => {
handle.abort();
match handle.await {
// The task finished in the race between the timeout firing and
// the abort landing — honor its result, not a false wedge.
Ok(result) => ServeTeardown::Graceful(result),
Err(e) if e.is_cancelled() => ServeTeardown::TimedOutAborted,
Err(e) => ServeTeardown::Panicked(e.into_panic()),
}
}
}
}
/// Allocate a process-unique temp directory for a spawned node's repositories
/// without pulling in the `tempfile` dev-dependency (which is unavailable to
/// the library crate under `--features test-harness`).
fn unique_repos_dir() -> PathBuf {
static COUNTER: AtomicU64 = AtomicU64::new(0);
let n = COUNTER.fetch_add(1, Ordering::Relaxed);
std::env::temp_dir().join(format!("gitlawb-deny-harness-{}-{}", std::process::id(), n))
}
/// Build an [`AppState`] over the given migrated pool, mirroring
/// `test_support::build_state` but with a real on-disk repository directory so
/// git smart-HTTP routes can serve. P2P is always disabled.
fn build_state(db: Arc<crate::db::Db>, pool: PgPool, repos_dir: PathBuf) -> AppState {
let keypair = Keypair::generate();
let scan_token_key = crate::state::AppState::derive_scan_token_key(&keypair);
let webhook_secret_key = crate::state::AppState::derive_webhook_secret_key(&keypair);
let task_cursor_key = crate::api::task_cursor::TaskCursorKey::derive(&keypair.to_seed());
let node_did = keypair.did();
let (ref_tx, _) = tokio::sync::broadcast::channel(1);
let (task_tx, _) = tokio::sync::broadcast::channel(1);
let schema = Arc::new(crate::graphql::build_schema(
db.clone(),
ref_tx.clone(),
task_tx.clone(),
task_cursor_key.clone(),
));
AppState {
config: Arc::new(Config::parse_from(["gitlawb-node"])),
db,
node_did,
node_keypair: Arc::new(keypair),
p2p: None,
http_client: Arc::new(reqwest::Client::new()),
ref_update_tx: ref_tx,
task_event_tx: task_tx,
graphql_schema: schema,
task_cursor_key,
machine_id: None,
repo_store: crate::git::repo_store::RepoStore::for_testing(repos_dir, pool),
rate_limiter: RateLimiter::new(100, Duration::from_secs(60)),
create_ip_rate_limiter: RateLimiter::new(1000, Duration::from_secs(3600)),
push_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
close_issue_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
federated_rate_limiter: RateLimiter::new(12, Duration::from_secs(60)),
task_read_rate_limiter: RateLimiter::new(1200, Duration::from_secs(3600)),
ipfs_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
ipfs_work_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
ipfs_max_history_walks: crate::api::ipfs::MAX_HISTORY_WALKS_PER_REQUEST,
ipfs_max_legacy_probes: crate::api::ipfs::MAX_LEGACY_PROBES_PER_REQUEST,
ipfs_legacy_scan_page_rows: crate::api::ipfs::LEGACY_SCAN_PAGE_ROWS,
ipfs_max_legacy_scan_rows: crate::api::ipfs::MAX_LEGACY_SCAN_ROWS_PER_REQUEST,
ipfs_max_legacy_scan_rule_bytes: crate::api::ipfs::MAX_LEGACY_SCAN_RULE_BYTES_PER_REQUEST,
ipfs_scan_token_key: Arc::new(scan_token_key),
webhook_secret_key: Arc::new(webhook_secret_key),
ipfs_max_served_object_bytes: crate::api::ipfs::MAX_SERVED_OBJECT_BYTES,
push_limiter_trust: TrustedProxy::None,
sync_trigger_rate_limiter: RateLimiter::new(60, Duration::from_secs(3600)),
peer_write_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
shutdown_tx: watch::channel(false).0,
git_read_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_write_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_push_advert_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_encrypt_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
pin_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
encrypt_inflight: crate::state::EncryptInflight::new(),
repo_write_leases: crate::state::RepoWriteLeases::new(8),
git_read_per_caller: crate::rate_limit::PerCallerConcurrency::with_default_max_keys(16),
git_push_advert_per_caller: crate::rate_limit::PerCallerConcurrency::with_default_max_keys(
8,
),
git_write_per_caller: crate::rate_limit::PerCallerConcurrency::with_default_max_keys(8),
git_ipfs_walk_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_ipfs_walk_per_caller: crate::rate_limit::PerCallerConcurrency::with_default_max_keys(
16,
),
git_bin: "git".to_string(),
}
}
/// Spawn a real node bound to `127.0.0.1:0` over the given (already-created,
/// empty) test pool. Runs the schema migrations, builds the router, and serves
/// it on a background task through the production `axum::serve` stack with
/// connect-info so per-IP layers key on the real peer. Returns once the socket
/// is bound and accepting connections.
pub async fn spawn_node(pool: PgPool) -> TestNode {
let db = Arc::new(crate::db::Db::for_testing(pool.clone()));
db.run_migrations()
.await
.expect("test schema migrations should apply");
let repos_dir = unique_repos_dir();
std::fs::create_dir_all(&repos_dir).expect("create temp repos dir");
let state = build_state(db.clone(), pool.clone(), repos_dir.clone());
let node_did = state.node_did.to_string();
let shutdown_tx = state.shutdown_tx.clone();
let mut shutdown_rx = shutdown_tx.subscribe();
let router = crate::server::build_router(state);
let listener = TcpListener::bind(("127.0.0.1", 0))
.await
.expect("bind ephemeral port");
let addr = listener.local_addr().expect("read bound addr");
let server = tokio::spawn(async move {
axum::serve(
listener,
router.into_make_service_with_connect_info::<SocketAddr>(),
)
.with_graceful_shutdown(async move {
let _ = shutdown_rx.changed().await;
})
.await
});
TestNode {
base_url: format!("http://{addr}"),
node_did,
shutdown_tx,
repos_dir,
db,
server: Some(server),
pool,
suppress_graceful_shutdown_on_drop: false,
}
}
impl TestNode {
/// Test-only: make `Drop` tear the serve task down through `handle.abort()`
/// alone by suppressing the graceful watch signal. Used to prove the abort
/// is load-bearing (that the database is released even when the graceful
/// chain is broken); no production or normal-teardown path calls this.
pub fn force_abort_only_teardown(&mut self) {
self.suppress_graceful_shutdown_on_drop = true;
}
/// Graceful async teardown: signal shutdown, join the serve task, close
/// the pool, remove the temp repos dir. Call this at the end of every
/// test. `#[sqlx::test]` issues `DROP DATABASE` as soon as the test
/// future returns, so the server (which owns pool clones inside
/// `AppState`/`Db`) must be gone and the pool closed before then, or the
/// cleanup fails and leaks per-test databases. A serve task that does not
/// exit within 10s panics loudly: a wedged graceful shutdown must fail
/// the test, not hang CI.
///
/// When `self` drops on return, `Drop` re-sends the signal and re-removes
/// the dir; both are idempotent, and the serve handle was already taken,
/// so the overlap is harmless.
pub async fn shutdown(self) {
self.shutdown_with_timeout(Duration::from_secs(10)).await;
}
/// Body of [`Self::shutdown`] with an injectable teardown timeout so a test
/// can drive the wedged-shutdown path without waiting the production 10s.
/// Production always calls this with 10s.
async fn shutdown_with_timeout(mut self, timeout: Duration) {
let _ = self.shutdown_tx.send(true);
let outcome = match self.server.take() {
Some(mut handle) => Some(join_or_abort(&mut handle, timeout).await),
None => None,
};
// Release the pool and temp dir on EVERY path before returning OR
// panicking. A wedged graceful shutdown is exactly when in-flight
// per-connection tasks still hold `PgPool` clones; aborting the serve
// future alone does not reliably drop them, so the explicit
// `pool.close()` (which closes the shared inner) is what frees them
// before `#[sqlx::test]` runs `DROP DATABASE`. Panicking before this
// (the previous behavior) skipped the close on the one path that needs
// it most (#194 F2).
self.pool.close().await;
let _ = std::fs::remove_dir_all(&self.repos_dir);
match outcome {
// Honest coverage contract: on axum 0.8.8,
// `WithGracefulShutdown::into_future` always resolves `Ok(())` (the
// shutdown signal is treated as success and `accept()` errors are
// retried forever rather than returned), so this `Err` arm is
// unreachable today - every existing harness test exercises only the
// `Ok` path here. Forward-compat insurance for a future axum where
// `serve` can fail: that should surface as a clear panic, not be
// silently discarded.
Some(ServeTeardown::Graceful(result)) => {
result.expect("axum::serve exited with an error");
}
Some(ServeTeardown::TimedOutAborted) => {
panic!(
"serve task did not exit within {timeout:?} of the shutdown \
signal (wedged graceful shutdown)"
);
}
Some(ServeTeardown::Panicked(payload)) => std::panic::resume_unwind(payload),
None => {}
}
}
/// Insert a repo owned by `owner_did` and return its repo id. Mirrors
/// `test_support::seed_repo` (the `disk_path` field is unused by the
/// integration path — `RepoStore` computes the on-disk path from
/// `repos_dir`/owner/name, see [`Self::seed_bare_repo`]).
pub async fn seed_repo(&self, owner_did: &str, name: &str, is_public: bool) -> String {
let now = Utc::now();
let record = RepoRecord {
id: Uuid::new_v4().to_string(),
name: name.to_string(),
owner_did: owner_did.to_string(),
description: None,
is_public,
default_branch: "main".to_string(),
created_at: now,
updated_at: now,
disk_path: format!("/tmp/{name}"),
forked_from: None,
machine_id: None,
};
self.db.create_repo(&record).await.expect("seed repo");
record.id
}
/// Mark `branch` as protected on `repo_id`, so a non-owner push to it is
/// rejected by the branch-protection gate (used by the deny-prober's
/// protected-push probe — #195, F3).
pub async fn seed_protected_branch(&self, repo_id: &str, branch: &str, created_by: &str) {
self.db
.protect_branch(repo_id, branch, created_by)
.await
.expect("seed protected branch");
}
/// Seed an open PR `number` in `repo_id`, authored by `author_did`. The
/// author-or-owner close gate (`close_pr`) loads the PR *before* it runs, so a
/// deny-prober row for `.../pulls/{number}/close` needs the PR to exist or a
/// stranger gets a 404 (absent entity) instead of the 403 the gate emits.
pub async fn seed_pr(&self, repo_id: &str, number: i64, author_did: &str) {
let now = Utc::now().to_rfc3339();
let pr = PullRequest {
id: Uuid::new_v4().to_string(),
repo_id: repo_id.to_string(),
number,
title: "seed pr".to_string(),
body: None,
author_did: author_did.to_string(),
source_branch: "feature".to_string(),
target_branch: "main".to_string(),
status: "open".to_string(),
merged_by_did: None,
merged_at: None,
created_at: now.clone(),
updated_at: now,
};
self.db.create_pr(&pr).await.expect("seed pr");
}
/// Seed issue `issue_id` (a git-ref JSON blob authored by `author_did`) in the
/// repo's bare on-disk store. Like [`Self::seed_pr`], the `close_issue`
/// owner-or-author gate loads the issue before it runs, so the deny-prober row
/// for `.../issues/{id}/close` needs the issue present to reach the 403.
pub fn seed_issue(&self, owner_did: &str, name: &str, issue_id: &str, author_did: &str) {
let slug = owner_did.replace([':', '/'], "_");
let bare = self.repos_dir.join(&slug).join(format!("{name}.git"));
let json = format!(r#"{{"author":"{author_did}"}}"#);
crate::git::issues::create_issue(&bare, issue_id, &json).expect("seed issue");
}
/// Add a path-scoped visibility rule restricting `path_glob` to
/// `reader_dids` (empty = only the owner). Mode B keeps object SHAs intact
/// and withholds blob content, which is what the read/replication gates
/// enforce.
pub async fn withhold_path(
&self,
repo_id: &str,
path_glob: &str,
reader_dids: &[String],
created_by: &str,
) {
self.db
.set_visibility_rule(
repo_id,
path_glob,
VisibilityMode::B,
reader_dids,
created_by,
)
.await
.expect("set visibility rule");
}
/// Record `pinned_cids` rows for every blob OID returned by
/// [`Self::seed_bare_repo`] so `/ipfs/{cid}` can resolve via provenance on
/// current main (the table is the primary lookup path). CIDs are computed
/// from raw object bytes exactly as the production pin path does
/// (`Cid::from_git_object_bytes`), not from the git oid hash.
/// Returns each seeded path's canonical CID for use in `GET /ipfs/{cid}`.
pub async fn seed_pinned_cids_for_repo(
&self,
owner_did: &str,
repo_name: &str,
repo_id: &str,
oids: &std::collections::HashMap<String, String>,
) -> std::collections::HashMap<String, String> {
let slug = owner_did.replace([':', '/'], "_");
let bare = self.repos_dir.join(&slug).join(format!("{repo_name}.git"));
let mut cids = std::collections::HashMap::new();
for (path, oid_hex) in oids {
if path == "HEAD" {
continue;
}
let (_ty, raw) = crate::git::store::read_object(&bare, oid_hex)
.expect("read seeded object")
.expect("seeded object exists in bare repo");
let preliminary = gitlawb_core::cid::Cid::from_git_object_bytes(&raw).to_string();
let cid = preliminary
.parse::<cid::CidGeneric<64>>()
.expect("parse production pin cid")
.to_string();
self.db
.record_pinned_cid_with_source(oid_hex, &cid, repo_id)
.await
.expect("seed pinned_cids row");
let roundtrip = self
.db
.oids_for_cid(&cid)
.await
.expect("oids_for_cid after seed");
assert!(
!roundtrip.is_empty(),
"pinned_cids seed for {path} did not round-trip cid {cid}"
);
cids.insert(path.clone(), cid);
}
cids
}
/// Create a real bare git repository on disk at the exact path
/// `RepoStore::acquire` reads (`<repos_dir>/<owner-slug>/<name>.git`), with
/// one commit on `main` containing `files` (`(path, contents)`). Returns the
/// blob OID of each path so callers can assert a withheld OID never appears
/// in served bytes (U8). Shells out to `git`, mirroring
/// `test_support`'s served-content seam.
/// `object_format` is `"sha1"` for the git smart-HTTP surfaces and
/// `"sha256"` for the content-addressed `/ipfs/{cid}` surface (whose CIDs
/// are the sha2-256 object ids).
pub fn seed_bare_repo(
&self,
owner_did: &str,
name: &str,
files: &[(&str, &str)],
object_format: &str,
) -> std::collections::HashMap<String, String> {
let run = |args: &[&str], cwd: &std::path::Path| {
// Test-harness-only fixture seeding, feature-gated out of the production
// binary; runs local git against a tempdir inside a test's own lifetime.
// allow-unbounded-git: test-only seeding, never on a request path holding a permit
let out = std::process::Command::new("git")
.args(args)
.current_dir(cwd)
.output() // allow-unbounded-git: test-harness fixture seeding only
.expect("git runs");
assert!(
out.status.success(),
"git {args:?} failed: {}",
String::from_utf8_lossy(&out.stderr)
);
String::from_utf8_lossy(&out.stdout).trim().to_string()
};
// Build a source work tree, then bare-clone it into the served path.
let src = self.repos_dir.join(format!("src-{name}"));
std::fs::create_dir_all(&src).expect("create src dir");
let fmt_arg = format!("--object-format={object_format}");
run(&["init", "-q", "-b", "main", &fmt_arg], &src);
run(&["config", "user.email", "t@t"], &src);
run(&["config", "user.name", "t"], &src);
for (path, contents) in files {
let full = src.join(path);
if let Some(parent) = full.parent() {
std::fs::create_dir_all(parent).expect("create file parent dir");
}
std::fs::write(&full, contents).expect("write seed file");
run(&["add", path], &src);
}
run(&["commit", "-q", "-m", "seed"], &src);
let mut oids = std::collections::HashMap::new();
for (path, _) in files {
let oid = run(&["rev-parse", &format!("HEAD:{path}")], &src);
oids.insert((*path).to_string(), oid);
}
// The HEAD commit oid, used as the `want` when driving upload-pack.
oids.insert("HEAD".to_string(), run(&["rev-parse", "HEAD"], &src));
let slug = owner_did.replace([':', '/'], "_");
let bare = self.repos_dir.join(&slug).join(format!("{name}.git"));
std::fs::create_dir_all(bare.parent().unwrap()).expect("create bare parent");
run(
&[
"clone",
"--bare",
"-q",
src.to_str().unwrap(),
bare.to_str().unwrap(),
],
&self.repos_dir,
);
oids
}
}
#[cfg(test)]
mod tests {
use super::{join_or_abort, ServeTeardown};
use std::time::Duration;
/// A wedged serve task (never returns within the window) must be ABORTED and
/// reaped on timeout, not detached. The previous teardown moved the
/// `JoinHandle` into `timeout` and dropped it on elapse, which detaches the
/// still-running task and leaks the test DB across `DROP DATABASE` (#194 F2).
/// Load-bearing: neutralize the abort in `join_or_abort` (return without
/// abort+await) and this flips RED — `handle.is_finished()` stays false
/// because the task was left running.
#[tokio::test]
async fn join_or_abort_aborts_a_wedged_task_on_timeout() {
let mut handle: tokio::task::JoinHandle<std::io::Result<()>> = tokio::spawn(async {
loop {
tokio::time::sleep(Duration::from_secs(3600)).await;
}
});
assert!(
matches!(
join_or_abort(&mut handle, Duration::from_millis(50)).await,
ServeTeardown::TimedOutAborted
),
"a task that never returns must be reported timed-out, not graceful"
);
assert!(
handle.is_finished(),
"the wedged serve task must be aborted and reaped, not detached"
);
}
/// A task that returns within the window is reported graceful, carrying its
/// `io::Result` — the normal teardown path.
#[tokio::test]
async fn join_or_abort_reports_graceful_when_task_exits_in_time() {
let mut handle: tokio::task::JoinHandle<std::io::Result<()>> =
tokio::spawn(async { Ok(()) });
assert!(
matches!(
join_or_abort(&mut handle, Duration::from_secs(5)).await,
ServeTeardown::Graceful(Ok(()))
),
"a prompt clean exit must be reported graceful"
);
}
}
-3
View File
@@ -54,7 +54,6 @@ pub(crate) async fn test_state(pool: PgPool) -> AppState {
/// `config::tests::enforce_owner_push_is_declared_true_independent_of_the_environment`,
/// which reads the declaration off the parser. Splitting the two keeps a parser
/// question and an authorization question from sharing one failure mode.
#[allow(dead_code)]
pub(crate) async fn test_state_with(
pool: PgPool,
configure: impl FnOnce(&mut crate::config::Config),
@@ -164,7 +163,6 @@ pub(crate) async fn app(pool: PgPool) -> Router {
/// Sets `Content-Type: application/json` — the API is JSON throughout, and
/// without it axum's `Json` extractor returns 415 before the handler runs
/// (which would make any JSON-body authz assertion a false pass).
#[allow(dead_code)]
pub(crate) fn signed_request_as(did: &str, method: Method, uri: &str, body: Body) -> Request<Body> {
Request::builder()
.method(method)
@@ -182,7 +180,6 @@ pub(crate) fn signed_request_as(did: &str, method: Method, uri: &str, body: Body
/// test pass for the wrong reason.) The accepted sockets are parked in the
/// spawned task, which dies with the test's runtime, so the peer never sees a
/// close mid-test.
#[allow(dead_code)]
pub(crate) async fn silent_http_endpoint() -> String {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let endpoint = format!("http://{}", listener.local_addr().unwrap());
File diff suppressed because it is too large. Load diff
+25 -19
View File
@@ -517,7 +517,7 @@ fn inv22_ipfs_walk_admission_reaches_every_blocking_site() {
/// drives the actual disconnect through a parked `release`. Same instrument the F3
/// gate above uses.
///
/// MUTATION (RED): move the `tokio::spawn(post_receive_replication_tail` call below
/// MUTATION (RED): move the `post_receive_replication_tail(` call below
/// `guard.release(` and the ordering assertion fails; take it out of the
/// `if disposition.spawn_tail` block and the failed-push assertion fails; gate
/// `spawn_tail`/`release_ok` on the outcome commit and
@@ -542,17 +542,21 @@ fn inv22_replication_tail_spawns_at_the_durability_boundary() {
// position (carve-out early, or durability-bound at release) must sit
// inside a `disposition.spawn_tail` gate and feed `landed_refs` —
// never the unfiltered `ref_updates`.
let spawn_count = production
.matches("tokio::spawn(post_receive_replication_tail(")
.count();
// Match the tail CALL, not the `tokio::spawn(...)` wrapper: production
// currently wraps it in `tokio::spawn(async move { ... })`, and keying on
// the wrapper's exact shape is what let this gate go blind on the refactor.
// The `pub(crate) async fn` definition is the only other text occurrence,
// filtered by the `fn ` that must precede it.
let spawn_positions: Vec<usize> = production
.match_indices("post_receive_replication_tail(")
.map(|(p, _)| p)
.filter(|&p| !production[..p].trim_end().ends_with("fn"))
.collect();
assert!(
spawn_count >= 1,
!spawn_positions.is_empty(),
"U5 gate missing: the replication tail must be spawned by git_receive_pack"
);
for (i, (pos, _)) in production
.match_indices("tokio::spawn(post_receive_replication_tail(")
.enumerate()
{
for (i, pos) in spawn_positions.iter().copied().enumerate() {
let gate = production[..pos].rfind("if disposition.spawn_tail").expect(
"U5 gate missing: every tail spawn must be gated on `if disposition.spawn_tail`",
);
@@ -578,13 +582,15 @@ fn inv22_replication_tail_spawns_at_the_durability_boundary() {
);
let call_end = production[pos..].find(");").unwrap() + pos;
let call = &production[pos..call_end];
// `landed` is the in-scope alias for `landed_refs` in the carve-out
// spawn; either name is the proven-landed set, `ref_updates` is not.
assert!(
call.contains("landed_refs"),
call.contains("landed"),
"U5 gate bypassed: tail #{i} must carry the proven-landed ref set, not unfiltered ref_updates"
);
}
let spawn = production
.find("tokio::spawn(post_receive_replication_tail(")
let spawn = *spawn_positions
.first()
.expect("U5 gate missing: the replication tail must be spawned by git_receive_pack");
let release = production
.find(".release(disposition.release_ok)")
@@ -715,11 +721,11 @@ fn inv26_step3_live_and_drain_share_apply_request_effects() {
}
/// #26 Split PR 1 step 4 — the periodic queue-lifecycle purge is
/// wired in `lib.rs` and the contract is enforced by the `idx_receive_pack_requests_completed_at`
/// wired in `main.rs` and the contract is enforced by the `idx_receive_pack_requests_completed_at`
/// partial index from v41.
///
/// Assertions:
/// 1. `lib.rs` calls `purge_request_queue` once at boot (well, on
/// 1. `main.rs` calls `purge_request_queue` once at boot (well, on
/// the spawn-task interval) and the spawn function exists.
/// 2. The DB helpers `purge_completed_receive_pack_requests` and
/// `purge_completed_pending_ref_transitions` exist in `db/mod.rs`.
@@ -729,14 +735,14 @@ fn inv26_step3_live_and_drain_share_apply_request_effects() {
/// introduce the state, but the invariant holds for the future).
#[test]
fn inv26_step4_queue_lifecycle_purge_is_wired() {
let boot_src = src("lib.rs");
let main_src = src("main.rs");
assert!(
boot_src.contains("spawn_queue_lifecycle_sweep"),
"lib.rs must spawn the periodic queue-lifecycle purge"
main_src.contains("spawn_queue_lifecycle_sweep"),
"main.rs must spawn the periodic queue-lifecycle purge"
);
assert!(
boot_src.contains("purge_request_queue"),
"lib.rs must call purge_request_queue on the periodic sweep"
main_src.contains("purge_request_queue"),
"main.rs must call purge_request_queue on the periodic sweep"
);
let db_src = src("db/mod.rs");
-235
View File
@@ -1,235 +0,0 @@
//! INV-8 deny assertion. A denial must be an actual refusal (4xx) whose body
//! and headers carry none of the data it is withholding, and must never be a 2xx
//! (an empty-200 rendered as success is the denial-as-success bug this guards).
//!
//! The check is split into a pure `check_denied` (unit-tested below with the
//! self-check scenarios) and an async `assert_denied` wrapper that reads a real
//! `reqwest::Response`.
/// Pure deny check. Returns `Err(reason)` when the response is not a clean
/// denial. `withheld` are exact tokens (OIDs, paths, private slugs) that must
/// not appear anywhere in the body; register short-OID prefixes and encoded
/// path forms as separate tokens at the call site (a raw substring scan cannot
/// normalize them).
pub fn check_denied(
status: u16,
body: &str,
expected: u16,
withheld: &[&str],
) -> Result<(), String> {
// The harness only ever expects denials; a non-4xx expectation is a
// programming error in the test, not a node behavior under test.
if !(400..500).contains(&expected) {
return Err(format!(
"test bug: assert_denied expected status {expected} is not a 4xx"
));
}
// INV-8: a denial rendered as success is the headline bug. Catch it
// explicitly so the failure message names it, rather than only failing the
// status-equality check below.
if (200..300).contains(&status) {
return Err(format!(
"denial rendered as success: got {status}, expected denial {expected}. \
body={body:?}"
));
}
if status != expected {
return Err(format!(
"expected denial status {expected}, got {status}. body={body:?}"
));
}
for token in withheld {
if token.is_empty() {
continue;
}
if body.contains(token) {
return Err(format!(
"withheld token {token:?} leaked in denial body: {body:?}"
));
}
}
Ok(())
}
/// Like [`check_denied`] but also scans serialized response headers for withheld
/// tokens. Call sites that only have a body should keep using `check_denied`.
pub fn check_denied_with_headers(
status: u16,
body: &str,
header_text: &str,
expected: u16,
withheld: &[&str],
) -> Result<(), String> {
check_denied(status, body, expected, withheld)?;
for token in withheld {
if token.is_empty() {
continue;
}
if header_text.contains(token) {
return Err(format!(
"withheld token {token:?} leaked in denial headers: {header_text:?}"
));
}
}
Ok(())
}
/// Replace each withheld token in `s` with `[REDACTED]` so a test-failure
/// witness does not itself leak the secret it caught.
fn redact_withheld(s: &str, withheld: &[&str]) -> String {
let mut result = s.to_string();
for token in withheld {
if !token.is_empty() {
result = result.replace(token, "[REDACTED]");
}
}
result
}
/// Drive a real response through the deny check and panic on failure. Reads the
/// full body once.
pub async fn assert_denied(resp: reqwest::Response, expected: u16, withheld: &[&str]) {
let status = resp.status().as_u16();
let header_text = resp
.headers()
.iter()
.filter_map(|(name, value)| {
value
.to_str()
.ok()
.map(|v| format!("{}: {}", name.as_str(), v))
})
.collect::<Vec<_>>()
.join("\n");
// A body-read failure must NOT fold to "" — an empty body passes the withheld
// scan vacuously, so an unreadable denial (a mid-body reset/truncation) would be
// certified leak-free without ever inspecting the bytes. Fail loud instead. A
// legitimately empty body is `Ok("")`, not `Err`, so the honest empty-401 denials
// still pass.
let body = resp
.text()
.await
.expect("read denial body: an unreadable denial cannot be certified leak-free");
if let Err(reason) = check_denied_with_headers(status, &body, &header_text, expected, withheld)
{
// Panic with a fixed string: CodeQL's taint analysis traces withheld
// tokens from check_denied through redact_withheld into the panic sink,
// even though redact_withheld strips them at runtime. A fixed message
// keeps no tainted data on the sink path. The test name identifies
// which probe failed; check_denied's unit tests cover the error detail.
let _ = redact_withheld(&reason, withheld);
panic!("deny assertion failed: status/body/header leak check");
}
}
#[cfg(test)]
mod tests {
use super::{check_denied, check_denied_with_headers, redact_withheld};
#[test]
fn clean_403_with_no_leak_passes() {
let r = check_denied(
403,
r#"{"error":"only the repo owner can do this"}"#,
403,
&["a1b2c3"],
);
assert!(r.is_ok(), "{r:?}");
}
#[test]
fn empty_200_is_flagged_as_denial_rendered_as_success() {
let r = check_denied(200, "", 403, &[]);
assert!(r.is_err());
assert!(r.unwrap_err().contains("rendered as success"));
}
#[test]
fn leaking_403_fails_and_names_the_token() {
let withheld = "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef";
let body = format!("not found near object {withheld}");
let r = check_denied(403, &body, 403, &[withheld]);
assert!(r.is_err());
assert!(r.unwrap_err().contains(withheld));
}
#[test]
fn wrong_status_fails() {
// A 404 when the owner gate's 403 was expected must not pass: it means
// an earlier layer denied, so the gate under test was never exercised.
let r = check_denied(404, "not found", 403, &[]);
assert!(r.is_err());
assert!(r.unwrap_err().contains("expected denial status 403"));
}
#[test]
fn non_4xx_expected_is_a_test_bug() {
// Guards the harness itself: expecting a non-denial status is a test
// authoring error, not a node behavior, and must be rejected.
let r = check_denied(403, "denied", 200, &[]);
assert!(r.is_err());
assert!(r.unwrap_err().contains("not a 4xx"));
}
#[test]
fn empty_withheld_token_never_false_positives() {
// An empty token must be skipped, never treated as "contained in every
// body" — otherwise every denial would spuriously fail as a leak.
let r = check_denied(403, "any body", 403, &[""]);
assert!(r.is_ok(), "{r:?}");
}
#[test]
fn leaking_header_fails_and_names_the_token() {
let withheld = "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef";
let headers = format!("X-Debug: object {withheld}");
let r = check_denied_with_headers(
404,
r#"{"error":"repository not found"}"#,
&headers,
404,
&[withheld],
);
assert!(r.is_err());
let err = r.unwrap_err();
assert!(err.contains("leaked in denial headers"));
assert!(err.contains(withheld));
}
#[test]
fn clean_headers_with_clean_body_passes() {
let withheld = "a1b2c3";
let r = check_denied_with_headers(
404,
r#"{"error":"repository not found"}"#,
"Content-Type: application/json",
404,
&[withheld],
);
assert!(r.is_ok(), "{r:?}");
}
#[test]
fn redact_withheld_replaces_secrets() {
let withheld = "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef";
let msg = format!("withheld token {withheld:?} leaked in body: near {withheld}");
let redacted = redact_withheld(&msg, &[withheld]);
assert!(
!redacted.contains(withheld),
"token must be redacted: {redacted}"
);
assert!(
redacted.contains("[REDACTED]"),
"redaction marker must appear: {redacted}"
);
}
#[test]
fn redact_withheld_leaves_clean_strings() {
let result = redact_withheld(
"no secrets here",
&["deadbeefdeadbeefdeadbeefdeadbeefdeadbeef"],
);
assert_eq!(result, "no secrets here");
}
}
-33
View File
@@ -1,33 +0,0 @@
//! Shared support for the real-node deny harness: the deny assertion (U4), the
//! RFC-9421 signing client (U2), and (via `gitlawb_node::test_harness`) the
//! node spawner (U3).
pub mod assert;
// U2 fixture state matrix + per-gate-class probe generators; driven by U3.
#[allow(dead_code)]
pub mod probe;
// U1 deny-bearing route registry; fields consumed by U2/U3 as they land.
#[allow(dead_code)]
pub mod routes;
pub mod signing;
/// A reqwest client with a bounded request timeout, so a wedged git subprocess or
/// route fails the suite rather than hanging it until CI kills the job (#195, F1).
/// The two real-node probes that drive git subprocesses use this; the inline
/// twins at the upload-pack and registry-sweep sites use the same pattern.
#[allow(dead_code)]
pub fn bounded_client() -> reqwest::Client {
reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(30))
.build()
.unwrap()
}
/// Build an anonymous (unsigned) GET request for `path` under `base_url`.
/// Extracted from inline call sites so URL construction goes through a
/// function boundary, matching the `signed_request` pattern in `support::signing`.
#[allow(dead_code)]
pub fn anon_get(client: &reqwest::Client, base_url: &str, path: &str) -> reqwest::RequestBuilder {
let url = format!("{base_url}{path}");
client.get(url)
}
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
@@ -1,32 +0,0 @@
//! Real RFC-9421 request signing for the deny harness (U2). Wraps
//! `gitlawb_core::http_sig::sign_request` (the same entry point the
//! git-remote-gitlawb helper uses in production) so a test can present a valid
//! signature for any identity, reaching the authenticated-but-wrong-owner path
//! that an injected-DID shortcut cannot.
use gitlawb_core::http_sig::sign_request;
use gitlawb_core::identity::Keypair;
/// Build a reqwest request carrying a valid RFC-9421 signature for `keypair`.
///
/// `path` is both appended to `base_url` to form the request URL and signed as
/// the `@path` component, so the value the server reconstructs and the value
/// that was signed are guaranteed identical. `body` is signed via its
/// content-digest and sent as the request body.
pub fn signed_request(
client: &reqwest::Client,
method: reqwest::Method,
base_url: &str,
path: &str,
body: Vec<u8>,
keypair: &Keypair,
) -> reqwest::RequestBuilder {
let sig = sign_request(keypair, method.as_str(), path, &body);
let url = format!("{base_url}{path}");
client
.request(method, url)
.header("content-digest", sig.content_digest)
.header("signature-input", sig.signature_input)
.header("signature", sig.signature)
.body(body)
}
+1 -30
View File
@@ -148,18 +148,6 @@ impl NodeClient {
self.send_signed("POST", path, body).await
}
/// POST with JSON body + RFC 9421 signing and an optional user-provided
/// iCaptcha proof, sent on the initial request.
pub async fn post_with_proof(
&self,
path: &str,
body: &[u8],
initial_proof: Option<&str>,
) -> Result<reqwest::Response> {
self.send_signed_with_proof("POST", path, body, initial_proof)
.await
}
/// PUT with RFC 9421 signing + transparent iCaptcha solve/retry.
pub async fn put(&self, path: &str, body: &[u8]) -> Result<reqwest::Response> {
self.send_signed("PUT", path, body).await
@@ -174,32 +162,16 @@ impl NodeClient {
/// `x-icaptcha-*` headers) solve it and retry the same signed request with
/// the proof header, up to [`MAX_ICAPTCHA_RETRIES`]. Emits an actionable
/// hint on a 401 "not an agent" (the old-CLI / unregistered failure mode).
/// Existing call path: no user-supplied proof on the first request.
async fn send_signed(
&self,
method: &str,
path: &str,
body: &[u8],
) -> Result<reqwest::Response> {
self.send_signed_with_proof(method, path, body, None).await
}
/// Sign + send a write. `initial_proof`, if provided, is attached to the
/// first request. On a 403 iCaptcha challenge advertised via
/// `x-icaptcha-*` headers, solve and retry up to MAX_ICAPTCHA_RETRIES.
async fn send_signed_with_proof(
&self,
method: &str,
path: &str,
body: &[u8],
initial_proof: Option<&str>,
) -> Result<reqwest::Response> {
let mut proof = initial_proof.map(str::to_owned);
let mut proof: Option<String> = None;
let mut attempts = 0;
loop {
let resp = self.send_once(method, path, body, proof.as_deref()).await?;
let status = resp.status();
if status == reqwest::StatusCode::UNAUTHORIZED
@@ -223,7 +195,6 @@ impl NodeClient {
continue;
}
}
return Ok(resp);
}
}
+1 -40
View File
@@ -22,11 +22,6 @@ pub struct RegisterArgs {
)]
pub node: String,
/// A pre-obtained iCaptcha proof token, forwarded on the register request
/// instead of making the client solve a fresh challenge.
#[arg(long, env = "GITLAWB_ICAPTCHA_PROOF")]
pub icaptcha_proof: Option<String>,
/// Capabilities to advertise (comma-separated)
#[arg(
long,
@@ -61,7 +56,7 @@ pub async fn run(args: RegisterArgs) -> Result<()> {
}))?;
let resp = client
.post_with_proof("/api/register", &body, args.icaptcha_proof.as_deref())
.post("/api/register", &body)
.await
.context("failed to connect to node")?;
@@ -165,7 +160,6 @@ mod tests {
capabilities: vec!["git:push".to_string(), "git:fetch".to_string()],
model: None,
dir: Some(dir.path().to_path_buf()),
icaptcha_proof: None,
})
.await
.unwrap();
@@ -179,36 +173,6 @@ mod tests {
assert_eq!(content["node"].as_str().unwrap(), server.url());
}
/// A supplied `--icaptcha-proof` must ride the register request as the
/// `x-icaptcha-proof` header so an enforce-mode node accepts it without a
/// fresh solve round-trip.
#[tokio::test]
async fn register_forwards_icaptcha_proof_header() {
let dir = TempDir::new().unwrap();
write_identity(&dir);
let mut server = mockito::Server::new_async().await;
let m = server
.mock("POST", "/api/register")
.match_header("x-icaptcha-proof", "test.supplied.proof")
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"message":"Welcome","ucan":"eyJhbGci.test.token","trust_score":0.5,"expires":"2026-12-31"}"#)
.create_async()
.await;
run(RegisterArgs {
node: server.url(),
capabilities: vec!["git:push".to_string()],
model: None,
dir: Some(dir.path().to_path_buf()),
icaptcha_proof: Some("test.supplied.proof".to_string()),
})
.await
.unwrap();
m.assert();
}
/// `gl register` READS the identity from the parent of `GITLAWB_KEY`, so it has
/// to WRITE the bootstrap token there too. Sending it to `~/.gitlawb` instead is
/// silent split-brain: registration prints success, and every command that later
@@ -237,7 +201,6 @@ mod tests {
capabilities: vec!["git:push".to_string()],
model: None,
dir: None,
icaptcha_proof: None,
})
.await
.unwrap();
@@ -272,7 +235,6 @@ mod tests {
capabilities: vec!["git:push".to_string()],
model: None,
dir: Some(dir.path().to_path_buf()),
icaptcha_proof: None,
})
.await;
@@ -294,7 +256,6 @@ mod tests {
capabilities: vec![],
model: None,
dir: Some(dir.path().to_path_buf()),
icaptcha_proof: None,
})
.await;