Compare commits

...
Author SHA1 Message Date
jatmnandClaude Opus 5.5 e3aee00e9d ci: feed ci-surfaces NUL-delimited paths and pin its mapping in a test
The mapper read newline-delimited `git diff --name-only` output. In that
mode git C-quotes any path with non-ASCII, tab, double-quote or newline
characters, so crates/gl/src/é.rs arrived as "crates/gl/src/\303\251.rs",
matched no surface, and a PR whose only Rust change was such a file ran no
Rust job (Café.swift likewise skipped Swift CodeQL). Newline-delimited
input also cannot carry a path that contains a newline.

Both workflows now pass `git diff -z` output and the mapper reads it with
`read -r -d ''`; input that is not NUL-terminated is rejected (exit 2)
instead of being misread. Mapping output for every ASCII path is
unchanged (checked against the last 43 commit ranges).

scripts/test-ci-surfaces.sh pins one must-not-skip case per surface row,
the crate fan-out and widening rules, --all, the wiring rows, quoted
names, and rejection of newline-delimited input. PR Checks runs it in the
`changed surfaces` job before mapping, on every event, so a broken mapper
turns the run red rather than silently narrowing it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-04 10:08:10 -08:00
jatmnandClaude Opus 5.5 91adb5c863 ci: test gitlawb-node when bootstrap-peers.json changes
ci-surfaces.sh attributed Rust crates only by crates/<dir>/ paths, but
gitlawb-node also compiles in the repo-root bootstrap-peers.json through
include_str!. A PR that changed only that file therefore ran the Docker
smoke (docker build + --version, which embeds the JSON without parsing
it) and skipped cargo test, so embedded_seed_list_parses_successfully
("embedded bootstrap-peers.json must always parse") no longer guarded it.

Add a path -> crate table for files outside crates/ that a crate
compiles in, routed through the same check as crates/<dir>/ paths (an
unknown crate still widens to --workspace). bootstrap-peers.json now
yields rust=true with -p gitlawb-node and keeps its Docker trigger; every
other mapper output is unchanged.

The header also said anything the script cannot attribute precisely
widens the run. That holds only inside the Rust surface; a path no
surface lists (docs, README, infra/, other workflows) runs no PR Checks
job, as AGENTS.md says. Reword it to the two tiers the script implements.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-04 09:18:43 -08:00
jatmnandClaude Opus 5.5 37a0f60afc docs: say which changes widen PR CI to the whole workspace
ci-surfaces.sh has two widening tiers. A root Cargo.toml/Cargo.lock/.cargo
change (or toolchain config, or an unknown crate dir) runs the Rust jobs on
--workspace, while the other jobs still follow their own inputs. Only the CI
wiring itself (pr-checks.yml, codeql.yml, ci-surfaces.sh) and non-PR events
run every surface. AGENTS.md and the script header described both tiers as
"everything" / "the full suite" and left "CI wiring" undefined, so an edit
to, say, release.yml read as if it ran the full PR Checks suite; it runs
none of those jobs and only CodeQL's Actions analysis.

Comment and docs only; the mapper's output is unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-04 08:55:40 -08:00
jatmnandClaude Opus 5.5 888c4fd9dc ci: run PR checks and CodeQL only for the surfaces a PR touches
Every PR ran every PR Checks job on the whole workspace, and CodeQL default
setup analyzed every language, including a ~13 min macOS Swift autobuild
(55 min when it hung on main) for an 8-file macos-app/ that 4 of the last
~280 commits touched.

scripts/ci-surfaces.sh maps a PR's changed paths to the jobs that validate
them, and scopes cargo test / build --release to the touched crates plus
their transitive reverse dependencies (derived from crates/*/Cargo.toml).
Root Cargo.toml/Cargo.lock/.cargo, the CI wiring itself, and unknown crate
dirs widen to the full suite. Push to main, merge_group, and the release
flow's workflow_dispatch always run everything on the whole workspace.

codeql.yml replaces default setup with the same scoping per language and a
full analysis on push to main and weekly. It needs default setup disabled.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-04 08:38:33 -08:00
JATMNandClaude Opus 5.5 623b600835 fix(deps): bump libp2p to pick up hickory 0.26 and serde_with 3.22 (#518)
hickory-proto/hickory-resolver 0.25.2 were pulled in only by libp2p-dns
0.44. libp2p-dns 0.45 moves to hickory 0.26 but needs libp2p-core 0.44
and libp2p-identity 0.3, so the node's direct libp2p crates move to the
matching release line (core 0.44, gossipsub 0.50.1, identify 0.48,
identity 0.3, kad 0.49, quic 0.14, swarm 0.48, dns 0.45). multiaddr goes
to 0.19 to match libp2p-core and avoid compiling two copies. No source
changes were needed.

Lockfile: hickory-proto/resolver 0.25.2 -> 0.26.3 (GHSA-q2qq-hmj6-3wpp /
RUSTSEC-2026-0119, GHSA-6f2x-v7q7-m7m5), serde_with 3.18.0 -> 3.22.0
(GHSA-7gcf-g7xr-8hxj).

With the fix in, drop the RUSTSEC-2026-0118/0119 ignores from
.cargo/audit.toml as #76 requires.

Closes #76

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-11-04 16:37:21 +00:00
beardthelionandKevin Codex 9bda2e103c fix(node): authenticate gossip ref-update events before writing them (#325)
* feat(node): sign and verify gossip ref-update events

Add an optional sig field to RefUpdateEvent carrying a base64 Ed25519
signature over the event's own fields, plus signing_bytes, the single
producer of signing input on both sides, and sign/verify helpers.

The field is optional and skipped when absent, following the owner_did
precedent from #144, so an event from a signing node still deserializes
on a peer that predates this change. A nested envelope would have broken
that and is deliberately not used.

Verification resolves the key from the claimed node_did, so a signature
that does not bind the DID it claims is refused. Non-did:key and
unresolvable DIDs are refused with the peers-table denial sentences,
built from PeerWriteDenied rather than retyped so the two surfaces
cannot drift apart.

A golden test pins the signing input: a field reorder changes what every
peer signs, which no round-trip test can see.

Helpers are not wired into the ingest or emit paths yet.

* fix(node): authenticate gossip ref-update events before writing them

The gossipsub ingest path accepted a ref-update from any peer and wrote
it, while the HTTP route carrying the same payload checks the signature
against the claimed DID, requires a known peer, and validates the repo
slug. Extract the path into ingest_ref_update and apply the same three
gates in that order, so both transports answer the same input the same
way.

A present signature must verify in either mode. An absent one is refused
when GITLAWB_REQUIRE_SIGNED_PEER_WRITES is set and otherwise accepted
with a warning naming the flag, so peers that predate signing keep
federating until the fleet is ready. The DID-method refusal runs right
after parse in both modes and reuses the peers-table denial sentences.

Rejection writes nothing: neither a ref-update row nor a sync-queue
entry. Those are separate sinks and the tests assert them separately.

from_peer's comment now says it records the forwarder, not the author.

* fix(node): bound gossip ingest per source and sign outbound events

Brake the ingest path at 60 events per minute per forwarding peer,
checked ahead of the parse and the signature work. Verifying a signature
is the expensive step, so a brake behind it would sell an unauthenticated
flood the CPU the brake exists to protect. Rejection writes nothing to
either sink.

Sign events on the way out, so a fleet running with enforcement on can
verify what its peers publish. A signing failure warns and skips the
publish rather than emitting something a verifying peer would drop.

The emit-to-ingest round trip is asserted end to end, which is what
proves both sides agree on the signing input by execution rather than by
inspection.

* test(node): cover the did-method gate and correct overclaiming comments

The did-method gate had no load-bearing test: deleting it left the suite
green, because the only non-did:key case was signed, so verification
resolved the DID internally and returned the same refusal. Add the case
only the gate decides, an unsigned non-did:key event from a registered
peer with enforcement off, and correct the comment that claimed the
other test covered it.

Assert the limiter's source bound, which no test pinned, and make the
backward-compat test able to see an added field by parsing under a
deny_unknown_fields reader as well as a permissive one.

Say plainly at the peers check that authentication is not authorization:
a self-registered did:key passes it, so what the gate buys is blocking an
unregistered DID and, with the signature, blocking impersonation of an
existing peer. Drop the claim that the unsigned-acceptance warning is
covered by a test, because it is not.

An outcome of Accepted now means the row was stored; a failed write
reports itself instead of reading as success.

* fix(node): charge the ingest budget to the author, not the mesh edge

A single 60-per-forwarder budget was both too tight and pointed at the
wrong principal. Pushes emit one gossip event per ref, so a push touching
more than 60 refs lost its tail with no signal to the publisher, and
because gossipsub re-shares mesh-wide, a flood routed through an honest
neighbour spent that neighbour's budget on every receiver.

Split it. A loose pre-parse bound on the forwarding peer still caps the
CPU spent before anything is parsed or verified, which is all a
free-to-mint identity can honestly buy. A tighter bound on the
authenticated node_did sits after verification and the peer gate, so the
two durable writes are charged to whoever actually authored them, and it
is sized to admit a large push whole.

Both rejections now name the budget and its window, so a dropped
ref-update says which bound refused it rather than vanishing.

Replace the per-event full peers scan with a keyed lookup on the same
column; upsert_peer shares it.

A push above the author bound still loses its tail, now loudly. The two
transports continue to disagree about events per push: gossip publishes
per ref while sync/notify batches.

* docs(node): the did-method gate runs after the parse, not before it

Two test comments called it the pre-parse did-method gate. The parse is
above it and the gate is below; what is true is that it runs before the
signature branch, in both enforcement modes. The pre-parse label belongs
to the forwarder brake, which is genuinely the first statement, and the
nearby correct usage is where the wrong one came from.

* fix(node): charge the gossip write budget only to a proven author

An unsigned ref-update event's node_did is asserted, not proven. The author
budget was debited on that key in both signature arms, so a flood of unsigned
events naming a registered peer drained that peer's budget and its own genuine
signed updates were shed. The attacker chose the victim.

The author budget is now charged only when the signature verified. Unsigned
compatibility traffic charges a separate forwarder-keyed bucket sized at
GOSSIP_UNSIGNED_SOURCE_MAX_EVENTS (1500), above the per-author cap because a
forwarder aggregates many authors, and below the pre-parse brake; the
compile-time assert now pins all three in one chain. The shed carries its own
IngestOutcome variant so a forwarder is never named as an author.

validate_repo_slug moves above both the signature match and peer_exists, so a
malformed event costs no database round trip rather than merely being charged
to nobody. A test-only peer_exists tally proves that placement, which an
outcome assertion cannot distinguish.

* fix(node): version the signed ref-update format and refuse unknown versions

The signed payload carried no version and verify_ref_update knew one
representation, so the next signed-field addition would partition the mesh
mid-rollout: an upgraded publisher signs bytes an older receiver cannot
reproduce, and the older receiver reports a genuine event as a bad signature.

RefUpdateEvent now carries v, skipped when zero, so a v0 event's wire and
signing bytes stay byte-identical to the versionless form. GOLDEN_SIGNING_BYTES
is unchanged and every signature already issued still verifies. A frozen
artifact signed before the field existed is checked into the suite and verified
without being re-signed, since a signature a test just produced is
self-consistent by construction and proves nothing about compatibility.

An event whose version this build cannot interpret is now refused in its own
words, ahead of every other gate: no field below an unknown version has known
meaning. Without that, a v1 event was admitted outright rather than merely
mislabelled.

* docs(node): describe the real signing handoff at the publish site

The comment claimed signing was future work and the event went out unsigned.
It does not: the swarm loop signs via p2p::signed_publish_bytes immediately
before publishing, and skips the publish entirely when signing fails. Leaving
sig None here is correct, and the comment now says why rather than describing
a state the code left behind.

* docs: GITLAWB_REQUIRE_SIGNED_PEER_WRITES spans both transports

The flag gates gossip ref-update ingest as well as the HTTP announce and sync
routes, but README and .env.example still scoped it to those two routes. An
operator who upgraded the HTTP peers and then enabled it would silently drop
gossip from every publisher still running an old build, with no error visible
on the publisher's side because gossip carries no response.

Documents the cross-transport behavior and the one ordering requirement that
follows: upgrade every gossip publisher before enabling.

* fix(review): count gossip ingest outcomes and sweep the ingest limiters

Every IngestOutcome reached tracing and nothing else, so gossip-ingest health
was invisible on /metrics, the only surface this daemon exposes to a dashboard
or an alert. Adds gitlawb_gossip_ingest_events_total{outcome} alongside the
existing sync-queue and webhook counters, with the label derived from an
exhaustive match so a seventh variant fails to compile rather than shipping
uncounted. The reason strings stay in the log line; they are sender-shaped and
would make the label set unbounded.

The three ingest limiters were built as locals in the swarm task, so
sweep_rate_limiters could not reach them and their keys stayed resident until
the 200k ceiling. The swarm loop now sweeps them on its own interval, and
cleanup iterates a destructured array so a fourth limiter breaks the build
instead of being silently missed, which is the failure sweep_rate_limiters'
own comment records against the /ipfs limiter.

* test(node): close four gaps the review found in this PR's own guards

The frozen signing constants pinned only the all-Some optional shape, but the
sole publish site always emits cert_id None, so no constant covered the
encoding every real event uses. Proven during review: adding
skip_serializing_if to cert_id left the suite green while the signing input
changed for 100% of production events. A second constant now pins the
null-carrying form.

The legacy artifact's provenance assertion claimed re-capturing it from current
code was what the assertion refused. It was not: skip-when-zero means a
regenerated artifact also carries no version key, so the check passed and the
claim was unfounded. A SHA-256 over the frozen bytes does the job the message
promised, and the message now states only what it checks.

WriteFailed was the one outcome variant no test constructed, so the
independent-writes property it exists to express was unexercised. Both
directions are now driven by failing a real write.

The publish arm's body is extracted so the exact bytes handed to gossipsub are
asserted signed and accepted by ingest under enforcement. The select! dispatch
and the require_signed threading from main.rs still need a live swarm and are
named as uncovered rather than implied covered.

* fix(review): close the remaining findings from the round on this PR

The unsigned-acceptance warning fired before two gates that could still drop
the event, so a message the operator reads as an admission could be followed by
a forwarder-budget shed or an unknown-peer refusal. It now fires only once the
event is genuinely admitted, and a test pins that it fires exactly once, which
nothing checked before.

The publish site hardcoded the format version while the ingest gate compared
against the constant, so emitter and gate could drift apart silently. The
publish site now reads the constant.

The known-peer gate changed a federation precondition without documenting it: a
gossip publisher must also be a known peer on the receiving node, via bootstrap
or a prior announce, or its events are dropped in both flag modes. That is now
in the README beside the rollout ordering, derived from what the ingest path
does rather than from another docstring.

Six tests each hand-rolled the same both-modes rejection loop; they now share
one helper. The log-capture helper was left under a tmp_ prefix from the round
that added it and is renamed, since scaffolding vocabulary should not ship.

* fix(review): report unsigned gossip admissions as their own outcome and metric (#325)

An unsigned event that survives the rolling-upgrade window was returned as
IngestOutcome::Accepted, so `gitlawb_gossip_ingest_events_total{outcome="accepted"}`
mixed signature-verified admissions with unauthenticated legacy traffic. An
operator could not tell whether the fleet still relied on the compatibility
allowance or authenticated gossip was healthy.

Add IngestOutcome::UnsignedAdmitted with its own `unsigned_admitted` metric
label, returned when an unsigned event passes every gate and the writes land.
`Accepted` is now reserved for signature-verified events. Propagate through the
swarm-loop logging and the tests that asserted `Accepted` for unsigned
admissions.

Both guards proven load-bearing by execution: reverting the return-site change
goes RED on flag_off_unsigned_known_peer_event_is_accepted ("got Accepted"),
and giving UnsignedAdmitted the "accepted" label goes RED on
every_ingest_outcome_carries_a_distinct_metric_label.

* fix(node): shed an over-budget gossip author before the peer lookup (#325)

The author brake sat below `db.peer_exists`, so a signed author already past its
budget still paid a Postgres round trip before being refused. That is a brake
behind the work it bounds: the lookup is done for a request that was never going
to be admitted, and a signed author can drive it at the pre-parse source rate,
further with peer-id rotation.

The brake now answers first, and the answer is a READ rather than the charge.
`check` inserts a window for a key it has never seen, so probing with it here
would let a flood of self-minted signed DIDs occupy the bounded author map before
`peer_exists` could refuse them, and legitimate new authors would be shed once it
filled. That would have made the remedy worse than the finding. `is_over_budget`
allocates nothing, reports an unseen key as within budget, and leaves the charge
itself below the lookup, unmoved. The only behaviour that changes is WHEN an
already-over-budget author is refused.

Two guards, both proven load-bearing by execution. Removing the early shed makes
`an_over_budget_author_is_shed_without_a_peer_lookup` red on the call counter,
which observes the lookup not happening rather than restating the ordering.
Swapping the read for `check` makes `the_early_author_check_does_not_track_an_unseen_did`
red on the map size, which is the must-not that keeps the brake from becoming its
own memory-fill surface.

Found by a cross-family refute pass on the pushed head.

* chore: retrigger checks after a CodeQL infrastructure failure

The swift analysis job on run 32041989078 failed in "Set up job", before
checkout or any analysis step. GitHub refuses a rerun on default-setup
(dynamic) runs, so a new head is the only way to get a fresh scan.

* fix(deps): bump h2 to 0.4.19 for RUSTSEC-2026-0258

The branch lockfile still pinned h2 0.4.13, so cargo audit failed on
every push while main had already moved on. This is lockfile-only; no
source change in the gossip auth work.

* chore: re-trigger code quality check

---------

Co-authored-by: Kevin Codex <[email protected]>
2026-11-04 16:34:10 +00:00
JATMNandClaude Opus 5.5 151d688387 fix(deps): bump aws-smithy-json to 0.62.7 (GHSA-8ffr-xgwf-xj56) (#516)
Cargo.lock resolved aws-smithy-json 0.61.9 (via aws-sdk-s3 1.119.0) and
0.62.5 (via aws-config and the sso/ssooidc/sts clients). Both are in the
range affected by GHSA-8ffr-xgwf-xj56 / CVE-2026-18140 (<= 0.62.6):
uncontrolled recursion when skipping unknown keys in deeply nested JSON.

0.62.7 requires aws-smithy-runtime-api ^1.12.3, so the AWS SDK moves as
a set to the newest releases that still build on our 1.91 MSRV (they
declare rust-version 1.91.1; later releases need 1.94.1):
aws-config 1.8.18, aws-sdk-s3 1.137.0 (which itself requires
aws-smithy-json ^0.62.7), aws-sdk-sso 1.102.0, aws-sdk-ssooidc 1.104.0,
aws-sdk-sts 1.107.0, aws-sigv4 1.4.5, and matching aws-smithy-* crates.

aws-sigv4 1.4.5 moves to p256 0.13, which drops crypto-bigint 0.4.9 from
the lockfile. Update the RUSTSEC-2026-0387 note in .cargo/audit.toml to
match; the ignore is still needed for crypto-bigint 0.5.5.

Lockfile and audit-config comment only; no manifest or source changes.

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-11-04 15:14:11 +00:00
JATMNandClaude Opus 5.5 4d024e01bc fix(deps): resolve cargo audit failures on main (#514)
* fix(deps): resolve cargo audit failures on main

Update quinn-proto 0.11.19, rustls 0.23.46, rustls-webpki 0.103.16,
webpki-roots 1.0.11, zstd 0.13.4, zstd-safe 7.4.0 and zstd-sys 2.1.2 in
Cargo.lock to clear RUSTSEC-2026-0338..0342, 0346..0349, 0381, 0382,
0391, 0398, 0403 and 0410.

Ignore RUSTSEC-2026-0387 (crypto-bigint): the fix is only in 0.7.x, and
no dependent compatible with our tree uses it. alloy-signer (through
2.5.0) pins elliptic-curve 0.13, and aws-sigv4 >=1.5 requires Rust
1.94.1, above the 1.91 MSRV.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(audit): document RUSTSEC-2026-0387 reachability and graph-based removal

Correct the aws-sigv4 boundary (1.6.4 is the first release on
crypto-bigint 0.7; 1.5.x-1.6.3 still need ^0.5.4), mark the ignore as
reachable through the k256 PrivateKeySigner and SigV4a signing paths,
and tie removal to Cargo.lock having no crypto-bigint below 0.7.6.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-11-04 12:26:02 +08:00
beardthelion 6588590813 chore: update rustls 0.23.37 → 0.23.45 to resolve RUSTSEC-2026-0285 (#455) 2026-11-04 01:23:10 +00:00
17 changed files with 4402 additions and 626 deletions

No files matched your search

+19 -15
View File
@@ -8,24 +8,9 @@
# The audit job stays hard-fail: only the advisories explicitly listed below are
# suppressed, so any NEW advisory still breaks CI. Each entry below has no
# available upstream fix; everything fixable is fixed in Cargo.lock instead.
#
# Tracking: the hickory ignores are reachable accepted risk. A scheduled re-check
# (.github/workflows/audit-schedule.yml) runs without these ignores and fails if
# the lockfile moves off the pinned vulnerable versions while the ignores remain.
[advisories]
ignore = [
# hickory-proto 0.25.2 (pulled by libp2p-dns 0.44 -> hickory-resolver 0.25).
# REACHABLE: the node wraps its QUIC transport in the system DNS resolver
# (crates/gitlawb-node/src/p2p/mod.rs:238), so a malicious DNS response can
# trigger these DoS paths. No fix is available: the latest libp2p-dns on
# crates.io is 0.44 and pins hickory 0.25; the fix requires hickory >=0.26.1.
# REMOVE both once libp2p-dns ships a release using hickory 0.26 (verified
# 2026-06-21: cargo update -p hickory-proto locks 0 packages; still 0.25.2).
# Tracking: https://github.com/Gitlawb/node/issues/76
"RUSTSEC-2026-0118", # hickory NSEC3 closest-encloser unbounded loop
"RUSTSEC-2026-0119", # hickory O(n^2) name-compression CPU exhaustion
# rsa 0.9.10 (Marvin timing sidechannel). Present in Cargo.lock because sqlx
# resolves sqlx-mysql unconditionally, and sqlx-mysql depends on rsa. cargo
# audit scans the lockfile, so it flags rsa even though it is NOT linked into
@@ -35,4 +20,23 @@ ignore = [
# ever built with the `mysql` feature, or any other consumer of rsa enters
# the build, at which point it becomes a real reachable advisory.
"RUSTSEC-2023-0071", # rsa Marvin attack (no fix; not linked in our build)
# crypto-bigint 0.5.5 (secret-dependent branches in operations meant
# to be constant-time). REACHABLE accepted risk, on two paths:
# - alloy-signer -> elliptic-curve 0.13 -> crypto-bigint 0.5: the k256
# PrivateKeySigner that signs transactions in `gl name`
# (crates/gl/src/name.rs) and the node operator
# (crates/gitlawb-node/src/operator.rs).
# - aws-sigv4 (`sigv4a` feature, crates/gitlawb-node/Cargo.toml) ->
# crypto-bigint 0.5 (directly and via p256 0.13): SigV4a key
# derivation, used only for requests to endpoints that require SigV4a
# (e.g. S3 Multi-Region Access Points).
# Exploiting it needs local timing observation of those signing operations.
# The fix is crypto-bigint >=0.7.6, and no compatible dependent uses it:
# every alloy-signer release (through 2.5.0) requires elliptic-curve ^0.13,
# and aws-sigv4 1.6.4 is the first release on crypto-bigint 0.7 (1.5.x -
# 1.6.3 still require ^0.5.4), but it needs Rust 1.94.1, above our 1.91 MSRV
# (verified 2026-11-03). REMOVE only when Cargo.lock has no crypto-bigint
# below 0.7.6: `grep -A1 'name = "crypto-bigint"' Cargo.lock`.
"RUSTSEC-2026-0387", # crypto-bigint timing variability (no compatible fix)
]
+6 -2
View File
@@ -88,8 +88,12 @@ GITLAWB_P2P_PORT=7546
# GITLAWB_BOOTSTRAP_DISABLE_SEEDS=true for isolated dev/test networks.
GITLAWB_BOOTSTRAP_PEERS=
GITLAWB_BOOTSTRAP_DISABLE_SEEDS=false
# Require signatures on peer announce/sync write routes. Keep false during
# rolling upgrades so existing live nodes can still communicate.
# Require a peer write to prove its DID on every transport: RFC 9421 signatures
# on the announce/sync routes, and an Ed25519 payload signature on gossip
# ref-update events. Keep false during rolling upgrades so existing live nodes
# can still communicate. Upgrade every gossip-publishing peer to a build that
# signs events before setting this true, or their updates are dropped on
# arrival with no error visible on their side.
GITLAWB_REQUIRE_SIGNED_PEER_WRITES=false
# Require the authenticated pusher to be the repo owner on git-receive-pack.
+100
View File
@@ -0,0 +1,100 @@
name: CodeQL
# Advanced-setup replacement for CodeQL default setup. Default setup analyzes
# every detected language on every PR, including Swift (macos-app/), which
# autobuilds on a macOS runner for ~13 minutes per PR and has hung for 55 on
# main; that single job was the longest check on most PRs. Here a PR analyzes
# only the languages it touches (scripts/ci-surfaces.sh), while push to main
# and the weekly schedule analyze every language, so main's code-scanning
# baseline stays complete.
#
# Requires CodeQL default setup to be DISABLED (Settings -> Code security ->
# Code scanning): GitHub rejects advanced-setup uploads while it is enabled.
#
# A PR that skips a language may show "configuration not found" on the CodeQL
# check for that language; that is informational and expected.
on:
pull_request:
branches: [main]
push:
branches: [main]
schedule:
- cron: "17 4 * * 1"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: codeql-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
changes:
name: changed languages
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
codeql: ${{ steps.surfaces.outputs.codeql }}
matrix: ${{ steps.surfaces.outputs.codeql_matrix }}
steps:
- name: Check out repository
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
fetch-depth: 2
- name: Map changed paths to languages
id: surfaces
env:
EVENT_NAME: ${{ github.event_name }}
run: |
set -euo pipefail
if [ "$EVENT_NAME" = "pull_request" ]; then
# -z: literal paths; plain --name-only C-quotes unusual names.
git diff -z --name-only --no-renames HEAD^1 HEAD > changed-paths
tr '\0' '\n' < changed-paths >&2
scripts/ci-surfaces.sh < changed-paths > surfaces.out
else
scripts/ci-surfaces.sh --all > surfaces.out
fi
cat surfaces.out
grep -E '^(codeql|codeql_matrix)=' surfaces.out >> "$GITHUB_OUTPUT"
analyze:
name: Analyze (${{ matrix.language }})
needs: changes
if: needs.changes.outputs.codeql == 'true'
runs-on: ${{ matrix.runner }}
# Normal runs take <=14 min (Swift autobuild is the longest); cap a hung
# autobuild instead of letting it hold a runner for an hour.
timeout-minutes: 30
permissions:
contents: read
security-events: write
actions: read
packages: read
strategy:
fail-fast: false
matrix: ${{ fromJSON(needs.changes.outputs.matrix) }}
steps:
- name: Check out repository
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- name: Initialize CodeQL
uses: github/codeql-action/init@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
with:
languages: ${{ matrix.language }}
build-mode: ${{ matrix.build-mode }}
- name: Autobuild
if: matrix.build-mode == 'autobuild'
uses: github/codeql-action/autobuild@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
with:
category: "/language:${{ matrix.language }}"
+98 -3
View File
@@ -30,8 +30,62 @@ env:
RUST_BACKTRACE: 1
jobs:
# Decide which checks this event needs. On pull_request, each job below runs
# only when the PR touches the surface it validates, and the cargo test /
# release-build jobs are scoped to the touched crates plus their reverse
# dependencies (scripts/ci-surfaces.sh). Every other event (push to main,
# merge_group, the release flow's workflow_dispatch) runs the full suite, so
# main and the merge queue keep whole-workspace coverage and main-red-alert
# still sees a complete run. A skipped job reports as "skipped", not green.
changes:
name: changed surfaces
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
rust: ${{ steps.surfaces.outputs.rust }}
rust_packages: ${{ steps.surfaces.outputs.rust_packages }}
windows: ${{ steps.surfaces.outputs.windows }}
windows_packages: ${{ steps.surfaces.outputs.windows_packages }}
docker: ${{ steps.surfaces.outputs.docker }}
audit: ${{ steps.surfaces.outputs.audit }}
core_deps: ${{ steps.surfaces.outputs.core_deps }}
release_versioning: ${{ steps.surfaces.outputs.release_versioning }}
release_tag: ${{ steps.surfaces.outputs.release_tag }}
steps:
- name: Check out repository
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
# The pull_request checkout is the test-merge commit; depth 2 brings
# its first parent (the base tip) so the diff is exactly this PR.
fetch-depth: 2
# Every job's gate comes from the mapper, so pin its mapping first; a
# failure here fails this job and the run is red, not silently narrowed.
- name: Test the surface mapper
run: scripts/test-ci-surfaces.sh
- name: Map changed paths to surfaces
id: surfaces
env:
EVENT_NAME: ${{ github.event_name }}
run: |
set -euo pipefail
if [ "$EVENT_NAME" = "pull_request" ]; then
# -z: literal paths; plain --name-only C-quotes unusual names.
git diff -z --name-only --no-renames HEAD^1 HEAD > changed-paths
tr '\0' '\n' < changed-paths >&2
scripts/ci-surfaces.sh < changed-paths > surfaces.out
else
scripts/ci-surfaces.sh --all > surfaces.out
fi
cat surfaces.out
cat surfaces.out >> "$GITHUB_OUTPUT"
fmt-clippy:
name: fmt + clippy
needs: changes
if: needs.changes.outputs.rust == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
@@ -57,6 +111,8 @@ jobs:
release-tag-validation:
name: release tag validation
needs: changes
if: needs.changes.outputs.release_tag == 'true'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
@@ -70,6 +126,8 @@ jobs:
test:
name: test (${{ matrix.toolchain }})
needs: changes
if: needs.changes.outputs.rust == 'true'
runs-on: ubuntu-latest
timeout-minutes: 45
strategy:
@@ -114,10 +172,17 @@ jobs:
- name: cargo test
env:
DATABASE_URL: postgres://postgres:postgres@localhost:5432/gitlawb_test
run: cargo test --locked --workspace
# `--workspace`, or `-p <crate>...` for the crates a PR touches plus
# every crate that depends on them.
PACKAGES: ${{ needs.changes.outputs.rust_packages }}
run: |
# shellcheck disable=SC2086 # PACKAGES is a word list by design
cargo test --locked $PACKAGES
build-release:
name: build --release
needs: changes
if: needs.changes.outputs.rust == 'true'
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
@@ -137,10 +202,20 @@ jobs:
key: release
- name: cargo build --release
run: cargo build --locked --release --workspace
env:
PACKAGES: ${{ needs.changes.outputs.rust_packages }}
run: |
# shellcheck disable=SC2086 # PACKAGES is a word list by design
cargo build --locked --release $PACKAGES
audit:
name: cargo audit
# cargo audit reads only Cargo.lock (+ .cargo/audit.toml), so a PR that
# leaves both unchanged cannot change its verdict; a newly published
# advisory against main is caught by the push-to-main run and the weekly
# Scheduled Audit instead of turning every unrelated PR red.
needs: changes
if: needs.changes.outputs.audit == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
@@ -200,6 +275,8 @@ jobs:
msrv:
name: MSRV (Rust 1.91)
needs: changes
if: needs.changes.outputs.rust == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
@@ -223,6 +300,12 @@ jobs:
docker-build:
name: Docker build smoke test
# Runs when the image's own inputs change: the Dockerfile, the manifests
# its dependency layer copies, the lockfile, bootstrap-peers.json. A
# source-only Rust change is already compiled by `build --release` (stable)
# and `MSRV` (1.91, the image's toolchain); push to main still runs this.
needs: changes
if: needs.changes.outputs.docker == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
@@ -239,6 +322,8 @@ jobs:
test-windows:
name: test (windows, non-blocking)
needs: changes
if: needs.changes.outputs.windows == 'true'
runs-on: windows-latest
timeout-minutes: 45
# Non-blocking, mirroring the informational `beta` lane above. We ship `gl`
@@ -266,7 +351,13 @@ jobs:
key: windows
- name: cargo test (shipped Windows crates)
run: cargo test --locked -p gl -p git-remote-gitlawb
shell: bash
env:
# `-p gl -p git-remote-gitlawb`, or whichever of the two a PR affects.
PACKAGES: ${{ needs.changes.outputs.windows_packages }}
run: |
# shellcheck disable=SC2086 # PACKAGES is a word list by design
cargo test --locked $PACKAGES
# gitlawb-core is embedded by every consumer (gl, git-remote-gitlawb, the node
# daemon), so it must stay lean. This gate fails if gitlawb-core's normal
@@ -276,6 +367,8 @@ jobs:
# cannot do. Regen instructions live in the allowlist header.
core-deps-purity:
name: gitlawb-core dependency purity
needs: changes
if: needs.changes.outputs.core_deps == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
@@ -308,6 +401,8 @@ jobs:
# not its author knew this gate existed.
release-versioning:
name: release versioning coverage
needs: changes
if: needs.changes.outputs.release_versioning == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
+1 -1
View File
@@ -40,5 +40,5 @@ When a node denies a request, surface the denial to the user. Never render a den
## Toolchain and CI reality
- MSRV is Rust 1.91 (`rust-version` in the workspace `Cargo.toml`). The MSRV CI job runs `cargo check --workspace --all-targets` on exactly 1.91, so check that an API you rely on is stable there; the full test suite runs on stable.
- The PR workflow runs these jobs on every PR: `cargo fmt --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo test --workspace` (against Postgres), a release build, `cargo audit`, the MSRV check, and a Docker build. The `main` branch ruleset currently requires one approving review but does not require these checks to pass before merge; a failing job should block review approval in practice, but nothing enforces that mechanically today.
- The PR workflow's jobs are `cargo fmt --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo test` (against Postgres), a release build, `cargo audit`, the MSRV check, and a Docker build. On a PR, `scripts/ci-surfaces.sh` runs each job only when the PR touches its inputs, and scopes `cargo test` and the release build to the touched crates plus every crate that depends on them. Root `Cargo.toml`/`Cargo.lock`/`.cargo/` changes run the Rust jobs (fmt, clippy, tests, release build, MSRV) on the whole workspace; every other job still runs only when its own inputs change. Edits to the CI wiring itself (`pr-checks.yml`, `codeql.yml`, `scripts/ci-surfaces.sh`), push to `main`, and the merge queue run every job on the whole workspace. Edits to any other workflow run none of these jobs on a PR; CodeQL only analyzes them as Actions code. Run `cargo test --workspace` locally regardless. The same script limits CodeQL (`codeql.yml`) to the languages a PR touches. The `main` branch ruleset currently requires one approving review but does not require these checks to pass before merge; a failing job should block review approval in practice, but nothing enforces that mechanically today.
- Conventional commit titles (`feat:`, `fix:`, `docs:`, ...) are required by convention and drive releases, but no CI job checks them; a bad title fails review, not the pipeline.
Generated
+925 -546
View File
File diff suppressed because it is too large. Load diff
+7 -1
View File
@@ -369,6 +369,12 @@ GITLAWB_REQUIRE_SIGNED_PEER_WRITES=true
`POST /api/v1/sync/trigger` is not part of the staged rollout: it always requires a signature in both config modes and returns 401 without one, because each call drives an O(peers) outbound fan-out.
The flag is not HTTP-only. It also gates inbound gossip ref-update events on the libp2p mesh, which carry an Ed25519 payload signature rather than RFC 9421 headers because there are no HTTP headers to sign. Set to `true`, an unsigned gossip event is dropped. Set to `false`, an unsigned event is accepted with a warning during the rolling-upgrade window, and the write budget it consumes is charged to the forwarding peer rather than to the DID it claims, since an unsigned event's claimed DID is asserted and not proven. A present-but-invalid signature is refused in both modes; that is forgery, not an un-upgraded peer.
Because the flag now spans both transports, the rollout order matters in one direction: **upgrade every gossip-publishing peer to a build that signs events before you set this to `true`.** Enabling it while an old publisher is still live drops that publisher's ref-updates on arrival, and the publisher sees no error, because gossip has no response to carry one. Upgrading the HTTP peers alone is not sufficient.
There is a second precondition on gossip ingest that the flag does not control. A publisher's `node_did` must already exist in the receiving node's peers table, or the event is dropped as an unknown peer DID. This holds in both modes, signed and unsigned alike: a valid signature proves key possession, not membership. Rows reach that table over HTTP, either from `GITLAWB_BOOTSTRAP_PEERS` when this node contacts a bootstrap peer and records the DID it reports, or from a prior `POST /api/v1/peers/announce`. So a peer that only ever joined the libp2p mesh, with no HTTP announce and no bootstrap contact in either direction, will have its ref-updates dropped even with a good signature and the flag off. If gossip is silently not landing from a peer you can see on the mesh, check that its DID is in `GET /api/v1/peers` on the receiving side first.
---
## Configuration
@@ -392,7 +398,7 @@ Important node settings:
| `GITLAWB_BOOTSTRAP_PEERS` | Comma-separated HTTP peer URLs. |
| `GITLAWB_P2P_BOOTSTRAP` | Comma-separated libp2p multiaddrs. |
| `GITLAWB_BOOTSTRAP_DISABLE_SEEDS` | Disable embedded seed peers for isolated dev/test networks. |
| `GITLAWB_REQUIRE_SIGNED_PEER_WRITES` | Require signed peer announce/sync writes. Defaults to `false` during the staged rollout below. |
| `GITLAWB_REQUIRE_SIGNED_PEER_WRITES` | Require peer writes to prove their DID on every transport: RFC 9421 signatures on the announce/sync routes, and an Ed25519 payload signature on gossip ref-update events. Defaults to `false` during the staged rollout below. Upgrade every gossip publisher before enabling, or their updates are dropped with no error on their side. |
| `GITLAWB_ENFORCE_OWNER_PUSH` | Require the authenticated pusher to be the repo owner on `git-receive-pack`. **Defaults to `true`.** A `did:key` signature is authentication, not authorization — anyone can mint a key and sign — so with this off every signed caller may push to every repository, private ones included. Delegated and CI keys count as non-owners: a UCAN `git/push` capability is verified but not yet honored for authorization, so they cannot push while this is on. Set `false` only for a rolling upgrade; see [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md). |
| `GITLAWB_AUTO_SYNC` | Enable automatic sync from known peers. |
| `GITLAWB_MAX_PACK_BYTES` | Max git pack body size for smart-HTTP routes. |
+9 -9
View File
@@ -44,14 +44,14 @@ dirs-next = "2"
# withholds a canonically-equivalent path regardless of NFC/NFD form (#101).
unicode-normalization = "0.1"
reqwest = { workspace = true }
libp2p-core = "0.43"
libp2p-gossipsub = "0.49.4"
libp2p-identify = "0.47"
libp2p-identity = { version = "0.2", features = ["ed25519", "peerid", "rand"] }
libp2p-kad = "0.48"
libp2p-quic = { version = "0.13", features = ["tokio"] }
libp2p-swarm = { version = "0.47", features = ["macros", "tokio"] }
multiaddr = "0.18"
libp2p-core = "0.44"
libp2p-gossipsub = "0.50.1"
libp2p-identify = "0.48"
libp2p-identity = { version = "0.3", features = ["ed25519", "peerid", "rand"] }
libp2p-kad = "0.49"
libp2p-quic = { version = "0.14", features = ["tokio"] }
libp2p-swarm = { version = "0.48", features = ["macros", "tokio"] }
multiaddr = "0.19"
futures = "0.3"
aws-sdk-s3 = { version = "1", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
aws-config = { version = "1", features = ["behavior-version-latest"] }
@@ -72,7 +72,7 @@ alloy = { version = "1", default-features = false, features = [
"network",
"rpc-types-eth",
] }
libp2p-dns = { version = "0.44.0", features = ["tokio"] }
libp2p-dns = { version = "0.45.0", features = ["tokio"] }
[dev-dependencies]
mockito = "1"
+14
View File
@@ -2758,6 +2758,11 @@ async fn post_receive_replication_tail(
if announce {
if let Some(p2p) = &p2p_handle {
p2p.publish_ref_update(crate::p2p::RefUpdateEvent {
// Named, not literal: the ingest gate compares `v`
// against this same constant, so a bump has to move
// both ends together rather than leaving the emitter
// on a version the gate no longer accepts.
v: crate::p2p::CURRENT_REF_UPDATE_VERSION,
node_did: node_did_str.clone(),
pusher_did: pusher_did_clone.clone(),
repo: repo_slug.clone(),
@@ -2768,6 +2773,15 @@ async fn post_receive_replication_tail(
timestamp: chrono::Utc::now().to_rfc3339(),
cert_id: None,
cid: cid.map(|s| s.to_string()),
// Left unsigned here on purpose. The swarm loop
// signs the event with the node keypair via
// `p2p::signed_publish_bytes` immediately before it
// publishes, and skips the publish outright if
// signing fails, so the wire always carries a
// signature even though this construction site does
// not. Setting one here would be signed over a
// payload the publisher has not finished building.
sig: None,
})
.await;
}
+3 -1
View File
@@ -67,7 +67,9 @@ pub struct Config {
#[arg(long, env = "GITLAWB_BOOTSTRAP_PEERS", value_delimiter = ',')]
pub bootstrap_peers: Vec<String>,
/// Require RFC 9421 signatures on peer announce/sync write routes.
/// Require a peer write to prove its DID on any transport: RFC 9421
/// signatures on the peer announce/sync write routes, and a payload
/// signature on gossip ref-update events.
/// Keep false during rolling upgrades so existing live nodes can still gossip.
#[arg(
long,
+53 -9
View File
@@ -2571,15 +2571,7 @@ impl Db {
// The read-then-write race is benign: if a row appears in between,
// this write becomes an UPDATE and the UnprovenRepoint guard below
// still refuses any http_url change.
PeerWriteAuthority::Unproven
if !sqlx::query_scalar::<_, bool>(
"SELECT EXISTS(SELECT 1 FROM peers WHERE did = $1)",
)
.bind(did)
.fetch_one(&self.pool)
.await
.unwrap_or(false) =>
{
PeerWriteAuthority::Unproven if !self.peer_exists(did).await.unwrap_or(false) => {
match did.parse::<gitlawb_core::did::Did>() {
// Only reachable from the bootstrap announce-back in main.rs,
// which passes the contacted peer's raw JSON string; the
@@ -2709,6 +2701,21 @@ impl Db {
Ok(())
}
/// Whether a DID has a row in `peers`, by keyed lookup.
///
/// For callers that only need the membership answer. `list_peers` fetches
/// and materializes every row, so using it as a membership test on a hot
/// path (the gossip ingest gate) makes the cost of one event grow with the
/// size of the table.
pub async fn peer_exists(&self, did: &str) -> Result<bool> {
Ok(
sqlx::query_scalar::<_, bool>("SELECT EXISTS(SELECT 1 FROM peers WHERE did = $1)")
.bind(did)
.fetch_one(&self.pool)
.await?,
)
}
pub async fn list_peers(&self) -> Result<Vec<PeerRecord>> {
let rows = sqlx::query(
"SELECT did, http_url, last_seen, last_ping_ok, announced_at
@@ -7864,6 +7871,41 @@ mod peer_authority_tests {
db
}
/// `peer_exists` answers the membership question the gossip ingest gate
/// asks on every event, so both answers are pinned here: a registered DID
/// is true, and an unregistered one is false rather than an error or a
/// prefix match on a registered DID.
#[sqlx::test]
async fn peer_exists_answers_both_ways(pool: sqlx::PgPool) {
let db = db(pool).await;
let did = VICTIM_DID;
assert!(
!db.peer_exists(did).await.unwrap(),
"an empty peers table must answer false, not error"
);
db.upsert_peer(
did,
"https://peer.example.com",
PeerWriteAuthority::Proven(did),
)
.await
.unwrap();
assert!(db.peer_exists(did).await.unwrap());
assert!(
!db.peer_exists(OTHER_DID).await.unwrap(),
"a different DID must not match"
);
assert!(
!db.peer_exists(&VICTIM_DID[..VICTIM_DID.len() - 1])
.await
.unwrap(),
"the lookup is an equality test, not a prefix test"
);
}
/// The whole row, read back through `list_peers` rather than raw SQL, so a
/// case that claims "unchanged" is comparing every column a consumer sees.
async fn row(db: &Db, did: &str) -> Option<(String, String, Option<String>, bool, String)> {
@@ -8280,6 +8322,7 @@ mod peer_authority_tests {
/// | `a_legacy_row_can_still_refresh_its_liveness` (db/mod.rs) | test-only. Seeds a PRE-GATE row by raw SQL on purpose: `upsert_peer` cannot create one, since the gate it is testing refuses exactly that DID. The fixture models what a deployed table already holds |
/// | `gossip_ping_round_requires_two_failures_before_persisting_unreachable` (main.rs) | test-only fixture seed. Raw SQL because the test drives the readiness HYSTERESIS, which needs a row already at `last_ping_ok = TRUE` before the round runs; it never exercises the announce gate |
/// | `manual_ping_uses_readiness_without_mutating_federation_gate` (api/peers.rs) | test-only fixture seed, same shape and same reason: the row under test must pre-exist so the assertion is about what the ping does NOT rewrite |
/// | `seed_peer` (p2p/mod.rs) | test-only fixture seed. Raw SQL so the gossip ingest tests can drive the known-peer gate directly, including the did:web case `upsert_peer` would refuse |
///
/// And the `upsert_peer` CALL-SITE authority table, which the ledger above
/// structurally cannot hold, because the bootstrap site issues no SQL of its own
@@ -8377,6 +8420,7 @@ mod peers_table_writer_guard {
("prune_non_public_peers", 1),
("prune_self_peers", 1),
("seed_local_peer", 1),
("seed_peer", 1),
("upsert_peer", 2),
];
+4 -2
View File
@@ -115,7 +115,7 @@ async fn main() -> Result<()> {
}
// Load or generate the node's identity keypair
let keypair = load_or_create_keypair(&config)?;
let keypair = Arc::new(load_or_create_keypair(&config)?);
// Sealing key for the legacy-scan continuation tokens, DERIVED from the identity
// just loaded so it is the same key after a restart (see `derive_scan_token_key`).
let scan_token_key = AppState::derive_scan_token_key(&keypair);
@@ -265,6 +265,8 @@ async fn main() -> Result<()> {
Arc::clone(&db),
config.auto_sync,
shutdown_rx,
Arc::clone(&keypair),
config.require_signed_peer_writes,
)
.await
{
@@ -412,7 +414,7 @@ async fn main() -> Result<()> {
config: Arc::new(config.clone()),
db,
node_did: node_did.clone(),
node_keypair: Arc::new(keypair),
node_keypair: keypair,
p2p: p2p_handle,
http_client,
ref_update_tx,
+46
View File
@@ -11,6 +11,8 @@
//! `gitlawb_sync_queue_processed_total{status}`
//! * are webhooks reaching their endpoints? —
//! `gitlawb_webhook_deliveries_total{result}`
//! * is inbound gossip being admitted or shed, and for which reason?
//! `gitlawb_gossip_ingest_events_total{outcome}`
//! * how big are the packs we're sending and receiving? —
//! `gitlawb_pack_size_bytes`
//! * a single `gitlawb_info{version, did}` gauge = 1, for joins/dashboards
@@ -49,6 +51,7 @@ static AUTH_SUCCESSES: OnceLock<IntCounterVec> = OnceLock::new();
static AUTH_FAILURES: OnceLock<IntCounterVec> = OnceLock::new();
static SYNC_PROCESSED: OnceLock<IntCounterVec> = OnceLock::new();
static WEBHOOK_DELIVERIES: OnceLock<IntCounterVec> = OnceLock::new();
static GOSSIP_INGEST: OnceLock<IntCounterVec> = OnceLock::new();
static PACK_SIZE: OnceLock<Histogram> = OnceLock::new();
static PEERS_CONNECTED: OnceLock<IntGauge> = OnceLock::new();
@@ -168,6 +171,21 @@ fn init_inner(version: &str, node_did: &str) {
.set(webhook_deliveries)
.expect("set WEBHOOK_DELIVERIES once");
let gossip_ingest = IntCounterVec::new(
Opts::new(
"gitlawb_gossip_ingest_events_total",
"Total inbound gossip ref-update events and what the ingest path decided about each",
),
&["outcome"],
)
.expect("gitlawb_gossip_ingest_events_total definition");
registry
.register(Box::new(gossip_ingest.clone()))
.expect("register gitlawb_gossip_ingest_events_total");
GOSSIP_INGEST
.set(gossip_ingest)
.expect("set GOSSIP_INGEST once");
let pack_size = Histogram::with_opts(
HistogramOpts::new(
"gitlawb_pack_size_bytes",
@@ -270,6 +288,33 @@ pub fn record_webhook_delivery(result: &str) {
}
}
/// Record what the gossip ingest path decided about one inbound ref-update.
/// `outcome` ∈ {accepted, unsigned_admitted, write_failed, rejected,
/// source_rate_limited, author_rate_limited, unsigned_source_rate_limited}.
///
/// `accepted` is reserved for signature-verified events. An unsigned event that
/// survives the rolling-upgrade window is `unsigned_admitted`, so the
/// authenticated-admission rate is not silently padded by legacy compatibility
/// traffic; an operator can tell whether the fleet still relies on the
/// compatibility allowance.
///
/// The three shed reasons stay separate labels rather than one `rate_limited`
/// because they answer different operator questions: `source_rate_limited` is a
/// forwarding peer over the pre-parse brake, `unsigned_source_rate_limited` is
/// that same forwarder relaying unproven traffic, and `author_rate_limited`
/// names a principal a signature actually proved. Collapsing them would make a
/// flood of unauthenticated garbage indistinguishable from one registered peer
/// pushing too hard, which is exactly the distinction an alert needs.
///
/// The label is the variant, never the free-form reason string that
/// `Rejected`/`WriteFailed` carry: those are attacker-influenced and would blow
/// up the label cardinality of a process-wide registry.
pub fn record_gossip_ingest(outcome: &str) {
if let Some(c) = GOSSIP_INGEST.get() {
c.with_label_values(&[outcome]).inc();
}
}
/// Record a pack body size observation (bytes).
pub fn observe_pack_size(bytes: f64) {
if let Some(h) = PACK_SIZE.get() {
@@ -365,6 +410,7 @@ mod tests {
record_auth_failure("test/route", "test_reason");
record_sync_processed("done");
record_webhook_delivery("ok");
record_gossip_ingest("accepted");
observe_pack_size(1024.0);
set_peers_connected(0);
}
File diff suppressed because it is too large. Load diff
+17
View File
@@ -143,6 +143,23 @@ impl RateLimiter {
false
}
/// The configured key ceiling. Tests use it to assert that a limiter a
/// production factory builds is actually bounded, which flooding the real
/// ceiling in distinct keys would be far too slow to show.
#[cfg(test)]
pub fn max_keys(&self) -> usize {
self.max_keys
}
/// The configured per-key request cap. Tests use it to assert that a
/// limiter a production factory builds carries the cap it is documented
/// with; re-asserting the constant on its own cannot tell whether the
/// factory passed that constant or a different one.
#[cfg(test)]
pub fn max_requests(&self) -> usize {
self.max_requests
}
pub async fn cleanup(&self) {
let now = Instant::now();
let mut state = self.state.lock().await;
+221
View File
@@ -0,0 +1,221 @@
#!/usr/bin/env bash
#
# Map a pull request's changed paths to the CI surfaces they touch, so each
# check in pr-checks.yml and codeql.yml runs only when its inputs changed and
# Rust jobs test only the affected crates.
#
# Usage:
# git diff -z --name-only --no-renames BASE HEAD | scripts/ci-surfaces.sh
# scripts/ci-surfaces.sh --all # every surface, whole workspace
#
# Input is NUL-terminated paths (`git diff -z`). Without -z, git C-quotes any
# path with non-ASCII, tab, quote or newline characters ("crates/gl/\303\251.rs"),
# which would match no surface and skip every job; newline-delimited input
# also cannot carry a path that contains a newline. Input that does not end in
# NUL (e.g. a plain `git diff --name-only`) is rejected rather than misread.
# scripts/test-ci-surfaces.sh pins the mapping.
#
# Prints GITHUB_OUTPUT lines (key=value). Each surface runs only for the paths
# listed for it below, so a path no surface lists (docs, README, infra/, any
# other workflow) runs no PR Checks job. Within the Rust surface, a change this
# script cannot attribute to specific crates widens rather than narrows: a
# global Rust input or an unknown crate directory runs the Rust jobs on the
# whole workspace. The CI wiring itself (pr-checks.yml, codeql.yml, this
# script) runs every surface.
# Non-PR events (push to main, merge_group, workflow_dispatch, schedule) are
# expected to pass --all, so main and the merge queue keep full coverage.
#
# Runnable from anywhere in the repo.
set -euo pipefail
ROOT="$(git rev-parse --show-toplevel)"
all=false
if [ "${1:-}" = "--all" ]; then
all=true
elif [ "$#" -ne 0 ]; then
echo "usage: $0 [--all] < NUL-terminated-changed-paths" >&2
exit 2
fi
changed=()
if [ "$all" = false ]; then
path=""
while IFS= read -r -d '' path; do
[ -n "$path" ] && changed+=("$path")
done
# read leaves any unterminated tail in $path.
if [ -n "$path" ]; then
echo "ci-surfaces.sh: input is not NUL-terminated; pipe \`git diff -z --name-only\`" >&2
exit 2
fi
fi
# Workspace crates, derived from crates/*/Cargo.toml so a new crate is picked
# up without editing this script. dir -> package name, and dir -> the crate
# dirs it depends on through `path = "../<dir>"` (normal, dev, or build deps).
declare -A pkg_of=() deps_of=()
for manifest in "$ROOT"/crates/*/Cargo.toml; do
dir="$(basename "$(dirname "$manifest")")"
pkg_of[$dir]="$(sed -n 's/^name *= *"\([^"]*\)".*/\1/p' "$manifest" | head -n1)"
deps_of[$dir]="$(sed -n 's#.*path *= *"\.\./\([^"/]*\)".*#\1#p' "$manifest" | tr '\n' ' ')"
done
# Files outside crates/ that a crate compiles in (include_str!/include_bytes!),
# path -> crate dir. A change to one is a change to that crate, so its tests
# run. Candidates (keep those that resolve outside crates/<dir>/):
# git grep -nE 'include_(str|bytes)!\("(\.\./){2,}' crates
declare -A embedded_in=(
[bootstrap-peers.json]=gitlawb-node
)
# Mark a crate dir as touched; a dir this checkout does not know (removed or
# renamed crate) widens the Rust jobs to the whole workspace.
touch_crate() {
if [ -n "${pkg_of[$1]:-}" ]; then
touched[$1]=1
else
full_rust=true
fi
}
full_rust=false
rust=false
docker=false
audit=false
core_deps=false
release_versioning=false
release_tag=false
cq_actions=false
cq_js=false
cq_rust=false
cq_swift=false
declare -A touched=()
for path in "${changed[@]}"; do
case "$path" in
# The CI wiring itself: validate everything it gates.
.github/workflows/pr-checks.yml | .github/workflows/codeql.yml | scripts/ci-surfaces.sh)
all=true ;;
# Workspace-wide Rust inputs.
Cargo.toml | Cargo.lock | .cargo/* | rust-toolchain | rust-toolchain.toml \
| rustfmt.toml | .rustfmt.toml | clippy.toml | .clippy.toml)
full_rust=true ;;
crates/*/*)
dir="${path#crates/}"
touch_crate "${dir%%/*}" ;;
*)
if [ -n "${embedded_in[$path]:-}" ]; then
touch_crate "${embedded_in[$path]}"
fi ;;
esac
case "$path" in
Dockerfile | .dockerignore | bootstrap-peers.json | Cargo.toml | Cargo.lock \
| .cargo/* | rust-toolchain | rust-toolchain.toml | crates/*/Cargo.toml)
docker=true ;;
esac
case "$path" in
Cargo.lock | .cargo/audit.toml) audit=true ;;
esac
case "$path" in
Cargo.toml | Cargo.lock | crates/gitlawb-core/Cargo.toml \
| ci/gitlawb-core-allowed-deps.txt | scripts/check-gitlawb-core-deps.sh)
core_deps=true ;;
esac
case "$path" in
Cargo.toml | crates/*/Cargo.toml | release-please-config.json \
| .release-please-manifest.json | scripts/check-release-versioning.sh)
release_versioning=true ;;
esac
case "$path" in
scripts/resolve-release-tag.sh | scripts/test-resolve-release-tag.sh)
release_tag=true ;;
esac
case "$path" in
.github/workflows/* | .github/actions/*) cq_actions=true ;;
esac
case "$path" in
npm/* | *.js | *.mjs | *.cjs | *.jsx | *.ts | *.mts | *.cts | *.tsx) cq_js=true ;;
esac
case "$path" in
macos-app/* | *.swift) cq_swift=true ;;
esac
done
if [ "$all" = true ]; then
full_rust=true
docker=true
audit=true
core_deps=true
release_versioning=true
release_tag=true
cq_actions=true
cq_js=true
cq_swift=true
fi
# Affected crates: the touched ones plus everything that depends on them,
# transitively, so a gitlawb-core change still tests every consumer.
declare -A affected=()
for dir in "${!touched[@]}"; do affected[$dir]=1; done
grew=true
while [ "$grew" = true ]; do
grew=false
for dir in "${!deps_of[@]}"; do
[ -n "${affected[$dir]:-}" ] && continue
for dep in ${deps_of[$dir]}; do
if [ -n "${affected[$dep]:-}" ]; then
affected[$dir]=1
grew=true
break
fi
done
done
done
if [ "$full_rust" = true ] || [ "${#affected[@]}" -gt 0 ]; then
rust=true
cq_rust=true
fi
rust_packages=""
windows_packages=""
if [ "$full_rust" = true ]; then
rust_packages="--workspace"
windows_packages="-p gl -p git-remote-gitlawb"
elif [ "$rust" = true ]; then
for dir in $(printf '%s\n' "${!affected[@]}" | sort); do
rust_packages+="${rust_packages:+ }-p ${pkg_of[$dir]}"
case "${pkg_of[$dir]}" in
gl | git-remote-gitlawb) windows_packages+="${windows_packages:+ }-p ${pkg_of[$dir]}" ;;
esac
done
fi
windows=false
[ -n "$windows_packages" ] && windows=true
languages=()
[ "$cq_actions" = true ] && languages+=('{"language":"actions","build-mode":"none","runner":"ubuntu-latest"}')
[ "$cq_js" = true ] && languages+=('{"language":"javascript-typescript","build-mode":"none","runner":"ubuntu-latest"}')
[ "$cq_rust" = true ] && languages+=('{"language":"rust","build-mode":"none","runner":"ubuntu-latest"}')
[ "$cq_swift" = true ] && languages+=('{"language":"swift","build-mode":"autobuild","runner":"macos-latest"}')
codeql_matrix="{\"include\":[$(IFS=,; echo "${languages[*]}")]}"
codeql=false
[ "${#languages[@]}" -gt 0 ] && codeql=true
cat <<EOF
rust=$rust
rust_packages=$rust_packages
windows=$windows
windows_packages=$windows_packages
docker=$docker
audit=$audit
core_deps=$core_deps
release_versioning=$release_versioning
release_tag=$release_tag
codeql=$codeql
codeql_matrix=$codeql_matrix
EOF
+121
View File
@@ -0,0 +1,121 @@
#!/usr/bin/env bash
# Fixture test for scripts/ci-surfaces.sh, which alone decides which PR Checks
# jobs and CodeQL languages run. Pins at least one must-not-skip case per
# surface row, the crate fan-out and widening rules, and paths that git would
# C-quote without -z. Run from pr-checks.yml before the mapper is used.
set -euo pipefail
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
mapper="$repo_root/scripts/ci-surfaces.sh"
failures=0
# map PATH... : run the mapper on the paths as `git diff -z` would emit them.
map() {
if [ "$#" -gt 0 ]; then printf '%s\0' "$@"; fi | "$mapper"
}
# check CASE OUTPUT KEY=VALUE... : each KEY must have exactly VALUE.
check() {
local name="$1" out="$2" want key
shift 2
for want in "$@"; do
key="${want%%=*}"
if ! grep -qxF -- "$want" <<<"$out"; then
printf 'FAIL %s: want %s, got %s\n' "$name" "$want" \
"$(grep -E "^${key}=" <<<"$out" || echo '<missing>')" >&2
failures=$((failures + 1))
fi
done
}
# has_packages CASE OUTPUT PKG... : rust_packages must include each -p PKG.
has_packages() {
local name="$1" out="$2" pkgs pkg
shift 2
pkgs=" $(sed -n 's/^rust_packages=//p' <<<"$out") "
for pkg in "$@"; do
if [[ "$pkgs" != *" -p $pkg "* ]]; then
printf 'FAIL %s: rust_packages lacks -p %s (got%s)\n' "$name" "$pkg" "$pkgs" >&2
failures=$((failures + 1))
fi
done
}
nothing=(rust=false rust_packages= windows=false windows_packages= docker=false
audit=false core_deps=false release_versioning=false release_tag=false
codeql=false 'codeql_matrix={"include":[]}')
everything=(rust=true rust_packages=--workspace windows=true
'windows_packages=-p gl -p git-remote-gitlawb' docker=true audit=true
core_deps=true release_versioning=true release_tag=true codeql=true)
all_languages='codeql_matrix={"include":[{"language":"actions","build-mode":"none","runner":"ubuntu-latest"},{"language":"javascript-typescript","build-mode":"none","runner":"ubuntu-latest"},{"language":"rust","build-mode":"none","runner":"ubuntu-latest"},{"language":"swift","build-mode":"autobuild","runner":"macos-latest"}]}'
rust_only='codeql_matrix={"include":[{"language":"rust","build-mode":"none","runner":"ubuntu-latest"}]}'
# Docs-only and empty input run nothing.
check "docs only" "$(map README.md docs/x.md infra/x.tf)" "${nothing[@]}"
check "empty input" "$(map)" "${nothing[@]}"
# A crate change tests that crate and its reverse deps, nothing else.
check "gl source" "$(map crates/gl/src/main.rs)" \
rust=true 'rust_packages=-p gl' windows=true 'windows_packages=-p gl' \
docker=false audit=false "$rust_only"
out="$(map crates/gitlawb-core/src/lib.rs)"
check "gitlawb-core source" "$out" rust=true windows=true \
'windows_packages=-p git-remote-gitlawb -p gl' "$rust_only"
has_packages "gitlawb-core fan-out" "$out" gitlawb-core icaptcha-client \
gitlawb-attest git-remote-gitlawb gl gitlawb-node
check "icaptcha-client source" "$(map crates/icaptcha-client/src/lib.rs)" \
'rust_packages=-p gl -p icaptcha-client' 'windows_packages=-p gl'
# Anything the Rust surface cannot attribute to known crates widens.
check "unknown crate dir" "$(map crates/no-such-crate/src/lib.rs)" \
rust=true rust_packages=--workspace docker=false
check "root Cargo.lock" "$(map Cargo.lock)" \
rust=true rust_packages=--workspace docker=true audit=true core_deps=true \
release_versioning=false
check "toolchain file" "$(map rust-toolchain.toml)" rust=true rust_packages=--workspace
# Files compiled into a crate from outside crates/.
check "bootstrap-peers.json" "$(map bootstrap-peers.json)" \
rust=true 'rust_packages=-p gitlawb-node' windows=false docker=true
# One must-not-skip trigger per non-Rust row.
check "Dockerfile" "$(map Dockerfile)" docker=true rust=false
check "crate manifest" "$(map crates/gl/Cargo.toml)" \
docker=true release_versioning=true 'rust_packages=-p gl'
check "audit config" "$(map .cargo/audit.toml)" audit=true rust_packages=--workspace
check "core allowlist" "$(map ci/gitlawb-core-allowed-deps.txt)" core_deps=true rust=false
check "release-please config" "$(map release-please-config.json)" \
release_versioning=true rust=false
check "release tag resolver" "$(map scripts/resolve-release-tag.sh)" release_tag=true rust=false
check "other workflow" "$(map .github/workflows/release.yml)" "${nothing[@]:0:9}" \
codeql=true 'codeql_matrix={"include":[{"language":"actions","build-mode":"none","runner":"ubuntu-latest"}]}'
check "npm" "$(map npm/package.json)" codeql=true \
'codeql_matrix={"include":[{"language":"javascript-typescript","build-mode":"none","runner":"ubuntu-latest"}]}'
check "swift" "$(map macos-app/Sources/App.swift)" rust=false codeql=true \
'codeql_matrix={"include":[{"language":"swift","build-mode":"autobuild","runner":"macos-latest"}]}'
# Editing the CI wiring runs every surface, exactly like --all.
for wiring in .github/workflows/pr-checks.yml .github/workflows/codeql.yml scripts/ci-surfaces.sh; do
check "wiring $wiring" "$(map "$wiring")" "${everything[@]}" "$all_languages"
done
check "--all" "$("$mapper" --all </dev/null)" "${everything[@]}" "$all_languages"
# Names git C-quotes without -z (non-ASCII, tab, quote, newline) still map.
for name in $'\xc3\xa9.rs' $'a\tb.rs' 'a"b.rs' $'a\nb.rs'; do
check "quoted name $(printf %q "$name")" "$(map "crates/gl/src/$name")" \
rust=true 'rust_packages=-p gl'
done
check "non-ASCII swift" "$(map $'macos-app/Sources/Caf\xc3\xa9.swift')" codeql=true \
'codeql_matrix={"include":[{"language":"swift","build-mode":"autobuild","runner":"macos-latest"}]}'
# Newline-delimited input (git diff without -z) is rejected, not misread.
if printf 'crates/gl/src/main.rs\n' | "$mapper" >/dev/null 2>&1; then
echo "FAIL newline-delimited input: mapper accepted it" >&2
failures=$((failures + 1))
fi
if [ "$failures" -ne 0 ]; then
echo "$failures ci-surfaces mapping check(s) failed" >&2
exit 1
fi
echo "ci-surfaces mapping: all checks passed"