Compare commits

...
Author SHA1 Message Date
beardthelion f577142a35 ci: trigger the PR checks on the retargeted base 2026-11-04 16:48:32 -06:00
beardthelion 5d182be9c8 test(node): pass a libp2p identity to p2p::start in the gossip delivery tests 2026-11-04 16:24:34 -06:00
beardthelion 955023e328 fix(node): dedupe the ref-update row on the replay key, not on a random id
The reservation releases on a failed write so a republish can repair the
half that never landed. But each ingest minted a fresh row id, so a retry
after a partial failure (row stored, queue write failed) stored the same
signed content a second time. A replay in a degraded state could repeat a
landed side effect until a budget refused it.

The row id for a verified event is now derived from the replay key, so the
retry's insert dedupes on the existing ON CONFLICT instead of duplicating.
The queue entry still retries, which is the repair the release exists for.
Unsigned events keep a random id: they carry no replay key by design, and
deduplicating unauthenticated content would let an attacker pre-send a
victim's expected event and have the genuine one dropped.

Two existing tests pinned the duplicate write and now assert the deduped
shape: the partial-failure republish repairs the queue entry while the
row stays at one, and the restart readmission's residual effect is a
duplicate queue entry (a refetch), not a second record. The new test
asserts both the retry reaches the writes (WriteFailed, not Replayed) and
the row count stays at one.
2026-11-04 16:18:21 -06:00
beardthelion c65bd616d5 fix(node): harden the gossip rejection path against unauthenticated bytes
Two findings on the pre-auth reject path:

- An unsigned event under require_signed was resolved and slug-validated
  before the flag refused it, so a hostile flood bought DID resolution and
  a rejection reason built from attacker-chosen fields under only the
  loose pre-parse brake. The unsigned refusal now runs immediately after
  the version gate, before any field is read.

- The swarm loop logged IngestOutcome::Rejected and WriteFailed reasons
  verbatim, and several producers embed wire fields (the claimed node_did
  in resolve failures, for one). A signed event carrying a control-bearing
  node_did reached the operator log with escape sequences intact and no
  length bound. Both warn sites now render through sanitize_for_log, so
  every present and future Rejected producer is bounded and control-free
  at the one sink.

The live-mesh test now also publishes a signed event with a 4096-char
control-bearing node_did and asserts the warn stream shows the refusal
with no escape byte and no run past the sanitizer ceiling. A unit test
pins the new ordering by asserting the exact fixed refusal for an
unsigned event with a hostile node_did.
2026-11-04 16:18:21 -06:00
beardthelion 8dcb4934ff test(node): drive the live-mesh publish and refusal path end to end
Two real swarms on QUIC loopback with real signatures and real Postgres:
a signed ref-update from one node is admitted and stored by the other, a
re-delivery of the same signed content under a malleated wire encoding is
refused by the replay guard, and an unsigned event is refused under
require_signed. Both refusals are asserted in the warn stream, and a
control event landing after them proves the mesh still delivers.

This closes the seams ref_update_publish_args names as needing a live
swarm: the PublishRefUpdate select! arm dispatching to sign+publish, and
require_signed reaching the receive path. It needs a test-only command
variant, PublishRawRefUpdate, since the production publish path signs and
serializes itself and can never put a malleated or unsigned event on the
wire.

Still uncovered and named in the test's doc comment: the run_ingest_sweep
select! arm on its 300-second interval, and the main.rs half of the flag
wiring above p2p::start.
2026-11-04 16:18:21 -06:00
beardthelion c7e8293aeb test(node): pass the replay guard to the post-fork shed tests
The two ingest tests added on the target branch after this PR forked
(an_over_budget_author_is_shed_without_a_peer_lookup and
the_early_author_check_does_not_track_an_unseen_did) still call
ingest_ref_update with the pre-guard signature. Thread a fresh
ReplayGuard through each call, matching every other test site.
2026-11-04 16:18:21 -06:00
Kevin Codex 904e49f3bb fix(node): render rejected gossip timestamps bounded before logging
DateTime::parse_from_rfc3339 accepts arbitrarily many fractional-second
digits, so a signed timestamp can parse successfully and still carry
megabytes of attacker-chosen wire bytes into the TooOld / TooFarFuture
refusal details, which the swarm loop logs verbatim. Only the
parser-error branch sanitized. The root cause was treating successful
parsing as proof the wire representation is safe to expose.

The TooOld and TooFarFuture variants now carry the PARSED instant and
the detail renders its canonical to_rfc3339() form - bounded by this
build's formatter and control-free by construction - while the
unparseable arm keeps sanitize_for_log on the wire string. All three
arms go through one freshness_refusal_detail helper so the invariant
has a single home, plus a regression test driving an overlong but
parseable fractional stamp through both directions.
2026-11-04 16:18:21 -06:00
beardthelion 16adbd8ade fix(review): reconcile the replay guard with the unsigned-admission split
Integration only, from rebasing this branch onto #325's current head. #325 now
returns IngestOutcome::UnsignedAdmitted where it used to return Accepted for an
unsigned rolling-upgrade admission, and this branch predates that split.

Three sites. The seen-set bypass test asserted Accepted on both unsigned
deliveries and on the stale-timestamp case; both now expect UnsignedAdmitted,
and the properties under test are unchanged, that unsigned bytes are admitted
twice rather than deduplicated and that the freshness window does not reach
them. The warn-only-on-admission test's budget-spent case gained the
ReplayGuard argument the signature now takes, with a fresh guard because that
case drives an unsigned event the replay block skips.

The ingest match also grows an arm rather than changing one: `None if unsigned`
returns UnsignedAdmitted without settling a reservation, since the replay block
is gated on `verified` and an unsigned admission never holds one.
2026-11-04 16:18:21 -06:00
beardthelion 70cefe3b0c fix(core): stop an attacker-supplied created timestamp from overflowing the window check
created is parsed as an unrestricted i64 straight from the Signature-Input
header, so the sender picks it. At i64::MIN the past-side subtraction overflows,
which panics a debug build and wraps a release one into a value that can read as
inside the window. Confirmed by execution before the fix: 'attempt to subtract
with overflow'.

Saturating subtraction answers correctly at both ends, since a timestamp that
far out is refused by whichever side it saturates toward. A test drives all four
extremes of the type and asserts each is refused by a named direction; reverting
to plain subtraction reddens it on the overflow.

The symmetric abs() form this replaced had the same hazard, so splitting the
window into two comparisons did not introduce it, but it did not remove it
either. Found by a cross-family review pass after six same-family reviewers and
I had all read the line.
2026-11-04 16:18:21 -06:00
beardthelion ccebc3fa54 test(node): close the replay guard's remaining proof gaps, and fix the sibling window
Seven tests for the six gaps a review found. The reservation's release path was
reasoned rather than executed: the drop guard was only ever proven through an
early refusal, while the case its own doc comment names, a transient write
failure burning the event's slot, was never driven. Both write directions now
are. An expired entry at capacity must be replaced in place rather than answer
Saturated, confirm on an already-swept entry is pinned as a deliberate no-op,
and the single-critical-section shape is now driven concurrently rather than
asserted sequentially. Restart behavior was documented but untested; a fresh
guard readmits a seen event and the freshness window still bounds it, which is
the composition that makes the restart exposure finite.

check_created in gitlawb-core used a symmetric abs() window, so a request
stamped 299 seconds ahead was accepted and a signer could roughly double a
signature's effective validity by stamping forward. It is now two comparisons
like the gossip path, 300s late and 60s early, with the error naming the
direction so a fast peer and a slow one need different operator action. There
was no future-direction test at all; there are now five covering both. Every
caller was checked and none depended on the symmetry.

Adding a freshness window obliges auditing the siblings, which is how this one
surfaced: the repo argued both ways in two files for a week.
2026-11-04 16:18:21 -06:00
beardthelion 04ba7a6792 fix(review): bound the saturated guard's rescan and the log it writes
At capacity the inline sweep ran on every event, so a full O(capacity) retain
happened under the lock once per message and reclaimed nothing when nothing had
expired. The guard that bounds replay became a CPU amplifier in exactly the
state an attacker drives toward. The sweep is now rate limited to once a second;
the periodic sweep still reclaims on its own cadence, so only the redundant
rescans go away.

An unparseable timestamp was echoed into the refusal detail verbatim. That value
is attacker-controlled and arbitrary length, so it reached a warn! as both a
log-injection and an unbounded-size sink. Only that arm needs sanitizing; the
other two ran through the parser first.

The capacity rationale cited a count of registered DIDs, which this same file
says elsewhere an attacker mints freely through the announce path, so it was not
a bound at all. It now cites the bound that is real: reaching saturation costs a
hundred thousand durable rows and as many sync enqueues inside one retention
horizon, which the database makes loud.

ingest_now was read twice per ingest while its own doc comment claimed the two
layers share one reading, which is the invariant the retention derivation rests
on. Now read once and passed to both.

Also: the Unparseable outcome is driven through ingest rather than only as a
pure function, the periodic sweep is observable without a live swarm, the
restart exposure is written down where the rest of the tradeoffs already are,
and the saturation-counter lock covers every test that can reach Saturated
rather than the two I first found.
2026-11-04 16:18:21 -06:00
beardthelion 46474bcd23 fix(node): bound replay of authenticated gossip ref-update events
Wires the freshness check and the replay guard into ingest, immediately after
signature verification and above the author debit. The lower placement, just
above the writes, stops the duplicate row and the duplicate sync but still lets
every replay drain the victim author's budget, which is the harm the defect
names: a captured signature replayed 500 times empties the victim's window and
their next genuine push is refused.

The guard runs on the verified path only. Unsigned event bytes are predictable,
so applying it there would let an attacker pre-send a victim's expected event
and have the genuine one dropped as a replay, which is a censorship primitive
rather than a defense.

A replay flood still costs a parse and one Ed25519 verify, because the guard
has to sit below verification for the reason above. What it removes is the
peer_exists round trip, the victim's author debit, the ref-update row and the
sync enqueue.

The existing author-budget test signed one event and ingested the same bytes
five hundred times, so it had to re-sign per iteration to keep exercising the
budget under a shared guard. Giving it a fresh guard per call would have kept
it green while gutting the property it exists to prove. Its over-budget probe
needed the same treatment, since the replay gate sits above the author gate and
would have refused the burst's last bytes before the budget assertion ran.

Both saturation tests now take a shared lock so each keeps an exact assertion
on a process-wide counter; a lower bound would stay green if the fail-open
branch ever double-counted.
2026-11-04 16:18:21 -06:00
beardthelion 9f4efcc90b feat(node): add the freshness check and replay guard, not yet wired
Builds the two layers that bound replay of an authenticated ref-update, with
nothing calling them yet; the ingest path is wired in the next commit.

The freshness check is deliberately two-directional rather than an absolute
delta: an abs() window admits an event stamped up to the window ahead and pins
its seen-set slot until the clock catches up. Producers were enumerated before
settling the unparseable arm; the sole production publish site emits RFC-3339,
so an unparseable timestamp is refused rather than admitted.

The replay guard keys on SHA-256 of the canonical signing bytes, not the raw
wire bytes, because one signature verifies against many encodings and only the
canonical form collapses them to a single key. A golden digest is frozen for
the pre-version artifact, and the same constant is asserted for its v-injected
twin, so the collapse is pinned rather than described. Reservations settle
through a drop guard so only a confirmed entry outlives the ingest call, which
keeps a transient write failure from permanently burning an event's slot.

Replayed and StaleTimestamp are separate outcomes because they diagnose
different conditions, a mesh replay against a broken clock or a healing
partition, and folding them would be the same observability lie the unsigned
shed variant already exists to avoid.
2026-11-04 16:18:20 -06:00
3 changed files with 3326 additions and 62 deletions

No files matched your search

+106 -4
View File
@@ -122,15 +122,48 @@ impl HttpSignature {
})
}
/// Reject if the `created` timestamp is more than 5 minutes from now.
/// Reject a `created` timestamp outside the acceptance window, which is
/// ASYMMETRIC: up to 5 minutes into the past, but only 60 seconds into the
/// future.
///
/// The two directions are two comparisons rather than one distance, because
/// they are not the same event. A signature that arrives late is ordinary:
/// clocks run slow, requests queue, and networks are not instant, so the
/// past side keeps the full 300-second allowance. A signature stamped ahead
/// of this node's clock is not ordinary, and folding both sides into an
/// `abs()` let a signer extend its own validity: signed at T with
/// `created = T + 300`, it stayed acceptable until roughly T + 600, so the
/// effective window was double the one this function advertises. The
/// forward tolerance exists only to absorb honest drift between two hosts,
/// which is what 60 seconds buys; it is the same allowance the gossip
/// path's freshness check uses for the same reason.
pub fn check_created(&self) -> Result<()> {
const MAX_AGE_SECS: i64 = 300;
const MAX_FUTURE_SECS: i64 = 60;
let now = Utc::now().timestamp();
let skew = (now - self.created).abs();
if skew > 300 {
// Saturating, not plain subtraction. `created` is parsed as an
// unrestricted i64 straight from the Signature-Input header, so a
// sender picks it: at i64::MIN the past-side subtraction overflows,
// which panics a debug build and wraps a release one into an absurd
// value that can read as acceptable. Saturating gives the answer the
// check wants at both extremes, since a timestamp that far away is
// refused by whichever side it saturates toward.
let age = now.saturating_sub(self.created);
if age > MAX_AGE_SECS {
return Err(Error::HttpSignature(format!(
"clock skew too large: {skew}s (max 300s)"
"clock skew too large: created {age}s in the past (max {MAX_AGE_SECS}s)"
)));
}
let ahead = self.created.saturating_sub(now);
if ahead > MAX_FUTURE_SECS {
return Err(Error::HttpSignature(format!(
"clock skew too large: created {ahead}s in the future (max {MAX_FUTURE_SECS}s)"
)));
}
Ok(())
}
@@ -456,6 +489,75 @@ mod tests {
assert!(sig.check_created().is_err());
}
/// Build a parseable signature whose `created` is `offset` seconds from now
/// (negative for the past), so a test can pin one side of the window.
fn sig_created_at_offset(offset: i64) -> HttpSignature {
let kp = Keypair::generate();
let did = kp.did();
let created = Utc::now().timestamp() + offset;
let sig_input = format!(
r#"sig1=("@method" "@path" "content-digest");keyid="{did}";alg="ed25519";created={created}"#
);
HttpSignature::parse(&sig_input, "sig1=:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA:").unwrap()
}
/// Build a signature at an ABSOLUTE `created`, so a test can drive the
/// extremes an offset from now cannot reach.
fn sig_created_absolute(created: i64) -> HttpSignature {
let kp = Keypair::generate();
let did = kp.did();
let sig_input = format!(
r#"sig1=("@method" "@path" "content-digest");keyid="{did}";alg="ed25519";created={created}"#
);
HttpSignature::parse(&sig_input, "sig1=:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA:").unwrap()
}
/// `created` is parsed as an unrestricted i64 from a header the sender
/// writes, so the window check has to survive both ends of the type. Before
/// the saturating form this panicked with "attempt to subtract with
/// overflow" on the past side, which is a debug-build crash any client
/// could trigger by sending one header value.
#[test]
fn extreme_created_values_are_refused_rather_than_overflowing() {
for created in [i64::MIN, i64::MIN + 1, i64::MAX - 1, i64::MAX] {
let err = sig_created_absolute(created)
.check_created()
.expect_err("an extreme created must be refused, not accepted");
let msg = err.to_string();
assert!(
msg.contains("past") || msg.contains("future"),
"created={created} must be refused by a named direction, got: {msg}"
);
}
}
#[test]
fn created_301s_in_past_rejected() {
let err = sig_created_at_offset(-301).check_created().unwrap_err();
assert!(err.to_string().contains("past"), "{err}");
}
#[test]
fn created_299s_in_past_accepted() {
assert!(sig_created_at_offset(-299).check_created().is_ok());
}
#[test]
fn created_61s_in_future_rejected() {
let err = sig_created_at_offset(61).check_created().unwrap_err();
assert!(err.to_string().contains("future"), "{err}");
}
#[test]
fn created_30s_in_future_accepted() {
assert!(sig_created_at_offset(30).check_created().is_ok());
}
#[test]
fn created_exactly_now_accepted() {
assert!(sig_created_at_offset(0).check_created().is_ok());
}
#[test]
fn fresh_signature_passes_clock_skew() {
let kp = Keypair::generate();
+57 -3
View File
@@ -13,6 +13,9 @@
//! `gitlawb_webhook_deliveries_total{result}`
//! * is inbound gossip being admitted or shed, and for which reason?
//! `gitlawb_gossip_ingest_events_total{outcome}`
//! * is the gossip replay seen-set full, and therefore admitting events it
//! cannot deduplicate?
//! `gitlawb_gossip_replay_guard_saturated_total`
//! * how big are the packs we're sending and receiving? —
//! `gitlawb_pack_size_bytes`
//! * a single `gitlawb_info{version, did}` gauge = 1, for joins/dashboards
@@ -35,8 +38,8 @@
use std::sync::OnceLock;
use prometheus::{
Encoder, Histogram, HistogramOpts, IntCounterVec, IntGauge, IntGaugeVec, Opts, Registry,
TextEncoder,
Encoder, Histogram, HistogramOpts, IntCounter, IntCounterVec, IntGauge, IntGaugeVec, Opts,
Registry, TextEncoder,
};
/// The single, process-wide metrics registry. Initialized by [`init`].
@@ -52,6 +55,7 @@ static AUTH_FAILURES: OnceLock<IntCounterVec> = OnceLock::new();
static SYNC_PROCESSED: OnceLock<IntCounterVec> = OnceLock::new();
static WEBHOOK_DELIVERIES: OnceLock<IntCounterVec> = OnceLock::new();
static GOSSIP_INGEST: OnceLock<IntCounterVec> = OnceLock::new();
static GOSSIP_REPLAY_GUARD_SATURATED: OnceLock<IntCounter> = OnceLock::new();
static PACK_SIZE: OnceLock<Histogram> = OnceLock::new();
static PEERS_CONNECTED: OnceLock<IntGauge> = OnceLock::new();
/// 1 when the node started without its p2p identity because the key storage
@@ -191,6 +195,18 @@ fn init_inner(version: &str, node_did: &str) {
.set(gossip_ingest)
.expect("set GOSSIP_INGEST once");
let gossip_replay_guard_saturated = IntCounter::with_opts(Opts::new(
"gitlawb_gossip_replay_guard_saturated_total",
"Inbound gossip ref-updates admitted without being recorded because the replay seen-set was at capacity",
))
.expect("gitlawb_gossip_replay_guard_saturated_total definition");
registry
.register(Box::new(gossip_replay_guard_saturated.clone()))
.expect("register gitlawb_gossip_replay_guard_saturated_total");
GOSSIP_REPLAY_GUARD_SATURATED
.set(gossip_replay_guard_saturated)
.expect("set GOSSIP_REPLAY_GUARD_SATURATED once");
let pack_size = Histogram::with_opts(
HistogramOpts::new(
"gitlawb_pack_size_bytes",
@@ -307,7 +323,8 @@ pub fn record_webhook_delivery(result: &str) {
/// Record what the gossip ingest path decided about one inbound ref-update.
/// `outcome` ∈ {accepted, unsigned_admitted, write_failed, rejected,
/// source_rate_limited, author_rate_limited, unsigned_source_rate_limited}.
/// source_rate_limited, author_rate_limited, unsigned_source_rate_limited,
/// replayed, stale_timestamp}.
///
/// `accepted` is reserved for signature-verified events. An unsigned event that
/// survives the rolling-upgrade window is `unsigned_admitted`, so the
@@ -332,6 +349,38 @@ pub fn record_gossip_ingest(outcome: &str) {
}
}
/// Record one inbound gossip ref-update that the replay seen-set could not
/// record because it was at capacity.
///
/// Label-less on purpose: there is one degradation mode and nothing about it
/// varies per event, so a label would only add cardinality.
///
/// This is an alert-worthy signal, not a debug counter, and it is the ONLY way
/// the degraded state is visible. At capacity the guard fails open, which means
/// the event is admitted and `gitlawb_gossip_ingest_events_total{outcome=
/// "accepted"}` counts it exactly as it counts a healthy admission. Fail-open is
/// the right policy (a saturated set that dropped all fresh gossip would convert
/// a loud resource attack into quiet mesh-wide censorship), but it is not a free
/// one: while this counter is moving, a captured signature can again be replayed
/// against the freshness window's full 10 minutes, spending a `peer_exists`
/// round trip and a debit from the victim author's budget on every replay.
pub fn record_gossip_replay_guard_saturated() {
if let Some(c) = GOSSIP_REPLAY_GUARD_SATURATED.get() {
c.inc();
}
}
/// Test-only: current `gitlawb_gossip_replay_guard_saturated_total` value (0 if
/// the registry is not initialized). Lets the seen-set tests assert the
/// saturation signal fires without scraping the encoded text.
#[cfg(test)]
pub fn replay_guard_saturated_count_for_test() -> u64 {
GOSSIP_REPLAY_GUARD_SATURATED
.get()
.map(|c| c.get())
.unwrap_or(0)
}
/// Record a pack body size observation (bytes).
pub fn observe_pack_size(bytes: f64) {
if let Some(h) = PACK_SIZE.get() {
@@ -434,6 +483,11 @@ mod tests {
record_auth_failure("test/route", "test_reason");
record_sync_processed("done");
record_webhook_delivery("ok");
// `record_gossip_replay_guard_saturated` is deliberately NOT called
// here. It is a label-less process-wide counter and the seen-set test
// that asserts it asserts a before/after delta, so an increment from a
// second test running concurrently in this binary would be a flake with
// no diagnostic value.
record_gossip_ingest("accepted");
observe_pack_size(1024.0);
set_peers_connected(0);
File diff suppressed because it is too large. Load diff