Compare commits

..
Author SHA1 Message Date
beardthelion 34dc3993fd chore(node): retrigger checks after code-scanning alert triage (#555) 2026-11-09 16:04:37 -06:00
beardthelion 0b18a9fc7c chore(node): retrigger checks after code-scanning alert triage (#555) 2026-11-09 15:37:59 -06:00
beardthelion 57cb16ed06 fix(node): bound mirror preflight probes separately, align operator docs (#555) 2026-11-09 14:06:39 -06:00
beardthelion f399a280f6 Merge remote-tracking branch 'origin/main' into pr-347 2026-11-09 14:05:00 -06:00
4cd0a6a306 fix(node)!: gate agent-task reads behind visibility rules (#553)
* fix(node): gate agent-task reads behind the same visibility rules as repo data

list_tasks and get_task had no authorization at all: any anonymous caller
could enumerate every task on the node, including another party's
repo-less task, its ucan_token, and its payload (#268). Add task_visible,
mirroring the repo read-visibility gate already used by the ref-updates
feed: the delegator and assignee can always read their own task, a
repo-scoped task follows that repo's normal visibility rules, and a task
naming no repo (or a repo this node doesn't host) is visible only to its
delegator/assignee. Both REST and GraphQL now route through the same
collect_visible_tasks/get_visible_task collectors so the two surfaces
cannot drift, and neither read path echoes ucan_token back, since the
holder already received it via the create/claim response.

Fixes #268

* fix(node): query the task limit-clamp test as the delegator

tasks_limit_ceiling_clamped_to_200 seeded 201 repo-less tasks and read
them back anonymously, expecting all 200. That read is exactly the
enumeration #268 closes, so the new visibility gate correctly returns
none of them and the test went red. The clamp ceiling is what this test
pins, not the gate, so query as the tasks' delegator, who can legitimately
see all 201 rows.

Refs #268

* fix(node): scope task visibility lookups to the page's repos

collect_visible_tasks loaded every repo on the node and every visibility
rule in order to gate at most 200 tasks, so an anonymous request paid for
the whole node's repo and rule set. Narrow both lookups to the repo ids the
fetched page actually names, and skip them when no task names a repo.

The deduped repo snapshot stays the source of truth for resolving a
repo_id: it collapses mirror and canonical pairs and omits quarantined
repos, and an id missing from it has to keep failing closed. Resolving ids
straight from the repos table would surface exactly those withheld rows.

Add GraphQL denial tests as well. Nothing pinned that the task resolvers
delegate to the shared collectors, so a resolver that queried the database
directly would not have gone red.

Refs #268

* Preserve authorized task reads across shipped clients.

Refs #268

* Bound denied task history scans.

Refs #268

* Signal incomplete task scans with recoverable cursors and map read errors to AppError.

Refs #268

* Stop disclosing denied task rows in the scan-wall cursor and add GraphQL pagination state.

Refs #268

* Normalize task cursors and gate mutation endpoints behind visibility.

Canonicalize RFC 3339 timestamps in parse_after_cursor to handle URL-decoded spaces, reject mixed cursor alias families, gate complete_task and fail_task behind get_visible_task so unreadable tasks 404 instead of leaking existence with 403, and only flag incomplete when hitting candidate ceilings on full SQL batches.

Refs #268

* Update complete_task test in test_support to exercise both 404 on unreadable and 403 on non-assignee tasks.

Refs #268

* Bind task to repo id in complete_task test in test_support.

Refs #268

* Preserve timestamp fractional width and anchor continuation tests on legitimate cursors.

Refs #268

* Match on Value::Object to extract task ID in GraphQL continuation test.

Refs #268

* Remove unused mut binding on visible_newer in GraphQL query test.

Refs #268

* Hide unauthorized claim and keep assigned tasks from being stolen

Gate REST and GraphQL claim behind the same visibility check as complete and fail, refuse claim when another assignee already holds the task, and only broadcast publicly visible task events. Treat a full list page as incomplete when more candidates remain. Surface HTTP errors from CLI and MCP claim and complete helpers.

Refs #327

Co-Authored-By: cairn-code <[email protected]>

* Keep task-list incomplete for the scan ceiling only.

A full visible page was flagged incomplete whenever the SQL batch was full, so the first page of any list with more than 200 candidates looked stalled. Align the GraphQL claim test with the visibility gate's not-found message.

Refs #268

* fix(node): pin claim guards and map mutation errors through AppError

Review required tests that go red if the pre-assigned claim predicate or
the anonymous announce gate is deleted, and incomplete must not stay
true when the candidate stream is exhausted at the scan ceiling. Route
claim, complete, and fail through AppError so closed-pool outages stay
503 and 404s match the read envelope.

Refs #268

* fix(node): match bare and did:key assignee forms in claim and list

create_task stores the supplied assignee unchanged, so a raw SQL
equality check drops a designated assignee who presents the other
did:key form. Compare the normalized key so claim and filtered list
agree with did_matches.

Refs #268

* fix(node): check task creation status and add assignee expression index

- Add error_for_status() to cmd_create and task_create MCP tool
- Update test_create_task_server_error to assert failure on 500
- Add migration v18 creating expression index idx_agent_tasks_assignee_key matching ASSIGNEE_DID_CASE_SQL
- Add did:web:z6Mkfoo single-residual shape to parity boundary matrix

Refs #327

# Conflicts:
#	crates/gitlawb-node/src/db/mod.rs

* fix(node)!: Replace task cursors with opaque continuation tokens

The task read path treated visibility, pagination, and error vocabulary as
separate edits, so each one broke where they met. Rework them as one contract.

A raw (created_at, id) cursor forced a choice between two broken options: it
could name the last visible row, and then a denied window longer than the
1,000-candidate scan budget was unpageable forever; or it could name the last
examined row, and then a denied read leaked the id and timestamp of a task
GET /tasks/{id} otherwise 404s. Continuation tokens remove the choice. They
carry the last examined candidate, so paging always advances a full scan budget
per request, and they are encrypted and authenticated under a node-derived key,
so the caller learns nothing from one and cannot forge one naming a row of
their choosing. Encryption is a synthetic-IV construction over the hmac/sha2
pair already used for webhook signatures, so it adds no dependency and needs no
randomness source.

Making the token the only accepted cursor also gives the ordering key one
domain. agent_tasks.created_at is TEXT and compared as TEXT, so a caller-typed
'...Z' and '...+00:00' denote one instant but sort differently, and a client
could silently skip or repeat same-time rows. The token carries the stored
string verbatim, so the value compared is always one the server wrote. The raw
after_*/cursor_* pairs are removed rather than kept alongside it, since a second
domain is the bug.

Separate the two facts the old single incomplete flag conflated: has_more says
candidates remain, incomplete says this page is short only because the
authorization scan hit its ceiling. Both REST and GraphQL now return has_more,
incomplete, and next_cursor from the shared collector, and REST echoes the limit
it actually applied so a clamped request is visible as clamped.

Have gl task list and MCP task_list follow next_cursor instead of issuing one
request: --limit 500 returned a successful but silently truncated 200 rows.
Following is bounded by a page cap and a no-progress guard, and a run stopped by
either reports an explicit incomplete result with a resume cursor.

Route claimTask, completeTask, and failTask through the same task_write_conflict
classifier the REST handlers use, via curated helpers in the graphql module so
the map_err source guard still holds. A claim race or stale finish reached
GraphQL clients as a generic database error while REST clients got an actionable
conflict; genuine sqlx faults stay opaque on both.

Refs #327

* fix(node): End the candidate stream only after the batch is examined

A short SQL batch means no rows exist past it, not that every row in it was
examined. When the page filled mid-batch the collector treated the two as the
same, marked the stream ended, and suppressed the continuation, so every row
after the one that filled the page was unreachable. The equal-timestamp paging
tests caught it: three rows with a limit of one returned only the first.

Track how much of each batch was consumed and end the stream only when the
whole of a short batch has been examined. Otherwise leave `has_more` to the
probe row, which resumes from the last examined candidate.

Refs #327

* fix(gl): Reject a non-positive task-list limit and fault the write under test

A `--limit 0` reached the node, which clamped it to zero and answered with
an empty page marked complete, so an invalid request read as proof that no
tasks exist. Reject a non-positive limit in `fetch_tasks()`, the helper the
CLI and MCP share, so the guard cannot drift between the two surfaces.

`task_write_sql_faults_stay_opaque` did not exercise what it named. Dropping
`updated_at` also broke the SELECT in `get_task()`, so the fault surfaced
from the `get_visible_task()` pre-check through `graphql_app_err` and never
reached `graphql_claim_conflict`. A `BEFORE UPDATE` trigger keeps every read
valid and faults only inside `Db::claim_task`, and the test now also asserts
that a write-time fault is not reclassified as a claim race.

Refs #268

* fix(node): Bind task cursors to the caller and brake the task read routes

A continuation token names the last candidate a scan examined, not the last
row it returned, so it encodes how far that scan got under one caller's
visibility. The MAC bound the page filter but not the presenting identity,
so resuming a token as a different caller started the scan past rows that
caller was entitled to read and dropped them from the answer with nothing
to signal the loss. Bind the caller's normalized DID into the MAC, with
anonymous flagged absent rather than encoded as empty. Normalization goes
through normalize_owner_key so the two spellings of one did:key identity
bind identically, matching did_matches on the read path: a caller who
presents the other form of their own DID keeps their own page. A mismatched
token renders the existing single rejection message, so this adds no oracle.

GET /api/v1/tasks and GET /api/v1/tasks/{id} are anonymously reachable, and
the visibility gate costs a task lookup plus deduped-repo and
visibility-rule queries before it can return the opaque 404. An
unauthenticated prober therefore pays nothing while the node pays per
request, whether or not the id exists. Attach the per-IP limiter already
used on /ipfs/{cid}, configurable through GITLAWB_TASK_READ_RATE_LIMIT and
swept by the periodic task like every other per-key limiter.

Refs #268

* fix(node): Brake anonymous GraphQL task reads on the same per-IP bucket

The per-IP brake added for the task read routes covered only
/api/v1/tasks*, so an anonymous caller reached the same
collect_visible_tasks and get_visible_task gate over /graphql with no
bucket at all. The fence had an open lane beside it.

Carry the brake as GraphQL request data and debit it in the tasks and
task resolvers rather than layering rate_limit_by_ip onto the GraphQL
router: /graphql is one endpoint for every operation, so a router layer
would charge unrelated queries and every mutation against the task-read
bucket. Debiting per resolved field also prices an aliased query
honestly, since ten aliased tasks fields run the gate ten times.

Extract RATE_LIMIT_MESSAGE so the GraphQL surface, which cannot return a
429 status inside a 200 envelope, refuses with the same text the REST
routes use.

/graphql/ws serves the query root as well and stays unbraked; closing it
needs a WebSocketUpgrade handler and is left for a follow-up.

Refs #268
Refs #327

* fix(gl,node): parse typed task pages, track cursor cycles, and normalize assignee filter MAC

* fix(node): Cap aliased GraphQL task reads per request and derive has_more from visible rows

- Cap aliased GraphQL task read fields per request using an atomic counter
  on TaskReadBrake (MAX_GRAPHQL_TASK_READS_PER_REQUEST = 5).
- Derive has_more in collect_visible_tasks by scanning for bounded_limit + 1
  visible rows, eliminating the un-gated keyset probe that could leak the
  presence of trailing denied tasks.
- Add regression tests covering aliased GraphQL capping and trailing denied
  task has_more privacy.

Refs #327

* fix(node): Check database continuation when scan budget ends on exact batch boundary

When candidate scanning reaches MAX_TASK_SCAN_CANDIDATES without finding
a target_visible row and the final batch was full, probe the database for
rows beyond the scan position so an exhausted candidate stream is not
erroneously marked incomplete.

Refs #327

* fix(node): Document and pin the bound on the task scan-ceiling probe

The scan-ceiling branch of collect_visible_tasks settles has_more with an
un-gated LIMIT 1 probe, so a caller can learn whether any row - readable
or not - trails the position the scan stopped at. Withholding the probe
does not remove that bit: enumeration past a denied window longer than
one scan budget requires handing back a continuation, and following that
continuation returns the same terminal page one round trip later.

State what the probe discloses (one bit, only at server-chosen positions
a full scan budget apart, reachable only through a MAC'd cursor, never a
denied row's id, payload or ucan_token) and pin it end to end. Also
correct the comment above the branch, which claimed has_more never comes
from an un-gated probe while the code below it did exactly that.

Refs #327

* Index task keyset pages so the scan ceiling bounds database work.

Optional IS NULL predicates kept the planner from using a created_at/id order, so every list_tasks_keyset batch could sort a growing match set before LIMIT. Dedicated per-domain SQL plus v28 indexes make the candidate ceiling a database bound.

Refs #327

* Reject oversized task pages so list never returns more than the requested limit.

fetch_tasks asked for the remaining total, then appended every row on a valid-shaped page. A remote that sent more tasks than want could make gl and MCP expose more than --limit. Treat that page as protocol-invalid before any extra row is kept.

Refs #327

* Enforce task page limits on legacy responses and drop redundant single-column indexes.

Refs #327

* Address review feedback on task-read authorization gating and client safety.

- Wire TaskReadBrake into /graphql/ws subscriptions and verify per-request field caps and per-IP rate limiting over WebSocket connections.
- Define open-claim eligibility separately from read visibility so unassigned tasks on readable/unscoped domains can be claimed without making task bodies enumerable.
- Propagate identity errors on explicit key directories in CLI and MCP instead of silently falling back to anonymous mode.
- Enforce response byte limits before deserializing task pages, validate row schema and page-local uniqueness before row commit, and sanitize continuation cursors in terminal diagnostics.
- Add GraphQL denial assertions for repo-less tasks and token isolation.

Refs #268
Refs #327

* Address review feedback on task-read authorization gating, WebSocket budgets, and test framing.

Refs #395

* Address review feedback on quarantined task claims, WebSocket auth, and MCP limit parsing.

Refs #395

* Reject forged signatures during WebSocket authentication upgrade.

Refs #395

* Accumulate forged WebSocket handshake response until headers terminate.

Refs #395

* Withhold quarantined-repo tasks on all read surfaces before party checks.

Refs #395

* docs: Align task-read documentation with enforced visibility and rate limits.

Refs #395

* docs: Distinguish task list and get behavior in rate-limit guidance.

Refs #395

* fix(node): Batch task quarantine checks to bound read costs.

Refs #395

* docs(node): clarify the shared task-read budget

Document that GraphQL task queries over HTTP and WebSocket debit the same per-IP hourly budget as REST task reads.

Refs #395

* fix(node,gl): bound scoped repo lookup and validate MCP cursor

Bound repository access in list_repos_deduped_by_ids to requested groups via idx_repos_owner_key_name, leaving dedup_cte unscoped. Strictly validate MCP task_list cursor argument before network dispatch.

Refs #396

* test(db): verify scoped lookup work in prepared plans

Measure discarded rows and verify actual custom and generic prepared-plan counters, including the previous full-index-scan regression.

Fixup for a0ea4c78e4.
Refs #396

* docs(db): remove stray dedup comment escapes

Restore readable root-path notation in the deduplicated listing documentation.

Fixup for a0ea4c78e4.
Refs #396

* fix(node): address #464 review findings

- Hold migration 27/28 numbering: reservation comment now names the
  competing Gitlawb/node#384 range (27-35) and states the numbering is
  on hold pending #384. No renumber.
- Restore crates/gl/src/identity.rs to LF endings so the diff shows
  only the added load_optional_keypair function.
- Drop the dead single-column idx_agent_tasks_assignee_key build from
  v27 (v28 builds the keyset-ordered variants directly); adjust the
  v27 test to assert the raw index is dropped and the expression
  index is never built.
- Align task_claimable with claim_task's conditional write: drop the
  delegator short-circuit and correct the docstring.
- Pin error_for_status on the gl task list path with a 403 mock that
  serves a well-formed empty page, so the test goes red if the call
  is removed.

* test(node): pin the delegator deny on a pre-assigned claim

- Add claim_task_delegator_cannot_claim_preassigned_task: pins opaque 404
  for delegator claiming a pre-assigned task (regression guard for P2)
- Fix test_list_tasks_server_error_surfaces regex: /api/v1/tasks\? ->
  /api/v1/tasks\? (escaped ? for literal question mark)
- Assert actual reqwest StatusCode::FORBIDDEN instead of matching "403"
  in error string (avoids spurious pass from random mock port)

* test(node): exercise streamed byte-budget guard with chunked body

Use with_chunked_body instead of with_body so mockito sends no
Content-Length, pinning the no-length streamed accumulation guard.
Assert the "exceeded" message to distinguish from the declared-length
check.

* fix(node): pin declared-length byte-budget arm in oversized content-length test

Addresses beardthelion review finding (2026-09-26): test_read_task_page_json_oversized_content_length now asserts on the "(declared" arm the way the chunked twin asserts "(exceeded", so the Content-Length pre-check at crates/gl/src/task.rs:211 is genuinely pinned. Verified the pin by neutering the pre-check (test goes red), then restored. fmt clean.

* fix(node): fail closed on claim for unresolvable repo

task_claimable returned true for an unassigned task whose repo_id is
slash-form (mirror row) or names a repo this node does not host, while
task_visible fails closed on the same rows. A signed stranger could claim
such a task even though reads 404. Return false for a Some(repo_id) the
node cannot resolve to a locally-hosted repo, matching task_visible's
fail-closed check; keep the None arm open (repo-less tasks claimable by
design). Adds a regression test asserting a stranger's claim 404s and
leaves the assignee unchanged for both row classes.

* fix(node): pad sealed task cursor to fixed width

The scan-ceiling token seals the last examined row's (created_at, id) as
variable-length JSON under a length-preserving keystream, so token length
leaked the withheld row's field widths: a 32-char to_rfc3339() rendering
mints a 148-char token where a 35-char one mints 161 chars. Pad the
serialized payload with trailing JSON whitespace to a fixed 128-byte
bucket before the SIV tag is computed; decode tolerates the padding
(serde_json ignores trailing whitespace) and tokens minted before this
change still verify. REST and GraphQL share task_cursor::encode, so one
fix covers both.

Adds token_length_does_not_vary_with_position_width: asserts identical
token length across narrow/wide timestamp positions and that the padded
token still decodes to the exact stored position.

* fix(node): address beardthelion round-six P3 findings on task cursor

- Pin the encode docstring compat claim: add unpadded_legacy_token_still_verifies
  test minting a token via the pre-padding recipe (siv + apply_keystream with
  no padding) and asserting decode returns the position.
- Pin the PAYLOAD_BUCKET round-up claim: add oversized_payload_rounds_up_to_next_bucket
  test with a 140-char id crossing the 128-byte bucket, asserting round-trip
  at two buckets.
- Correct timestamp-width figures: to_rfc3339() renders 25 chars with no
  fractional seconds (not 32); 32 is the microsecond width.

* fix(node): address beardthelion round-seven P3 findings on task cursor

- Correct wire-form docstring to v1.<iv>.<body>
- Fix payload width figures: 101 bytes total, to_rfc3339() precision ladder
- Tighten oversized-payload test to assert exactly 2*PAYLOAD_BUCKET
- Pin legacy back-compat test with a frozen v1 token fixture

* test(node): address beardthelion round-eight P3 findings on task cursor

- Pin the one-bucket width: round_trips_position_verbatim now asserts the
  decoded body of a normal token is exactly PAYLOAD_BUCKET, catching a pad
  that skips next-multiple or a drifted bucket constant. Every other width
  assertion here is bucket-relative, so only an absolute check goes red.

- Assert the frozen fixture is the unpadded shape:
  unpadded_legacy_token_still_verifies decodes the body segment and asserts
  raw.len() < PAYLOAD_BUCKET, so a literal re-minted under the padded
  recipe would retire the back-compat pin loudly instead of passing
  silently.

- Narrow the recipe-pin claim: the docstring now names what the frozen
  literal actually binds (domains, MAC input, keystream, segment layout,
  field names) instead of claiming padding and serialization changes break
  the test. The padding-width claim stays with
  oversized_payload_rounds_up_to_next_bucket.

* fix(node): trim stored assignee on every task gate; align SECURITY.md task-read wording

* test(node): pin mirror-row list-vs-get asymmetry; fix plan-test bind (#553)

- EXPLAIN helper binds normalize_owner_key(assignee) to match the
  production list_tasks_keyset bind.
- Pin the deliberate asymmetry jatmn asked about: a delegator sees a
  slash-form (mirror-hosted) task in a list, while get fails closed for
  every caller because the row has no local repo backing. Notes in
  get_visible_task and a regression test keep it from drifting.
- Repair test-mod drift from the main merge: build_schema gains its
  TaskCursorKey arg at three sites and list_tasks calls move to
  list_tasks_keyset.

* test(node): reconcile tasks complexity contract with TaskPageType (#553)

The four GraphQL task/brake tests failed on this branch: contract tests
still queried the flat tasks { id } shape that TaskPageType replaced, and
the alias-brake tests ran default-size lists whose six aliases exceed
GRAPHQL_MAX_COMPLEXITY before the brake can debit.

- Contract tests now select through items { ... } with refreshed cost
  arithmetic.
- Brake tests use tasks(limit: 1), keeping six aliases under the budget
  so the 6th field reaches TaskReadBrake and fails RATE_LIMIT_MESSAGE.
  Complexity stays the outer gate (#552); the brake stays the per-request
  field-count gate.

* docs: note mirror-hosted task list-vs-get asymmetry (#553)

Option 3 for slash-form mirror repo_id rows was pinned in code and test at
6c0926de; this adds the operator-facing half jatmn asked for. README and
SECURITY task-read bullets now state that mirror rows list for party readers
while get-by-id and GraphQL task fail closed for all callers.

* ci: retrigger codeql scan after alert 86 dismissal

---------

Co-authored-by: euxarist <[email protected]>
Co-authored-by: cairn-code <[email protected]>
Co-authored-by: cairn-intern <[email protected]>
Co-authored-by: Marlowe <[email protected]>
2026-11-09 19:02:54 +00:00
beardthelion 400939d827 test(node): scrub inherited git env in mirror fixture spawns (#555)
The repository_local_* fixture tests ran plain Command::new("git") and
inherited any GIT_DIR/GIT_CONFIG_* state from an outer checkout, so a
fixture spawn could act on the wrong repository. A shared fixture_git
helper now sets the cwd and removes git environment vars; all five spawn
sites use it.
2026-11-09 12:44:46 -06:00
beardthelion 0fbc0888fe fix(node): reject special-use upstream literals at configure time (#555)
validate_mirror_upstream_url used only the peer public-host classifier, so
an IP literal the fetch worker's stricter permit denies (192.0.2/24,
198.18/15, 2001:db8::/32, ...) could be stored and then hard-fail on every
scheduled scan. The IANA special-use permit moves to peers::
mirror_upstream_ip_permitted, shared by configure validation and the
resolve-path check; the negative test set now covers the denied literals.
2026-11-09 12:40:38 -06:00
0919cf7c62 fix(core): ignore invalid signatures in satisfies_threshold (#349) (#554)
* fix(core): ignore invalid signatures in satisfies_threshold (#349)

RefUpdateCert::satisfies_threshold called verify_all, which fails closed
on the first malformed or invalid signature entry. Signature entries live
outside the signed body, so any third party can append a junk entry with
no key material — appending one denied an otherwise valid certificate, a
DoS on the threshold check.

Add a lenient valid_signers() helper that skips entries failing to parse,
decode, or verify, and have satisfies_threshold count distinct valid
signers from it. verify_all stays strict (fail-closed) so its API and the
tampered_signature_fails_verify_all test are unchanged. A forged entry
naming a real maintainer DID with a bad signature is skipped, so it cannot
inflate the count either.

Adds tests: appended junk entries do not deny a valid 2-of-2 cert, and a
forged maintainer signature is not counted toward the threshold.

Closes #349

* fix(core): bound the tolerant signature scan before cryptographic work

satisfies_threshold verified every entry, so a peer could append
arbitrarily many syntactically valid entries with bogus signatures and
buy attacker-controlled CPU and allocation on every threshold check.
Bound each stage ahead of the loop: reject a list over
MAX_SIGNATURE_ENTRIES loudly (a prefix scan would let junk starve real
signatures out of the window), refuse to base64-decode any encoded
signature that is not exactly the 86 chars of a 64-byte Ed25519
signature, skip non-maintainer and already-counted signers on a string
compare before any curve work, and return as soon as the threshold is
met.

* docs(cert): correct ENCODED_SIG_LEN length-gate comment (#554)

The comment claimed anything longer cannot decode to a valid signature.
The same 64 bytes appear in padded or non-url-safe encodings; the gate is a
canonical-form DoS bound for the tolerant scan, not a crypto impossibility.

---------

Co-authored-by: andreolf <[email protected]>
Co-authored-by: Kevin Codex <[email protected]>
2026-11-09 17:42:13 +00:00
beardthelion e5f73f7ff7 Merge remote-tracking branch 'origin/main' into pr-347
# Conflicts:
#	.env.example
#	crates/gitlawb-node/src/config.rs
#	crates/gitlawb-node/src/main.rs
2026-11-09 11:36:27 -06:00
beardthelionandRapyBus S.A.C. 05afd68e41 bootstrap-peers: add moonwake-ledger with a dialable p2p multiaddr (#559)
Co-authored-by: RapyBus S.A.C. <[email protected]>
2026-11-09 17:21:54 +00:00
beardthelion 6c971e7ea0 ci(audit): ignore RUSTSEC-2026-0459, unlinked libsqlite3-sys advisory (#564)
The advisory covers bundled SQLite in libsqlite3-sys 0.30.1, which sits in
Cargo.lock only because sqlx resolves sqlx-sqlite unconditionally. sqlx builds
postgres-only, so the bundled C source is never compiled or linked. The fix
needs libsqlite3-sys >= 0.39 but sqlx-sqlite 0.8.6 pins ^0.30.1; the waiver is
documented to retire if the sqlite feature ever enters the build.
2026-11-09 16:54:05 +00:00
beardthelion 10d5b98b2b fix(gl): report quickstart registration failure when the node returns no UCAN (#355) (#556)
A 2xx with a non-JSON or ucan-less body still printed "Registered
successfully / UCAN saved to ..." while nothing was written, and left a
stale ucan.json from another node in place. Print the success lines only
when a UCAN was actually persisted.
2026-11-09 16:11:08 +00:00
beardthelion eaab0f64b8 Merge remote-tracking branch 'origin/main' into pr-347
# Conflicts:
#	crates/gitlawb-node/src/api/peers.rs
#	crates/gitlawb-node/src/config.rs
#	crates/gitlawb-node/src/db/mod.rs
2026-11-08 21:39:11 -06:00
KRATOS d34aac8c2a fix(node): close inbound mirror review blockers
Origin-Session: local-d6a143 | Codex | 24 prompts
2026-08-15 16:14:59 +05:30
KRATOS 111d1ccd6b fix(node): harden scheduled mirror fetches
Origin-Session: local-d6a143 | Codex | 24 prompts
2026-08-15 15:41:50 +05:30
KRATOS 5cad709659 feat(node): add scheduled inbound mirror worker
Origin-Session: local-d6a143 | Codex | 18 prompts
2026-08-15 15:15:39 +05:30
KRATOS a6d8459002 fix(node): close remaining mirror review findings
Origin-Session: local-d6a143 | Codex | 21 prompts
2026-08-15 09:47:18 +05:30
KRATOS 1b7639f0d7 fix(node): align peer URL validation message
Origin-Session: local-d6a143 | Codex | 20 prompts
2026-08-15 00:33:01 +05:30
KRATOS 5b9e06d501 fix(node): close mirror review blockers
Origin-Session: local-d6a143 | Codex | 20 prompts
2026-08-15 00:25:14 +05:30
KRATOS 92997b3695 fix(node): address mirror state review findings
Origin-Session: local-d6a143 | Codex | 16 prompts
2026-08-14 22:22:19 +05:30
KRATOS ea34d84c3a fix(node): reject null mirror transition state
Origin-Session: local-d6a143 | Codex | 14 prompts
2026-08-14 21:44:27 +05:30
KRATOS 5cc7e7a564 feat(node): persist upstream mirror transition state
Origin-Session: local-d6a143 | Codex | 12 prompts
2026-08-14 21:28:17 +05:30
11 changed files with 1740 additions and 86 deletions

No files matched your search

+12
View File
@@ -58,4 +58,16 @@ ignore = [
# (verified 2026-11-03). REMOVE only when Cargo.lock has no crypto-bigint
# below 0.7.6: `grep -A1 'name = "crypto-bigint"' Cargo.lock`.
"RUSTSEC-2026-0387", # crypto-bigint timing variability (no compatible fix)
# libsqlite3-sys 0.30.1 (bundled SQLite WAL uninitialized-memory disclosure,
# CVE-2026-119375). Present in Cargo.lock because sqlx resolves sqlx-sqlite
# unconditionally; sqlx is built postgres-only
# (crates/gitlawb-node/Cargo.toml), so sqlx-sqlite and its bundled SQLite C
# source are never compiled or linked (`cargo tree -i sqlx-sqlite` prints
# nothing) and there is no reachable WAL open to attack. The fix requires
# libsqlite3-sys >= 0.39, but sqlx-sqlite 0.8.6 pins ^0.30.1; a compatible
# bump needs a sqlx upgrade, not a lockfile refresh. REMOVE this ignore if
# sqlx is ever built with the `sqlite` feature or any other libsqlite3-sys
# consumer enters the build.
"RUSTSEC-2026-0459", # libsqlite3-sys bundled SQLite CVE (not linked in our build)
]
+15
View File
@@ -372,6 +372,21 @@ GITLAWB_TRUSTED_PROXY=
# Enable automatic background sync from known peers
GITLAWB_AUTO_SYNC=false
# ── External upstream mirroring ───────────────────────────────────────────
# Default-off scheduled fetch worker for repositories configured as INBOUND.
# Enabling requires GITLAWB_ENFORCE_OWNER_PUSH=true; the node refuses to start
# otherwise. The worker updates branches and tags atomically while preserving
# Gitlawb-owned review/internal refs. Enabling also requires Git >= 2.37 on
# the host for http.curloptResolve DNS pinning; the node refuses to start
# when the installed Git is older.
GITLAWB_UPSTREAM_MIRROR_ENABLED=false
# Pause after each complete scan. Must be 5..=86400 seconds.
GITLAWB_UPSTREAM_MIRROR_INTERVAL_SECS=300
# Deadline for the external fetch child only; the preflight probes run on
# their own short bound. Must be 1..=3153600000 seconds.
GITLAWB_UPSTREAM_MIRROR_FETCH_TIMEOUT_SECS=600
# Keyset page size; every page is processed before the next scan delay.
GITLAWB_UPSTREAM_MIRROR_PAGE_SIZE=100
# ── Reconciliation sweep ─────────────────────────────────────────────────
# Periodic durability sweep: re-derives the public pin set and the withheld-blob
# recovery set each hour and fills gaps so a dropped replication job never means
+9 -1
View File
@@ -1,7 +1,7 @@
{
"$comment": "Canonical seed list for the Twigpine network. Merged with GITLAWB_BOOTSTRAP_PEERS at startup. PRs to add public nodes welcome.",
"version": 1,
"updated": "2026-06-30",
"updated": "2026-10-02",
"peers": [
{
"name": "gitlawb",
@@ -42,6 +42,14 @@
"http_url": "https://rapybus-gitlawb.com",
"p2p_multiaddr": null,
"added": "2026-05-16"
},
{
"name": "moonwake-ledger",
"operator": "Moonwake Ledger",
"did": "did:key:z6Mkh9AZ46Z3RkGZfFCom5vzX2535f54d6G2o2ZGkCqjwW8K",
"http_url": "https://node.moonwakeledger.com",
"p2p_multiaddr": null,
"added": "2026-10-02"
}
]
}
+228 -11
View File
@@ -41,6 +41,30 @@ use crate::{Error, Result};
/// The certificate type discriminant. Always `"gitlawb/ref-update/v1"`.
pub const CERT_TYPE: &str = "gitlawb/ref-update/v1";
/// Exact url-safe unpadded base64 length of a 64-byte Ed25519 signature.
/// The tolerant threshold scan requires this canonical unpadded length before
/// attempting decode: the length gate is a DoS bound that keeps an
/// attacker-sized sig string from costing allocation before the signature is
/// rejected. A longer string is not meaningless (the same 64 bytes can appear
/// in padded or non-url-safe encodings) but no in-tree signer produces one,
/// so the scan skips non-canonical entries rather than decoding them.
const ENCODED_SIG_LEN: usize = 86;
/// Hard ceiling on signature entries a certificate may carry into the
/// tolerant threshold scan. A legitimate certificate holds one entry per
/// maintainer (a re-sign after key rotation may briefly leave a second), so
/// this is far above any real signer set, while turning the scan's worst case
/// from "one curve verification per attacker-appended entry" into a small
/// constant. Rejection is loud (`Err`), not a silent prefix scan: scanning
/// only the first N entries would let junk placed ahead of the real
/// signatures starve them out of the window, which is the #349 denial again
/// in positional form. An attacker positioned to inflate the list past this
/// ceiling can already suppress the certificate outright. The loud
/// rejection does deny a legitimately over-long certificate; that is an
/// explicit, detectable denial above the cap rather than a silent window
/// junk placement could exploit.
const MAX_SIGNATURE_ENTRIES: usize = 64;
/// A signature on a ref-update certificate.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CertSignature {
@@ -138,9 +162,15 @@ impl RefUpdateCert {
Ok(())
}
/// Verify all signatures on this certificate.
/// Verify all signatures on this certificate, failing closed.
///
/// Returns the list of DIDs whose signatures are valid.
/// On success returns the list of DIDs whose signatures are valid. This is
/// strict: it returns an error if *any* signature entry is malformed or
/// invalid. Because signature entries live outside the signed body (anyone
/// can append one without key material), do not use this to make a
/// threshold decision on an untrusted certificate — a single appended junk
/// entry would reject the whole cert. Use [`Self::satisfies_threshold`],
/// which ignores invalid entries and counts only the valid signers.
pub fn verify_all(&self) -> Result<Vec<Did>> {
use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine};
let signing_bytes = self.body.to_signing_bytes()?;
@@ -169,21 +199,65 @@ impl RefUpdateCert {
/// from the provided set of authorized maintainer DIDs.
///
/// Counts distinct signer DIDs, not signature entries: a repeated
/// signature from the same maintainer counts once.
/// signature from the same maintainer counts once. Malformed and invalid
/// signature entries are skipped, not fatal — entries live outside the
/// signed body, so any third party can append junk without key material,
/// and short-circuiting on the first bad one (as [`Self::verify_all`]
/// does) would let that junk deny an otherwise valid certificate (#349).
///
/// `threshold` must be at least 1. Zero is rejected instead of being
/// treated as "no signatures required": a caller that derives the
/// threshold from configuration and gets 0 would otherwise accept any
/// certificate, whether its signatures verify or not.
/// Tolerance must not become an unbounded verification path (#349 review
/// round 2): entries are attacker-extensible, so the work done for them is
/// bounded BEFORE the loop, not discovered inside it:
///
/// - a list over [`MAX_SIGNATURE_ENTRIES`] is rejected outright (see the
/// constant for why loud rejection beats a silent prefix scan);
/// - an encoded signature is length-gated before base64 work — a 64-byte
/// Ed25519 signature is exactly [`ENCODED_SIG_LEN`] characters in
/// url-safe unpadded base64, so an oversized blob never allocates;
/// - entries whose signer is not in `maintainers`, or is already counted,
/// are skipped on a string compare, before any decode or curve work —
/// so even within the cap, junk naming strangers costs no crypto;
/// - the scan returns as soon as the threshold is met.
pub fn satisfies_threshold(&self, maintainers: &[Did], threshold: usize) -> Result<bool> {
use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine};
if threshold == 0 {
return Err(Error::RefCert("threshold must be at least 1".to_string()));
}
if self.signatures.len() > MAX_SIGNATURE_ENTRIES {
return Err(Error::RefCert(format!(
"certificate carries {} signature entries (max {})",
self.signatures.len(),
MAX_SIGNATURE_ENTRIES
)));
}
let signing_bytes = self.body.to_signing_bytes()?;
let mut counted: HashSet<&Did> = HashSet::new();
let valid = self.verify_all()?;
let distinct_signers: HashSet<&Did> =
valid.iter().filter(|d| maintainers.contains(d)).collect();
Ok(distinct_signers.len() >= threshold)
for cert_sig in &self.signatures {
if counted.contains(&cert_sig.signer) || !maintainers.contains(&cert_sig.signer) {
continue;
}
if cert_sig.sig.len() != ENCODED_SIG_LEN {
continue;
}
let Ok(vk) = cert_sig.signer.to_verifying_key() else {
continue;
};
let Ok(sig_bytes_vec) = URL_SAFE_NO_PAD.decode(&cert_sig.sig) else {
continue;
};
let Ok(sig_bytes) = <[u8; 64]>::try_from(sig_bytes_vec) else {
continue;
};
if verify(&vk, &signing_bytes, &sig_bytes).is_ok() {
counted.insert(&cert_sig.signer);
if counted.len() >= threshold {
return Ok(true);
}
}
}
Ok(counted.len() >= threshold)
}
/// Validate the certificate structure (not signatures).
@@ -416,6 +490,149 @@ mod tests {
assert!(!cert.satisfies_threshold(&maintainers, 2).unwrap());
}
/// An attacker can append junk signature entries to a cert (they live
/// outside the signed body, so no key material is needed). `satisfies_threshold`
/// must ignore them and still count the real signers — before the fix,
/// `verify_all`'s `?` on the junk made the whole check error out, denying a
/// valid 2-of-2 cert.
#[test]
fn satisfies_threshold_ignores_appended_junk_signatures() {
let kp1 = Keypair::generate();
let kp2 = Keypair::generate();
let mut cert = RefUpdateCert::new(
kp1.did(),
"refs/heads/main".to_string(),
dummy_hash('0'),
dummy_hash('a'),
1,
&kp1,
)
.unwrap();
cert.countersign(&kp2).unwrap();
// Junk sits AHEAD of the real entries so the scan must actually skip
// it: appended last, the threshold-2 early return at the second real
// signature would fire before any junk is examined and the test would
// stay green even if the skip logic never ran.
cert.signatures.insert(
0,
CertSignature {
signer: kp1.did(),
sig: "!!!not-base64!!!".to_string(),
},
);
// Valid base64 but only 3 bytes — fails the 64-byte length check.
cert.signatures.insert(
0,
CertSignature {
signer: kp2.did(),
sig: "AAAA".to_string(),
},
);
// Exactly ENCODED_SIG_LEN of valid base64 decoding to garbage —
// exercises the decode+verify path before the skip, not just the
// length gate.
cert.signatures.insert(
0,
CertSignature {
signer: kp1.did(),
sig: "A".repeat(ENCODED_SIG_LEN),
},
);
let maintainers = vec![kp1.did(), kp2.did()];
assert!(cert.satisfies_threshold(&maintainers, 2).unwrap());
}
/// A junk entry that names a real maintainer DID but carries a signature
/// that does not verify must not be counted — otherwise appending garbage
/// under a maintainer's DID could forge a threshold.
#[test]
fn satisfies_threshold_does_not_count_a_forged_maintainer_signature() {
let kp1 = Keypair::generate();
let kp2 = Keypair::generate();
let mut cert = RefUpdateCert::new(
kp1.did(),
"refs/heads/main".to_string(),
dummy_hash('0'),
dummy_hash('a'),
1,
&kp1,
)
.unwrap();
// Well-formed 64-byte signature from kp2, but over unrelated bytes, so
// it fails verification against the cert body.
cert.signatures.push(CertSignature {
signer: kp2.did(),
sig: kp2.sign_b64(b"unrelated bytes"),
});
let maintainers = vec![kp1.did(), kp2.did()];
// Only kp1 actually signed the body: 2-of-2 must fail, 1-of-2 holds.
assert!(!cert.satisfies_threshold(&maintainers, 2).unwrap());
assert!(cert.satisfies_threshold(&maintainers, 1).unwrap());
}
/// The tolerant scan must not be an unbounded verification path: a list
/// past MAX_SIGNATURE_ENTRIES is rejected loudly before any curve work,
/// while a list at the cap still verifies normally.
#[test]
fn satisfies_threshold_rejects_an_oversized_signature_list() {
let kp = Keypair::generate();
let mut cert = RefUpdateCert::new(
kp.did(),
"refs/heads/main".to_string(),
dummy_hash('0'),
dummy_hash('a'),
1,
&kp,
)
.unwrap();
let junk = || CertSignature {
signer: Keypair::generate().did(),
sig: "AAAA".to_string(),
};
while cert.signatures.len() < MAX_SIGNATURE_ENTRIES {
cert.signatures.push(junk());
}
let maintainers = vec![kp.did()];
// At the cap: still fine, the real signature counts.
assert!(cert.satisfies_threshold(&maintainers, 1).unwrap());
// One past the cap: loud rejection, no tolerant scan.
cert.signatures.push(junk());
let err = cert.satisfies_threshold(&maintainers, 1).unwrap_err();
assert!(err.to_string().contains("signature entries"));
}
/// An entry whose encoded signature is not the exact base64 length of a
/// 64-byte Ed25519 signature is skipped before any base64 decode — an
/// attacker-sized sig string must cost neither allocation nor curve work,
/// and must not affect the verdict.
#[test]
fn satisfies_threshold_skips_an_oversized_encoded_signature() {
let kp1 = Keypair::generate();
let kp2 = Keypair::generate();
let mut cert = RefUpdateCert::new(
kp1.did(),
"refs/heads/main".to_string(),
dummy_hash('0'),
dummy_hash('a'),
1,
&kp1,
)
.unwrap();
// Names a real maintainer, valid base64 alphabet, but megabytes long:
// the length gate must refuse it without decoding.
cert.signatures.push(CertSignature {
signer: kp2.did(),
sig: "A".repeat(2 * 1024 * 1024),
});
let maintainers = vec![kp1.did(), kp2.did()];
assert!(cert.satisfies_threshold(&maintainers, 1).unwrap());
assert!(!cert.satisfies_threshold(&maintainers, 2).unwrap());
}
#[test]
fn threshold_zero_is_rejected() {
let kp = Keypair::generate();
+116 -55
View File
@@ -84,7 +84,7 @@ pub(crate) const PUBLIC_HTTP_URL_REQUIREMENT: &str = "must be a public http(s) U
/// sits at a prefix-length-dependent offset (RFC 6052 §2.2) — so they return
/// `None`. Any caller that needs them blocked must reject the wider
/// `64:ff9b::/32` itself; `is_public_http_url` does this in its native-v6 arm.
fn embedded_ipv4(v6: std::net::Ipv6Addr) -> Option<std::net::Ipv4Addr> {
pub(crate) fn embedded_ipv4(v6: std::net::Ipv6Addr) -> Option<std::net::Ipv4Addr> {
if let Some(v4) = v6.to_ipv4_mapped().or_else(|| v6.to_ipv4()) {
return Some(v4);
}
@@ -104,6 +104,116 @@ fn embedded_ipv4(v6: std::net::Ipv6Addr) -> Option<std::net::Ipv4Addr> {
None
}
/// Whether an IP literal is outside the local/private ranges that an
/// attacker-controlled outbound URL must never reach. Kept separate from URL
/// parsing so callers that resolve a DNS name themselves can apply the same
/// classification to every answer and then pin the approved addresses.
pub(crate) fn is_public_ip(ip: std::net::IpAddr) -> bool {
// Reject loopback/unspecified on the literal as given — catches `::1` and
// `::` before the IPv4-folding below (`::1`.to_ipv4() would otherwise map
// to a non-loopback `0.0.0.1`).
if ip.is_loopback() || ip.is_unspecified() {
return false;
}
// Fold any IPv6 literal that embeds an IPv4 address (mapped, compatible,
// 6to4, NAT64) down to that IPv4 so the v4 range checks catch
// loopback/private addresses smuggled in via an IPv6 encoding, then
// re-check loopback/unspecified.
let ip = match ip {
std::net::IpAddr::V6(v6) => embedded_ipv4(v6).map(std::net::IpAddr::V4).unwrap_or(ip),
v4 => v4,
};
if ip.is_loopback() || ip.is_unspecified() {
return false;
}
match ip {
std::net::IpAddr::V4(v4) => {
let o = v4.octets();
// RFC1918 private, link-local, CGNAT (100.64.0.0/10), or the
// RFC1122 "this host" block 0.0.0.0/8 (never a valid destination;
// 0.0.0.0 itself is already caught by the is_unspecified check).
if v4.is_private()
|| v4.is_link_local()
|| (o[0] == 100 && (o[1] & 0xc0) == 64)
|| o[0] == 0
{
return false;
}
}
std::net::IpAddr::V6(v6) => {
let s = v6.segments();
// fc00::/7 (unique-local), fe80::/10 (link-local), or fec0::/10
// (site-local, deprecated by RFC 3879; the link-local mask does
// not cover it).
if (s[0] & 0xfe00) == 0xfc00 || (s[0] & 0xffc0) == 0xfe80 || (s[0] & 0xffc0) == 0xfec0 {
return false;
}
// Any NAT64 address (64:ff9b::/32) that is not the cleanly
// decodable well-known /96 — e.g. the RFC 8215 local-use
// 64:ff9b:1::/48 — carries a translated target whose embedded v4
// sits at a prefix-length-dependent offset (RFC 6052 §2.2).
// Rather than risk a wrong decode across every prefix length we
// reject the whole NAT64 space here. The well-known /96 was already
// folded to its v4 above and never reaches this arm.
if s[0] == 0x0064 && s[1] == 0xff9b {
return false;
}
}
}
true
}
/// Mirror-upstream variant of [`is_public_ip`]: in addition to the shared
/// peer classification, rejects IANA special-purpose ranges that are
/// non-global, locally meaningful, or unsafe as owner-controlled outbound
/// fetch endpoints. For IP-literal upstreams, configure-time URL validation
/// and the fetch-time egress policy share this classifier, so a stored
/// literal can never fail the worker it is scheduled for. Hostname upstreams
/// are only shape-checked at configure time: each fetch resolves DNS and
/// applies this classifier to the answers, so a hostname whose answers turn
/// non-public still fails at fetch time.
pub(crate) fn mirror_upstream_ip_permitted(ip: std::net::IpAddr) -> bool {
if !is_public_ip(ip) {
return false;
}
let classified = match ip {
std::net::IpAddr::V6(v6) => embedded_ipv4(v6).map(std::net::IpAddr::V4).unwrap_or(ip),
_ => ip,
};
match classified {
std::net::IpAddr::V4(v4) => {
let [a, b, c, _] = v4.octets();
// IANA special-purpose ranges that are not globally reachable but
// may be routed inside labs, clouds, or operator networks. They are
// unsuitable for an owner-controlled outbound fetch even though
// `Ipv4Addr::is_private` does not classify them as RFC1918.
!(a >= 224
|| (a == 192 && b == 0 && c == 0)
|| (a == 192 && b == 0 && c == 2)
|| (a == 192 && b == 88 && c == 99)
|| (a == 198 && matches!(b, 18 | 19))
|| (a == 198 && b == 51 && c == 100)
|| (a == 203 && b == 0 && c == 113))
}
std::net::IpAddr::V6(v6) => {
let s = v6.segments();
// IANA special-purpose ranges that are non-global, locally
// meaningful, or unsafe to treat as ordinary forge endpoints.
// Reject the complete IETF protocol-assignment /23 rather than
// trying to maintain its narrow globally reachable exceptions;
// a public forge has no reason to depend on those anycast ranges.
!(v6.is_multicast()
|| (s[0] & 0xffc0) == 0xfec0
|| (s[0] == 0x0100 && s[1] == 0 && s[2] == 0 && s[3] == 0)
|| (s[0] == 0x0100 && s[1] == 0 && s[2] == 0 && s[3] == 1)
|| (s[0] == 0x2001 && (s[1] & 0xfe00) == 0)
|| (s[0] == 0x2001 && s[1] == 0x0db8)
|| (s[0] == 0x3fff && (s[1] & 0xf000) == 0)
|| s[0] == 0x5f00)
}
}
}
/// Whether a peer `http_url` is a public http(s) endpoint safe to register.
/// Rejects non-http(s) schemes, loopback/unspecified/private/link-local IPs,
/// and `localhost` / `.localhost` / `.local` / `.internal` hostnames. Used at
@@ -137,60 +247,9 @@ pub fn is_public_http_url(raw: &str) -> bool {
// before parsing as an IP.
let ip_candidate = host.trim_start_matches('[').trim_end_matches(']');
if let Ok(ip) = ip_candidate.parse::<std::net::IpAddr>() {
// Reject loopback/unspecified on the literal as given — catches `::1`
// and `::` before the IPv4-folding below (`::1`.to_ipv4() would
// otherwise map to a non-loopback `0.0.0.1`).
if ip.is_loopback() || ip.is_unspecified() {
return false;
}
// Fold any IPv6 literal that embeds an IPv4 address (mapped, compatible,
// 6to4, NAT64) down to that IPv4 so the v4 range checks catch
// loopback/private addresses smuggled in via an IPv6 encoding, then
// re-check loopback/unspecified.
let ip = match ip {
std::net::IpAddr::V6(v6) => embedded_ipv4(v6).map(std::net::IpAddr::V4).unwrap_or(ip),
v4 => v4,
};
if ip.is_loopback() || ip.is_unspecified() {
return false;
}
match ip {
std::net::IpAddr::V4(v4) => {
let o = v4.octets();
// RFC1918 private, link-local, CGNAT (100.64.0.0/10), or the
// RFC1122 "this host" block 0.0.0.0/8 (never a valid destination;
// 0.0.0.0 itself is already caught by the is_unspecified check).
if v4.is_private()
|| v4.is_link_local()
|| (o[0] == 100 && (o[1] & 0xc0) == 64)
|| o[0] == 0
{
return false;
}
}
std::net::IpAddr::V6(v6) => {
let s = v6.segments();
// fc00::/7 (unique-local), fe80::/10 (link-local), or fec0::/10
// (site-local, deprecated by RFC 3879 and so never a legitimate
// peer address, but still routable on networks that kept it).
if (s[0] & 0xfe00) == 0xfc00
|| (s[0] & 0xffc0) == 0xfe80
|| (s[0] & 0xffc0) == 0xfec0
{
return false;
}
// Any NAT64 address (64:ff9b::/32) that is not the cleanly
// decodable well-known /96 — e.g. the RFC 8215 local-use
// 64:ff9b:1::/48 — carries a translated target whose embedded v4
// sits at a prefix-length-dependent offset (RFC 6052 §2.2).
// Rather than risk a wrong decode across every prefix length we
// reject the whole NAT64 space here. The well-known /96 was
// already folded to its v4 above and never reaches this arm.
if s[0] == 0x0064 && s[1] == 0xff9b {
return false;
}
}
}
// IP literals share the single DNS/literal classification in
// `is_public_ip`; hostname-only rules stay local to this function.
return is_public_ip(ip);
} else if !host.contains('.') {
// A name with no dot at all. The `.local` / `.internal` rules above are
// suffix checks, so a single label slips both, and the resolver decides
@@ -616,6 +675,8 @@ mod tests {
fn rejects_deprecated_site_local_v6() {
// fec0::/10, deprecated by RFC 3879. The link-local mask (0xffc0 ==
// 0xfe80) does not cover it: 0xfec0 & 0xffc0 is 0xfec0.
assert!(!super::is_public_ip("fec0::1".parse().unwrap()));
assert!(!super::is_public_ip("feff:ffff::1".parse().unwrap()));
for bad in ["http://[fec0::1]/", "http://[feff:ffff::1]:7545"] {
assert!(!is_public_http_url(bad), "{bad:?} must be rejected");
}
+92
View File
@@ -152,6 +152,46 @@ pub struct Config {
#[arg(long, env = "GITLAWB_AUTO_SYNC", default_value_t = false)]
pub auto_sync: bool,
/// Periodically fetch configured INBOUND repositories from their external
/// HTTPS upstream. Disabled by default: enabling this also requires
/// `GITLAWB_ENFORCE_OWNER_PUSH=true`, so an upstream-authoritative mirror
/// cannot accept writes from arbitrary signed identities between fetches.
#[arg(long, env = "GITLAWB_UPSTREAM_MIRROR_ENABLED", default_value_t = false)]
pub upstream_mirror_enabled: bool,
/// Pause between complete upstream-mirror scans. A scan is sequential and
/// paginated; the pause begins only after the prior scan has finished, so a
/// slow forge cannot create overlapping worker cycles.
#[arg(
long,
env = "GITLAWB_UPSTREAM_MIRROR_INTERVAL_SECS",
default_value_t = 300,
value_parser = clap::value_parser!(u64).range(5..=86_400)
)]
pub upstream_mirror_interval_secs: u64,
/// Maximum wall-clock time for one external `git fetch`. The child runs in
/// its own process group and is fully reaped on timeout. The preflight
/// URL-rewrite and local-config probes run on their own short bounds and
/// do not consume this budget.
#[arg(
long,
env = "GITLAWB_UPSTREAM_MIRROR_FETCH_TIMEOUT_SECS",
default_value_t = 600,
value_parser = clap::value_parser!(u64).range(1..=GIT_SERVICE_TIMEOUT_SECS_MAX)
)]
pub upstream_mirror_fetch_timeout_secs: u64,
/// Number of durable INBOUND rows read per database page. Every page is
/// processed before the scan sleeps; this bounds query memory without
/// permanently starving repositories beyond the first page.
#[arg(
long,
env = "GITLAWB_UPSTREAM_MIRROR_PAGE_SIZE",
default_value_t = 100,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_000)
)]
pub upstream_mirror_page_size: usize,
/// Enable the periodic reconciliation sweep that re-derives pin/seal sets
/// and fills durability gaps. Defaults to true; set to false to disable
/// the sweep even when a pin backend (IPFS/Pinata) is configured.
@@ -858,6 +898,13 @@ impl Config {
Self::DB_LOCK_POOL_NON_PUSH_HEADROOM
));
}
if self.upstream_mirror_enabled && !self.enforce_owner_push {
return Err(
"GITLAWB_UPSTREAM_MIRROR_ENABLED requires GITLAWB_ENFORCE_OWNER_PUSH=true: \
an upstream-authoritative mirror must not accept writes from arbitrary signed identities"
.to_string(),
);
}
let main_floor = Self::DB_POOL_APP_HEADROOM as u64;
if (self.db_max_connections as u64) < main_floor {
return Err(format!(
@@ -1057,6 +1104,51 @@ pub(crate) fn validate_web_url(raw: &str) -> Result<(), String> {
mod tests {
use super::*;
#[test]
fn upstream_mirror_worker_is_default_off_and_bounded() {
let mut config = Config::parse_from(["gitlawb-node"]);
assert!(!config.upstream_mirror_enabled);
assert_eq!(config.upstream_mirror_interval_secs, 300);
assert_eq!(config.upstream_mirror_fetch_timeout_secs, 600);
assert_eq!(config.upstream_mirror_page_size, 100);
config.validate().unwrap();
assert!(
Config::try_parse_from(["gitlawb-node", "--upstream-mirror-interval-secs", "4",])
.is_err()
);
assert!(
Config::try_parse_from(["gitlawb-node", "--upstream-mirror-page-size", "0",]).is_err()
);
}
#[test]
fn upstream_mirror_worker_requires_owner_push_enforcement() {
// enforce_owner_push defaults true on main, so the only unsafe shape
// is the explicit opt-out: mirror enabled plus enforcement refused.
let mut unsafe_config = Config::parse_from([
"gitlawb-node",
"--upstream-mirror-enabled",
"--enforce-owner-push",
"false",
]);
let error = unsafe_config.validate().unwrap_err();
assert!(error.contains("GITLAWB_ENFORCE_OWNER_PUSH=true"));
Config::parse_from([
"gitlawb-node",
"--upstream-mirror-enabled",
"--enforce-owner-push",
"true",
])
.validate()
.unwrap();
// The default posture alone also validates: enforcement is on.
Config::parse_from(["gitlawb-node", "--upstream-mirror-enabled"])
.validate()
.unwrap();
}
#[test]
fn lock_pool_size_defaults_to_40_and_rejects_zero() {
assert_eq!(
+171 -1
View File
@@ -118,6 +118,17 @@ pub struct RepoMirrorState {
pub updated_at: DateTime<Utc>,
}
/// One canonical repository eligible for a scheduled external-forge fetch.
/// The worker keyset-pages these rows by `repo_id`; transition metadata is not
/// exposed because the selecting query admits only stable INBOUND state.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct InboundMirrorTarget {
pub repo_id: String,
pub owner_did: String,
pub name: String,
pub upstream_url: String,
}
#[cfg_attr(not(test), allow(dead_code))]
impl RepoMirrorState {
fn validate(&self) -> Result<()> {
@@ -137,7 +148,7 @@ impl RepoMirrorState {
}
#[cfg_attr(not(test), allow(dead_code))]
fn validate_mirror_upstream_url(raw: &str) -> Result<reqwest::Url> {
pub(crate) fn validate_mirror_upstream_url(raw: &str) -> Result<reqwest::Url> {
if raw.is_empty() || raw.len() > 2048 {
anyhow::bail!("mirror upstream URL must contain 1 to 2048 bytes");
}
@@ -152,6 +163,16 @@ fn validate_mirror_upstream_url(raw: &str) -> Result<reqwest::Url> {
if !crate::api::peers::is_public_http_url(raw) {
anyhow::bail!("mirror upstream URL must use a public host");
}
// An IP-literal host must also pass the fetch worker's stricter permit;
// otherwise the row stores fine but every scheduled fetch hard-fails.
if let Some(host) = url.host_str() {
let literal = host.trim_start_matches('[').trim_end_matches(']');
if let Ok(ip) = literal.parse::<std::net::IpAddr>() {
if !crate::api::peers::mirror_upstream_ip_permitted(ip) {
anyhow::bail!("mirror upstream URL uses an unsuitable literal host");
}
}
}
if url.host_str().is_none() || url.path().trim_matches('/').is_empty() {
anyhow::bail!("mirror upstream URL must include a host and repository path");
}
@@ -1888,6 +1909,54 @@ impl Db {
row.map(row_to_repo_mirror_state).transpose()
}
/// Keyset-page stable INBOUND mirrors for the external-forge worker.
/// Transitioning/outbound rows are excluded at the database boundary so a
/// cycle cannot intentionally schedule them. The worker rechecks the state
/// after taking the repo write lock to close the selection-to-fetch window.
pub async fn list_inbound_mirror_targets(
&self,
after_repo_id: Option<&str>,
limit: i64,
) -> Result<Vec<InboundMirrorTarget>> {
let rows = sqlx::query(
"SELECT id, owner_did, name, upstream_url
FROM repos
WHERE mirror_status = 'inbound'
AND upstream_url IS NOT NULL
AND (
(mirror_transition_id IS NULL AND mirror_transition_phase IS NULL)
OR (mirror_transition_id IS NOT NULL
AND mirror_transition_phase = 'completed')
)
AND ($1::text IS NULL OR id > $1)
ORDER BY id ASC
LIMIT $2",
)
.bind(after_repo_id)
.bind(limit.clamp(1, 1_000))
.fetch_all(&self.pool)
.await?;
rows.into_iter()
.map(|row| {
Ok(InboundMirrorTarget {
repo_id: row
.try_get("id")
.context("reading inbound mirror repository id")?,
owner_did: row
.try_get("owner_did")
.context("reading inbound mirror owner DID")?,
name: row
.try_get("name")
.context("reading inbound mirror repository name")?,
upstream_url: row
.try_get("upstream_url")
.context("reading inbound mirror upstream URL")?,
})
})
.collect()
}
/// Register a mirrored repo from a peer in the local DB so git smart HTTP can serve it.
/// Uses INSERT OR IGNORE (SQLite) / ON CONFLICT DO NOTHING (Postgres) so it's idempotent.
pub async fn upsert_mirror_repo(
@@ -5497,6 +5566,12 @@ mod mirror_state_tests {
"https://forge.internal/Gitlawb/node.git",
"https://[::ffff:127.0.0.1]/Gitlawb/node.git",
"https://[64:ff9b::7f00:1]/Gitlawb/node.git",
// Special-use literals the peer classifier accepts but the fetch
// worker's stricter permit denies; configure and fetch must agree.
"https://192.0.2.1/org/repo.git",
"https://198.18.0.1/org/repo.git",
"https://203.0.113.5/org/repo.git",
"https://[2001:db8::1]/org/repo.git",
] {
assert!(
validate_mirror_upstream_url(invalid).is_err(),
@@ -5713,6 +5788,101 @@ mod mirror_state_tests {
assert_eq!(stored.updated_at, original_updated_at);
}
#[sqlx::test]
async fn inbound_worker_pages_all_and_only_stable_inbound_mirrors(pool: PgPool) {
let db = migrated_db(pool).await;
let ids = [
Uuid::parse_str("00000000-0000-0000-0000-000000000001").unwrap(),
Uuid::parse_str("00000000-0000-0000-0000-000000000002").unwrap(),
Uuid::parse_str("00000000-0000-0000-0000-000000000003").unwrap(),
];
for (index, id) in ids.into_iter().enumerate() {
let mut existing = repo(id);
existing.name = format!("mirror-{index}");
db.create_repo(&existing).await.unwrap();
db.configure_inbound_mirror(
&existing.id,
&format!("https://github.example/org/mirror-{index}.git"),
)
.await
.unwrap();
}
// The middle row simulates a transition selected by the next B1 slice.
// It must not be scheduled while authority is changing.
sqlx::query(
"UPDATE repos
SET mirror_status = 'transitioning_inbound_to_outbound',
mirror_transition_id = $2,
mirror_transition_phase = 'queued'
WHERE id = $1",
)
.bind(ids[1].to_string())
.bind(Uuid::new_v4())
.execute(db.pool())
.await
.unwrap();
let first = db.list_inbound_mirror_targets(None, 1).await.unwrap();
assert_eq!(first.len(), 1);
assert_eq!(first[0].repo_id, ids[0].to_string());
assert_eq!(first[0].name, "mirror-0");
assert_eq!(first[0].owner_did, "did:key:z6MkMirrorOwner");
assert_eq!(
first[0].upstream_url,
"https://github.example/org/mirror-0.git"
);
let second = db
.list_inbound_mirror_targets(Some(&first[0].repo_id), 1)
.await
.unwrap();
assert_eq!(second.len(), 1);
assert_eq!(second[0].repo_id, ids[2].to_string());
let exhausted = db
.list_inbound_mirror_targets(Some(&second[0].repo_id), 1)
.await
.unwrap();
assert!(exhausted.is_empty());
}
/// The second stable shape the CHECK admits: an inbound row whose
/// transition completed keeps its audit metadata and must still be
/// scheduled. Only in-flight phases make a row unstable.
#[sqlx::test]
async fn inbound_worker_pages_completed_stable_mirror_rows(pool: PgPool) {
let db = migrated_db(pool).await;
let id = Uuid::parse_str("00000000-0000-0000-0000-000000000004").unwrap();
let mut existing = repo(id);
existing.name = "mirror-completed".to_string();
db.create_repo(&existing).await.unwrap();
db.configure_inbound_mirror(
&existing.id,
"https://github.example/org/mirror-completed.git",
)
.await
.unwrap();
sqlx::query(
"UPDATE repos
SET mirror_transition_id = $2,
mirror_transition_phase = 'completed'
WHERE id = $1",
)
.bind(id.to_string())
.bind(Uuid::new_v4())
.execute(db.pool())
.await
.unwrap();
let page = db.list_inbound_mirror_targets(None, 10).await.unwrap();
assert!(
page.iter().any(|t| t.repo_id == id.to_string()),
"completed-stable inbound row must be scheduled: {page:?}"
);
}
#[sqlx::test]
async fn malformed_mirror_row_returns_an_error_instead_of_panicking(pool: PgPool) {
let row = sqlx::query(
+101 -14
View File
@@ -43,6 +43,53 @@ const WALK_TIMEOUT: Duration = Duration::from_secs(600);
#[cfg(unix)]
const WATCHDOG_TERM_GRACE: Duration = Duration::from_secs(1);
/// Build a Git child command, optionally with an isolated environment for an
/// owner-selected HTTPS destination. The isolated form intentionally drops
/// ambient proxy, credential-helper, askpass, and Git config environment that
/// could otherwise forward node-operator secrets to the selected upstream.
fn git_child_command(
git_bin: &str,
args: &[&str],
repo_path: &Path,
isolated_https: bool,
isolated_config: &[(&str, &str)],
) -> std::process::Command {
let mut command = std::process::Command::new(git_bin);
command
.args(args)
.current_dir(repo_path)
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped());
if isolated_https {
let path = std::env::var_os("PATH");
command.env_clear();
if let Some(path) = path {
command.env("PATH", path);
}
command
.env("GIT_CONFIG_NOSYSTEM", "1")
.env(
"GIT_CONFIG_GLOBAL",
if cfg!(windows) { "NUL" } else { "/dev/null" },
)
.env("GIT_TERMINAL_PROMPT", "0")
.env("GIT_ASKPASS", "false")
.env("SSH_ASKPASS", "false")
.env("LANG", "C")
.env("LC_ALL", "C");
if !isolated_config.is_empty() {
command.env("GIT_CONFIG_COUNT", isolated_config.len().to_string());
for (index, (key, value)) in isolated_config.iter().enumerate() {
command
.env(format!("GIT_CONFIG_KEY_{index}"), key)
.env(format!("GIT_CONFIG_VALUE_{index}"), value);
}
}
}
command
}
/// Run one git child under a shared `deadline` with process-group teardown,
/// BLOCKING, and return its stdout. The child runs in its own process group; a
/// watchdog thread SIGTERMs (lets git clean up its `*.lock` files), then SIGKILLs,
@@ -81,24 +128,22 @@ fn child_terminated_without_reaping(pid: i32) -> bool {
}
#[cfg(unix)]
pub(crate) fn run_bounded_git_raw(
fn run_bounded_git_raw_impl(
git_bin: &str,
args: &[&str],
repo_path: &Path,
stdin_bytes: &[u8],
deadline: Instant,
isolated_https: bool,
isolated_config: &[(&str, &str)],
) -> Result<(std::process::ExitStatus, Vec<u8>, Vec<u8>)> {
use std::io::{Read, Write};
use std::os::unix::process::CommandExt;
use std::sync::mpsc::RecvTimeoutError;
let label = args.first().copied().unwrap_or("git");
let mut child = std::process::Command::new(git_bin)
.args(args)
.current_dir(repo_path)
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
let mut command = git_child_command(git_bin, args, repo_path, isolated_https, isolated_config);
let mut child = command
.process_group(0)
.spawn()
.with_context(|| format!("failed to spawn git {label}"))?;
@@ -208,6 +253,51 @@ pub(crate) fn run_bounded_git_raw(
Ok((status, out, err))
}
pub(crate) fn run_bounded_git_raw(
git_bin: &str,
args: &[&str],
repo_path: &Path,
stdin_bytes: &[u8],
deadline: Instant,
) -> Result<(std::process::ExitStatus, Vec<u8>, Vec<u8>)> {
run_bounded_git_raw_impl(git_bin, args, repo_path, stdin_bytes, deadline, false, &[])
}
/// Run a bounded Git child without inheriting ambient Git/proxy/credential
/// configuration. Reserved for owner-selected external HTTPS destinations;
/// ordinary local Git operations retain their existing environment.
pub(crate) fn run_bounded_git_raw_isolated_https(
git_bin: &str,
args: &[&str],
repo_path: &Path,
stdin_bytes: &[u8],
deadline: Instant,
) -> Result<(std::process::ExitStatus, Vec<u8>, Vec<u8>)> {
run_bounded_git_raw_impl(git_bin, args, repo_path, stdin_bytes, deadline, true, &[])
}
/// Isolated HTTPS runner with command-scope Git configuration supplied as
/// separate key/value environment entries. Unlike `git -c key=value`, this
/// preserves valid URL subsection keys whose path contains `=`.
pub(crate) fn run_bounded_git_raw_isolated_https_with_config(
git_bin: &str,
args: &[&str],
repo_path: &Path,
stdin_bytes: &[u8],
deadline: Instant,
config: &[(&str, &str)],
) -> Result<(std::process::ExitStatus, Vec<u8>, Vec<u8>)> {
run_bounded_git_raw_impl(
git_bin,
args,
repo_path,
stdin_bytes,
deadline,
true,
config,
)
}
/// Bounded git returning only stdout, `bail!`ing on any nonzero exit. The thin
/// wrapper the walk callers use. Probes that must distinguish exit classes —
/// `git cat-file` absence vs an object-store access failure — call
@@ -241,23 +331,20 @@ pub(crate) fn run_bounded_git(
/// the Unix version's signature and result semantics so every caller compiles on all
/// targets (#174).
#[cfg(not(unix))]
pub(crate) fn run_bounded_git_raw(
fn run_bounded_git_raw_impl(
git_bin: &str,
args: &[&str],
repo_path: &Path,
stdin_bytes: &[u8],
deadline: Instant,
isolated_https: bool,
isolated_config: &[(&str, &str)],
) -> Result<(std::process::ExitStatus, Vec<u8>, Vec<u8>)> {
use std::io::{Read, Write};
use std::sync::mpsc::RecvTimeoutError;
let label = args.first().copied().unwrap_or("git");
let mut child = std::process::Command::new(git_bin)
.args(args)
.current_dir(repo_path)
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
let mut child = git_child_command(git_bin, args, repo_path, isolated_https, isolated_config)
.spawn()
.with_context(|| format!("failed to spawn git {label}"))?;
+17
View File
@@ -24,6 +24,7 @@ mod state;
mod sync;
#[cfg(test)]
mod test_support;
mod upstream_mirror;
mod visibility;
mod webhooks;
@@ -681,6 +682,22 @@ async fn main() -> Result<()> {
info!("auto-sync worker started");
}
// External-forge mirroring is a separate, default-off worker. Startup
// validation already requires owner-only pushes when it is enabled; the
// worker additionally verifies the Git runtime can pin DNS answers before
// any URL reaches `git fetch`.
if config.upstream_mirror_enabled {
upstream_mirror::start(
Arc::clone(&state.db),
Arc::clone(&state.config),
state.repo_store.clone(),
state.repo_write_leases.clone(),
Arc::clone(&state.git_write_semaphore),
state.subscribe_shutdown(),
)
.context("starting upstream mirror worker")?;
}
// Periodic reconciliation sweep: re-derives pin/seal sets and fills gaps
// so a dropped replication job never means data loss.
{
+968
View File
@@ -0,0 +1,968 @@
//! Scheduled external-forge fetches for canonical INBOUND mirrors.
//!
//! This worker is deliberately separate from `sync`, which mirrors repositories
//! between Gitlawb peers. External upstream URLs are owner-controlled data that
//! eventually reach `git fetch`, so each cycle resolves DNS itself, validates
//! every address, and pins the approved answers into Git/libcurl with
//! `http.curloptResolve`. Redirects, proxies, credential helpers, submodule
//! recursion, and non-HTTPS protocols are disabled at the command boundary.
use anyhow::{Context, Result};
use std::collections::BTreeSet;
use std::net::IpAddr;
use std::path::Path;
use std::sync::Arc;
use std::time::{Duration, Instant};
use tracing::{info, warn};
use crate::config::Config;
use crate::db::{Db, InboundMirrorTarget, MirrorStatus, MirrorTransitionPhase};
use crate::git::repo_store::RepoStore;
use crate::state::{repo_identity_key, RepoWriteLeases};
const DNS_TIMEOUT: Duration = Duration::from_secs(10);
const MAX_PINNED_ADDRESSES: usize = 32;
const MIN_CURL_OPT_RESOLVE_GIT: (u64, u64) = (2, 37);
#[derive(Debug, Clone, Copy)]
struct EgressPolicy;
#[derive(Debug, Clone, PartialEq, Eq)]
struct PinnedUpstream {
url: String,
curlopt_resolve: Option<String>,
}
impl EgressPolicy {
async fn resolve(&self, url: reqwest::Url) -> Result<PinnedUpstream> {
let raw_host = url.host_str().context("mirror upstream URL has no host")?;
let connect_host = raw_host
.trim_start_matches('[')
.trim_end_matches(']')
.to_ascii_lowercase();
let policy_host = normalize_policy_host(&connect_host);
reject_always_local_hostname(&policy_host)?;
let port = url
.port_or_known_default()
.context("mirror upstream URL has no known port")?;
if let Ok(ip) = connect_host.parse::<IpAddr>() {
self.validate_addresses(&policy_host, &[ip])?;
return Ok(PinnedUpstream {
url: url.to_string(),
// An IP literal performs no DNS lookup, so there is no rebinding
// window to pin. TLS still verifies the literal from the URL.
curlopt_resolve: None,
});
}
let resolved = tokio::time::timeout(
DNS_TIMEOUT,
tokio::net::lookup_host((connect_host.as_str(), port)),
)
.await
.context("mirror upstream DNS lookup timed out")?
.context("resolving mirror upstream host")?;
let addresses: BTreeSet<IpAddr> = resolved.map(|addr| addr.ip()).collect();
if addresses.is_empty() {
anyhow::bail!("mirror upstream host resolved to no addresses");
}
let addresses: Vec<IpAddr> = addresses.into_iter().collect();
self.validate_addresses(&policy_host, &addresses)?;
let pinned = addresses
.iter()
.map(|ip| match ip {
IpAddr::V4(v4) => v4.to_string(),
IpAddr::V6(v6) => format!("[{v6}]"),
})
.collect::<Vec<_>>()
.join(",");
Ok(PinnedUpstream {
url: url.to_string(),
curlopt_resolve: Some(format!("+{connect_host}:{port}:{pinned}")),
})
}
fn validate_addresses(&self, host: &str, addresses: &[IpAddr]) -> Result<()> {
if addresses.is_empty() || addresses.len() > MAX_PINNED_ADDRESSES {
anyhow::bail!(
"mirror upstream host must resolve to 1..={MAX_PINNED_ADDRESSES} addresses"
);
}
for &ip in addresses {
if !address_is_permitted(ip) {
anyhow::bail!("mirror upstream host {host:?} resolved to disallowed address {ip}");
}
}
Ok(())
}
}
fn normalize_policy_host(host: &str) -> String {
host.trim_end_matches('.').to_ascii_lowercase()
}
fn reject_always_local_hostname(host: &str) -> Result<()> {
if host.is_empty()
|| host == "localhost"
|| host.ends_with(".localhost")
|| host.ends_with(".local")
|| host.ends_with(".internal")
{
anyhow::bail!("mirror upstream host is local-only and cannot be fetched");
}
Ok(())
}
fn address_is_permitted(ip: IpAddr) -> bool {
crate::api::peers::mirror_upstream_ip_permitted(ip)
}
fn parse_git_version(output: &str) -> Option<(u64, u64)> {
let version = output.split_whitespace().find(|part| {
part.as_bytes()
.first()
.is_some_and(|byte| byte.is_ascii_digit())
})?;
let mut parts = version.split('.');
Some((parts.next()?.parse().ok()?, parts.next()?.parse().ok()?))
}
fn ensure_safe_git_runtime(git_bin: &str) -> Result<()> {
// Bounded runner like every other git spawn on this path: --version does
// no I/O of its own, but the child still takes a permit and the deadline.
let (status, stdout, stderr) = crate::git::visibility_pack::run_bounded_git_raw_isolated_https(
git_bin,
&["--version"],
&std::env::temp_dir(),
b"",
Instant::now() + Duration::from_secs(10),
)?;
if !status.success() {
anyhow::bail!(
"{git_bin} --version failed: {}",
String::from_utf8_lossy(&stderr)
.chars()
.take(256)
.collect::<String>()
);
}
let stdout = String::from_utf8_lossy(&stdout);
let version = parse_git_version(&stdout).context("parsing git version")?;
if version < MIN_CURL_OPT_RESOLVE_GIT {
anyhow::bail!(
"upstream mirroring requires Git 2.37+ for DNS pinning; found {}.{}",
version.0,
version.1
);
}
Ok(())
}
#[derive(Debug, Clone, PartialEq, Eq)]
struct FetchCommand {
config: Vec<(String, String)>,
args: Vec<String>,
}
fn fetch_command(upstream: &PinnedUpstream) -> FetchCommand {
let http_scope = |name: &str| format!("http.{}.{name}", upstream.url);
let credential_scope = |name: &str| format!("credential.{}.{name}", upstream.url);
let mut config = vec![
("http.proxy".to_string(), "".to_string()),
("http.followRedirects".to_string(), "false".to_string()),
// Reset any inherited multi-valued resolver entries before adding the
// address set validated for this exact fetch.
("http.curloptResolve".to_string(), "".to_string()),
// Never forward operator or repository-scoped HTTP credentials to an
// owner-selected forge. Empty values reset Git's multi-valued headers
// and disable cookie state for this invocation.
("http.extraHeader".to_string(), "".to_string()),
("http.cookieFile".to_string(), "".to_string()),
("http.saveCookies".to_string(), "false".to_string()),
("http.sslVerify".to_string(), "true".to_string()),
// URL-specific configuration outranks generic `http.*` values even
// when the generic value is command-scoped. Repeat every security
// control at the exact validated URL so a repository-local scoped
// setting cannot re-enable redirects/proxies/headers/cookies or disable
// TLS. Key/value environment entries avoid `git -c` splitting a valid
// URL path that contains `=`.
(http_scope("proxy"), "".to_string()),
(http_scope("followRedirects"), "false".to_string()),
(http_scope("curloptResolve"), "".to_string()),
(http_scope("extraHeader"), "".to_string()),
(http_scope("cookieFile"), "".to_string()),
(http_scope("saveCookies"), "false".to_string()),
(http_scope("sslVerify"), "true".to_string()),
];
if let Some(resolve) = &upstream.curlopt_resolve {
config.extend([
("http.curloptResolve".to_string(), resolve.clone()),
(http_scope("curloptResolve"), resolve.clone()),
]);
}
config.extend([
("credential.helper".to_string(), "".to_string()),
(credential_scope("helper"), "".to_string()),
("credential.interactive".to_string(), "false".to_string()),
("core.askPass".to_string(), "false".to_string()),
("fetch.recurseSubmodules".to_string(), "false".to_string()),
// Do not let a smart server or repository-local config introduce a
// second, unvalidated pack/bundle URL outside the pinned upstream.
("fetch.uriprotocols".to_string(), "".to_string()),
("fetch.bundleURI".to_string(), "".to_string()),
("protocol.allow".to_string(), "never".to_string()),
("protocol.https.allow".to_string(), "always".to_string()),
]);
let args = vec![
"fetch".to_string(),
"--atomic".to_string(),
"--force".to_string(),
"--prune".to_string(),
"--prune-tags".to_string(),
"--no-auto-maintenance".to_string(),
"--no-recurse-submodules".to_string(),
"--no-write-fetch-head".to_string(),
upstream.url.clone(),
"+refs/heads/*:refs/heads/*".to_string(),
"+refs/tags/*:refs/tags/*".to_string(),
];
FetchCommand { config, args }
}
fn reject_local_url_rewrite(
git_bin: &str,
repo_path: &Path,
upstream_url: &str,
deadline: Instant,
) -> Result<()> {
let args = ["ls-remote", "--get-url", upstream_url];
let (status, stdout, stderr) = crate::git::visibility_pack::run_bounded_git_raw_isolated_https(
git_bin, &args, repo_path, b"", deadline,
)?;
if !status.success() {
let stderr = String::from_utf8_lossy(&stderr);
anyhow::bail!(
"checking mirror upstream URL rewrite failed: {}",
stderr.chars().take(4_096).collect::<String>()
);
}
let expanded = std::str::from_utf8(&stdout)
.context("Git returned a non-UTF-8 expanded mirror upstream URL")?
.trim_end_matches(['\r', '\n']);
if expanded != upstream_url {
anyhow::bail!("repository-local Git config attempted to rewrite the mirror upstream URL");
}
Ok(())
}
fn reject_sensitive_local_config(git_bin: &str, repo_path: &Path, deadline: Instant) -> Result<()> {
let args = [
"config",
"--local",
"--includes",
"--get-regexp",
r"^(http|credential|url)\.",
];
let (status, stdout, stderr) = crate::git::visibility_pack::run_bounded_git_raw_isolated_https(
git_bin, &args, repo_path, b"", deadline,
)?;
match status.code() {
Some(1) if stdout.is_empty() => Ok(()),
Some(0) => anyhow::bail!(
"repository-local network or credential Git config is not allowed for upstream mirrors"
),
_ => {
let stderr = String::from_utf8_lossy(&stderr);
anyhow::bail!(
"checking repository-local Git config failed: {}",
stderr.chars().take(4_096).collect::<String>()
)
}
}
}
/// Budget for each preflight probe inside [`run_fetch`] (the URL-rewrite
/// check and the sensitive-local-config scan). Both run local git children
/// against the on-disk repo, so a short bound is generous; keeping them on
/// their own deadlines means `upstream_mirror_fetch_timeout_secs` covers
/// only the external pack transfer it documents.
const PREFLIGHT_PROBE_TIMEOUT: Duration = Duration::from_secs(30);
fn run_fetch(
git_bin: &str,
repo_path: &Path,
upstream: &PinnedUpstream,
timeout: Duration,
) -> Result<()> {
let command = fetch_command(upstream);
let args: Vec<&str> = command.args.iter().map(String::as_str).collect();
let config: Vec<(&str, &str)> = command
.config
.iter()
.map(|(key, value)| (key.as_str(), value.as_str()))
.collect();
let rewrite_deadline = Instant::now()
.checked_add(PREFLIGHT_PROBE_TIMEOUT)
.context("mirror URL-rewrite probe timeout is not representable")?;
reject_local_url_rewrite(git_bin, repo_path, &upstream.url, rewrite_deadline)?;
let config_deadline = Instant::now()
.checked_add(PREFLIGHT_PROBE_TIMEOUT)
.context("mirror local-config probe timeout is not representable")?;
reject_sensitive_local_config(git_bin, repo_path, config_deadline)?;
let deadline = Instant::now()
.checked_add(timeout)
.context("upstream mirror fetch timeout is not representable")?;
let (status, _stdout, stderr) =
crate::git::visibility_pack::run_bounded_git_raw_isolated_https_with_config(
git_bin, &args, repo_path, b"", deadline, &config,
)?;
if !status.success() {
let stderr = String::from_utf8_lossy(&stderr);
let stderr = stderr.chars().take(4_096).collect::<String>();
anyhow::bail!("git upstream fetch failed: {stderr}");
}
Ok(())
}
#[derive(Clone)]
struct Worker {
db: Arc<Db>,
config: Arc<Config>,
repo_store: RepoStore,
repo_write_leases: RepoWriteLeases,
git_write_semaphore: Arc<tokio::sync::Semaphore>,
policy: EgressPolicy,
git_bin: String,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum FetchOutcome {
Updated,
SkippedStale,
}
impl Worker {
async fn fetch_target(&self, target: InboundMirrorTarget) -> Result<FetchOutcome> {
let url = crate::db::validate_mirror_upstream_url(&target.upstream_url)?;
let upstream = self.policy.resolve(url).await?;
let repo_identity = repo_identity_key(&target.owner_did, &target.name);
let steal_after = Duration::from_secs(
self.config
.upstream_mirror_fetch_timeout_secs
.saturating_mul(2)
.saturating_add(60),
);
let _lease = self
.repo_write_leases
.acquire(&repo_identity, steal_after)
.await
.context("upstream mirror write-lease waiter cap reached")?;
let _write_permit = Arc::clone(&self.git_write_semaphore)
.acquire_owned()
.await
.context("upstream mirror write semaphore closed")?;
let acquire_timeout = Duration::from_secs(self.config.git_acquire_timeout_secs);
let guard = tokio::time::timeout(
acquire_timeout,
self.repo_store
.acquire_write(&target.owner_did, &target.name),
)
.await
.context("upstream mirror write-lock acquisition timed out")??;
let repo_path = guard.path().to_path_buf();
// A row can leave INBOUND state after selection. Recheck only after
// both the process-local lease and the cluster-wide advisory write lock
// are held. Every future transition path must take these same locks
// before changing authority, so no other node can commit a transition
// between this check and the fetch.
let current = match self.db.get_repo_mirror_state(&target.repo_id).await {
Ok(Some(current)) => current,
Ok(None) => {
let _ = guard.release(false).await;
anyhow::bail!("inbound mirror state disappeared before fetch");
}
Err(error) => {
let _ = guard.release(false).await;
return Err(error);
}
};
// Both stable shapes the CHECK admits: never transitioned, or a
// completed transition whose metadata is retained as audit. Only an
// in-flight transition makes the target stale.
let stable = (current.transition_id.is_none() && current.transition_phase.is_none())
|| (current.transition_id.is_some()
&& current.transition_phase == Some(MirrorTransitionPhase::Completed));
if current.status != MirrorStatus::Inbound
|| !stable
|| current.upstream_url != target.upstream_url
{
let _ = guard.release(false).await;
info!(repo_id = %target.repo_id, "upstream mirror state changed before fetch; skipping stale target");
return Ok(FetchOutcome::SkippedStale);
}
let git_bin = self.git_bin.clone();
let fetch_timeout = Duration::from_secs(self.config.upstream_mirror_fetch_timeout_secs);
let fetch = tokio::task::spawn_blocking(move || {
run_fetch(&git_bin, &repo_path, &upstream, fetch_timeout)
})
.await
.context("upstream mirror fetch task panicked")?;
let success = fetch.is_ok();
let outcome = guard.release(success).await;
// A refused publish (fenced or unknowable upload) must not report
// Updated; surface it as the failure it is before mapping the fetch.
outcome.into_result()?;
fetch.map(|()| FetchOutcome::Updated)
}
async fn scan_once(&self) {
let mut cursor: Option<String> = None;
loop {
let targets = match self
.db
.list_inbound_mirror_targets(
cursor.as_deref(),
self.config.upstream_mirror_page_size as i64,
)
.await
{
Ok(targets) => targets,
Err(error) => {
warn!(err = %error, "failed to list inbound mirror targets");
return;
}
};
if targets.is_empty() {
return;
}
let page_len = targets.len();
cursor = targets.last().map(|target| target.repo_id.clone());
for target in targets {
let repo_id = target.repo_id.clone();
let upstream_url = target.upstream_url.clone();
match self.fetch_target(target).await {
Ok(FetchOutcome::Updated) => {
info!(repo_id, upstream = %upstream_url, "upstream mirror fetch completed")
}
Ok(FetchOutcome::SkippedStale) => {}
Err(error) => {
warn!(repo_id, upstream = %upstream_url, err = %error, "upstream mirror fetch failed")
}
}
}
if page_len < self.config.upstream_mirror_page_size {
return;
}
}
}
}
/// Validate the opt-in runtime and spawn the scheduled worker. The worker runs
/// one scan immediately, sleeps only after a complete scan, and exits between
/// scans when graceful shutdown is signalled. An in-flight Git child is bounded
/// and reaped before the worker observes shutdown.
pub fn start(
db: Arc<Db>,
config: Arc<Config>,
repo_store: RepoStore,
repo_write_leases: RepoWriteLeases,
git_write_semaphore: Arc<tokio::sync::Semaphore>,
mut shutdown_rx: tokio::sync::watch::Receiver<bool>,
) -> Result<()> {
ensure_safe_git_runtime("git")?;
let policy = EgressPolicy;
let worker = Worker {
db,
config: Arc::clone(&config),
repo_store,
repo_write_leases,
git_write_semaphore,
policy,
git_bin: "git".to_string(),
};
tokio::spawn(async move {
info!(
interval_secs = config.upstream_mirror_interval_secs,
"upstream mirror worker started"
);
loop {
worker.scan_once().await;
tokio::select! {
_ = tokio::time::sleep(Duration::from_secs(config.upstream_mirror_interval_secs)) => {}
changed = shutdown_rx.changed() => {
if changed.is_err() || *shutdown_rx.borrow() {
info!("upstream mirror worker stopped");
return;
}
}
}
}
});
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use clap::Parser;
use std::path::PathBuf;
async fn test_worker(
pool: sqlx::PgPool,
git_bin: PathBuf,
page_size: usize,
) -> (Worker, Arc<Db>, tempfile::TempDir) {
let db = Arc::new(Db::for_testing(pool.clone()));
db.run_migrations().await.unwrap();
let repos = tempfile::tempdir().unwrap();
let mut config = Config::parse_from([
"gitlawb-node".to_string(),
"--upstream-mirror-enabled".to_string(),
"--enforce-owner-push".to_string(),
"--upstream-mirror-fetch-timeout-secs".to_string(),
"5".to_string(),
"--upstream-mirror-page-size".to_string(),
page_size.to_string(),
]);
config.validate().unwrap();
let config = Arc::new(config);
let worker = Worker {
db: Arc::clone(&db),
config,
repo_store: RepoStore::for_testing(repos.path().to_path_buf(), pool),
repo_write_leases: RepoWriteLeases::new(8),
git_write_semaphore: Arc::new(tokio::sync::Semaphore::new(4)),
policy: EgressPolicy,
git_bin: git_bin.to_string_lossy().into_owned(),
};
(worker, db, repos)
}
async fn add_inbound_mirror(
worker: &Worker,
db: &Db,
id: &str,
name: &str,
upstream_url: &str,
) -> InboundMirrorTarget {
let now = chrono::Utc::now();
let record = crate::db::RepoRecord {
id: id.to_string(),
name: name.to_string(),
owner_did: "did:key:z6MkMirrorWorker".to_string(),
description: None,
is_public: true,
default_branch: "main".to_string(),
created_at: now,
updated_at: now,
disk_path: format!("/test/{id}.git"),
forked_from: None,
machine_id: None,
};
db.create_repo(&record).await.unwrap();
worker
.repo_store
.init(&record.owner_did, &record.name)
.await
.unwrap();
let state = db
.configure_inbound_mirror(&record.id, upstream_url)
.await
.unwrap();
InboundMirrorTarget {
repo_id: record.id,
owner_did: record.owner_did,
name: record.name,
upstream_url: state.upstream_url,
}
}
#[cfg(unix)]
fn fake_git(repos: &tempfile::TempDir) -> (PathBuf, PathBuf) {
use std::os::unix::fs::PermissionsExt;
let git = repos.path().join("fake-git");
let calls = repos.path().join("fetch-calls");
let script = format!(
r#"#!/bin/sh
if [ "$1" = "ls-remote" ]; then
printf '%s\n' "$3"
exit 0
fi
if [ "$1" = "config" ]; then
# Match real Git when no sensitive repository-local keys are configured.
exit 1
fi
case "$*" in
*fail.git*) printf '%s\n' fail >> '{}'; exit 7 ;;
*) printf '%s\n' ok >> '{}'; exit 0 ;;
esac
"#,
calls.display(),
calls.display()
);
std::fs::write(&git, script).unwrap();
let mut permissions = std::fs::metadata(&git).unwrap().permissions();
permissions.set_mode(0o700);
std::fs::set_permissions(&git, permissions).unwrap();
(git, calls)
}
#[test]
fn git_version_gate_requires_curlopt_resolve_support() {
assert_eq!(parse_git_version("git version 2.37.0"), Some((2, 37)));
assert_eq!(
parse_git_version("git version 2.39.5 (Apple Git-154)"),
Some((2, 39))
);
assert_eq!(parse_git_version("not git"), None);
assert!((2, 36) < MIN_CURL_OPT_RESOLVE_GIT);
}
#[test]
fn every_resolved_address_must_be_public() {
let policy = EgressPolicy;
let public_v4: IpAddr = "8.8.8.8".parse().unwrap();
let private_v4: IpAddr = "10.2.3.4".parse().unwrap();
let private_v6: IpAddr = "fd00::20".parse().unwrap();
assert!(policy
.validate_addresses("public.example", &[public_v4])
.is_ok());
assert!(policy
.validate_addresses("private.example", &[private_v4, private_v6])
.is_err());
assert!(policy
.validate_addresses("mixed.example", &[public_v4, private_v4])
.is_err());
let too_many = (1..=MAX_PINNED_ADDRESSES + 1)
.map(|last| IpAddr::V4(std::net::Ipv4Addr::new(8, 8, 0, last as u8)))
.collect::<Vec<_>>();
assert!(policy
.validate_addresses("oversized.example", &too_many)
.is_err());
for never in [
"127.0.0.1",
"169.254.169.254",
"192.0.2.1",
"198.18.0.1",
"198.51.100.1",
"203.0.113.10",
"::1",
"::ffff:198.18.0.1",
"2002:c612:1::",
"64:ff9b::c612:1",
"100::1",
"100:0:0:1::1",
"2001::1",
"2001:2::1",
"2001:db8::1",
"fec0::1",
"fe80::1",
"3fff::1",
"5f00::1",
] {
let ip = never.parse().unwrap();
assert!(!address_is_permitted(ip), "{never} must stay denied");
}
}
#[test]
fn local_only_hostnames_are_always_rejected() {
for invalid in [
"",
"localhost",
"forge.localhost",
"forge.local",
"ghe.internal",
] {
assert!(
reject_always_local_hostname(invalid).is_err(),
"{invalid:?}"
);
}
assert!(reject_always_local_hostname("github.example").is_ok());
}
#[test]
fn fetch_command_pins_dns_and_only_updates_branches_and_tags() {
let upstream = PinnedUpstream {
url: "https://github.example/org/repo.git".to_string(),
curlopt_resolve: Some("+github.example:443:8.8.8.8,[2606:4700:4700::1111]".to_string()),
};
let command = fetch_command(&upstream);
let has_config = |key: &str, value: &str| {
command
.config
.contains(&(key.to_string(), value.to_string()))
};
assert!(has_config("http.followRedirects", "false"));
assert!(has_config("http.proxy", ""));
assert!(has_config("http.extraHeader", ""));
assert!(has_config("http.cookieFile", ""));
assert!(has_config("http.saveCookies", "false"));
assert!(has_config("http.sslVerify", "true"));
assert!(has_config(
"http.curloptResolve",
"+github.example:443:8.8.8.8,[2606:4700:4700::1111]"
));
assert!(has_config(
"http.https://github.example/org/repo.git.followRedirects",
"false"
));
assert!(has_config(
"http.https://github.example/org/repo.git.proxy",
""
));
assert!(has_config(
"http.https://github.example/org/repo.git.extraHeader",
""
));
assert!(has_config(
"http.https://github.example/org/repo.git.curloptResolve",
"+github.example:443:8.8.8.8,[2606:4700:4700::1111]"
));
assert!(has_config(
"credential.https://github.example/org/repo.git.helper",
""
));
assert!(has_config("protocol.allow", "never"));
assert!(has_config("protocol.https.allow", "always"));
assert!(has_config("fetch.uriprotocols", ""));
assert!(has_config("fetch.bundleURI", ""));
assert!(!command
.config
.iter()
.any(|(key, _)| key.ends_with("sslCert") || key.ends_with("sslKey")));
assert!(command.args.contains(&"--atomic".to_string()));
assert!(command
.args
.contains(&"+refs/heads/*:refs/heads/*".to_string()));
assert!(command
.args
.contains(&"+refs/tags/*:refs/tags/*".to_string()));
assert!(!command.args.iter().any(|arg| arg == "+refs/*:refs/*"));
}
/// Spawn `git` inside a fixture repo with inherited repository-location
/// and config environment scrubbed, so a test run inside an outer Git
/// checkout or custom config can never leak into the fixture.
fn fixture_git(dir: &std::path::Path) -> std::process::Command {
let mut cmd = std::process::Command::new("git");
cmd.current_dir(dir);
for var in [
"GIT_DIR",
"GIT_WORK_TREE",
"GIT_CONFIG",
"GIT_CONFIG_GLOBAL",
"GIT_CONFIG_SYSTEM",
"GIT_CONFIG_NOSYSTEM",
"GIT_CONFIG_COUNT",
"GIT_EXEC_PATH",
"GIT_SSH",
"GIT_SSH_COMMAND",
"GIT_ASKPASS",
"SSH_ASKPASS",
] {
cmd.env_remove(var);
}
cmd
}
#[test]
fn repository_local_instead_of_rewrite_is_rejected() {
let repo = tempfile::tempdir().unwrap();
let init = fixture_git(repo.path())
.args(["init", "--bare"])
.output()
.unwrap();
assert!(init.status.success());
let config = fixture_git(repo.path())
.args([
"config",
"url.https://evil.example/.insteadOf",
"https://github.example/",
])
.output()
.unwrap();
assert!(config.status.success());
let error = reject_local_url_rewrite(
"git",
repo.path(),
"https://github.example/org/repo.git",
Instant::now() + Duration::from_secs(5),
)
.unwrap_err();
assert!(error.to_string().contains("attempted to rewrite"));
}
#[test]
fn repository_local_network_and_credential_config_is_rejected() {
let repo = tempfile::tempdir().unwrap();
let init = fixture_git(repo.path())
.args(["init", "--bare"])
.output()
.unwrap();
assert!(init.status.success());
reject_sensitive_local_config("git", repo.path(), Instant::now() + Duration::from_secs(5))
.unwrap();
let config = fixture_git(repo.path())
.args(["config", "http.sslCert", "/tmp/untrusted-client.pem"])
.output()
.unwrap();
assert!(config.status.success());
let error = reject_sensitive_local_config(
"git",
repo.path(),
Instant::now() + Duration::from_secs(5),
)
.unwrap_err();
assert!(error.to_string().contains("network or credential"));
}
#[cfg(unix)]
#[test]
fn hardened_real_git_reaches_https_transport_for_equals_path() {
let repo = tempfile::tempdir().unwrap();
let init = fixture_git(repo.path())
.args(["init", "--bare"])
.output()
.unwrap();
assert!(init.status.success());
// This is intentionally a loopback test seam around `run_fetch`, below
// the egress-policy layer. Reaching the listener proves the generated
// Git config neither dies on an empty client certificate nor splits the
// valid `=` path at the command-config boundary. The listener closes
// immediately, so the expected final result is a TLS transport error.
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let port = listener.local_addr().unwrap().port();
listener.set_nonblocking(true).unwrap();
let accepted = std::thread::spawn(move || {
let deadline = Instant::now() + Duration::from_secs(3);
while Instant::now() < deadline {
match listener.accept() {
Ok((_stream, _address)) => return true,
Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => {
std::thread::sleep(Duration::from_millis(10));
}
Err(error) => panic!("accepting real-Git transport probe: {error}"),
}
}
false
});
let upstream = PinnedUpstream {
url: format!("https://127.0.0.1:{port}/org/repo=mirror.git"),
curlopt_resolve: None,
};
let error = run_fetch("git", repo.path(), &upstream, Duration::from_secs(5)).unwrap_err();
assert!(
accepted.join().unwrap(),
"real Git must reach the HTTPS transport; got {error:#}"
);
let error = error.to_string();
assert!(!error.contains("credential missing host field"));
assert!(!error.contains("invalid key"));
}
#[tokio::test]
async fn literal_addresses_are_checked_without_dns() {
let policy = EgressPolicy;
let allowed = policy
.resolve(reqwest::Url::parse("https://8.8.8.8/org/repo.git").unwrap())
.await
.unwrap();
assert_eq!(allowed.curlopt_resolve, None);
for denied in [
"127.0.0.1",
"10.2.3.4",
"169.254.169.254",
"192.0.2.1",
"198.18.0.1",
"203.0.113.10",
"[::1]",
"[100::1]",
"[100:0:0:1::1]",
"[2001::1]",
"[2001:db8::1]",
"[fec0::1]",
"[fd00::20]",
"[3fff::1]",
"[5f00::1]",
] {
let url = reqwest::Url::parse(&format!("https://{denied}/org/repo.git")).unwrap();
assert!(
policy.resolve(url).await.is_err(),
"{denied} must stay denied"
);
}
}
#[sqlx::test]
async fn stale_target_is_rechecked_under_the_write_locks(pool: sqlx::PgPool) {
let missing_git = PathBuf::from("/definitely/missing/git");
let (worker, db, _repos) = test_worker(pool, missing_git, 1).await;
let target = add_inbound_mirror(
&worker,
&db,
"00000000-0000-0000-0000-000000000001",
"stale",
"https://8.8.8.8/org/stale.git",
)
.await;
sqlx::query(
"UPDATE repos
SET mirror_status = 'outbound'
WHERE id = $1",
)
.bind(&target.repo_id)
.execute(db.pool())
.await
.unwrap();
let outcome = worker.fetch_target(target).await.unwrap();
assert_eq!(outcome, FetchOutcome::SkippedStale);
assert!(worker.repo_write_leases.is_empty());
}
#[cfg(unix)]
#[sqlx::test]
async fn scan_pages_past_one_failure_and_fetches_the_next_repo(pool: sqlx::PgPool) {
let bootstrap = tempfile::tempdir().unwrap();
let (git, calls) = fake_git(&bootstrap);
let (worker, db, _repos) = test_worker(pool, git, 1).await;
add_inbound_mirror(
&worker,
&db,
"00000000-0000-0000-0000-000000000001",
"fails",
"https://8.8.8.8/org/fail.git",
)
.await;
add_inbound_mirror(
&worker,
&db,
"00000000-0000-0000-0000-000000000002",
"succeeds",
"https://8.8.8.8/org/ok.git",
)
.await;
worker.scan_once().await;
let calls = std::fs::read_to_string(calls).unwrap();
assert_eq!(calls.lines().collect::<Vec<_>>(), ["fail", "ok"]);
assert!(worker.repo_write_leases.is_empty());
}
}
+11 -4
View File
@@ -108,11 +108,18 @@ pub async fn run(args: QuickstartArgs) -> Result<()> {
"saved_at": chrono::Utc::now().to_rfc3339(),
});
std::fs::write(&ucan_path, serde_json::to_string_pretty(&record)?)?;
let trust = payload["trust_score"].as_f64().unwrap_or(0.0);
println!(" ✓ Registered successfully");
println!(" Trust score: {trust:.2}");
println!(" UCAN saved to {}", ucan_path.display());
} else {
// A 2xx without a usable ucan is not a registration; the
// response may not even be JSON. Saying success here would
// also leave a stale ucan.json from another node in place.
println!(" ✗ Registration returned no UCAN (unexpected response body)");
// No retry hint here: a `gl register` instruction without
// this invocation's --dir would load a different identity.
}
let trust = payload["trust_score"].as_f64().unwrap_or(0.0);
println!(" ✓ Registered successfully");
println!(" Trust score: {trust:.2}");
println!(" UCAN saved to {}", ucan_path.display());
println!();
}
Ok(resp) => {