Compare commits

..
Author SHA1 Message Date
Kevin Codex e9da517fde Merge pull request #240 from Gitlawb/release-please--branches--main--components--gitlawb-node
chore(main): release 0.7.0
2026-07-23 08:27:35 +08:00
Kevin Codex 9ca94c4d47 Merge pull request #241 from Gitlawb/ci/gitlawb-core-dep-purity
ci(core): add dependency-purity allowlist gate for gitlawb-core
2026-07-23 07:06:43 +08:00
beardthelion c0ad500b55 ci(core): allowlist curve25519-dalek-derive
Proc-macro helper for curve25519-dalek (already allowlisted). Newer cargo
versions list it under `cargo tree --edges normal` while older ones do not;
the allowlist is a superset so it passes on both. Caught by core-deps-purity
on the first CI run of this branch.
2026-07-22 12:36:25 -05:00
beardthelion e9d6aadd58 ci(core): add dependency-purity allowlist gate for gitlawb-core
gitlawb-core is embedded by gl, git-remote-gitlawb, and the node daemon,
so it must stay lean. Add a CI gate that fails if its normal (non-dev,
non-build) transitive dependency tree gains any crate not on an explicit
allowlist.

The allowlist is exhaustive-by-construction: a new heavy dependency reds
CI whether or not anyone thought to ban it, which a denylist cannot do.
The checker resolves like the build/test jobs (no --locked, since the
committed lockfile can lag the manifests) and snapshots/restores
Cargo.lock so a local run leaves no working-tree side effects.

- ci/gitlawb-core-allowed-deps.txt: the 83 current normal deps, plus the
  regeneration command and rationale in the header
- scripts/check-gitlawb-core-deps.sh: recompute and diff; hard-fail on a
  non-allowlisted crate, note-only on a stale (removed) allowlist entry
- pr-checks.yml: new hard-fail core-deps-purity job, matching the pinned
  action SHAs and style of the sibling jobs
2026-07-22 12:04:34 -05:00
Kevin Codex b484a24955 Merge pull request #237 from Gitlawb/fix/issue-228-windows-ci
ci: add a non-blocking Windows test lane for the shipped CLIs
2026-07-22 23:28:53 +08:00
github-actions[bot] 32006b26a2 chore(main): release 0.7.0 2026-07-22 15:28:32 +00:00
Kevin Codex b75aa2f1da Merge pull request #236 from Gitlawb/fix/gl-acceptance-findings
Fix/gl acceptance findings
2026-07-22 23:28:09 +08:00
Kevin Codex c681af70db fix(gl): anchor cert --verify to a trusted issuer; pin canonical payload form
Settles the review's design question: a valid signature alone proves
internal consistency — the payload was signed by whatever key the
certificate itself names — so a hostile source could self-sign and
pass. --verify now also requires the issuing node_did to match a
trusted anchor: --expect-node when given, else the DID of the node
being queried; unanchorable (node info unreachable, no --expect-node)
fails closed. The displayed verdict wording says what it proves.

Also from review:
- A frozen-literal test pins gl's payload serialization to the node's
  canonical byte form, so serialization drift (new field, a
  preserve_order feature unifying across the workspace) fails a test
  instead of silently rendering every real certificate INVALID.
- The printed unalias remediation uses the idempotent-safe
  'unalias gl 2>/dev/null' form everywhere (doctor, install.sh, brew
  caveats), matching what doctor already suggested.
- Acknowledged, deliberately left: bash-side alias detection and
  127/8-range loopback matching (noted as out of scope for a
  warning-level heuristic).
2026-07-22 21:43:44 +08:00
Kevin Codex 3fa38dce4e fix(gl): address review findings — non-fatal DID check, exact-host loopback, detached HEAD, multi-URL remotes, command-aware unalias
- cert show: the node-DID comparison is contextual; a node-info fetch
  error now prints a note instead of failing the command after the
  certificate and verdict were already shown.
- doctor: loopback detection parses the URL host exactly — substring
  matching misread localhost.example and localhost-in-path URLs.
- init: a detached HEAD gets its own hint (switch to a branch first)
  instead of a guessed 'git push gitlawb main'.
- status: every fetch AND push URL of each remote is inspected — a
  gitlawb:// push URL behind an https fetch URL now counts.
- doctor/install.sh: unalias detection is command-aware — comments,
  'unalias global', and gl-only-in-a-trailing-comment no longer
  suppress the warning (BSD-grep-compatible pattern; ordering vs
  oh-my-zsh load documented as accepted slack). Eight new test cases.
2026-07-22 21:17:38 +08:00
t c71bba5cd1 ci: add a non-blocking Windows test lane for the shipped CLIs (#228)
We ship `gl` and `git-remote-gitlawb` to Windows users via release.yml but run
no Windows tests: pr-checks.yml is ubuntu-only, and the single windows-latest in
the repo is the release build step, which never runs `cargo test`. So changes to
the shipped Windows CLIs (e.g. the multi-round fetch fix in #192) reach Windows
users with zero test runs on Windows.

Add a `test-windows` job that runs `cargo test -p gl -p git-remote-gitlawb` on
windows-latest with `continue-on-error: true`, mirroring the informational beta
lane: a Windows-only failure is visible on the PR but does not block merge. The
two crates are the client CLIs and depend on no database, so the job needs no
Postgres service (service containers do not run on windows-latest regardless);
git is preinstalled on the runner for the real_git_fetch harness.

This makes the #[cfg(windows)] regressions from #192 load-bearing instead of
never-run. Un-gating the currently #[cfg(unix)] harness tests is the deferred
follow-on in #228.
2026-07-22 08:01:06 -05:00
Kevin Codex e3df51d76c fix: warn about the oh-my-zsh gl alias at install time
install.sh detects the common shadowing setup (omz git plugin, no
unalias in ~/.zshrc) and prints the fix right after installing; the
generated Homebrew formula gains a caveats block saying the same, so
brew install/upgrade surfaces it too. Catching this at install time
beats letting the first gl invocation print git's error.
2026-07-22 20:51:23 +08:00
Kevin Codex 0765c42fef feat(gl): doctor warns when a shell alias shadows the gl binary
oh-my-zsh's default git plugin ships gl='git pull', silently shadowing
gl in every interactive zsh — the symptom is the baffling 'fatal: not a
git repository'. Aliases beat PATH, so no installer can fix it; doctor
now detects the setup (explicit alias gl= in ~/.zshrc, or the omz git
plugin without an unalias) and prescribes the one-line fix. Best-effort
heuristic, warning-level, covered by four unit tests.
2026-07-22 20:51:23 +08:00
Kevin Codex 94c61f7942 fix(gl): doctor treats a reachable local node as configuration, not failure
A localhost GITLAWB_NODE red-flagged every self-hosted and dev-harness
setup even when the node was healthy; the connectivity check already
fails loudly when the configured node is unreachable. The env check now
passes with an explanatory note.
2026-07-22 20:37:33 +08:00
Kevin Codex b0a2ed8308 feat(gl): actually verify certificate signatures in gl cert show
show printed the signing payload and told the user to verify offline
themselves. It now rebuilds the node's canonical payload (field order
mirrors gitlawb-node cert issuance), derives the Ed25519 key from the
certificate's node DID, verifies, and prints a VALID/INVALID verdict;
--verify makes the exit code reflect it for scripting. Round-trip +
tamper + malformed-signature unit tests included.
2026-07-22 20:37:33 +08:00
Kevin Codex cb42a4830d fix(gl): make gl init's push guidance match the repo's actual state
- git init -b main (plain-init fallback for git <2.28) so a fresh repo
  can't land on master while every hint says main.
- The closing hint checks HEAD: with no commits it prints the
  add/commit step first — a bare 'git push gitlawb main' at that point
  fails with 'src refspec main does not match any', the exact trap the
  acceptance test walked into. The current branch name is used in the
  hint either way.
2026-07-22 20:37:33 +08:00
Kevin Codex 2d88937fba fix(gl): detect gitlawb remotes beyond origin in gl status
gl init names its remote 'gitlawb', but detect_gitlawb_remote() only
ever read origin — so gl status reported 'not in a gitlawb repo' in the
very repo gl init had just set up. Scan all remotes, preferring origin
(clones) then gitlawb, then any other gitlawb:// URL. (#231 acceptance
test finding)
2026-07-22 20:37:33 +08:00
Kevin Codex e14c2d3f3f Merge pull request #230 from Gitlawb/ci/merge-race-guards
ci: guard against semantic merge races breaking main
2026-07-22 16:35:22 +08:00
Kevin Codex 4465beb0c2 ci: make the red-main alert a state tracker, not an event echo
Address review: the alerter reflected CI events rather than main's
current state, which could both cry wolf and stay silent.

- Freshness gate: a run only opens or closes anything when its commit is
  still main's tip — re-running an old red run after a fix has landed no
  longer opens a false stop-ship issue (and a stale green can't close a
  current one).
- Trigger covers timed_out and startup_failure — both are red mains that
  previously alerted nothing. cancelled stays excluded as a human abort.
- Recovery: a green run on the tip comments and closes open ci-main-red
  issues, turning the alert into a current-state signal.
- queue: max — the single-slot pending queue let a later completion
  displace a pending red alert under burst merges.
- Issue lookup filters out pull requests carrying the label.
- github-script converged on v8.0.0 (SHA verified against the tag) in
  pr-triage too.
2026-07-22 16:20:15 +08:00
Kevin Codex 82efd60229 ci: serialize main-red-alert runs to prevent duplicate issues
Two near-simultaneous red-main events could both pass the open-issue
lookup and each create a ci-main-red issue (or 422 racing createLabel).
A workflow concurrency group with cancel-in-progress: false serializes
them; the default single-slot pending queue is enough since a superseded
alert's state is re-read by the newest run anyway.
2026-07-22 16:18:38 +08:00
Kevin Codex e8f193a5df ci: guard against semantic merge races breaking main
Two individually-green PRs can break main in combination when the second
merges on checks that ran against a stale base (#113 + #145 compiled fine
apart, but #145 added a field to ReceivedRefUpdate that #113's new test
initializer never set — E0063 on merged main, fixed by #229).

- pr-checks: add merge_group trigger so the suite can gate a GitHub merge
  queue, which re-validates every PR against the true merged result.
- main-red-alert: new workflow that opens/updates a labeled issue whenever
  a push-to-main PR Checks run fails, so a red main is loud instead of
  silent (release-please PRs inherit main breakage, as #182 showed).
2026-07-22 16:18:38 +08:00
Kevin Codex 07d76ba556 Merge pull request #235 from Gitlawb/ci/docker-native-arm
ci: build release Docker images natively per-arch, merge with a manifest
2026-07-22 15:37:45 +08:00
Kevin Codex 05e434d906 ci: lowercase the ghcr image reference for the digest push
buildx's --output name= does no lowercasing (unlike metadata-action),
so ghcr.io/Gitlawb/node is an invalid reference and the per-arch digest
push would fail outright. Compute the lowercase image once in the
resolve step and reuse it, matching the manifest job.
2026-07-22 15:23:42 +08:00
Kevin Codex 7593540f62 ci: build release Docker images natively per-arch, merge with a manifest
The v0.6.0 release run's multi-arch Docker build wedged for 3+ hours
emulating the arm64 Rust release build under QEMU, and — because the
workflow concurrency group is cancel-in-progress:false — every queued
release run sat blocked behind it.

- docker is now a matrix: amd64 on ubuntu-latest, arm64 on
  ubuntu-24.04-arm — both native, mirroring release-binaries. Each leg
  smoke-tests the image (--version must report the released version)
  and pushes by digest only.
- docker-manifest stitches the digests into the :X.Y.Z, :X.Y and
  :latest tags with buildx imagetools, so a half-finished matrix can
  never publish a partially-tagged image.
- workflow_dispatch grows docker_backfill_tag (and npm_backfill_tag
  becomes optional): each channel backfills independently for an
  already-cut release — needed to publish the missing v0.6.0 image.
2026-07-22 13:13:37 +08:00
Kevin Codex ccc2cf46f2 Merge pull request #233 from Gitlawb/ci/npm-trusted-publishing
ci: switch npm publishing to trusted publishing (OIDC) + manual backfill
2026-07-22 12:28:33 +08:00
Kevin Codex 1fddc81b7c ci: skip release-please on manual npm backfill dispatch
A backfill run must be a narrow npm-only operation. Without an event
guard, workflow_dispatch also ran release-please with contents+PR write
permissions and could create or advance a release PR as a side effect.
npm-publish is unaffected: its !cancelled() condition tolerates the
skipped need, and all other jobs already gate on release_created.
2026-07-22 12:10:25 +08:00
Kevin Codex 62d73c4165 ci: switch npm publishing to trusted publishing (OIDC) + manual backfill
The npm-publish job silently skipped every release since 0.4.x because
NPM_TOKEN was never configured — the guard made the job green while
@gitlawb/gl sat at 0.3.7 on the registry. Replace token auth entirely:

- npm Trusted Publishing: the job's GitHub OIDC identity (id-token: write)
  is exchanged with the registry; no long-lived secret to configure, leak,
  or rotate. Provenance is attested automatically. All five @gitlawb
  packages exist on the registry now, so OIDC no longer has the
  first-publish bootstrap problem that forced token auth in #127.
- Fail loud: the secret guard is gone. If a package's trusted publisher
  isn't configured on npmjs.com, the publish errors visibly instead of
  green-skipping.
- setup-node 24 + npm >=11.5.1 (OIDC exchange support; older npm falls
  back to absent token auth with a confusing error).
- workflow_dispatch backfill: publish an already-cut GitHub release
  (e.g. v0.6.0) to npm by tag — binaries come from the existing release
  assets, so only the npm-publish job runs.
2026-07-22 11:50:44 +08:00
Kevin Codex 2f622ae5fa Merge pull request #182 from Gitlawb/release-please--branches--main--components--gitlawb-node
chore(main): release 0.6.0
2026-07-22 09:32:24 +08:00
github-actions[bot] f9506b466e chore(main): release 0.6.0 2026-07-22 09:08:39 +08:00
Kevin Codex d5ec4695f4 test(node): add missing owner_did field to fails-closed seed update (#229)
PRs #113 and #145 raced: #145 added owner_did to ReceivedRefUpdate and
updated its own call sites, but the seed initializer in
list_repo_events_fails_closed_when_repo_lookup_errors (added by #113)
was never updated, so main fails to compile test targets (E0063).
This breaks clippy --all-targets, both test jobs, and MSRV on CI,
including the release PR #182.

owner_did stays None here on purpose: the test exercises the legacy
None-branch slug fallback.

Co-authored-by: Kevin Codex <[email protected]>
2026-07-22 09:03:07 +08:00
euxaristia 98d728c325 fix(gl): point doctor's version check at Gitlawb/node, not the frozen Gitlawb/releases repo (#198)
* fix(gl): point doctor's version check at Gitlawb/node, not the frozen Gitlawb/releases repo

check_version queried https://api.github.com/repos/gitlawb/releases/releases/latest.
Gitlawb/releases is a separate repo that stopped receiving tags after v0.3.8
(2026-03-30). Releases have shipped directly on Gitlawb/node since v0.4.0, via
release-please, and install.sh already treats Gitlawb/node as the canonical
release source. The old query target meant gl doctor's version check could
never again detect a real update.

Points the check at Gitlawb/node instead, matching install.sh's own default,
and makes the GitHub API base injectable so the fix is covered by a test
against a mocked server rather than the real GitHub API.

Closes #197

* fix(gl): check HTTP status before parsing the release response; strengthen up-to-date test

Addresses CodeRabbit review on #198:
- check_version now checks resp.status().is_success() before parsing the body,
  so a GitHub rate-limit or other API error surfaces as its actual HTTP status
  instead of a generic "could not parse latest tag".
- test_check_version_up_to_date now also asserts the detail message, so the
  test can't silently pass on an unmatched mock request (mockito's fallback for
  an unhandled request is a 501, which check_version also maps to Ok).

* test(gl): add unit test for non-2xx check_version response

* fix(gl): remove trailing blank line at EOF in doctor.rs

cargo fmt --all -- --check fails on the extra terminal blank line, and the
fmt + clippy workflow runs that check as a required gate.
2026-07-22 08:33:11 +08:00
euxaristia b6877a6598 fix(core): honor wildcard capabilities in Ucan::can, matching is_attenuated_by (#200)
* fix(core): honor wildcard capabilities in Ucan::can, matching is_attenuated_by

Ucan::can did strict equality on both resource and action, so a validly
delegated wildcard capability (with: "*", or can: "*" / "repo/admin") could
never match a concrete runtime request even though is_attenuated_by treats
those same values as wildcards during delegation-chain verification.

Not a security issue: the mismatch only produces false negatives (denying a
capability the delegation chain actually grants), never false positives.
Dormant today since no production code path calls can() yet — auth's
caller_authorized_to_push is pure did_matches(), with the UCAN-capability
check noted as future (Phase 2) work.

Makes can() mirror is_attenuated_by's semantics: the stored capability plays
the "parent" role, so with == "*" matches any resource and can == "*" or
"repo/admin" matches any action.

Closes #199

* test: add can_honors_action_wildcard unit test for Ucan::can
2026-07-22 08:32:29 +08:00
beardthelionandt b07d41ea97 fix: strip Cf bidi/format controls in terminal-bound error sanitizers (#183) (#188)
* feat(core): add shared is_bidi_format predicate for terminal sanitizers (#183)

Terminal-bound error sanitizers must strip Cf bidi/format controls as well as Cc control bytes (INV-6), or a right-to-left override in attacker-influenceable bytes can visually reorder the displayed line (Trojan Source, CWE-451). gl already did this with a local is_bidi_format; two sibling sanitizers shipped with only the Cc half (#183). Centralize the reordering-subset predicate here so all consumers reuse one audited definition instead of re-deriving the set. Deliberately excludes legitimate Cf chars (ZWJ, soft hyphen).

* refactor(gl): consume shared is_bidi_format from gitlawb-core (#183)

Replace gl's local is_bidi_format copy with the centralized gitlawb-core predicate so exactly one definition exists in the workspace and the three terminal sanitizers cannot drift apart. Behavior-preserving; the existing strip test is unchanged and a must-not-over-strip guard (ASCII + Arabic U+0627 + ZWJ U+200D survive) pins the reordering-only scope.

* fix(git-remote): strip Cf bidi controls from terminal-bound error bodies (#183)

safe_error_body_excerpt filtered only Cc control bytes, so a Cf bidi override in a hostile or MITM'd node's HTTP error body reached the operator's terminal and could visually reorder the line (Trojan Source, CWE-451). Drop the reordering Cf subset via the shared gitlawb-core::sanitize::is_bidi_format, without setting pending_space so stripped chars leave no phantom gap. Covered per bidi class plus a transport-path proof through handle_connect, RED observed before the fix.

* fix(icaptcha): strip Cf bidi controls from terminal-bound error excerpts (#183)

sanitize_excerpt filtered only Cc, so a Cf bidi override in a node-advertised iCaptcha service's reason/body reached the terminal on the bail! path (Trojan Source, CWE-451). Drop the reordering Cf subset via the shared gitlawb-core predicate, without pending_space. The existing sanitize_strips_control_chars_and_escapes test asserted only is_control() and so was vacuous for this class (why the gap shipped); a dedicated per-class bidi test now covers it, RED observed before the fix.

* style: apply rustfmt to #183 sanitizer additions

* test(#183): pin a leading-position bidi char in the strip tests

The per-class strip tests placed bidi chars only mid-string. A bidi char at index 0 exercises the empty-excerpt/out path (where the pending_space guard behaves differently), confirming a leading override still yields no phantom leading space. Traced correct; now asserted.

* test(review): pin that bidi chars don't consume the length cap (#183)

Correctness, adversarial, and testing reviewers all flagged that no test locks in the drop-before-count ordering: a dropped bidi char must not consume MAX_ERR_CHARS budget. Behavior verified correct by execution during review; this test guards it against regression (interleaved bidi across the cap still yields exactly MAX_ERR_CHARS visible chars, no phantom spaces).

---------

Co-authored-by: t <t@t>
2026-07-22 08:29:49 +08:00
beardthelionandt c7427e943d test(icaptcha): assert obtain_proof consumes the mocked service (#184)
* test(icaptcha): assert obtain_proof consumes the mocked service

Add an integration test that runs obtain_proof against a mockito server and
asserts both /v1/challenge and /v1/answer were consumed, so the flow is proven
to call the service rather than short-circuit or make a live call. Adds mockito
as a dev-dependency.

* test(icaptcha): block non-mock egress so the no-live-call guard is real (#184)

The mock-consumed assertions only prove the two mock endpoints were hit; they
cannot see a request that also leaks the DID, answer, or API key to DEFAULT_URL
or any other origin, so the test's stated "no live call" guarantee was untested.

Blackhole every non-loopback destination for the flow: NO_PROXY lets the
loopback mock through while ALL_PROXY routes any other host to a closed port, so
the client contacting anything but the injected mock fails the request. Verified
load-bearing: pointing cfg.url at DEFAULT_URL makes obtain_proof fail with a
proxy tunnel error (RED), and the loopback-mock path still passes (GREEN).

* test(icaptcha): blackhole scheme-specific proxy vars so the no-live-call guard covers https

The mock-consumed guard blackholed non-loopback egress with ALL_PROXY only, but
reqwest honors HTTPS_PROXY/https_proxy (and the http equivalents) OVER ALL_PROXY
for https URLs. On a runner whose environment has a working HTTPS_PROXY, a
regression could still leak the DID/answer/API key to the https DEFAULT_URL while
both loopback mock calls are consumed and the test stayed green. Override the
scheme-specific proxy vars (both cases) to the closed port too; NO_PROXY keeps the
loopback mock direct. Single #[test] per process, so the override needs no restore.

Verified: a throwaway repro showed reqwest routing an https request to HTTPS_PROXY
past the ALL_PROXY blackhole (the gap); with the fix the override sends that egress
to the closed port even when a working HTTPS_PROXY is pre-set; the guard still goes
RED when cfg.url points off-loopback.

---------

Co-authored-by: t <t@t>
2026-07-22 08:25:53 +08:00
GravireiandGravirei 69a62e51c2 fix(node): carry full owner DID on ref-update wire event (#144) (#145)
* fix(node): carry full owner DID on ref-update wire event (#144)

* test(node): add ref-update owner_did round-trip, DB, and API tests (#144)

* fix(node): move owner_did migration to v10, add upgrade-path test

* fix(node): add owner_did to GraphQL, broadcast, and REST feed surfaces (#144)

* fix(node): address review feedback — drop ipfs regression, fix doc, improve migration test, use local owner_did

* fix(node): fix migration v10 test to actually start from v9 schema; complete owner_did API assertion

* fix(node): show local owner on global feed, assert nullable in migration test, add owner_did echo assertions

* fix(node): prefer stored wire owner_did over lossy slug-local resolution; mirror projection in GraphQL

* fix: address review findings — restore iCaptcha PoW/origin matching, fix feed owner_did projections

* fix: match icaptcha-client Cargo.toml ordering with upstream/main

sha2 was at line 16 in our branch but at line 18 in upstream/main.
The 3-way merge in CI's pull/145/merge checkout kept both lines,
causing a 'duplicate key' error. Moving sha2 after tracing matches
upstream/main exactly, so the merge commit will have a single entry.

* fix(node): bind wire owner_did to local repo before surfacing as trusted

The peer-supplied owner_did (HTTP notify + gossipsub receive) and the
repo slug arrive over untrusted paths, yet REST and GraphQL projected the
wire value as trusted feed ownership.

- P1: add Db::resolve_ref_update_owner_did so a wire owner_did is echoed
  only when it matches the canonical owner of the actual local repo at that
  slug; otherwise it is dropped (returns null). The canonical DID is
  returned, never the raw wire bytes.
- P3: legacy owner_did = None rows are now attributed via an exact,
  normalized, unique local match (get_repo), not the prefix-tolerant drop
  gate, so a cross-method slug collision cannot synthesize the wrong owner.
- Unify REST (global + per-repo gossip) and GraphQL on the shared resolver
  so the two surfaces cannot drift. Add tests for forged-wire drop and
  exact-legacy attribution.

* style(node): apply cargo fmt to owner_did resolver

* fix: batch owner_did resolution, propagate errors, preserve full DID for mirrors

P2 batch: replace per-row join_all with resolve_ref_update_owner_dids
that issues at most one get_repo query per distinct local repo in the
page rather than one per event row. Applies to both REST and GraphQL
surfaces.

P2 error: REST handler now propagates DB errors (rather than silently
coercing them to null) by calling the batch method with ?.

P2 mirror: when the validated wire DID carries a full method prefix
(e.g. did:key:z...) and the matching repo is a mirror storing only
the bare key, return the full wire DID instead of the bare key to
preserve the DID-method contract.

P1/P3 already addressed in fb553e6.

* fix: rebase onto current main and use single batch SQL for owner DID resolution

Rebased from stale 3ec8cbf base onto current upstream/main (ad7c2b2),
dropping the already-merged iCaptcha work from the diff. The PR diff
is now only the 11 owner-DID files.

The batch owner-DID resolver now issues a single set-based SQL query
with one OR condition per distinct slug, replacing the loop of per-key
get_repo calls. A 200-row page of 200 distinct remote slugs now costs
1 SQL round-trip instead of 200.

P1/P3 already addressed in c64c082. P2 mirror, error propagation,
and the earlier batch refactor preserved in the rebased commits.

---------

Co-authored-by: Gravirei <[email protected]>
2026-07-22 08:24:48 +08:00
beardthelionandt ee8d7c03b2 fix(node): gate /hooks and sibling read surfaces leaking private repo metadata (#94) (#113)
* fix(node): owner-gate list_webhooks to stop leaking webhook URLs (#94)

list_webhooks sat on the optional_signature read group with no auth or
ownership check, so any unauthenticated caller who knew a repo's owner/name
could enumerate its webhooks (target URL, created_by_did, events; only the
secret was redacted).

Gate it read-visibility-then-owner, mirroring create_webhook/delete_webhook:
headerless -> 401, non-reader of a private repo -> 404 (existence hidden,
uniform with the sibling read surfaces), non-owner of a public repo -> 403.
Add a list_webhooks row to the authz_guard drift table so the gate can't
silently regress.

* fix(node): read-visibility-gate list_replicas for private repos (#94)

list_replicas sat on optional_signature with no visibility check, so a
private repo's replica URLs were enumerable by anyone who knew owner/name.
Replica lists are a documented public mirror-discovery surface, so gate on
read visibility (authorize_repo_read) rather than owner-only: a public repo
stays anonymously listable, a private repo returns 404 to anyone who cannot
read it.

register_replica registers non-owner DIDs, and a replica operator is not a
visibility reader, so a non-owner replica operator of a private repo gets
404 — the intended contract, pinned by the new test. Add the authz_guard
row.

* fix(node): read-visibility-gate list_protected_branches (#94)

list_protected_branches sat on optional_signature with no visibility check,
leaking a private repo's branch names to any unauthenticated caller. Gate on
read visibility (authorize_repo_read at the root, INV-2 root-listing
exception): public repos stay anonymously listable, private repos 404 to a
non-reader. Add the authz_guard row.

* fix(node): gate list_repo_events for local private repos, keep gossip-only (#94)

list_repo_events served a private repo's ref certificates and gossip ref
metadata (ref names, SHAs) to any unauthenticated caller. The handler also
tolerates a repo it knows only via gossip (no local row) and legitimately
serves its events, so a blanket get_repo->authorize_repo_read swap would have
404'd that federation path.

Gate only the locally-hosted branch: when get_repo returns a record, run
visibility_check on it at the root and 404 on deny before fetching either the
cert or gossip events. A gossip-only repo (no local row) keeps its existing
public federation-feed behavior; the global /api/v1/events/ref-updates feed
remains a separately tracked deferral. Add the authz_guard row (visibility_check
marker, since the gate is direct to avoid a second get_repo).

* fix(gl): sign the webhook list request now that GET /hooks is owner-gated (#94)

`gl webhook list` built its NodeClient with no keypair, so the GET went out
anonymously. With the node now owner-gating GET /hooks, that request returns
401. Thread a --dir through the List subcommand and load the keypair, matching
`gl webhook create`/`delete`, so the listing is signed as the owner.

(Replica and protected-branch listings stay anonymous for public repos under
read-visibility gating, so their gl clients need no change.)

* fix(review): sign the webhook list GET (get_signed), not plain get (#94)

Code review caught that NodeClient::get() never attaches the RFC 9421
signature even when a keypair is loaded — only get_signed/post/put/delete
sign. The earlier gl webhook-list fix loaded the keypair but still called
get(), so `gl webhook list` (and the MCP webhook_list tool) 401'd against
the now-owner-gated endpoint. Switch both to get_signed.

Harden the two cmd_list tests to require a Signature/Signature-Input header on
the mock (verified: they fail against plain get(), pass with get_signed) — the
missing assertion that let the bug through. Add the untested owner-of-private-
repo webhooks path (200 + redaction). Fix the stale module doc and the
authz_guard bucket comment for list_webhooks.

* fix(gl): sign read-visibility GETs so private-repo owners can read their own repos (#94)

The #94 server change read-visibility-gates GET /replicas and
/branches/protected: public repos stay anonymous, private repos require the
owner/reader to authenticate. Four gl read commands sent unsigned GETs and so
broke (404) or silently degraded (empty output) for a private repo's owner:
gl protect list, gl repo owner (silent empty protected list), gl repo info
(silent missing replica count), gl repo replicas (and it had no --dir flag).

Add NodeClient::get_maybe_signed — signs when an identity is present, stays
anonymous otherwise (mirrors the conditional signing of post/put/delete) —
and route the four call sites through it. Add --dir to `gl repo replicas`.
Harden the protect-list tests to assert the request is signed (verified: they
fail on plain get()). Public-repo anonymous reads are unchanged.

* fix(review): sign the primary repo fetch in gl repo info/owner too (#94)

Round-2 review caught that cmd_info and cmd_owner loaded a keypair and signed
their replica/protected-branch sub-fetch but left the PRIMARY GET
/api/v1/repos/{owner}/{name} on plain get(). get_repo is read-visibility-gated,
so for a private repo that primary fetch 404'd first and the command bailed
before reaching the signed sub-fetch — the half-fix never actually let an owner
inspect their own private repo. Route both primary fetches through
get_maybe_signed.

Add the missing signing guards: harden test_cmd_owner_is_owner to require a
signature on both mocks, and add tests for cmd_replicas and cmd_info (both were
untested) asserting the request is signed when an identity is present.

Out of scope (pre-existing, gated before #94, untouched here): gl repo commits,
gl pr list/view/diff, and the MCP repo_get/repo_commits/pr_* tools call other
already-gated read endpoints unsigned and are broken for private repos
independently of this change.

* test(review): execute the previously-reasoned-only paths (#94)

Close the vetting gaps from the review:
- node: a listed reader (non-owner) passes the read gate but is still refused
  the webhook list (403), and headerless on an existing-private vs an absent
  repo both 401 — proving the 401 fires before any lookup, so it is not an
  existence oracle.
- node: the read-visibility surfaces (replicas, protected branches) admit a
  listed reader who is not the owner (the allow-list branch of visibility_check),
  while a non-reader stranger still 404s.
- gl: get_maybe_signed's anonymous fallback (no identity -> unsigned GET) is now
  exercised, RED-proven by flipping it to get_signed (which signs and fails the
  no-signature-header assertion).

* test(review): execute the MCP dispatch and gl->node signature seams (#94)

Close the last two reasoned-only paths by execution:

- MCP webhook_list: a call_tool("webhook_list", ...) dispatch test against a
  mock node asserts the /hooks request carries a signature (get_signed).
  RED-proven: flipping the arm back to get() fails the header assertion.

- gl->node end-to-end: a real RFC-9421 signature produced exactly as the gl
  client's get_signed does (gitlawb_core::http_sig::sign_request over GET +
  empty body) is run through the node's actual optional_signature middleware,
  which verifies it and authorizes the owner (200 + webhook body). RED-proven:
  signing over POST while sending GET makes the node reject it. This stitches
  the gl signing side and the node verifying side in one test rather than
  mocking one end.

* fix(review): fail closed when the repo lookup errors in list_repo_events (#94)

`get_repo(...).await.ok().flatten()` collapsed a lookup Err into None, so a
transient DB failure skipped the read-visibility gate and the handler served a
private repo's gossip ref-update events. Propagate with `?` instead: an error
now maps to AppError::Internal (500) and fails closed, while a genuine Ok(None)
(repo not hosted locally) stays the intentional ungated pass-through.

Add a regression test that forces a get_repo error (drop the column its SELECT
reads) and asserts 500 with no secret in the body. The injection is
self-verifying: it asserts the lookup succeeds pre-drop and errors post-drop, so
the test can't pass vacuously if get_repo's projection changes later.

* fix(node,gl): close residual gaps in repo-read gating (#113)

Address jatmn's review feedback on the read-surface gate:

- gl webhook: resolve owner from the signing identity, not the node DID.
  cmd_create/list/delete all routed through the local resolve_owner helper,
  which only ever returned the node root DID, so for a repo owned by the
  local identity the owner-gated /hooks route returned 404/403. Drop that
  helper and reuse repo::resolve_owner_repo_pair (accepts owner/repo, else
  derives from --dir/keypair). Fixed across all three subcommands so
  create-writes and list/delete-reads stay on the same owner.

- gl webhook list: check the HTTP status before reading `webhooks`. A
  401/403/404 body has no webhooks field and previously printed
  "No webhooks" and exited 0, hiding the rejection; now bails with the
  server message like cmd_create.

- visibility::is_owner: delegate to api::did_matches so the read gate and
  require_repo_owner share one normalization. Fixes the under-match (bare
  owner stored, full did:key caller, as mirror rows) that denied the owner
  with the opaque 404, and the cross-method over-match.

- authz_guard: pin both halves of list_webhooks (authorize_repo_read and
  require_repo_owner) so removing the existence-hiding gate fails the drift
  test.

cargo fmt/clippy clean; 644 workspace tests pass.

* test(node)+fix(gl): close events reader-coverage gap, align label-list signing

Self-review follow-ups on the read-gate work:

- list_repo_events: add list_repo_events_admits_listed_reader, pinning the
  allow-list branch of the events gate (a listed non-owner reader gets 200
  with the cert; a non-reader stranger 404s with no leak). Mirrors the
  read_visibility_admits_listed_reader coverage the sibling replica and
  protected-branch surfaces already had; the events handler was the only
  gated read surface missing it.

- gl repo: cmd_label_list now loads an identity and uses get_maybe_signed
  instead of a bare anonymous get(), matching cmd_replicas/cmd_info. Public
  repos stay anonymously listable; a private-repo owner can read their own
  labels once /labels is read-visibility-gated, with no silent client drift.

fmt/clippy clean; 645 workspace tests pass.

* fix(review): pin the label-list signing and the events 404 sha-no-leak

Final-review hardening of the prior two follow-ups:

- gl repo label-list tests: require the signature/signature-input headers on
  the signed-branch mocks, and add an anonymous (no-identity) test asserting
  the header is absent. Without the header assertion the tests passed even
  with a bare get(), so the signing change was not actually pinned
  (RED-verified: reverting to get() now fails both signed-branch tests).

- list_repo_events_admits_listed_reader: also assert the cert sha (not just
  the ref name) is absent from the stranger's 404 body, matching the
  list_repo_events_is_read_visibility_gated leak-check convention.

fmt/clippy clean; 646 workspace tests pass.

* fix(gl): error on 2xx webhook-list body missing the webhooks array (#113)

A 2xx /hooks response without a `webhooks` array could hide a node/API
contract regression. cmd_list now errors with context instead of mapping
it to "No webhooks". An empty-but-present array stays a valid no-webhooks
result. Adds a test for the missing-array case.

* fix(gl,node): scope MCP webhook_list owner to keypair, gate list_labels on read visibility (#113)

- MCP webhook_list: default owner to the signing keypair's short DID (explicit
  owner arg still overrides) instead of the node root DID, so the signed,
  owner-gated /hooks request resolves to the caller's own repo. Mirrors the
  gl webhook list fix. resolve_owner and webhook_create/delete left untouched.
- list_labels: gate on authorize_repo_read so a private repo's label names 404
  for non-readers (public repos stay anonymously listable), matching the sibling
  read surfaces and making the existing client comment true. Added the
  authz_guard marker and a read-visibility integration test.
- gl label list: check HTTP status before reading the body so the now-gated
  404 surfaces as an error instead of printing "No labels" and exiting 0,
  matching the sibling list clients (replicas, protected branches).

* fix(gl): surface denied repo info reads instead of fabricating a repo (#113)

cmd_info parsed the GET /api/v1/repos/{owner}/{name} body without checking
the HTTP status, so a private-repo JSON 404 was rendered as a fake repo
summary (owner / ? / public / main) and exited 0. Capture the status and
bail on non-2xx, matching the sibling list commands.

Also assert the anonymous 404 label-list body does not leak the label name,
closing the last unread-body branch in the read-visibility gate test.

* test(node): drop #113-gated handlers from KNOWN_UNGATED; sync lockfile to 0.4.0

The repo-scoped egress guard's allowlist (added in #122, merged after this
branch was opened) still listed the read surfaces this branch gates. Now that
list_labels, list_repo_events, list_webhooks, list_replicas, and
list_protected_branches are read-visibility gated, the staleness assert
requires removing them from the allowlist. The #120 surfaces (certs, issues,
bounties, stars) stay listed; git_info_refs stays until #119 lands. Lockfile
versions follow the 0.4.0 release the rebase landed on.

* fix(gl): fail the MCP webhook_list tool on non-2xx node responses

The MCP webhook_list path deserialized the response body and returned it as a
successful tool result without checking the HTTP status, so a 401/403/404 JSON
error (missing identity, wrong owner, private or deleted repo) was reported as
success with the error body as the result. Check status().is_success() before
deserializing and surface the node's message as a tool error, matching the
`gl webhook list` path. Covered by a non-2xx test (RED without the check).

* fix(gl): read the protected_branches key in repo owner

gl repo owner parsed the protected-branch response as a bare array or a
"branches" key, but the node returns {"protected_branches":[...],"count":N},
so it always printed "Protected: (none)" even when branches were protected.
Extract the parse into parse_protected_branches matching the gl protect list
shape, and update the owner tests to mock the real schema instead of bare
arrays (which is why the bug went uncaught).

* test(gl): cover the deny/negative arms reconciled by the merge

The merge changed three gl branches that no existing test executed (they
were only reasoned): resolve_owner_did with no local identity, cmd_info on
an authenticated-but-denied 403, and get_maybe_signed with no keypair. Add
a test per branch asserting the must-not case (error/unsigned, never a
fabricated repo card). Verified load-bearing by mutation: disabling the
cmd_info status gate turns the 403 test red.

---------

Co-authored-by: t <t@t>
2026-07-22 08:24:09 +08:00
69 changed files with 5107 additions and 18764 deletions

No files matched your search

+2 -60
View File
@@ -109,73 +109,15 @@ GITLAWB_MAX_PACK_BYTES=2147483648
# Max seconds a served git upload-pack / receive-pack (clone / push) may run
# before it is aborted with a 504. Bounds a hung git that would otherwise pin a
# worker and, on push, the repo write lock. Also bounds both info/refs
# advertisements, the withheld-blob pack build, and the push-side candidate
# discovery (rev-list / cat-file), all reaped via process-group teardown (#174).
# Default 600.
# worker and, on push, the repo write lock. Does NOT cover the info/refs
# advertisement or the withheld-blob path, which remain unbounded. Default 600.
GITLAWB_GIT_SERVICE_TIMEOUT_SECS=600
# Max seconds the storage-ACQUISITION phase of a served git op may run before the
# request is shed with a 503, separate from the git-run timeout above. A
# concurrency permit is taken before this phase and GITLAWB_GIT_SERVICE_TIMEOUT_SECS
# only starts once git spawns, so without this a stalled backend (a hung Tigris
# HEAD/GET, or a hung pg advisory-lock iteration on push) pins the permit and drains
# the pool until every later request 503s. On expiry the permit is released
# (fail-closed). Kept separate because acquisition and git execution are distinct
# cost centers. Must be positive; set very large to effectively disable. Default 30.
GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS=30
# Max concurrent git READ ops (upload-pack + the upload-pack info/refs
# advertisement) served at once, a global pool separate from the push pool below.
# The anon receive-pack info/refs advertisement has its OWN pool (see below), not
# this one. Over-cap sheds a clean 503 + Retry-After. Anonymous reads draw from
# here, so pair it with GITLAWB_MAX_CONCURRENT_READS_PER_CALLER (below) so one
# caller cannot monopolize the pool. Default 128.
GITLAWB_MAX_CONCURRENT_GIT_OPS=128
# Max concurrent git-receive-pack (push) POST operations, in a pool separate from
# the read pool (GITLAWB_MAX_CONCURRENT_GIT_OPS) so anonymous reads cannot shed an
# authenticated push at admission. The anon receive-pack info/refs advertisement
# runs in a SEPARATE pool of the same size (disjoint from this one), so an
# advertisement flood cannot shed a push either. Over-cap sheds a 503 +
# Retry-After. Default 32.
GITLAWB_MAX_CONCURRENT_GIT_PUSHES=32
# Max concurrent read ops (upload-pack + the upload-pack info/refs advertisement)
# a single caller may hold, so one caller cannot monopolize the read pool. Keyed
# on the resolved SOURCE IP, never the DID: a signature does not move a caller off
# this cap. The source-IP key is only as granular as GITLAWB_TRUSTED_PROXY below:
# left unset, a node behind an edge/NAT keys all callers on the edge IP and this
# collapses to one global cap. Set GITLAWB_TRUSTED_PROXY for per-client keying; a
# high-fanout caller (CI behind one NAT) then needs the operator to raise this.
# Default 16.
GITLAWB_MAX_CONCURRENT_READS_PER_CALLER=16
# ── Push rate limiting (git-receive-pack flood brake) ─────────────────────
# Max receive-pack requests (info/refs advertisement + push POST) per client
# IP per hour. 0 disables. Default 600.
GITLAWB_PUSH_RATE_LIMIT=600
# ── /ipfs/{cid} visibility-walk admission (#174) ──────────────────────────
# GET /ipfs/{cid} runs a per-repo full-history git walk in a blocking thread to
# decide whether the caller may read a path-scoped blob. It is publicly reachable,
# so it is bounded to keep a permissionless caller from fanning out unbounded
# concurrent walks and exhausting blocking-pool threads + PIDs.
# Max concurrent /ipfs walks across all callers (a pool of its own, disjoint from
# the served-git pools). Over-cap sheds a 503. Default 32.
GITLAWB_MAX_CONCURRENT_IPFS_WALKS=32
# Max concurrent /ipfs walks a single SOURCE IP may hold (keyed like the git
# per-caller caps via GITLAWB_TRUSTED_PROXY; reject-before-insert bounded map).
# Default 4.
GITLAWB_IPFS_WALK_PER_SOURCE=4
# Max legacy (NULL-provenance) repos probed per single /ipfs request, bounding the
# scan-fallback fan-out (git cat-file per candidate repo) for an anonymous caller. A
# truncated scan sheds a retryable 503, never a false 404. Default 256.
GITLAWB_IPFS_MAX_REPOS_WALKED=256
# Max /ipfs/{cid} requests per client IP per hour (route flood brake, distinct
# from the concurrency caps above). 0 disables. Default 600.
GITLAWB_IPFS_RATE_LIMIT=600
# ── Creation rate limiting (repo/agent/issue/PR flood brake) ──────────────
# Max creation requests (POST /api/v1/repos, /api/register, fork, issues,
# pulls) per client IP per hour, in addition to the per-DID limit. The per-DID
+121
View File
@@ -0,0 +1,121 @@
# Tracks main's CI state as a labeled issue: opens (or updates) a ci-main-red
# issue when a push-to-main run of PR Checks ends red, and closes it when a
# subsequent run on main's tip goes green.
#
# Why: two individually-green PRs can break main in combination when the
# second merges on checks that ran against a stale base (semantic merge
# race — #113 + #145 broke test compilation, found only via #229). A red main
# is otherwise silent, and release-please PRs inherit the breakage.
#
# The alert reflects main's CURRENT STATE, not raw CI events: runs whose
# commit is no longer main's tip are ignored (a re-run of an old red run
# after a fix has landed must not cry wolf), and a green run on the tip
# closes any open alert.
name: Main red alert
on:
workflow_run:
workflows: ["PR Checks"]
types: [completed]
branches: [main]
permissions:
issues: write
contents: read # getBranch tip check
# Serialize runs: concurrent completions would race the label/issue lookups
# into duplicates (or 422 on createLabel). queue: max keeps every pending
# completion instead of letting a later (possibly green-skipped) arrival
# displace a pending red alert under burst merges; it is only incompatible
# with cancel-in-progress: true, which we don't use.
concurrency:
group: main-red-alert
cancel-in-progress: false
queue: max
jobs:
track:
name: Track main CI state
# timed_out (the test job runs 45 min against Postgres) and
# startup_failure (malformed workflow) are red too — only cancelled is
# excluded as a deliberate human abort. success runs are needed to close
# the alert once main recovers.
if: >-
github.event.workflow_run.event == 'push' &&
contains(fromJSON('["failure", "timed_out", "startup_failure", "success"]'), github.event.workflow_run.conclusion)
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Reconcile the broken-main issue with main's current state
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
with:
script: |
const run = context.payload.workflow_run;
const label = 'ci-main-red';
// Freshness gate: only the run for main's current tip may open or
// close anything. Re-runs of superseded commits are events about
// the past, not statements about main's state.
const { data: branch } = await github.rest.repos.getBranch({
...context.repo, branch: 'main',
});
if (branch.commit.sha !== run.head_sha) {
core.info(`run ${run.id} is for ${run.head_sha}, main is at ${branch.commit.sha} — ignoring stale run`);
return;
}
// listForRepo returns PRs carrying the label too — those must
// never be mistaken for the tracking issue.
const open = (await github.rest.issues.listForRepo({
...context.repo, state: 'open', labels: label, per_page: 100,
})).data.filter(item => !item.pull_request);
if (run.conclusion === 'success') {
const body = `Recovered: [PR Checks](${run.html_url}) is green on main at \`${run.head_sha}\`.`;
for (const issue of open) {
await github.rest.issues.createComment({
...context.repo, issue_number: issue.number, body,
});
await github.rest.issues.update({
...context.repo, issue_number: issue.number, state: 'closed', state_reason: 'completed',
});
}
return;
}
const headline = run.head_commit?.message?.split('\n')[0] ?? '';
const body = [
`[PR Checks run](${run.html_url}) concluded \`${run.conclusion}\` on main at \`${run.head_sha}\`.`,
'',
`> ${headline}`,
'',
'Likely a semantic merge race: each merged PR was green on a stale base.',
'Fix main before merging anything else — release-please PRs inherit this breakage.',
'',
"This issue closes automatically when a run on main's tip goes green.",
].join('\n');
try {
await github.rest.issues.getLabel({ ...context.repo, name: label });
} catch (e) {
if (e.status !== 404) throw e;
await github.rest.issues.createLabel({
...context.repo,
name: label,
color: 'b60205',
description: 'CI is failing on main',
});
}
if (open.length > 0) {
await github.rest.issues.createComment({
...context.repo, issue_number: open[0].number, body,
});
} else {
await github.rest.issues.create({
...context.repo,
title: 'CI is red on main',
body,
labels: [label],
});
}
+64
View File
@@ -2,6 +2,10 @@ name: PR Checks
on:
pull_request:
# Run in the merge queue so a PR is re-validated against the true merged
# result before it lands. Guards against semantic merge races where two
# individually-green PRs break main in combination (#113 + #145 → #229).
merge_group:
push:
branches:
- main
@@ -212,3 +216,63 @@ jobs:
- name: Smoke test --version
run: docker run --rm gitlawb-node:ci-test --version
test-windows:
name: test (windows, non-blocking)
runs-on: windows-latest
timeout-minutes: 45
# Non-blocking, mirroring the informational `beta` lane above. We ship `gl`
# and `git-remote-gitlawb` to Windows users (release.yml) with zero Windows
# test coverage, so run their suites here to make Windows-only regressions
# VISIBLE on the PR without gating merge (#228). These two are the client
# CLIs and depend on no database, so no Postgres service is needed (service
# containers do not run on windows-latest regardless). `git` is preinstalled
# on windows-latest, which the real_git_fetch harness relies on.
continue-on-error: true
steps:
- name: Check out repository
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- name: Set up Rust toolchain
uses: dtolnay/rust-toolchain@b3b07ba8b418998c39fb20f53e8b695cdcc8de1b # stable
with:
toolchain: stable
- name: Cache cargo
uses: Swatinem/rust-cache@98c8021b550208e191a6a3145459bfc9fb29c4c0 # v2.8.0
with:
key: windows
- name: cargo test (shipped Windows crates)
run: cargo test -p gl -p git-remote-gitlawb
# gitlawb-core is embedded by every consumer (gl, git-remote-gitlawb, the node
# daemon), so it must stay lean. This gate fails if gitlawb-core's normal
# (non-dev, non-build) dependency tree gains any crate not on the allowlist in
# ci/gitlawb-core-allowed-deps.txt. Exhaustive-by-construction: a new heavy
# dependency reds CI whether or not anyone thought to ban it, which a denylist
# cannot do. Regen instructions live in the allowlist header.
core-deps-purity:
name: gitlawb-core dependency purity
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Check out repository
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- name: Set up Rust toolchain
uses: dtolnay/rust-toolchain@b3b07ba8b418998c39fb20f53e8b695cdcc8de1b # stable
with:
toolchain: stable
- name: Cache cargo
uses: Swatinem/rust-cache@98c8021b550208e191a6a3145459bfc9fb29c4c0 # v2.8.0
with:
key: core-deps-purity
- name: Check gitlawb-core dependency allowlist
run: bash scripts/check-gitlawb-core-deps.sh
+1 -1
View File
@@ -24,7 +24,7 @@ jobs:
timeout-minutes: 10
steps:
- name: Label and guide
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
with:
script: |
const pr = context.payload.pull_request;
+223 -52
View File
@@ -4,6 +4,20 @@ on:
push:
branches:
- main
# Manual backfill: republish an already-cut release to a channel that
# missed it. Fill in only the channel(s) you want — an empty input leaves
# that channel's jobs skipped. npm reuses the existing release's binary
# assets; docker rebuilds the image from the tag and re-tags the manifest.
workflow_dispatch:
inputs:
npm_backfill_tag:
description: "Existing release tag to publish to npm (e.g. v0.6.0)"
required: false
type: string
docker_backfill_tag:
description: "Existing release tag to publish to ghcr (e.g. v0.6.0)"
required: false
type: string
concurrency:
group: release-${{ github.ref }}
@@ -14,7 +28,10 @@ permissions: {}
jobs:
release-please:
if: ${{ github.repository == 'Gitlawb/node' }}
# Skipped on manual dispatch: a backfill must never create or advance a
# release PR as a side effect (npm-publish tolerates the skipped need via
# !cancelled(), and every other job gates on release_created == 'true').
if: ${{ github.repository == 'Gitlawb/node' && github.event_name != 'workflow_dispatch' }}
name: Release Please
runs-on: ubuntu-latest
permissions:
@@ -31,24 +48,61 @@ jobs:
with:
token: ${{ secrets.GITHUB_TOKEN }}
# Each architecture builds NATIVELY (amd64 on ubuntu-latest, arm64 on
# ubuntu-24.04-arm) and pushes by digest; docker-manifest below stitches the
# digests into the tagged multi-arch image. No QEMU: emulating the arm64
# Rust release build wedged the v0.6.0 run for 3+ hours and, via the
# workflow concurrency group, blocked every queued release run behind it.
docker:
name: Build & Push Docker Image
name: Build & Push Docker Image (${{ matrix.arch }})
needs: release-please
if: ${{ needs.release-please.outputs.release_created == 'true' }}
runs-on: ubuntu-latest
# Runs on a fresh release, or on manual dispatch for an existing tag
# (release-please is skipped on dispatch, hence !cancelled()).
if: >-
${{ !cancelled() && (
needs.release-please.outputs.release_created == 'true' ||
(github.event_name == 'workflow_dispatch' && inputs.docker_backfill_tag != '')
) }}
strategy:
fail-fast: false
matrix:
include:
- arch: amd64
os: ubuntu-latest
platform: linux/amd64
- arch: arm64
os: ubuntu-24.04-arm
platform: linux/arm64
runs-on: ${{ matrix.os }}
permissions:
contents: read
packages: write
steps:
- name: Resolve release tag
id: rel
env:
DISPATCH_TAG: ${{ inputs.docker_backfill_tag }}
RELEASE_TAG: ${{ needs.release-please.outputs.tag_name }}
run: |
set -euo pipefail
TAG="${DISPATCH_TAG:-$RELEASE_TAG}"
case "$TAG" in
v[0-9]*) ;;
*) echo "::error::'$TAG' does not look like a release tag (vX.Y.Z)"; exit 1 ;;
esac
echo "tag=$TAG" >> "$GITHUB_OUTPUT"
echo "version=${TAG#v}" >> "$GITHUB_OUTPUT"
# ghcr requires a lowercase repository path, and unlike metadata-action,
# buildx's `--output name=` does no lowercasing — a mixed-case owner
# makes the digest push fail with "invalid reference format".
echo "image=ghcr.io/${GITHUB_REPOSITORY,,}" >> "$GITHUB_OUTPUT"
- name: Checkout release tag
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
ref: ${{ needs.release-please.outputs.tag_name }}
ref: ${{ steps.rel.outputs.tag }}
persist-credentials: false
- name: Set up QEMU
uses: docker/setup-qemu-action@29109295f81e9208d7d86ff1c6c12d2833863392 # v3.6.0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@b5ca514318bd6ebac0fb2aedd5d36ec1b5c232a2 # v3.10.0
@@ -59,45 +113,130 @@ jobs:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata
id: meta
uses: docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804 # v5.7.0
with:
images: ghcr.io/${{ github.repository }}
tags: |
type=semver,pattern={{version}},value=${{ needs.release-please.outputs.version }}
type=semver,pattern={{major}}.{{minor}},value=${{ needs.release-please.outputs.version }}
type=raw,value=latest
- name: Build and load locally (smoke)
uses: docker/build-push-action@14487ce63c7a62a4a324b0bfb37086795e31c6c1 # v6.16.0
with:
context: .
platforms: ${{ matrix.platform }}
load: true
tags: gitlawb-node:smoke
cache-from: type=gha
cache-from: type=gha,scope=docker-${{ matrix.arch }}
# Native runner on both arches, so the built image can always execute.
# Assert the released version so a stale artifact fails here, not in prod.
- name: Smoke test
run: docker run --rm gitlawb-node:smoke --version
env:
VERSION: ${{ steps.rel.outputs.version }}
run: |
set -euo pipefail
out="$(docker run --rm gitlawb-node:smoke --version)"
echo "$out"
grep -qF "$VERSION" <<<"$out" || {
echo "::error::image --version did not report expected version $VERSION (got: $out)"
exit 1
}
- name: Build and push (multi-arch)
# Push by digest only — tags are applied once by docker-manifest so a
# half-finished matrix can never publish a partially-tagged image.
# provenance:false keeps each push a plain single-arch manifest, which is
# what imagetools create expects to merge.
- name: Build and push by digest
id: push
uses: docker/build-push-action@14487ce63c7a62a4a324b0bfb37086795e31c6c1 # v6.16.0
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
platforms: ${{ matrix.platform }}
provenance: false
outputs: type=image,name=${{ steps.rel.outputs.image }},push-by-digest=true,name-canonical=true,push=true
cache-from: type=gha,scope=docker-${{ matrix.arch }}
cache-to: type=gha,mode=max,scope=docker-${{ matrix.arch }}
- name: Export digest
env:
DIGEST: ${{ steps.push.outputs.digest }}
run: |
set -euo pipefail
mkdir -p /tmp/digests
touch "/tmp/digests/${DIGEST#sha256:}"
- name: Upload digest
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: docker-digest-${{ matrix.arch }}
path: /tmp/digests/*
if-no-files-found: error
retention-days: 1
docker-manifest:
name: Publish Docker manifest
needs: [release-please, docker]
if: ${{ !cancelled() && needs.docker.result == 'success' }}
runs-on: ubuntu-latest
permissions:
packages: write
steps:
- name: Resolve release tag
id: rel
env:
DISPATCH_TAG: ${{ inputs.docker_backfill_tag }}
RELEASE_TAG: ${{ needs.release-please.outputs.tag_name }}
run: |
set -euo pipefail
TAG="${DISPATCH_TAG:-$RELEASE_TAG}"
case "$TAG" in
v[0-9]*) ;;
*) echo "::error::'$TAG' does not look like a release tag (vX.Y.Z)"; exit 1 ;;
esac
echo "tag=$TAG" >> "$GITHUB_OUTPUT"
echo "version=${TAG#v}" >> "$GITHUB_OUTPUT"
- name: Download digests
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
pattern: docker-digest-*
path: /tmp/digests
merge-multiple: true
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@b5ca514318bd6ebac0fb2aedd5d36ec1b5c232a2 # v3.10.0
- name: Log in to GitHub Container Registry
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3.4.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Create and push multi-arch manifest
env:
VERSION: ${{ steps.rel.outputs.version }}
run: |
set -euo pipefail
# ghcr requires a lowercase repository path.
IMAGE="ghcr.io/${GITHUB_REPOSITORY,,}"
MAJOR_MINOR="${VERSION%.*}"
digests=""
for f in /tmp/digests/*; do
digests="$digests $IMAGE@sha256:$(basename "$f")"
done
# shellcheck disable=SC2086
docker buildx imagetools create \
-t "$IMAGE:$VERSION" \
-t "$IMAGE:$MAJOR_MINOR" \
-t "$IMAGE:latest" \
$digests
docker buildx imagetools inspect "$IMAGE:$VERSION"
- name: Release summary
env:
VERSION: ${{ steps.rel.outputs.version }}
TAG: ${{ steps.rel.outputs.tag }}
run: |
{
echo "## Released ${{ needs.release-please.outputs.tag_name }}"
echo "## Released $TAG"
echo
echo "- Image: \`ghcr.io/${{ github.repository }}:${{ needs.release-please.outputs.version }}\`"
echo "- GitHub: https://github.com/${{ github.repository }}/releases/tag/${{ needs.release-please.outputs.tag_name }}"
echo "- Image: \`ghcr.io/${GITHUB_REPOSITORY,,}:$VERSION\` (linux/amd64 + linux/arm64)"
echo "- GitHub: https://github.com/$GITHUB_REPOSITORY/releases/tag/$TAG"
} >> "$GITHUB_STEP_SUMMARY"
release-binaries:
@@ -248,37 +387,62 @@ jobs:
npm-publish:
name: Publish to npm
needs: [release-please, release-binaries]
if: ${{ needs.release-please.outputs.release_created == 'true' }}
# Runs on a fresh release, or on manual dispatch for an existing tag
# (release-binaries is skipped on dispatch, hence !cancelled()). Publishing
# auth is npm Trusted Publishing (GitHub OIDC) — no NPM_TOKEN. Each
# @gitlawb package must have this repo + workflow (release.yml) configured
# as its trusted publisher on npmjs.com, or publish fails loudly here —
# deliberately loud: the silent secret-guard skip is how 0.4.x–0.6.0
# never reached npm.
if: >-
${{ !cancelled() && (
(needs.release-please.outputs.release_created == 'true' && needs.release-binaries.result == 'success') ||
(github.event_name == 'workflow_dispatch' && inputs.npm_backfill_tag != '')
) }}
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write # npm provenance
id-token: write # npm trusted publishing (OIDC) + provenance
steps:
- name: Guard on secret
id: guard
- name: Resolve release tag
id: rel
env:
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
DISPATCH_TAG: ${{ inputs.npm_backfill_tag }}
RELEASE_TAG: ${{ needs.release-please.outputs.tag_name }}
run: |
if [ -z "${NPM_TOKEN:-}" ]; then
echo "::warning::NPM_TOKEN is not set — skipping npm publish."
echo "enabled=false" >> "$GITHUB_OUTPUT"
else
echo "enabled=true" >> "$GITHUB_OUTPUT"
fi
set -euo pipefail
TAG="${DISPATCH_TAG:-$RELEASE_TAG}"
case "$TAG" in
v[0-9]*) ;;
*) echo "::error::'$TAG' does not look like a release tag (vX.Y.Z)"; exit 1 ;;
esac
echo "tag=$TAG" >> "$GITHUB_OUTPUT"
echo "version=${TAG#v}" >> "$GITHUB_OUTPUT"
- name: Checkout release tag
if: ${{ steps.guard.outputs.enabled == 'true' }}
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
ref: ${{ needs.release-please.outputs.tag_name }}
ref: ${{ steps.rel.outputs.tag }}
persist-credentials: false
# npm >= 11.5.1 performs the OIDC token exchange automatically when the
# package has a trusted publisher configured; older npm silently falls
# back to (absent) token auth and fails confusingly.
- name: Set up Node 24 + OIDC-capable npm
uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5.0.0
with:
node-version: "24"
- name: Ensure npm supports trusted publishing
run: |
set -euo pipefail
npm install -g npm@^11.5.1
npm --version
- name: Lay in release binaries
if: ${{ steps.guard.outputs.enabled == 'true' }}
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
VERSION: ${{ needs.release-please.outputs.version }}
TAG: ${{ needs.release-please.outputs.tag_name }}
VERSION: ${{ steps.rel.outputs.version }}
TAG: ${{ steps.rel.outputs.tag }}
run: |
set -euo pipefail
# npm platform package -> Rust target triple (unix only; Windows is not
@@ -305,9 +469,8 @@ jobs:
done
- name: Set versions
if: ${{ steps.guard.outputs.enabled == 'true' }}
env:
VERSION: ${{ needs.release-please.outputs.version }}
VERSION: ${{ steps.rel.outputs.version }}
run: |
set -euo pipefail
node -e '
@@ -327,13 +490,12 @@ jobs:
'
- name: Publish
if: ${{ steps.guard.outputs.enabled == 'true' }}
env:
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
VERSION: ${{ needs.release-please.outputs.version }}
VERSION: ${{ steps.rel.outputs.version }}
run: |
set -euo pipefail
echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > ~/.npmrc
# No token: npm exchanges this job's GitHub OIDC identity with the
# registry (trusted publishing); provenance is attested automatically.
# Platform packages first, then the wrapper (so its optionalDependencies resolve).
# Skip versions already on the registry so a rerun after a partial publish
# is idempotent instead of erroring on the first existing package.
@@ -429,6 +591,15 @@ jobs:
bin.install "git-remote-gitlawb"
end
def caveats
<<~CAVEATS
oh-my-zsh's git plugin aliases gl='git pull', which shadows this
binary in interactive shells. If \`gl\` prints "fatal: not a git
repository", run:
echo 'unalias gl 2>/dev/null' >> ~/.zshrc && source ~/.zshrc
CAVEATS
end
test do
assert_match version.to_s, shell_output("#{bin}/gl --version")
end
+1 -1
View File
@@ -1,3 +1,3 @@
{
".": "0.5.1"
".": "0.7.0"
}
+34
View File
@@ -1,5 +1,39 @@
# Changelog
## [0.7.0](https://github.com/Gitlawb/node/compare/v0.6.0...v0.7.0) (2026-07-22)
### Features
* **gl:** actually verify certificate signatures in gl cert show ([b0a2ed8](https://github.com/Gitlawb/node/commit/b0a2ed830821b5a77b0dafdcf5ce4b1ad904ec25))
* **gl:** doctor warns when a shell alias shadows the gl binary ([0765c42](https://github.com/Gitlawb/node/commit/0765c42feff47b29054b9a277881a7cd5bfd9135))
### Bug Fixes
* **gl:** address review findings — non-fatal DID check, exact-host loopback, detached HEAD, multi-URL remotes, command-aware unalias ([3fa38dc](https://github.com/Gitlawb/node/commit/3fa38dce4e435ef017f607d355266c2892161f8d))
* **gl:** anchor cert --verify to a trusted issuer; pin canonical payload form ([c681af7](https://github.com/Gitlawb/node/commit/c681af70dba3a76b11795d324f2ca0fecf7e42fd))
* **gl:** detect gitlawb remotes beyond origin in gl status ([2d88937](https://github.com/Gitlawb/node/commit/2d88937fbabf21013f28debb4d4f93270a4bb8cd))
* **gl:** doctor treats a reachable local node as configuration, not failure ([94c61f7](https://github.com/Gitlawb/node/commit/94c61f79421fa26ecc1bdf7c255e0c48e66a2048))
* **gl:** make gl init's push guidance match the repo's actual state ([cb42a48](https://github.com/Gitlawb/node/commit/cb42a4830d0d070165e2212f316d009ec716dbee))
* warn about the oh-my-zsh gl alias at install time ([e3df51d](https://github.com/Gitlawb/node/commit/e3df51d76c0275fde33954ff0af0aab95fd0cb7d))
## [0.6.0](https://github.com/Gitlawb/node/compare/v0.5.1...v0.6.0) (2026-07-22)
### Features
* **icaptcha-client:** solve the iCaptcha proof-of-work on answer ([#181](https://github.com/Gitlawb/node/issues/181)) ([c98b503](https://github.com/Gitlawb/node/commit/c98b503f90cade54ce9c588bd2373a4e3486f2fc))
### Bug Fixes
* **core:** honor wildcard capabilities in Ucan::can, matching is_attenuated_by ([#200](https://github.com/Gitlawb/node/issues/200)) ([b6877a6](https://github.com/Gitlawb/node/commit/b6877a65981dbf41c4a7838971eb5c826c7fcaf2))
* **gl:** point doctor's version check at Gitlawb/node, not the frozen Gitlawb/releases repo ([#198](https://github.com/Gitlawb/node/issues/198)) ([98d728c](https://github.com/Gitlawb/node/commit/98d728c325f68f0ee4a62e473dc484b7dd627504))
* **node:** carry full owner DID on ref-update wire event ([#144](https://github.com/Gitlawb/node/issues/144)) ([#145](https://github.com/Gitlawb/node/issues/145)) ([69a62e5](https://github.com/Gitlawb/node/commit/69a62e51c2d2f090f9cae88f4d8ce3b9017b7149))
* **node:** gate /hooks and sibling read surfaces leaking private repo metadata ([#94](https://github.com/Gitlawb/node/issues/94)) ([#113](https://github.com/Gitlawb/node/issues/113)) ([ee8d7c0](https://github.com/Gitlawb/node/commit/ee8d7c03b26a0b678cc90b8801aa340fe81da700))
* strip Cf bidi/format controls in terminal-bound error sanitizers ([#183](https://github.com/Gitlawb/node/issues/183)) ([#188](https://github.com/Gitlawb/node/issues/188)) ([b07d41e](https://github.com/Gitlawb/node/commit/b07d41ea97569d79d3f545ae5d43c373cd6f0f33))
## [0.5.1](https://github.com/Gitlawb/node/compare/v0.5.0...v0.5.1) (2026-07-10)
Generated
+1
View File
@@ -3821,6 +3821,7 @@ version = "0.4.0"
dependencies = [
"anyhow",
"gitlawb-core",
"mockito",
"reqwest",
"serde",
"serde_json",
+1 -8
View File
@@ -343,20 +343,13 @@ Important node settings:
| `GITLAWB_REQUIRE_SIGNED_PEER_WRITES` | Require signed peer announce/sync writes. |
| `GITLAWB_AUTO_SYNC` | Enable automatic sync from known peers. |
| `GITLAWB_MAX_PACK_BYTES` | Max git pack body size for smart-HTTP routes. |
| `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack, receive-pack, or `info/refs` advertisement may run before it is aborted (504). Default 600. Also bounds the withheld-blob classification walk (on both the upload-pack serve and receive-pack replication paths) and the push-side pin-candidate discovery (`rev-list` / `cat-file`), each reaped via process-group teardown at the deadline. |
| `GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS` | Max seconds the storage-acquisition phase (Tigris HEAD/GET, push advisory-lock) of a served git op may run before the request is shed with a 503, separate from the git-run timeout. The concurrency permit is released on expiry so a stalled backend cannot pin the pool. Default 30. |
| `GITLAWB_MAX_CONCURRENT_IPFS_WALKS` | Max concurrent `GET /ipfs/{cid}` visibility walks across all callers (own pool, disjoint from the served-git pools); over-cap sheds 503. Default 32. |
| `GITLAWB_IPFS_WALK_PER_SOURCE` | Max concurrent `/ipfs` walks a single source IP may hold. Default 4. |
| `GITLAWB_IPFS_MAX_REPOS_WALKED` | Max legacy (NULL-provenance) repos probed per `/ipfs/{cid}` request, bounding the scan-fallback fan-out. A truncated scan returns a retryable 503, not a false 404. Default 256. |
| `GITLAWB_IPFS_RATE_LIMIT` | Max `/ipfs/{cid}` requests per client IP per hour (route flood brake). 0 disables. Default 600. |
| `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack/receive-pack may run before it is aborted (504). Default 600. Does not bound `info/refs` or the withheld-blob path. |
| `GITLAWB_TIGRIS_BUCKET` | Optional S3/Tigris shared repo storage bucket. |
| `GITLAWB_PINATA_JWT` | Optional Pinata/IPFS warm-storage pinning. |
| `GITLAWB_IRYS_URL` | Optional Irys/Arweave permanent anchoring. |
Production note: change the default Postgres password before exposing a node publicly.
Legacy-pin window: releases before the CID-resolver work stored the provider CID (Kubo dag-pb / Pinata) as a pinned object's resolver key. The `/ipfs/{cid}` resolver now recomputes the raw-content CID from the object bytes and refuses to serve a key that does not match, so `GET /api/v1/ipfs/pins` can still advertise an unrepaired legacy CID that 404s. Such a row is repaired opportunistically the next time a push carries the object again (its key is rewritten to the raw CID, the old value kept in `legacy_provider_cid`), but git negotiation omits objects the node already has, so most legacy rows never re-enter a push delta. A deferred one-shot startup sweep, not this opportunistic path, is what fully retires the advertise-then-404 window. Rows whose object bytes are gone stay withheld.
---
## Optional node staking
+112
View File
@@ -0,0 +1,112 @@
# Dependency allowlist for the `gitlawb-core` crate.
#
# gitlawb-core is the shared-primitives crate every consumer embeds (the `gl`
# CLI, git-remote-gitlawb, the node daemon). It must stay lean so those
# consumers do not inherit daemon-weight dependencies. This file is the source
# of truth for the crates gitlawb-core's NORMAL (non-dev, non-build) transitive
# dependency closure is allowed to contain. CI (`scripts/check-gitlawb-core-deps.sh`,
# wired into the `core-deps-purity` job in .github/workflows/pr-checks.yml)
# fails if gitlawb-core's tree contains any crate not listed here.
#
# The allowlist is exhaustive-by-construction: a NEW dependency reds CI whether
# or not anyone thought to ban it, which a denylist cannot do.
#
# Measured in-workspace, so feature unification across the workspace can, in
# rare cases, surface a crate here that gitlawb-core would not pull standalone.
# An unexpected new entry is a prompt to check what pulled it in, not noise to
# rubber-stamp.
#
# Regenerate (from repo root, after an INTENTIONAL dependency change):
# cargo tree -p gitlawb-core --edges normal --prefix none \
# | sed -E 's/ v[0-9].*$//' \
# | grep -v '^gitlawb-core$' \
# | sort -u
# then paste the result below this header block. (The checker snapshots and
# restores Cargo.lock; a manual regen may refresh it — `git checkout Cargo.lock`
# afterward if you did not intend that.)
#
# Lines starting with `#` and blank lines are ignored by the checker.
aead
anyhow
base-x
base256emoji
base64
base64ct
block-buffer
cfg-if
chacha20
chacha20poly1305
chrono
cid
cipher
const-oid
const-str
core2
cpufeatures
crypto-common
crypto_box
crypto_secretbox
curve25519-dalek
curve25519-dalek-derive
data-encoding
data-encoding-macro
data-encoding-macro-internal
der
digest
ed25519
ed25519-dalek
equivalent
generic-array
getrandom
hashbrown
hex
iana-time-zone
indexmap
inout
itoa
libc
match-lookup
memchr
multibase
multihash
multihash-codetable
multihash-derive
multihash-derive-impl
num-traits
opaque-debug
pem-rfc7468
pkcs8
poly1305
ppv-lite86
proc-macro-crate
proc-macro2
quote
rand
rand_chacha
rand_core
salsa20
serde
serde_core
serde_derive
serde_json
sha2
signature
spki
subtle
syn
synstructure
thiserror
thiserror-impl
toml_datetime
toml_edit
toml_parser
typenum
unicode-ident
universal-hash
unsigned-varint
uuid
winnow
zerocopy
zeroize
zeroize_derive
zmij
+1 -1
View File
@@ -1,7 +1,7 @@
[package]
name = "git-remote-gitlawb"
description = "Git remote helper — enables 'git clone gitlawb://did:gitlawb:...'"
version = "0.5.1" # x-release-please-version
version = "0.7.0" # x-release-please-version
edition.workspace = true
rust-version.workspace = true
license.workspace = true
+83
View File
@@ -570,6 +570,13 @@ fn safe_error_body_excerpt(body: &str) -> String {
continue;
}
if gitlawb_core::sanitize::is_bidi_format(ch) {
// Drop Cf bidi/format controls (INV-6, #183) without marking
// pending_space: unlike a newline/tab they are not whitespace, so
// "a\u{202e}c" collapses to "ac" with no phantom gap.
continue;
}
if ch.is_whitespace() {
pending_space = true;
continue;
@@ -1159,6 +1166,55 @@ mod tests {
assert!(!message.contains('\n'));
}
#[test]
fn http_error_message_strips_bidi_format_controls() {
// Each Cf bidi class must be stripped (INV-6, #183), not just U+202E:
// override (RLO), embedding + pop (LRE/PDF), isolate (LRI/PDI), and marks
// (LRM/ALM). Asserted per code point so a partial predicate fails loudly.
// The visible words must survive and JOIN with no phantom space at the
// strip points: a bidi char is dropped like a control byte, but unlike a
// newline/tab it is not whitespace, so it must not set pending_space. The
// leading bidi char (index 0) also exercises the empty-excerpt path.
let body = "\u{202e}err\u{202a}\u{202c}\u{2066}\u{2069}\u{200e}\u{061c}ok";
let message = http_error_message(
"GET",
"/info/refs",
reqwest::StatusCode::BAD_GATEWAY,
body,
None,
);
for c in [
'\u{202e}', '\u{202a}', '\u{202c}', '\u{2066}', '\u{2069}', '\u{200e}', '\u{061c}',
] {
assert!(
!message.contains(c),
"bidi U+{:04X} leaked: {message:?}",
c as u32
);
}
assert!(
message.contains("errok"),
"words should join with no phantom space: {message:?}"
);
}
#[test]
fn http_error_message_preserves_legitimate_and_rtl_text() {
// Must not over-strip: plain text, a genuine RTL SCRIPT letter (Arabic
// U+0627, category Lo), and ZWJ (U+200D, a legitimate Cf char) survive.
let message = http_error_message(
"GET",
"/info/refs",
reqwest::StatusCode::BAD_GATEWAY,
"ok \u{0627}\u{200D}b",
None,
);
assert!(
message.ends_with("ok \u{0627}\u{200D}b"),
"legitimate text was altered: {message:?}"
);
}
#[test]
fn http_error_message_truncates_long_response_body() {
let body = "x".repeat(MAX_ERROR_BODY_CHARS + 100);
@@ -1194,6 +1250,33 @@ mod tests {
assert_eq!(requests.len(), 1);
}
#[test]
fn connect_error_strips_bidi_from_response_body() {
// Transport-path proof: a bidi override in the node's real HTTP error body
// must not reach the surfaced error text. Drives the full handle_connect
// error path, not just the safe_error_body_excerpt unit boundary (INV-6).
let (base_url, server) = serve_http(vec![TestResponse {
request_line: "GET /z6Mk/myrepo/info/refs?service=git-upload-pack HTTP/1.1",
status: "404 Not Found",
body: "not\u{202e}found",
}]);
let repo_base = format!("{base_url}/z6Mk/myrepo");
let mut stdin = std::io::Cursor::new(Vec::<u8>::new());
let err = handle_connect(&repo_base, "git-upload-pack", None, &mut stdin).unwrap_err();
let message = err.to_string();
assert!(
!message.contains('\u{202e}'),
"bidi override reached the terminal: {message:?}"
);
assert!(
message.contains("notfound"),
"body should survive with the bidi char stripped: {message:?}"
);
let _ = server.join();
}
#[test]
fn connect_post_error_includes_response_body() {
let (base_url, server) = serve_http(vec![
+1 -1
View File
@@ -1,7 +1,7 @@
[package]
name = "gitlawb-attest"
description = "External Attestation v1: pluggable provenance attachments for gitlawb ref-update certs"
version = "0.5.1" # x-release-please-version
version = "0.7.0" # x-release-please-version
edition.workspace = true
rust-version.workspace = true
license.workspace = true
+1 -1
View File
@@ -1,7 +1,7 @@
[package]
name = "gitlawb-core"
description = "Core cryptographic primitives for the gitlawb network: DIDs, CIDs, UCAN, HTTP Signatures, ref-update certificates"
version = "0.5.1" # x-release-please-version
version = "0.7.0" # x-release-please-version
edition.workspace = true
rust-version.workspace = true
license.workspace = true
-45
View File
@@ -64,19 +64,6 @@ impl Cid {
}
}
/// True when `s` parses as a CIDv1 with the raw codec — the exact shape
/// [`Cid::from_git_object_bytes`] produces and the `/ipfs` resolver looks up.
/// A legacy provider CID (Kubo dag-pb, Pinata CIDv0) parses to a different
/// version or codec and returns `false`, marking it an opportunistic-repair
/// candidate. Decidable from the string alone (no object bytes), so the pin path
/// can gate the byte-read/recompute cost on it and leave non-legacy rows at the
/// existing DB-only skip cost. An unparseable string is non-canonical (`false`).
pub fn is_raw_cidv1(s: &str) -> bool {
s.parse::<CidGeneric<64>>()
.map(|c| c.version() == cid::Version::V1 && c.codec() == RAW)
.unwrap_or(false)
}
impl fmt::Display for Cid {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.0)
@@ -190,38 +177,6 @@ mod tests {
assert!(result.is_err());
}
#[test]
fn is_raw_cidv1_classifies_codec_from_string() {
// The canonical resolver key: CIDv1 + raw codec → not a repair candidate.
let raw = Cid::from_git_object_bytes(b"blob 5\0hello");
assert!(
is_raw_cidv1(raw.as_str()),
"from_git_object_bytes output is CIDv1/raw"
);
// A CIDv0 (Pinata dag-pb legacy shape) → repair candidate.
assert!(
!is_raw_cidv1("QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG"),
"a CIDv0 dag-pb value is a legacy-repair candidate"
);
// A CIDv1 with the dag-pb codec (the Kubo above-block-size root) over the
// same multihash → still a repair candidate (codec, not just version).
let parsed = raw.as_str().parse::<CidGeneric<64>>().unwrap();
const DAG_PB: u64 = 0x70;
let dagpb = CidGeneric::<64>::new_v1(DAG_PB, *parsed.hash()).to_string();
assert!(
!is_raw_cidv1(&dagpb),
"a CIDv1 dag-pb value is a legacy-repair candidate"
);
// Garbage is non-canonical.
assert!(
!is_raw_cidv1("not-a-cid"),
"an unparseable string is non-canonical"
);
}
#[test]
fn sha256_hex_of_empty_input_is_well_known() {
// SHA-256("") is a fixed constant; verifies the hasher is wired correctly.
+1
View File
@@ -5,6 +5,7 @@ pub mod encrypt;
pub mod error;
pub mod http_sig;
pub mod identity;
pub mod sanitize;
pub mod ucan;
pub use error::Error;
+69
View File
@@ -0,0 +1,69 @@
//! Text sanitization predicates shared across the workspace.
//!
//! Untrusted peer/remote bytes that reach an operator's terminal (node error
//! bodies, node-advertised strings) must be stripped of BOTH `Cc` control bytes
//! (`char::is_control()`) AND `Cf` bidi/format controls before display: a
//! right-to-left override or directional isolate can visually reorder the line
//! and spoof the text (Trojan Source, CWE-451). This is INV-6.
//!
//! `is_bidi_format` is the single workspace definition of the second half of
//! that obligation. It was split out of `gl`'s local copy so the two terminal
//! sanitizers that shipped with only the `Cc` half (#183) — and any future
//! sanitizer — reuse one audited predicate instead of re-deriving the set.
/// True for the Unicode bidirectional/directional-isolate format characters
/// (the reordering subset of category `Cf`) that can visually reorder a terminal
/// line. These are NOT matched by [`char::is_control()`], which covers `Cc` only.
///
/// This is deliberately the reordering subset, not the whole `Cf` category:
/// legitimate `Cf` characters such as ZWJ (`U+200D`, emoji sequences) and the
/// soft hyphen (`U+00AD`) are NOT matched, so honest international text passes
/// through unchanged. Widening this set is a behavior change for every consumer.
///
/// Pair it with `!c.is_control()` in a terminal-bound sanitizer (INV-6); see the
/// wrappers in `gl`, `git-remote-gitlawb`, and `icaptcha-client`.
pub fn is_bidi_format(c: char) -> bool {
matches!(c,
'\u{200E}' | '\u{200F}' | '\u{061C}' // LRM, RLM, ALM
| '\u{202A}'..='\u{202E}' // LRE, RLE, PDF, LRO, RLO
| '\u{2066}'..='\u{2069}' // LRI, RLI, FSI, PDI
)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn strips_every_reordering_code_point() {
// Asserted individually so a partial predicate (e.g. one that covers the
// marks but not the isolates) fails loudly rather than passing.
for c in [
'\u{200E}', '\u{200F}', '\u{061C}', // LRM, RLM, ALM
'\u{202A}', '\u{202B}', '\u{202C}', '\u{202D}',
'\u{202E}', // LRE, RLE, PDF, LRO, RLO
'\u{2066}', '\u{2067}', '\u{2068}', '\u{2069}', // LRI, RLI, FSI, PDI
] {
assert!(
is_bidi_format(c),
"U+{:04X} must be a bidi-format char",
c as u32
);
}
}
#[test]
fn passes_legitimate_and_control_chars() {
// Must NOT over-strip: plain text, a genuine RTL SCRIPT letter (Lo, not a
// format char), ZWJ (legitimate Cf), and soft hyphen (legitimate Cf) are
// not bidi-format. ESC is a Cc control char — owned by is_control(), not
// this predicate, which is bidi-only.
for c in ['a', '\u{0627}', '\u{200D}', '\u{00AD}', '\u{1b}'] {
assert!(
!is_bidi_format(c),
"U+{:04X} must NOT be a bidi-format char",
c as u32
);
}
}
}
+67 -4
View File
@@ -187,11 +187,17 @@ impl Ucan {
}
/// Check if this UCAN grants a specific capability on a resource.
///
/// Mirrors `Capability::is_attenuated_by`'s wildcard semantics: a stored
/// capability of `with: "*"` or `can: "*"` / `"repo/admin"` covers any
/// requested resource/action, since a valid delegation chain can produce
/// exactly that capability (see `is_attenuated_by`).
pub fn can(&self, resource: &str, action: &str) -> bool {
self.payload
.att
.iter()
.any(|cap| cap.with == resource && cap.can == action)
self.payload.att.iter().any(|cap| {
let resource_ok = cap.with == resource || cap.with == "*";
let action_ok = cap.can == action || cap.can == "*" || cap.can == caps::REPO_ADMIN;
resource_ok && action_ok
})
}
/// Encode to a compact JSON string (the wire format).
@@ -312,6 +318,63 @@ mod tests {
assert!(!ucan.can("gitlawb://repos/test/repo", caps::PR_MERGE));
}
#[test]
fn can_honors_resource_wildcard() {
let issuer = Keypair::generate();
let audience = Keypair::generate().did();
let ucan = Ucan::issue(
&issuer,
audience,
vec![Capability::new("*", caps::GIT_PUSH)],
None,
)
.unwrap();
assert!(ucan.can("gitlawb://repos/test/repo", caps::GIT_PUSH));
assert!(ucan.can("gitlawb://repos/other/repo", caps::GIT_PUSH));
assert!(!ucan.can("gitlawb://repos/test/repo", caps::PR_MERGE));
}
#[test]
fn can_honors_repo_admin_action_wildcard() {
let issuer = Keypair::generate();
let audience = Keypair::generate().did();
let ucan = Ucan::issue(
&issuer,
audience,
vec![Capability::new(
"gitlawb://repos/test/repo",
caps::REPO_ADMIN,
)],
None,
)
.unwrap();
assert!(ucan.can("gitlawb://repos/test/repo", caps::GIT_PUSH));
assert!(ucan.can("gitlawb://repos/test/repo", caps::PR_MERGE));
assert!(!ucan.can("gitlawb://repos/other/repo", caps::GIT_PUSH));
}
#[test]
fn can_honors_action_wildcard() {
let issuer = Keypair::generate();
let audience = Keypair::generate().did();
let ucan = Ucan::issue(
&issuer,
audience,
vec![Capability::new("gitlawb://repos/test/repo", "*")],
None,
)
.unwrap();
assert!(ucan.can("gitlawb://repos/test/repo", caps::GIT_PUSH));
assert!(ucan.can("gitlawb://repos/test/repo", caps::PR_MERGE));
assert!(!ucan.can("gitlawb://repos/other/repo", caps::GIT_PUSH));
}
#[test]
fn bootstrap_ucan() {
let issuer = Keypair::generate();
+1 -1
View File
@@ -1,7 +1,7 @@
[package]
name = "gitlawb-node"
description = "The gitlawb node daemon — git hosting over HTTP with DID auth"
version = "0.5.1" # x-release-please-version
version = "0.7.0" # x-release-please-version
edition.workspace = true
rust-version.workspace = true
license.workspace = true
+75 -23
View File
@@ -138,9 +138,33 @@ pub async fn list_ref_updates(
let caller = auth.as_ref().map(|e| e.0 .0.as_str());
let updates = collect_visible_ref_updates(&state.db, None, limit, caller).await?;
// Resolve the trusted display owner_did per row. The stored wire value is
// untrusted (arrives over gossip / unsigned peer notify), so it is echoed
// only when it matches the canonical owner of the local repo the slug
// names (#P1); legacy None rows are attributed via an exact unique local
// match (#P3). Both surfaces share this resolver so they cannot drift.
// The batch resolver issues at most one query per distinct local repo
// rather than one per event row (#P2).
let raw_dids: Vec<Option<String>> = state
.db
.resolve_ref_update_owner_dids(
&updates
.iter()
.map(|u| (u.repo.as_str(), u.owner_did.as_deref()))
.collect::<Vec<_>>(),
)
.await?;
let owner_dids: Vec<serde_json::Value> = raw_dids
.into_iter()
.map(|o| o.map_or(serde_json::Value::Null, |s| serde_json::json!(s)))
.collect();
let events: Vec<serde_json::Value> = updates
.iter()
.map(|u| {
.enumerate()
.map(|(i, u)| {
let owner_did = owner_dids[i].clone();
serde_json::json!({
"id": u.id,
"node_did": u.node_did,
@@ -153,6 +177,7 @@ pub async fn list_ref_updates(
"cert_id": u.cert_id,
"received_at": u.received_at,
"from_peer": u.from_peer,
"owner_did": owner_did,
})
})
.collect();
@@ -225,6 +250,7 @@ pub async fn list_repo_events(
"pusher_did": c.pusher_did,
"node_did": c.node_did,
"timestamp": c.issued_at,
"owner_did": record.owner_did,
"source": "local",
})
})
@@ -240,28 +266,46 @@ pub async fn list_repo_events(
// collect_visible_ref_updates drops any row whose slug matches a local repo the
// caller cannot read (fail-closed), and propagates DB errors instead of swallowing
// them.
let gossip_events: Vec<serde_json::Value> =
collect_visible_ref_updates(&state.db, Some(&repo_id_str), limit, caller)
.await?
.iter()
.map(|u| {
serde_json::json!({
"type": "gossipsub",
"id": u.id,
"repo": u.repo,
"ref_name": u.ref_name,
"old_sha": u.old_sha,
"new_sha": u.new_sha,
"pusher_did": u.pusher_did,
"node_did": u.node_did,
"timestamp": u.timestamp,
"cert_id": u.cert_id,
"received_at": u.received_at,
"from_peer": u.from_peer,
"source": "gossipsub",
})
let gossip_updates =
collect_visible_ref_updates(&state.db, Some(&repo_id_str), limit, caller).await?;
let gossip_raw: Vec<Option<String>> = state
.db
.resolve_ref_update_owner_dids(
&gossip_updates
.iter()
.map(|u| (u.repo.as_str(), u.owner_did.as_deref()))
.collect::<Vec<_>>(),
)
.await?;
let gossip_owner_dids: Vec<serde_json::Value> = gossip_raw
.into_iter()
.map(|o| o.map_or(serde_json::Value::Null, |s| serde_json::json!(s)))
.collect();
let gossip_events: Vec<serde_json::Value> = gossip_updates
.iter()
.enumerate()
.map(|(i, u)| {
let owner_did = gossip_owner_dids[i].clone();
serde_json::json!({
"type": "gossipsub",
"id": u.id,
"repo": u.repo,
"ref_name": u.ref_name,
"old_sha": u.old_sha,
"new_sha": u.new_sha,
"pusher_did": u.pusher_did,
"node_did": u.node_did,
"timestamp": u.timestamp,
"cert_id": u.cert_id,
"received_at": u.received_at,
"from_peer": u.from_peer,
"owner_did": owner_did,
"source": "gossipsub",
})
.collect();
})
.collect();
// Merge both lists
let mut all_events: Vec<serde_json::Value> = cert_events;
@@ -326,6 +370,7 @@ mod ref_updates_feed_tests {
cert_id: None,
received_at: Utc::now().to_rfc3339(),
from_peer: "peer1".into(),
owner_did: None,
}
}
@@ -1105,7 +1150,14 @@ mod ref_updates_feed_tests {
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
assert_eq!(count(&body_json(resp).await), 2);
let body = body_json(resp).await;
assert_eq!(count(&body), 2);
for event in body["events"].as_array().unwrap() {
assert_eq!(
event["owner_did"], OWNER,
"each event must carry the local owner_did"
);
}
}
// Anon reads a quarantined mirror → 404 (withheld without disclosing existence
File diff suppressed because it is too large. Load diff
+2 -172
View File
@@ -61,14 +61,11 @@ pub async fn create_issue(
let json_str = serde_json::to_string(&issue)
.map_err(|e| AppError::BadRequest(format!("serialization error: {e}")))?;
// Shed 503 + Retry-After on an exhausted write-lock POOL instead of a generic
// git 500 (#173 F1). This path holds no admission permit, so it reaches the pool
// unthrottled; reuse the push handler's mapping so the two cannot drift.
let guard = state
.repo_store
.acquire_write(&record.owner_did, &record.name)
.await
.map_err(|e| crate::api::repos::acquire_write_app_error(&e, &repo))?;
.map_err(|e| AppError::Git(e.to_string()))?;
let disk_path = guard.path().to_path_buf();
let create_result = git_issues::create_issue(&disk_path, &issue_id, &json_str);
@@ -232,13 +229,11 @@ pub async fn close_issue(
.await?
.ok_or_else(|| AppError::RepoNotFound(format!("{owner}/{repo}")))?;
// Same capacity shed as create_issue above (#173 F1): an exhausted write-lock
// pool is a 503 + Retry-After, not a 500 git error.
let guard = state
.repo_store
.acquire_write(&record.owner_did, &record.name)
.await
.map_err(|e| crate::api::repos::acquire_write_app_error(&e, &repo))?;
.map_err(|e| AppError::Git(e.to_string()))?;
let disk_path = guard.path().to_path_buf();
// Owner OR issue author may close. The author lives in the issue's git-JSON
@@ -284,168 +279,3 @@ pub async fn close_issue(
Ok(Json(issue))
}
/// #173 F1 follow-up: the two issue write paths reach `acquire_write` holding NO
/// admission permit (unlike the push handler, which is capped by the git-push
/// semaphore), so they are the callers most likely to meet an exhausted write-lock
/// POOL under load. An exhausted pool is a capacity signal, so both must shed
/// 503 + Retry-After (`AppError::Overloaded`) the way the push handler does, not
/// report the generic 500 git error that says nothing about retrying.
#[cfg(test)]
mod lock_pool_shed_tests {
use super::*;
use axum::response::IntoResponse;
use sqlx::PgPool;
fn seed_repo(owner_did: &str, name: &str) -> crate::db::RepoRecord {
let now = Utc::now();
crate::db::RepoRecord {
id: Uuid::new_v4().to_string(),
name: name.to_string(),
owner_did: owner_did.to_string(),
description: None,
is_public: true,
default_branch: "main".to_string(),
created_at: now,
updated_at: now,
disk_path: format!("/tmp/{name}"),
forked_from: None,
machine_id: None,
}
}
/// State whose repo store draws write locks from a ONE-connection pool with a
/// short checkout timeout, so a single held guard exhausts it promptly rather
/// than at the pool default.
async fn one_connection_lock_pool_state(pool: &PgPool) -> AppState {
let mut state = crate::test_support::test_state(pool.clone()).await;
state.repo_store = crate::git::repo_store::RepoStore::new(
std::path::PathBuf::from("/tmp/gitlawb-issues-lockpool"),
None,
crate::git::repo_store::build_lock_pool(pool, 1, std::time::Duration::from_secs(1)),
);
state
}
/// The shed must be a real 503 carrying Retry-After, not just an internal enum
/// variant: assert on the rendered response so a remapping of `Overloaded` is
/// caught here too.
fn assert_sheds_503_with_retry_after(err: AppError, what: &str) {
let resp = err.into_response();
assert_eq!(
resp.status(),
StatusCode::SERVICE_UNAVAILABLE,
"{what}: an exhausted write-lock pool must shed 503, not a 500 git error"
);
assert_eq!(
resp.headers()
.get("retry-after")
.map(|v| v.to_str().unwrap()),
Some("1"),
"{what}: a capacity shed must tell the client when to retry"
);
}
/// RED-before/GREEN-after for `create_issue`. Both directions: the shed while the
/// only lock-pool connection is held by a guard on a DIFFERENT repo (so this is
/// pool capacity, not advisory-lock contention on this repo), and the must-not
/// case once that connection is back.
#[sqlx::test]
async fn create_issue_lock_pool_exhaustion_sheds_503_not_500(pool: PgPool) {
let owner = "did:key:zISSUECREATELOCKPOOLAAAAAAAAAAAAAAAAAAAA";
let state = one_connection_lock_pool_state(&pool).await;
state
.db
.create_repo(&seed_repo(owner, "lp-create"))
.await
.expect("seed repo");
let held = state
.repo_store
.acquire_write(owner, "other-repo")
.await
.expect("the first write takes the only lock-pool connection");
let shed = create_issue(
State(state.clone()),
Extension(AuthenticatedDid(owner.to_string())),
Path((owner.to_string(), "lp-create".to_string())),
Json(CreateIssueRequest {
title: "t".to_string(),
body: None,
signed_payload: None,
}),
)
.await;
let err = shed.expect_err("an exhausted lock pool must fail the call");
assert_sheds_503_with_retry_after(err, "create_issue");
// MUST-NOT: with the pool free again the call is not shed as capacity (it
// fails later on the nonexistent on-disk repo, which is a git 500).
held.release(false).await;
let admitted = create_issue(
State(state.clone()),
Extension(AuthenticatedDid(owner.to_string())),
Path((owner.to_string(), "lp-create".to_string())),
Json(CreateIssueRequest {
title: "t".to_string(),
body: None,
signed_payload: None,
}),
)
.await;
assert!(
!matches!(admitted, Err(AppError::Overloaded(_))),
"with the lock pool free, create_issue must not be shed as capacity; got {:?}",
admitted.err()
);
}
/// RED-before/GREEN-after for `close_issue`, same two directions.
#[sqlx::test]
async fn close_issue_lock_pool_exhaustion_sheds_503_not_500(pool: PgPool) {
let owner = "did:key:zISSUECLOSELOCKPOOLBBBBBBBBBBBBBBBBBBBBB";
let state = one_connection_lock_pool_state(&pool).await;
state
.db
.create_repo(&seed_repo(owner, "lp-close"))
.await
.expect("seed repo");
let held = state
.repo_store
.acquire_write(owner, "other-repo")
.await
.expect("the first write takes the only lock-pool connection");
let shed = close_issue(
State(state.clone()),
Extension(AuthenticatedDid(owner.to_string())),
Path((
owner.to_string(),
"lp-close".to_string(),
"deadbeef".to_string(),
)),
)
.await;
let err = shed.expect_err("an exhausted lock pool must fail the call");
assert_sheds_503_with_retry_after(err, "close_issue");
held.release(false).await;
let admitted = close_issue(
State(state.clone()),
Extension(AuthenticatedDid(owner.to_string())),
Path((
owner.to_string(),
"lp-close".to_string(),
"deadbeef".to_string(),
)),
)
.await;
assert!(
!matches!(admitted, Err(AppError::Overloaded(_))),
"with the lock pool free, close_issue must not be shed as capacity; got {:?}",
admitted.err()
);
}
}
+4
View File
@@ -71,6 +71,10 @@ pub async fn remove_label(
}
/// GET /api/v1/repos/:owner/:repo/labels
///
/// Read-visibility-gated (INV-2 root listing): a public repo's labels stay
/// anonymously listable; a private repo's label names are hidden (404) from
/// anyone who cannot read it at the root.
pub async fn list_labels(
State(state): State<AppState>,
Path((owner, name)): Path<(String, String)>,
+18 -26
View File
@@ -172,6 +172,7 @@ mod authz_guard {
let issues = include_str!("issues.rs");
let bounties = include_str!("bounties.rs");
let replicas = include_str!("replicas.rs");
let events = include_str!("events.rs");
let tasks = include_str!("tasks.rs");
let stars = include_str!("stars.rs");
let protect = include_str!("protect.rs");
@@ -187,6 +188,13 @@ mod authz_guard {
(pulls, "merge_pr", "require_repo_owner("),
(webhooks, "create_webhook", "require_repo_owner("),
(webhooks, "delete_webhook", "require_repo_owner("),
// Bucket A/B hybrid — list_webhooks is read-visibility THEN owner:
// authorize_repo_read 404s a non-reader of a private repo, then
// require_repo_owner 403s a non-owner of a public one. Both halves
// are pinned: the authorize_repo_read marker guards the read half
// (existence hiding) and require_repo_owner guards the owner half.
(webhooks, "list_webhooks", "authorize_repo_read("),
(webhooks, "list_webhooks", "require_repo_owner("),
(labels, "add_label", "require_repo_owner("),
(labels, "remove_label", "require_repo_owner("),
// Bucket A' — owner OR author (did_matches against the author)
@@ -200,11 +208,15 @@ mod authz_guard {
(issues, "create_issue", "authorize_repo_read("),
(bounties, "create_bounty", "authorize_repo_read("),
(repos, "fork_repo", "authorize_repo_read("),
// get_by_cid resolves each candidate (provenance path + legacy scan) through
// the shared `gate_and_serve` (#173 round 2); the gate markers themselves are
// asserted below. This row proves the delegation is real — the gate is
// actually reached, not dead code.
(ipfs, "get_by_cid", "gate_and_serve("),
// get_by_cid gates each iterated repo row directly via visibility_check
// (KTD2a: it must NOT route through authorize_repo_read's fuzzy re-resolve).
(ipfs, "get_by_cid", "visibility_check("),
// #94 sibling read surfaces: gate private-repo metadata on read
// visibility (public repos stay anonymous; private repos 404).
(replicas, "list_replicas", "authorize_repo_read("),
(protect, "list_protected_branches", "authorize_repo_read("),
(labels, "list_labels", "authorize_repo_read("),
(events, "list_repo_events", "authorize_repo_read("),
// Bucket C — signer-self: the acting DID is matched/bound to auth.0
(tasks, "create_task", "did_matches("),
(tasks, "claim_task", "did_matches("),
@@ -238,22 +250,6 @@ mod authz_guard {
"visibility::require_owner must use did_matches for DID-safe owner matching"
);
// The CID read surface (#173) enforces its gate inside the shared
// `gate_and_serve`, which BOTH the provenance path and the legacy scan call, so
// the markers must live there (the get_by_cid row above only proves delegation).
// The repo's own "/" visibility check (KTD2a — never authorize_repo_read's fuzzy
// re-resolve) and the quarantine hard-drop BEFORE visibility (INV-11) are both
// load-bearing: removing either re-opens a leak on the provenance path.
let gate_body = fn_body(ipfs, "gate_and_serve");
assert!(
gate_body.contains("visibility_check("),
"gate_and_serve must gate the CID read surface via visibility_check (KTD2a)"
);
assert!(
gate_body.contains("if quarantined"),
"gate_and_serve must hard-drop a quarantined repo before the visibility gate (INV-11)"
);
for (src, func, marker) in rows {
let body = fn_body(src, func);
assert!(
@@ -527,11 +523,7 @@ mod authz_guard {
// Repo-scoped reads known to be ungated today, each tracked by an issue.
// Remove an entry the moment its gate lands (the staleness assert enforces it).
let known_ungated: &[(&str, &str)] = &[
("list_webhooks", "#94 (PR #113)"),
("list_replicas", "PR #113"),
("list_protected_branches", "PR #113"),
];
let known_ungated: &[(&str, &str)] = &[];
let is_known = |n: &str| known_ungated.iter().any(|(k, _)| *k == n);
// Any one of these = the handler binds the caller to an authz decision: the
// first two are read gates, the rest are the write/owner/self-binding forms.
+5
View File
@@ -347,6 +347,10 @@ pub struct NotifyRequest {
pub timestamp: Option<String>,
#[serde(default)]
pub cert_id: Option<String>,
/// Full owner DID — added in #144 for DID-aware feed gating.
/// Optional for backward compat with older senders.
#[serde(default)]
pub owner_did: Option<String>,
}
pub async fn notify_sync(
@@ -391,6 +395,7 @@ pub async fn notify_sync(
node_did: req.node_did.clone(),
pusher_did: req.pusher_did.clone().unwrap_or_default(),
repo: req.repo.clone(),
owner_did: req.owner_did.clone(),
ref_name: req.ref_name.clone(),
old_sha: req.old_sha.clone().unwrap_or_default(),
new_sha: req.new_sha.clone(),
+7 -5
View File
@@ -79,12 +79,14 @@ pub async fn unprotect_branch(
pub async fn list_protected_branches(
State(state): State<AppState>,
Path((owner, repo)): Path<(String, String)>,
auth: Option<Extension<AuthenticatedDid>>,
) -> Result<Json<serde_json::Value>> {
let record = state
.db
.get_repo(&owner, &repo)
.await?
.ok_or_else(|| AppError::RepoNotFound(format!("{owner}/{repo}")))?;
// Read-visibility-gated (INV-2 root listing): a public repo's protected
// branches stay anonymously listable; a private repo's branch names are
// hidden (404) from anyone who cannot read it at the root.
let caller = auth.as_ref().map(|e| e.0 .0.as_str());
let (record, _rules) =
crate::api::authorize_repo_read(&state, &owner, &repo, caller, "/").await?;
let branches = state.db.list_protected_branches(&record.id).await?;
+1 -117
View File
@@ -209,14 +209,11 @@ pub async fn merge_pr(
return Err(AppError::BadRequest(format!("PR is already {}", pr.status)));
}
// Shed 503 + Retry-After on an exhausted write-lock POOL instead of a generic
// git 500 (#173 F1). Merging holds no admission permit, so it reaches the pool
// unthrottled; reuse the push handler's mapping so the two cannot drift.
let guard = state
.repo_store
.acquire_write(&record.owner_did, &record.name)
.await
.map_err(|e| crate::api::repos::acquire_write_app_error(&e, &name))?;
.map_err(|e| AppError::Git(e.to_string()))?;
let disk_path = guard.path().to_path_buf();
let merger_did = auth.0;
let merge_result = store::merge_branch(
@@ -427,116 +424,3 @@ pub async fn list_comments(
let comments = state.db.list_pr_comments(&pr.id).await?;
Ok(Json(serde_json::json!({ "comments": comments })))
}
/// #173 F1 follow-up: `merge_pr` reaches `acquire_write` holding NO admission permit
/// (unlike the push handler, which is capped by the git-push semaphore), so it is one
/// of the callers most likely to meet an exhausted write-lock POOL under load. An
/// exhausted pool is a capacity signal, so the merge must shed 503 + Retry-After
/// (`AppError::Overloaded`) the way the push handler does, not report the generic
/// 500 git error that says nothing about retrying.
#[cfg(test)]
mod lock_pool_shed_tests {
use super::*;
use axum::response::IntoResponse;
use sqlx::PgPool;
fn seed_repo(owner_did: &str, name: &str) -> crate::db::RepoRecord {
let now = Utc::now();
crate::db::RepoRecord {
id: Uuid::new_v4().to_string(),
name: name.to_string(),
owner_did: owner_did.to_string(),
description: None,
is_public: true,
default_branch: "main".to_string(),
created_at: now,
updated_at: now,
disk_path: format!("/tmp/{name}"),
forked_from: None,
machine_id: None,
}
}
/// RED-before/GREEN-after for `merge_pr`. Both directions: the shed while the only
/// lock-pool connection is held by a guard on a DIFFERENT repo (so this is pool
/// capacity, not advisory-lock contention on this repo), and the must-not case
/// once that connection is back.
#[sqlx::test]
async fn merge_pr_lock_pool_exhaustion_sheds_503_not_500(pool: PgPool) {
let owner = "did:key:zMERGELOCKPOOLOWNERAAAAAAAAAAAAAAAAAAAAA";
let mut state = crate::test_support::test_state(pool.clone()).await;
// One lock-pool connection with a short checkout timeout, so a single held
// guard exhausts it promptly rather than at the pool default.
state.repo_store = crate::git::repo_store::RepoStore::new(
std::path::PathBuf::from("/tmp/gitlawb-pulls-lockpool"),
None,
crate::git::repo_store::build_lock_pool(&pool, 1, std::time::Duration::from_secs(1)),
);
let repo = seed_repo(owner, "lp-merge");
let repo_id = repo.id.clone();
state.db.create_repo(&repo).await.expect("seed repo");
let now = Utc::now().to_rfc3339();
state
.db
.create_pr(&PullRequest {
id: Uuid::new_v4().to_string(),
repo_id: repo_id.clone(),
number: 1,
title: "lp".to_string(),
body: None,
author_did: owner.to_string(),
source_branch: "feature".to_string(),
target_branch: "main".to_string(),
status: "open".to_string(),
merged_by_did: None,
merged_at: None,
created_at: now.clone(),
updated_at: now,
})
.await
.expect("seed open PR");
let held = state
.repo_store
.acquire_write(owner, "other-repo")
.await
.expect("the first write takes the only lock-pool connection");
let shed = merge_pr(
State(state.clone()),
Extension(AuthenticatedDid(owner.to_string())),
Path((owner.to_string(), "lp-merge".to_string(), 1)),
)
.await;
let err = shed.expect_err("an exhausted lock pool must fail the call");
let resp = err.into_response();
assert_eq!(
resp.status(),
StatusCode::SERVICE_UNAVAILABLE,
"merge_pr: an exhausted write-lock pool must shed 503, not a 500 git error"
);
assert_eq!(
resp.headers()
.get("retry-after")
.map(|v| v.to_str().unwrap()),
Some("1"),
"merge_pr: a capacity shed must tell the client when to retry"
);
// MUST-NOT: with the pool free again the merge is not shed as capacity (it
// fails later on the nonexistent on-disk repo, which is a git 500).
held.release(false).await;
let admitted = merge_pr(
State(state.clone()),
Extension(AuthenticatedDid(owner.to_string())),
Path((owner.to_string(), "lp-merge".to_string(), 1)),
)
.await;
assert!(
!matches!(admitted, Err(AppError::Overloaded(_))),
"with the lock pool free, merge_pr must not be shed as capacity; got {:?}",
admitted.err()
);
}
}
+7 -6
View File
@@ -113,16 +113,17 @@ pub async fn unregister_replica(
}
/// GET /api/v1/repos/:owner/:repo/replicas
/// Public — returns the list of replicas (DID + URL + registration timestamp).
/// Read-visibility-gated: a PUBLIC repo's replica list stays anonymously
/// listable (mirror-discovery), but a PRIVATE repo's replica URLs are hidden
/// (404) from anyone who cannot read the repo at its root.
pub async fn list_replicas(
State(state): State<AppState>,
Path((owner, repo)): Path<(String, String)>,
auth: Option<Extension<AuthenticatedDid>>,
) -> Result<Json<serde_json::Value>> {
let record = state
.db
.get_repo(&owner, &repo)
.await?
.ok_or_else(|| AppError::RepoNotFound(format!("{owner}/{repo}")))?;
let caller = auth.as_ref().map(|e| e.0 .0.as_str());
let (record, _rules) =
crate::api::authorize_repo_read(&state, &owner, &repo, caller, "/").await?;
let replicas = state.db.list_replicas(&record.id).await?;
File diff suppressed because it is too large. Load diff
+22 -8
View File
@@ -1,8 +1,8 @@
//! Webhook CRUD API.
//!
//! POST /api/v1/repos/:owner/:repo/hooks — create (auth required)
//! GET /api/v1/repos/:owner/:repo/hooks — list
//! DELETE /api/v1/repos/:owner/:repo/hooks/:id — delete (auth required)
//! POST /api/v1/repos/:owner/:repo/hooks — create (owner only)
//! GET /api/v1/repos/:owner/:repo/hooks — list (owner only; auth required)
//! DELETE /api/v1/repos/:owner/:repo/hooks/:id — delete (owner only)
use axum::extract::{Extension, Path, State};
use axum::http::StatusCode;
@@ -73,12 +73,26 @@ pub async fn create_webhook(
pub async fn list_webhooks(
State(state): State<AppState>,
Path((owner, name)): Path<(String, String)>,
auth: Option<Extension<AuthenticatedDid>>,
) -> Result<Json<serde_json::Value>> {
let record = state
.db
.get_repo(&owner, &name)
.await?
.ok_or_else(|| AppError::RepoNotFound(format!("{owner}/{name}")))?;
// This route sits on `optional_signature`, so the DID is optional. Webhook
// callback URLs are owner-secret config and there is no anonymous form, so a
// headerless caller is rejected before any lookup (401, which fires for an
// existing-private and an absent repo alike, so it leaks no existence).
let Some(Extension(AuthenticatedDid(caller))) = auth else {
return Err(AppError::Unauthorized(
"listing webhooks requires authentication".into(),
));
};
// Read-visibility first, then owner. authorize_repo_read returns 404 on a
// visibility deny, so a non-reader of a private repo cannot tell it exists
// (uniform with the sibling read surfaces); require_repo_owner then yields
// 403 only for a non-owner of a public/readable repo, where existence is not
// secret.
let (record, _rules) =
crate::api::authorize_repo_read(&state, &owner, &name, Some(&caller), "/").await?;
crate::api::require_repo_owner(&record, &caller)?;
let hooks = state.db.list_webhooks(&record.id).await?;
// Redact secrets in list response
-18
View File
@@ -516,28 +516,10 @@ mod tests {
rate_limiter: RateLimiter::new(100, Duration::from_secs(60)),
create_ip_rate_limiter: RateLimiter::new(1000, Duration::from_secs(3600)),
push_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
ipfs_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
ipfs_work_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
ipfs_max_history_walks: crate::api::ipfs::MAX_HISTORY_WALKS_PER_REQUEST,
ipfs_max_legacy_probes: crate::api::ipfs::MAX_LEGACY_PROBES_PER_REQUEST,
ipfs_max_served_object_bytes: crate::api::ipfs::MAX_SERVED_OBJECT_BYTES,
push_limiter_trust: crate::rate_limit::TrustedProxy::None,
sync_trigger_rate_limiter: RateLimiter::new(60, Duration::from_secs(3600)),
peer_write_rate_limiter: RateLimiter::new(600, Duration::from_secs(3600)),
shutdown_tx: tokio::sync::watch::channel(false).0,
git_read_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_write_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_push_advert_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_encrypt_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
encrypt_inflight: crate::state::EncryptInflight::new(),
git_read_per_caller: crate::rate_limit::PerCallerConcurrency::with_default_max_keys(16),
git_push_advert_per_caller:
crate::rate_limit::PerCallerConcurrency::with_default_max_keys(8),
git_write_per_caller: crate::rate_limit::PerCallerConcurrency::with_default_max_keys(8),
git_ipfs_walk_semaphore: Arc::new(tokio::sync::Semaphore::new(64)),
git_ipfs_walk_per_caller:
crate::rate_limit::PerCallerConcurrency::with_default_max_keys(16),
git_bin: "git".to_string(),
}
}
+3 -502
View File
@@ -172,9 +172,9 @@ pub struct Config {
/// its process group torn down, in seconds. Bounds a git that neither
/// finishes nor disconnects. Must be positive; set it very large to
/// effectively disable the bound. Default: 600s (10 min), generous for large
/// clones. Also bounds the ref advertisement (`info/refs`) and the withheld-blob
/// pack build (`upload_pack_excluding`'s pack-objects stage), which now share the
/// same timeout + process-group teardown (#174).
/// clones. Does not cover the ref advertisement (`info/refs`) or the
/// withheld-blob fetch path (`upload_pack_excluding`, a blocking
/// `spawn_blocking` a tokio timeout cannot cancel); both remain unbounded.
#[arg(
long,
env = "GITLAWB_GIT_SERVICE_TIMEOUT_SECS",
@@ -183,27 +183,6 @@ pub struct Config {
)]
pub git_service_timeout_secs: u64,
/// Maximum wall-clock time the storage-acquisition phase of a served git
/// operation may run before the request is shed with a 503, in seconds. This
/// bounds `RepoStore::{acquire,acquire_fresh,acquire_write}` — the Tigris
/// HEAD/GET on a read/advert acquire and the advisory-lock retry loop (incl. a
/// per-iteration `pg_try_advisory_lock` that can block on a hung Postgres pool)
/// on a write acquire. A concurrency permit is taken BEFORE this phase, and
/// `git_service_timeout_secs` only starts once git spawns, so without this the
/// acquire phase is unbounded: a stalled backend pins the permit and drains the
/// pool until every later request 503s. On expiry the permit is released and a
/// bounded 503 + Retry-After is returned (fail-closed). Kept separate from
/// `git_service_timeout_secs` because acquisition and git execution are distinct
/// cost centers — one shared budget would let a slow acquire starve git. Must be
/// positive; set it very large to effectively disable the bound. Default: 30s.
#[arg(
long,
env = "GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS",
default_value_t = 30,
value_parser = clap::value_parser!(u64).range(1..)
)]
pub git_acquire_timeout_secs: u64,
/// Maximum connections in the PostgreSQL pool. This is a cap, not a floor
/// (connections open lazily). Size against the database server's
/// max_connections, remembering admin tooling opens its own pool.
@@ -255,198 +234,6 @@ pub struct Config {
value_parser = clap::value_parser!(u64).range(1..)
)]
pub db_retry_max_secs: u64,
/// Maximum number of served git operations (upload-pack / receive-pack /
/// info-refs) allowed to run concurrently. Beyond this the node sheds the
/// request with a clean 503 + Retry-After instead of spawning another git
/// subprocess and risking PID/thread exhaustion. Portable backstop: the
/// compose `pids_limit` is not present on Fly, whose connection-concurrency
/// cap is a different axis (500 connections each fan out to git +
/// pack-objects + threads). Size below the process budget with headroom.
///
/// This is the READ pool (`git_read_semaphore`): upload-pack and the UPLOAD-PACK
/// `info/refs` advertisement only. The authenticated push POST draws from a
/// separate write pool (`max_concurrent_git_pushes`) that anonymous reads can
/// never reach, and each read caller is additionally bounded by
/// `max_concurrent_reads_per_caller`, so an anonymous flood cannot shed the actual
/// push nor monopolize reads (#174). The anon-reachable RECEIVE-PACK `info/refs`
/// advertisement draws from its OWN dedicated pool (sized like the write pool but
/// disjoint), so an advertisement flood can never occupy a permit the
/// authenticated push POST needs at admission (#174).
///
/// A permit is held for the whole op. Every git subprocess that STREAMS is
/// duration-bounded and reaps its process group on disconnect: upload-pack,
/// receive-pack, and both info/refs advertisements run under
/// `git_service_timeout_secs` with `process_group(0)` teardown, and the
/// withheld-blob (`upload_pack_excluding`) pack-objects stage plus the push-side
/// candidate-discovery children (`rev-list` / `cat-file`) now run under the same
/// bounded runner with process-group teardown, so a stuck git child no longer
/// holds its slot indefinitely (#174 closed the duration/cancellation gaps this
/// comment previously tracked).
///
/// Default: 128. Must be between 1 and 1_048_576; the ceiling keeps the value
/// well under tokio's `Semaphore` permit limit so an oversized value is a
/// clean CLI error rather than a boot-time panic.
#[arg(
long,
env = "GITLAWB_MAX_CONCURRENT_GIT_OPS",
default_value_t = 128,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub max_concurrent_git_ops: usize,
/// Maximum number of concurrent `git-receive-pack` (push) operations. The
/// authenticated push POST draws from this dedicated pool, separate from
/// `max_concurrent_git_ops` (reads), so a flood of anonymous reads cannot shed an
/// authenticated push at admission (#174). The anon-reachable receive-pack
/// `info/refs` advertisement runs in a SEPARATE pool of the same size (derived
/// from this knob), disjoint from this one, so an advertisement flood cannot
/// occupy a POST's slot either (#174). Beyond this a push sheds a clean 503 +
/// Retry-After.
///
/// Default: 32. Must be between 1 and 1_048_576 (the ceiling keeps the value
/// under tokio's `Semaphore` permit limit so an oversized value is a clean CLI
/// error rather than a boot-time panic).
///
/// CONNECTION BUDGET. A push holds a Postgres connection from the node's separate
/// advisory-lock pool for the whole receive-pack, and that pool is sized from this
/// knob (this value + 8, clamped to 64 in `main.rs`). The node's total ceiling is
/// therefore `db_max_connections` (default 20) + the lock pool (default 40), i.e.
/// 60 by default, and at most `db_max_connections` + 64. Size BOTH against the
/// database server's `max_connections`: `db_max_connections`' own doc predates the
/// lock pool and no longer covers most of the node's connections. The +8 headroom
/// is shared with the three non-push `acquire_write` callers (`api/issues.rs` x2,
/// `api/pulls.rs`). Raising this knob past the clamp does NOT buy more lock-pool
/// connections; pushes beyond it wait briefly and then shed a 503 + Retry-After.
#[arg(
long,
env = "GITLAWB_MAX_CONCURRENT_GIT_PUSHES",
default_value_t = 32,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub max_concurrent_git_pushes: usize,
/// Maximum concurrent read operations (`upload-pack` and the upload-pack
/// `info/refs` advertisement) a single caller may hold at once, so one caller
/// cannot monopolize the `max_concurrent_git_ops` read pool (#174). Callers are
/// keyed on the RESOLVED SOURCE IP, never the DID — a signature does not move a
/// caller off this cap, so an authenticated client cannot mint DIDs to escape it.
/// IMPORTANT: the source-IP key is only as granular as `GITLAWB_TRUSTED_PROXY`.
/// Left unset (the default), a node behind an edge/NAT keys all callers on the
/// edge IP, so this cap collapses to a single global cap rather than per-client.
/// Set `GITLAWB_TRUSTED_PROXY` to key on the real client; a high-fanout caller (a
/// CI fleet behind one NAT) then needs the operator to raise this. Over-cap for a
/// caller sheds a clean 503 + Retry-After.
///
/// Default: 16. Must be between 1 and 1_048_576.
#[arg(
long,
env = "GITLAWB_MAX_CONCURRENT_READS_PER_CALLER",
default_value_t = 16,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub max_concurrent_reads_per_caller: usize,
/// Maximum number of concurrent `GET /ipfs/{cid}` requests that may run their
/// visibility walk at once. The publicly-reachable `/ipfs/{cid}` route runs
/// `allowed_blob_set_for_caller_bounded` in `spawn_blocking` — a full-history
/// git walk (up to `git_service_timeout_secs`) — for each candidate repo. It
/// draws from THIS pool, not any served-git pool: a distinct public cost center
/// on a distinct surface, so sharing a git pool would let anonymous /ipfs
/// traffic shed authenticated git ops (the auth-boundary trap). A permit is
/// held for the whole request (across the repo loop) so it reflects real
/// blocking-thread occupancy, not merely the tokio wait. Beyond this the request
/// sheds a clean 503 + Retry-After. Must be between 1 and 1_048_576; the ceiling
/// keeps the value under tokio's `Semaphore` permit limit so an oversized value
/// is a clean CLI error rather than a boot-time panic. Default: 32.
#[arg(
long,
env = "GITLAWB_MAX_CONCURRENT_IPFS_WALKS",
default_value_t = 32,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub max_concurrent_ipfs_walks: usize,
/// Maximum concurrent `/ipfs/{cid}` walk requests a single source may hold at
/// once, so one source cannot monopolize `max_concurrent_ipfs_walks` (#174).
/// Callers are keyed on the RESOLVED SOURCE IP (`client_key`/`GITLAWB_TRUSTED_PROXY`),
/// never the DID — `/ipfs` accepts any `did:key` via `optional_signature` with no
/// admission step, so keying on the DID would let one host mint disposable DIDs to
/// multiply its budget. A request with no resolvable key (no trusted header, no
/// peer) is bounded by the global pool only, never this sub-cap. Over-cap sheds a
/// clean 503 + Retry-After. Must be between 1 and 1_048_576. Default: 4.
#[arg(
long,
env = "GITLAWB_IPFS_WALK_PER_SOURCE",
default_value_t = 4,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub ipfs_walk_per_source: usize,
/// Per-request ceiling on the number of legacy (NULL-provenance) repos the
/// `/ipfs/{cid}` resolver's scan fallback will PROBE (`acquire` + `git cat-file
/// -t`) before giving up. The provenance path targets one repo; the legacy scan,
/// absent this bound, fans one anonymous request out to O(repos) subprocess spawns
/// and cold-cache fetches for a CID enumerable from the public pins index (#173,
/// INV-10). A truncated scan surfaces as a retryable 503, never a false 404. Wired
/// into `AppState::ipfs_max_legacy_probes` at construction; the history-walk ceiling
/// stays constant (`MAX_HISTORY_WALKS_PER_REQUEST`) and is NOT governed by this knob
/// (a smaller value would falsely 503 a provenanced request). Must be between 1 and
/// 1_048_576. Default: 256.
#[arg(
long,
env = "GITLAWB_IPFS_MAX_REPOS_WALKED",
default_value_t = crate::api::ipfs::MAX_LEGACY_PROBES_PER_REQUEST as usize,
value_parser = clap::builder::RangedU64ValueParser::<usize>::new().range(1..=1_048_576)
)]
pub ipfs_max_repos_walked: usize,
/// Per-client-IP rate limit for `GET /ipfs/{cid}`, in requests per hour. The
/// route is publicly reachable (`optional_signature`) and each request can drive
/// a full-history git walk, so it carries a per-IP flood brake in addition to the
/// concurrency cap above (a rate limit bounds request *rate*, the semaphore
/// bounds concurrent slow holds — different axes). Keyed on the resolved client
/// IP via `GITLAWB_TRUSTED_PROXY`. `0` disables. Default: 600.
///
/// This is the pure once-per-request ROUTE brake. The resolver's internal
/// per-probe/per-walk WORK budget is a SEPARATE bucket whose capacity is DERIVED
/// from this value (`AppState::ipfs_work_budget`), not a knob of its own; `0` here
/// disables that derived bucket too.
#[arg(long, env = "GITLAWB_IPFS_RATE_LIMIT", default_value_t = 600)]
pub ipfs_rate_limit: usize,
/// Rows the legacy provider-CID repair sweep reads per batch (U4, #173).
///
/// The sweep walks every `pinned_cids` row on the node once, repairing rows that
/// releases before this branch keyed on a PROVIDER CID (Kubo dag-pb / Pinata CIDv0)
/// instead of the raw-content resolver key. This bounds one batch, so the sweep can
/// never turn into a single unbounded table scan competing with request traffic.
/// Conservative on purpose: paired with the inter-batch delay below the default is
/// ~64 rows per minute, which finishes a large pin set in hours of idle background
/// work rather than one expensive burst. Must be between 1 and 100_000.
#[arg(
long,
env = "GITLAWB_PIN_REPAIR_SWEEP_BATCH",
default_value_t = 64,
value_parser = clap::builder::RangedU64ValueParser::<i64>::new().range(1..=100_000)
)]
pub pin_repair_sweep_batch: i64,
/// Seconds the legacy provider-CID repair sweep sleeps between batches (U4, #173).
///
/// Each batch costs an indexed range scan plus, for the legacy rows in it, a
/// `git cat-file` per row. The delay is what keeps that off the DB's and the disk's
/// critical path: the sweep is repairing rows that have been unresolvable since the
/// upgrade, so finishing slowly is fine and finishing fast at the cost of live
/// traffic is not. `0` disables the pause (test and one-off operational use only).
/// Must be between 0 and 86_400.
#[arg(
long,
env = "GITLAWB_PIN_REPAIR_SWEEP_DELAY_SECS",
default_value_t = 60,
value_parser = clap::builder::RangedU64ValueParser::<u64>::new().range(0..=86_400)
)]
pub pin_repair_sweep_delay_secs: u64,
}
impl Config {
@@ -485,290 +272,4 @@ mod tests {
Config::try_parse_from(["gitlawb-node", "--git-service-timeout-secs", "0"]).is_err()
);
}
#[test]
fn max_concurrent_git_ops_defaults_and_rejects_out_of_range() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).max_concurrent_git_ops,
128
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-git-ops", "8"])
.max_concurrent_git_ops,
8
);
// 0 permits would shed every served-git request with a 503; clap must reject it.
assert!(Config::try_parse_from(["gitlawb-node", "--max-concurrent-git-ops", "0"]).is_err());
// Above the ceiling would panic tokio's Semaphore::new at boot (permits >
// usize::MAX >> 3); clap must reject it as a clean CLI error instead.
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-git-ops", "1048577"])
.is_err()
);
// The ceiling itself is accepted.
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-git-ops", "1048576"])
.max_concurrent_git_ops,
1_048_576
);
}
#[test]
fn max_concurrent_git_pushes_defaults_and_rejects_out_of_range() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).max_concurrent_git_pushes,
32
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-git-pushes", "8"])
.max_concurrent_git_pushes,
8
);
// 0 permits would shed every push with a 503; clap must reject it.
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-git-pushes", "0"]).is_err()
);
// Above the ceiling would panic tokio's Semaphore::new at boot; clap rejects it.
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-git-pushes", "1048577"])
.is_err()
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-git-pushes", "1048576"])
.max_concurrent_git_pushes,
1_048_576
);
}
#[test]
fn max_concurrent_ipfs_walks_defaults_and_rejects_out_of_range() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).max_concurrent_ipfs_walks,
32
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-ipfs-walks", "4"])
.max_concurrent_ipfs_walks,
4
);
// 0 permits would shed every /ipfs walk with a 503; clap must reject it.
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-ipfs-walks", "0"]).is_err()
);
// Above the ceiling would panic tokio's Semaphore::new at boot; clap rejects it.
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-ipfs-walks", "1048577"])
.is_err()
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-ipfs-walks", "1048576"])
.max_concurrent_ipfs_walks,
1_048_576
);
}
/// U4 (#173): the repair sweep's bounds are conservative by default and a batch of
/// 0 (a sweep that walks nothing and never terminates) is a CLI error, not a
/// runtime hang. The delay does accept 0, for tests and one-off operational runs.
#[test]
fn pin_repair_sweep_knobs_default_conservatively() {
let c = Config::parse_from(["gitlawb-node"]);
assert_eq!(c.pin_repair_sweep_batch, 64);
assert_eq!(c.pin_repair_sweep_delay_secs, 60);
assert!(Config::try_parse_from(["gitlawb-node", "--pin-repair-sweep-batch", "0"]).is_err());
assert!(
Config::try_parse_from(["gitlawb-node", "--pin-repair-sweep-batch", "100001"]).is_err()
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--pin-repair-sweep-batch", "8"])
.pin_repair_sweep_batch,
8
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--pin-repair-sweep-delay-secs", "0"])
.pin_repair_sweep_delay_secs,
0
);
assert!(
Config::try_parse_from(["gitlawb-node", "--pin-repair-sweep-delay-secs", "86401"])
.is_err()
);
}
#[test]
fn ipfs_walk_per_source_defaults_and_rejects_out_of_range() {
assert_eq!(Config::parse_from(["gitlawb-node"]).ipfs_walk_per_source, 4);
assert_eq!(
Config::parse_from(["gitlawb-node", "--ipfs-walk-per-source", "2"])
.ipfs_walk_per_source,
2
);
// 0 would shed every /ipfs walk from a keyed source; clap must reject it.
assert!(Config::try_parse_from(["gitlawb-node", "--ipfs-walk-per-source", "0"]).is_err());
assert!(
Config::try_parse_from(["gitlawb-node", "--ipfs-walk-per-source", "1048577"]).is_err()
);
}
#[test]
fn ipfs_max_repos_walked_defaults_and_rejects_out_of_range() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).ipfs_max_repos_walked,
256
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--ipfs-max-repos-walked", "8"])
.ipfs_max_repos_walked,
8
);
// 0 would walk no repos (serve nothing); clap must reject it.
assert!(Config::try_parse_from(["gitlawb-node", "--ipfs-max-repos-walked", "0"]).is_err());
assert!(
Config::try_parse_from(["gitlawb-node", "--ipfs-max-repos-walked", "1048577"]).is_err()
);
}
/// The `GITLAWB_IPFS_MAX_REPOS_WALKED` knob must actually reach the legacy-probe
/// budget it advertises (R5, KTD5): production seeds `ipfs_max_legacy_probes` from
/// this helper, so the knob is a no-op unless the helper reflects it. RED while the
/// helper returns the hardcoded `MAX_LEGACY_PROBES_PER_REQUEST` (256 regardless of
/// the knob), GREEN once it reads the knob.
#[test]
fn ipfs_max_repos_walked_wires_the_legacy_probe_budget() {
use crate::state::AppState;
// Knob set to 1 → a one-probe legacy budget.
let one = Config::parse_from(["gitlawb-node", "--ipfs-max-repos-walked", "1"]);
assert_eq!(
AppState::ipfs_legacy_probe_budget(&one),
1,
"the knob must control the legacy-probe budget, not be ignored"
);
// Unset knob preserves the shipped 256-probe behaviour.
let default = Config::parse_from(["gitlawb-node"]);
assert_eq!(
AppState::ipfs_legacy_probe_budget(&default),
256,
"the default knob keeps the shipped 256-probe budget"
);
assert_eq!(
AppState::ipfs_legacy_probe_budget(&default),
crate::api::ipfs::MAX_LEGACY_PROBES_PER_REQUEST,
"the default budget equals the constant it replaced"
);
// Ceiling guard: the knob never governs the history-walk ceiling, which must
// stay at MAX_PIN_SOURCES + 1 or a provenanced full source set false-503s.
assert!(
crate::api::ipfs::MAX_HISTORY_WALKS_PER_REQUEST > crate::db::MAX_PIN_SOURCES as u32,
"the history-walk ceiling is independent of the repos-walked knob"
);
}
/// The `/ipfs` work-budget capacity is DERIVED from the route limit (R6, KTD6), with
/// a hard floor of one full legacy search per window (the effective
/// `ipfs_max_legacy_probes`). This guards the derived default so a single
/// default-config deep search never self-throttles mid-scan and recreates the F6
/// admit-then-429 for a legitimate caller. A `RateLimiter` sized to the derived
/// budget must admit the whole probe budget back to back.
#[test]
fn ipfs_work_budget_derives_from_route_limit_and_clears_the_probe_floor() {
use crate::state::AppState;
// Default config: derived work budget = max(route 600, probe budget 256) = 600,
// comfortably above the 256-probe floor.
let default = Config::parse_from(["gitlawb-node"]);
let budget = AppState::ipfs_work_budget(&default);
assert_eq!(budget, 600, "default derives max(route 600, probe 256)");
assert!(
budget >= AppState::ipfs_legacy_probe_budget(&default) as usize,
"the work budget must clear one full legacy search per window"
);
// Tight route limit (1): the floor lifts the work budget to the probe budget
// (256), NOT down to 1 — a single deep search still completes its full scan.
let tight = Config::parse_from(["gitlawb-node", "--ipfs-rate-limit", "1"]);
assert_eq!(
AppState::ipfs_work_budget(&tight),
256,
"a tight route limit is floored at the 256-probe budget, not clamped to 1"
);
// Raised probe budget lifts the floor with it (the work budget tracks the
// effective probe budget, not the constant).
let raised = Config::parse_from([
"gitlawb-node",
"--ipfs-rate-limit",
"10",
"--ipfs-max-repos-walked",
"1000",
]);
assert_eq!(
AppState::ipfs_work_budget(&raised),
1000,
"the floor tracks the operator-raised legacy-probe budget"
);
// 0 route limit disables the derived bucket too (a 0-capacity limiter admits all).
let disabled = Config::parse_from(["gitlawb-node", "--ipfs-rate-limit", "0"]);
assert_eq!(
AppState::ipfs_work_budget(&disabled),
0,
"route limit 0 disables the derived work bucket alongside the route brake"
);
// Behavioral floor: a limiter sized to the derived (tight-route) budget admits
// the whole probe budget back to back for one source, then sheds the next.
let budget = AppState::ipfs_work_budget(&tight);
let limiter =
crate::rate_limit::RateLimiter::new(budget, std::time::Duration::from_secs(3600));
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
rt.block_on(async {
for i in 0..budget {
assert!(
limiter.check("1.2.3.4").await,
"probe {i} of one full default-config scan must be admitted (no mid-scan throttle)"
);
}
assert!(
!limiter.check("1.2.3.4").await,
"the probe past the derived budget is shed"
);
});
}
#[test]
fn max_concurrent_reads_per_caller_defaults_and_rejects_out_of_range() {
assert_eq!(
Config::parse_from(["gitlawb-node"]).max_concurrent_reads_per_caller,
16
);
assert_eq!(
Config::parse_from(["gitlawb-node", "--max-concurrent-reads-per-caller", "4"])
.max_concurrent_reads_per_caller,
4
);
// 0 would shed every read from a keyed caller; clap must reject it.
assert!(
Config::try_parse_from(["gitlawb-node", "--max-concurrent-reads-per-caller", "0"])
.is_err()
);
assert!(Config::try_parse_from([
"gitlawb-node",
"--max-concurrent-reads-per-caller",
"1048577"
])
.is_err());
assert_eq!(
Config::parse_from([
"gitlawb-node",
"--max-concurrent-reads-per-caller",
"1048576"
])
.max_concurrent_reads_per_caller,
1_048_576
);
}
}
File diff suppressed because it is too large. Load diff
+1 -46
View File
@@ -41,18 +41,12 @@ pub enum AppError {
#[error("incomplete: {0}")]
Incomplete(String),
#[error("search incomplete: {0}")]
SearchIncomplete(String),
#[error("git error: {0}")]
Git(String),
#[error("git service timed out: {0}")]
Timeout(String),
#[error("server overloaded: {0}")]
Overloaded(String),
#[error("database error: {0}")]
Db(#[from] sqlx::Error),
@@ -144,15 +138,6 @@ impl IntoResponse for AppError {
AppError::Incomplete(msg) => {
(StatusCode::UNPROCESSABLE_ENTITY, "incomplete", msg.clone())
}
// A bounded search that could not complete (the CID resolver hit its
// legacy-probe or walk ceiling), distinct from the 404 that asserts a
// definitive not-found: absence was NOT proven, so the caller should
// retry rather than treat it as gone (#173, F2). 503, retryable.
AppError::SearchIncomplete(msg) => (
StatusCode::SERVICE_UNAVAILABLE,
"search_incomplete",
msg.clone(),
),
AppError::Git(msg) => (StatusCode::INTERNAL_SERVER_ERROR, "git_error", msg.clone()),
// 504, distinct from the 500 git_error and from the read-gate's 404 /
// the auth 401, so the client can tell a deadline from a failure.
@@ -162,12 +147,6 @@ impl IntoResponse for AppError {
DB_UNAVAILABLE_CODE,
DB_UNAVAILABLE_MESSAGE.into(),
),
// 503 with a Retry-After (attached after this match — the shared tail
// can't carry per-variant headers). This is the single place Overloaded
// becomes a response, so it can never ship a 503 without the retry hint.
AppError::Overloaded(msg) => {
(StatusCode::SERVICE_UNAVAILABLE, "overloaded", msg.clone())
}
AppError::Db(e) => (StatusCode::INTERNAL_SERVER_ERROR, "db_error", e.to_string()),
AppError::Internal(e) => (
StatusCode::INTERNAL_SERVER_ERROR,
@@ -181,21 +160,7 @@ impl IntoResponse for AppError {
"message": message,
}));
let mut resp = (status, body).into_response();
// Both retryable 503s advertise when to retry: Overloaded (capacity shed) and
// SearchIncomplete (a bounded CID search cut short by a cap — retry may complete
// it). They ride the shared tail above for body/status, so the header is attached
// here rather than in bespoke early returns, keeping each variant handled once.
if matches!(
self,
AppError::Overloaded(_) | AppError::SearchIncomplete(_)
) {
resp.headers_mut().insert(
axum::http::header::RETRY_AFTER,
axum::http::HeaderValue::from_static("1"),
);
}
resp
(status, body).into_response()
}
}
@@ -217,14 +182,4 @@ mod tests {
StatusCode::INTERNAL_SERVER_ERROR
);
}
#[test]
fn overloaded_maps_to_503_with_retry_after() {
let resp = AppError::Overloaded("x".into()).into_response();
assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE);
assert_eq!(
resp.headers().get("retry-after").unwrap().to_str().unwrap(),
"1"
);
}
}
+96 -238
View File
@@ -29,9 +29,9 @@
use std::collections::HashSet;
use std::path::{Path, PathBuf};
use std::time::{Duration, Instant};
use std::process::Command;
use anyhow::Result;
use anyhow::{bail, Context, Result};
/// Env var that forces the push path to always full-scan, bypassing the delta
/// optimization (KTD7 kill-switch). Reuses the already-tested fallback branch
@@ -64,21 +64,11 @@ pub enum PinCandidates {
/// Fail-closed: any condition where the introduced set cannot be safely
/// determined returns [`PinCandidates::FullScanRequired`] rather than a partial
/// set, so the caller full-scans instead of silently under-pinning.
pub fn resolve_push_delta(
repo_path: &Path,
new_tips: &[&str],
old_tips: &[&str],
git_bin: &str,
deadline: Instant,
) -> PinCandidates {
resolve_push_delta_inner(
repo_path,
new_tips,
old_tips,
force_full_scan(),
git_bin,
deadline,
)
pub fn resolve_push_delta(repo_path: &Path, new_tips: &[&str], old_tips: &[&str]) -> PinCandidates {
// KTD7 kill-switch: force the (already-tested) full-scan fallback. The env
// read is split out from the pure logic so the resolver stays unit-testable
// without touching process-global state.
resolve_push_delta_inner(repo_path, new_tips, old_tips, force_full_scan())
}
/// Whether the force-full-scan kill-switch env var is set.
@@ -93,8 +83,6 @@ fn resolve_push_delta_inner(
new_tips: &[&str],
old_tips: &[&str],
force_full_scan: bool,
git_bin: &str,
deadline: Instant,
) -> PinCandidates {
if force_full_scan {
tracing::debug!("{FORCE_FULL_SCAN_ENV} set — forcing full scan");
@@ -114,7 +102,7 @@ fn resolve_push_delta_inner(
// commit). Bare `cat-file -t` returns `tag` for an annotated tag, and
// `for-each-ref %(*objecttype)` peels only one level — neither is correct.
for tip in new_tips {
match peeled_object_type(repo_path, tip, git_bin, deadline) {
match peeled_object_type(repo_path, tip) {
Some(t) if t == "commit" => {}
other => {
tracing::debug!(
@@ -127,7 +115,7 @@ fn resolve_push_delta_inner(
}
}
match rev_list_delta(repo_path, new_tips, old_tips, git_bin, deadline) {
match rev_list_delta(repo_path, new_tips, old_tips) {
Ok(oids) => PinCandidates::Delta(oids),
Err(e) => {
tracing::debug!(err = %e, "push-delta rev-list failed — forcing full scan");
@@ -138,43 +126,41 @@ fn resolve_push_delta_inner(
/// Return the fully-peeled object type of `sha` (e.g. `commit`, `tree`,
/// `blob`), or `None` if the object is missing/unpeelable or git errored.
fn peeled_object_type(
repo_path: &Path,
sha: &str,
git_bin: &str,
deadline: Instant,
) -> Option<String> {
let peel = format!("{sha}^{{}}");
let out = crate::git::visibility_pack::run_bounded_git(
git_bin,
&["cat-file", "-t", &peel],
repo_path,
b"",
deadline,
)
.ok()?;
Some(String::from_utf8_lossy(&out).trim().to_string())
fn peeled_object_type(repo_path: &Path, sha: &str) -> Option<String> {
let output = Command::new("git")
.args(["cat-file", "-t", &format!("{sha}^{{}}")])
.current_dir(repo_path)
.output()
.ok()?;
if !output.status.success() {
return None;
}
Some(String::from_utf8_lossy(&output.stdout).trim().to_string())
}
/// Run `git rev-list --objects --no-object-names <new> --not <old>` and return
/// the bare OID set. Decides on `status.success()` *before* parsing stdout, so
/// a walk that prints a valid prefix then errors mid-walk is discarded.
fn rev_list_delta(
repo_path: &Path,
new_tips: &[&str],
old_tips: &[&str],
git_bin: &str,
deadline: Instant,
) -> Result<Vec<String>> {
fn rev_list_delta(repo_path: &Path, new_tips: &[&str], old_tips: &[&str]) -> Result<Vec<String>> {
let mut args: Vec<&str> = vec!["rev-list", "--objects", "--no-object-names"];
args.extend_from_slice(new_tips);
if !old_tips.is_empty() {
args.push("--not");
args.extend_from_slice(old_tips);
}
let out =
crate::git::visibility_pack::run_bounded_git(git_bin, &args, repo_path, b"", deadline)?;
let stdout = String::from_utf8_lossy(&out);
let output = Command::new("git")
.args(&args)
.current_dir(repo_path)
.output()
.context("failed to run git rev-list for push delta")?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
bail!("git rev-list failed: {stderr}");
}
let stdout = String::from_utf8_lossy(&output.stdout);
Ok(stdout
.lines()
.map(|l| l.trim().to_string())
@@ -189,19 +175,23 @@ fn rev_list_delta(
/// reconciliation sweep relies on. It returns *all* objects (including
/// unreachable/dangling ones), which is what the sweep needs to catch
/// stragglers — do not swap it for a reachability walk.
pub fn list_all_objects(repo_path: &Path, git_bin: &str, deadline: Instant) -> Result<Vec<String>> {
let out = crate::git::visibility_pack::run_bounded_git(
git_bin,
&[
pub fn list_all_objects(repo_path: &Path) -> Result<Vec<String>> {
let output = Command::new("git")
.args([
"cat-file",
"--batch-all-objects",
"--batch-check=%(objectname)",
],
repo_path,
b"",
deadline,
)?;
let stdout = String::from_utf8_lossy(&out);
])
.current_dir(repo_path)
.output()
.context("failed to run git cat-file")?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
bail!("git cat-file failed: {stderr}");
}
let stdout = String::from_utf8_lossy(&output.stdout);
Ok(stdout
.lines()
.map(|l| l.trim().to_string())
@@ -213,23 +203,23 @@ pub fn list_all_objects(repo_path: &Path, git_bin: &str, deadline: Instant) -> R
/// `--batch-check='%(objectname) %(objecttype)'`. The pin path's fail-closed
/// filter needs to tell blobs (content, withholdable) from commits/trees
/// (structural, never withheld) without typing the candidate list itself.
pub fn list_all_objects_with_type(
repo_path: &Path,
git_bin: &str,
deadline: Instant,
) -> Result<Vec<(String, String)>> {
let out = crate::git::visibility_pack::run_bounded_git(
git_bin,
&[
pub fn list_all_objects_with_type(repo_path: &Path) -> Result<Vec<(String, String)>> {
let output = Command::new("git")
.args([
"cat-file",
"--batch-all-objects",
"--batch-check=%(objectname) %(objecttype)",
],
repo_path,
b"",
deadline,
)?;
let stdout = String::from_utf8_lossy(&out);
])
.current_dir(repo_path)
.output()
.context("failed to run git cat-file")?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
bail!("git cat-file failed: {stderr}");
}
let stdout = String::from_utf8_lossy(&output.stdout);
Ok(stdout
.lines()
.filter_map(|l| {
@@ -245,12 +235,8 @@ pub fn list_all_objects_with_type(
/// fail-closed pin filter drops any candidate blob absent from the reachable,
/// visibility-allowed set; a dangling private blob is in this set but not the
/// allowed set, so it never replicates (#99).
pub fn all_blob_oids(
repo_path: &Path,
git_bin: &str,
deadline: Instant,
) -> Result<HashSet<String>> {
Ok(list_all_objects_with_type(repo_path, git_bin, deadline)?
pub fn all_blob_oids(repo_path: &Path) -> Result<HashSet<String>> {
Ok(list_all_objects_with_type(repo_path)?
.into_iter()
.filter(|(_, ty)| ty == "blob")
.map(|(oid, _)| oid)
@@ -286,22 +272,18 @@ pub async fn resolve_candidates_for_push(
repo_path: PathBuf,
new_tips: Vec<String>,
old_tips: Vec<String>,
git_bin: String,
timeout: Duration,
) -> PinCandidateSet {
tokio::task::spawn_blocking(move || {
// ONE shared deadline for the whole scan, per jatmn ("the same deadline").
let deadline = Instant::now() + timeout;
let new_refs: Vec<&str> = new_tips.iter().map(String::as_str).collect();
let old_refs: Vec<&str> = old_tips.iter().map(String::as_str).collect();
match resolve_push_delta(&repo_path, &new_refs, &old_refs, &git_bin, deadline) {
match resolve_push_delta(&repo_path, &new_refs, &old_refs) {
PinCandidates::Delta(objs) => {
tracing::info!(delta = objs.len(), repo = %repo_path.display(), "pin candidate set from push delta");
PinCandidateSet { candidates: objs, full_scan: false }
}
PinCandidates::FullScanRequired => {
tracing::warn!(repo = %repo_path.display(), "pin delta unavailable (non-commit tip, git error, or force-full-scan) — full-scan fallback");
match list_all_objects(&repo_path, &git_bin, deadline) {
match list_all_objects(&repo_path) {
Ok(objs) => PinCandidateSet { candidates: objs, full_scan: true },
Err(e) => {
tracing::warn!(repo = %repo_path.display(), err = %e, "full-scan fallback failed; pinning nothing this push (reconciliation sweep backstops)");
@@ -322,13 +304,8 @@ pub async fn resolve_candidates_for_push(
mod tests {
use super::*;
use std::collections::HashSet;
use std::process::Command;
use tempfile::TempDir;
fn td() -> std::time::Instant {
std::time::Instant::now() + std::time::Duration::from_secs(600)
}
/// Minimal git helper for building test repos.
struct Repo {
_td: TempDir,
@@ -386,10 +363,9 @@ mod tests {
let repo = Repo::new();
let c1 = repo.commit_file("a.txt", "one\n");
let c2 = repo.commit_file("b.txt", "two\n");
let got: HashSet<String> =
delta(resolve_push_delta(&repo.path, &[&c2], &[&c1], "git", td()))
.into_iter()
.collect();
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&c2], &[&c1]))
.into_iter()
.collect();
// The new blob b.txt and commit c2 are in the delta; the old blob a.txt
// and commit c1 are not.
let new_blob = repo.rev("HEAD:b.txt");
@@ -407,7 +383,7 @@ mod tests {
// genuinely new objects, never fewer.
let repo = Repo::new();
let c1 = repo.commit_file("a.txt", "one\n");
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&c1], &[], "git", td()))
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&c1], &[]))
.into_iter()
.collect();
assert!(got.contains(&c1));
@@ -423,15 +399,9 @@ mod tests {
// Rewrite history: reset to base, commit a different file.
repo.git(&["reset", "-q", "--hard", &base]);
let new_tip = repo.commit_file("c.txt", "three\n");
let got: HashSet<String> = delta(resolve_push_delta(
&repo.path,
&[&new_tip],
&[&old_tip],
"git",
td(),
))
.into_iter()
.collect();
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&new_tip], &[&old_tip]))
.into_iter()
.collect();
assert!(got.contains(&new_tip), "new tip in delta");
assert!(got.contains(&repo.rev(&format!("{new_tip}:c.txt"))));
// No error; force-push computes new-minus-old cleanly.
@@ -443,7 +413,7 @@ mod tests {
repo.commit_file("a.txt", "one\n");
// All updates were deletions => new_tips empty after the caller strips zeros.
assert_eq!(
resolve_push_delta(&repo.path, &[], &[ZERO], "git", td()),
resolve_push_delta(&repo.path, &[], &[ZERO]),
PinCandidates::Delta(Vec::new())
);
}
@@ -454,7 +424,7 @@ mod tests {
repo.commit_file("a.txt", "one\n");
let blob = repo.rev("HEAD:a.txt");
assert_eq!(
resolve_push_delta(&repo.path, &[&blob], &[], "git", td()),
resolve_push_delta(&repo.path, &[&blob], &[]),
PinCandidates::FullScanRequired,
"a blob tip must force full scan (rev-list would exit 0 and walk it)"
);
@@ -466,7 +436,7 @@ mod tests {
repo.commit_file("a.txt", "one\n");
let tree = repo.rev("HEAD^{tree}");
assert_eq!(
resolve_push_delta(&repo.path, &[&tree], &[], "git", td()),
resolve_push_delta(&repo.path, &[&tree], &[]),
PinCandidates::FullScanRequired
);
}
@@ -479,7 +449,7 @@ mod tests {
repo.git(&["tag", "-a", "treetag", "-m", "x", &tree]);
let tag = repo.rev("treetag");
assert_eq!(
resolve_push_delta(&repo.path, &[&tag], &[], "git", td()),
resolve_push_delta(&repo.path, &[&tag], &[]),
PinCandidates::FullScanRequired,
"annotated tag peeling to a tree must force full scan"
);
@@ -495,7 +465,7 @@ mod tests {
repo.git(&["tag", "-a", "t2", "-m", "x", &t1]);
let t2 = repo.rev("t2");
assert_eq!(
resolve_push_delta(&repo.path, &[&t2], &[], "git", td()),
resolve_push_delta(&repo.path, &[&t2], &[]),
PinCandidates::FullScanRequired
);
}
@@ -510,7 +480,7 @@ mod tests {
let c1 = repo.commit_file("a.txt", "one\n");
repo.git(&["tag", "-a", "rel", "-m", "release", &c1]);
let tag = repo.rev("rel");
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&tag], &[], "git", td()))
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&tag], &[]))
.into_iter()
.collect();
assert!(
@@ -531,7 +501,7 @@ mod tests {
let t1 = repo.rev("t1");
repo.git(&["tag", "-a", "t2", "-m", "x", &t1]);
let t2 = repo.rev("t2");
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&t2], &[], "git", td()))
let got: HashSet<String> = delta(resolve_push_delta(&repo.path, &[&t2], &[]))
.into_iter()
.collect();
assert!(
@@ -546,14 +516,9 @@ mod tests {
let repo = Repo::new();
let c1 = repo.commit_file("a.txt", "one\n");
let c2 = repo.commit_file("b.txt", "two\n");
let set = resolve_candidates_for_push(
repo.path.clone(),
vec![c2.clone()],
vec![c1.clone()],
"git".to_string(),
std::time::Duration::from_secs(600),
)
.await;
let set =
resolve_candidates_for_push(repo.path.clone(), vec![c2.clone()], vec![c1.clone()])
.await;
assert!(!set.full_scan, "happy-path delta is not a full scan");
let got: HashSet<String> = set.candidates.into_iter().collect();
let new_blob = repo.rev("HEAD:b.txt");
@@ -571,18 +536,8 @@ mod tests {
let repo = Repo::new();
repo.commit_file("a.txt", "one\n");
let blob = repo.rev("HEAD:a.txt");
let all: HashSet<String> = list_all_objects(&repo.path, "git", td())
.unwrap()
.into_iter()
.collect();
let set = resolve_candidates_for_push(
repo.path.clone(),
vec![blob],
vec![],
"git".to_string(),
std::time::Duration::from_secs(600),
)
.await;
let all: HashSet<String> = list_all_objects(&repo.path).unwrap().into_iter().collect();
let set = resolve_candidates_for_push(repo.path.clone(), vec![blob], vec![]).await;
assert!(set.full_scan, "non-commit tip is signalled as a full scan");
let got: HashSet<String> = set.candidates.into_iter().collect();
assert_eq!(got, all, "non-commit tip falls back to full repo scan");
@@ -594,7 +549,7 @@ mod tests {
repo.commit_file("a.txt", "one\n");
let bogus = "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef";
assert_eq!(
resolve_push_delta(&repo.path, &[bogus], &[], "git", td()),
resolve_push_delta(&repo.path, &[bogus], &[]),
PinCandidates::FullScanRequired,
"a missing/corrupt tip OID must force full scan"
);
@@ -609,7 +564,7 @@ mod tests {
let c1 = repo.commit_file("a.txt", "one\n");
let c2 = repo.commit_file("b.txt", "two\n");
let old_tree = repo.rev(&format!("{c1}^{{tree}}"));
let result = resolve_push_delta(&repo.path, &[&c2], &[&old_tree], "git", td());
let result = resolve_push_delta(&repo.path, &[&c2], &[&old_tree]);
match result {
PinCandidates::FullScanRequired => {} // safe: caller full-scans
PinCandidates::Delta(objs) => {
@@ -628,15 +583,10 @@ mod tests {
// branch2 from base advances independently
repo.git(&["checkout", "-q", "-b", "branch2", &base]);
let b2 = repo.commit_file("c.txt", "three\n");
let got: HashSet<String> = delta(resolve_push_delta(
&repo.path,
&[&b1, &b2],
&[&base, &base],
"git",
td(),
))
.into_iter()
.collect();
let got: HashSet<String> =
delta(resolve_push_delta(&repo.path, &[&b1, &b2], &[&base, &base]))
.into_iter()
.collect();
assert!(got.contains(&b1), "branch1 new commit");
assert!(got.contains(&b2), "branch2 new commit");
}
@@ -645,7 +595,7 @@ mod tests {
fn empty_repo_no_tips_yields_empty_delta() {
let repo = Repo::new();
assert_eq!(
resolve_push_delta(&repo.path, &[], &[], "git", td()),
resolve_push_delta(&repo.path, &[], &[]),
PinCandidates::Delta(Vec::new())
);
}
@@ -659,12 +609,12 @@ mod tests {
let repo = Repo::new();
let c1 = repo.commit_file("a.txt", "one\n");
assert_eq!(
resolve_push_delta_inner(&repo.path, &[&c1], &[], true, "git", td()),
resolve_push_delta_inner(&repo.path, &[&c1], &[], true),
PinCandidates::FullScanRequired
);
// And with the flag off, the same push yields a Delta.
assert!(matches!(
resolve_push_delta_inner(&repo.path, &[&c1], &[], false, "git", td()),
resolve_push_delta_inner(&repo.path, &[&c1], &[], false),
PinCandidates::Delta(_)
));
}
@@ -674,7 +624,7 @@ mod tests {
let repo = Repo::new();
repo.commit_file("a.txt", "one\n");
repo.commit_file("b.txt", "two\n");
let all = list_all_objects(&repo.path, "git", td()).unwrap();
let all = list_all_objects(&repo.path).unwrap();
// 2 commits + 2 trees + 2 blobs = 6 objects.
assert_eq!(all.len(), 6, "got: {all:?}");
}
@@ -692,7 +642,7 @@ mod tests {
std::fs::write(repo.path.join("orphan.bin"), b"dangling\n").unwrap();
let dangling = repo.git(&["hash-object", "-w", "orphan.bin"]);
let blobs = all_blob_oids(&repo.path, "git", td()).unwrap();
let blobs = all_blob_oids(&repo.path).unwrap();
assert!(blobs.contains(&reachable_blob), "reachable blob present");
assert!(
blobs.contains(&dangling),
@@ -702,7 +652,7 @@ mod tests {
assert!(!blobs.contains(&tree), "tree is not a blob");
// The typed lister tags each object; spot-check the dangling blob's type.
let typed = list_all_objects_with_type(&repo.path, "git", td()).unwrap();
let typed = list_all_objects_with_type(&repo.path).unwrap();
assert!(
typed
.iter()
@@ -710,96 +660,4 @@ mod tests {
"dangling object is typed as a blob"
);
}
#[cfg(unix)]
#[test]
fn list_all_objects_times_out_on_a_hung_git_instead_of_blocking() {
use std::os::unix::fs::PermissionsExt;
use std::time::{Duration, Instant};
let dir = tempfile::TempDir::new().unwrap();
// Fake git: hang on cat-file (bounded to 30s so a broken test can't wedge).
let fake = dir.path().join("fakegit");
std::fs::write(
&fake,
"#!/bin/sh\ncase \"$1\" in\n cat-file) i=0; while [ $i -lt 30 ]; do sleep 1; i=$((i+1)); done ;;\n *) : ;;\nesac\n",
)
.unwrap();
let mut perm = std::fs::metadata(&fake).unwrap().permissions();
perm.set_mode(0o755);
std::fs::set_permissions(&fake, perm).unwrap();
let git_bin = fake.to_str().unwrap().to_string();
let (tx, rx) = std::sync::mpsc::channel();
let path = dir.path().to_path_buf();
std::thread::spawn(move || {
let _ = tx.send(list_all_objects(
&path,
&git_bin,
Instant::now() + Duration::from_millis(150),
));
});
let res = rx.recv_timeout(Duration::from_secs(10)).expect(
"list_all_objects must return within the watchdog budget, not block on a hung git",
);
assert!(
res.is_err(),
"a hung git must make list_all_objects error out, not hang"
);
}
/// #174 finding 2: the DELTA-path children — `peeled_object_type` (cat-file),
/// `rev_list_delta` (rev-list) — and `list_all_objects_with_type` (cat-file) are
/// the remaining candidate-discovery execs (the common push path). Each must
/// return within the watchdog budget on a hung git rather than block and pin the
/// write permit. Revert any one to a bare `Command::output()` and its arm blocks
/// past the recv budget (RED).
#[cfg(unix)]
#[test]
fn delta_path_exec_fns_time_out_on_a_hung_git() {
use std::os::unix::fs::PermissionsExt;
use std::time::{Duration, Instant};
let dir = tempfile::TempDir::new().unwrap();
// Fake git hangs on BOTH cat-file and rev-list (bounded 30s so a broken
// test can't leak a permanent orphan or wedge the suite).
let fake = dir.path().join("fakegit");
std::fs::write(
&fake,
"#!/bin/sh\ncase \"$1\" in\n cat-file|rev-list) i=0; while [ $i -lt 30 ]; do sleep 1; i=$((i+1)); done ;;\n *) : ;;\nesac\n",
)
.unwrap();
let mut perm = std::fs::metadata(&fake).unwrap().permissions();
perm.set_mode(0o755);
std::fs::set_permissions(&fake, perm).unwrap();
let git_bin = fake.to_str().unwrap().to_string();
let path = dir.path().to_path_buf();
// Run `f` on a thread and require it to RETURN (not block) within 10s.
fn returns_within<T: Send + 'static>(f: impl FnOnce() -> T + Send + 'static) -> T {
let (tx, rx) = std::sync::mpsc::channel();
std::thread::spawn(move || {
let _ = tx.send(f());
});
rx.recv_timeout(Duration::from_secs(10)).expect(
"a delta-path exec fn must return within the watchdog budget, not block on a hung git",
)
}
let dl = || Instant::now() + Duration::from_millis(150);
let sha = "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef";
let (p, g, d) = (path.clone(), git_bin.clone(), dl());
assert!(
returns_within(move || peeled_object_type(&p, sha, &g, d)).is_none(),
"peeled_object_type must time out to None on a hung cat-file"
);
let (p, g, d) = (path.clone(), git_bin.clone(), dl());
assert!(
returns_within(move || rev_list_delta(&p, &[sha], &[], &g, d)).is_err(),
"rev_list_delta must error out on a hung rev-list"
);
let (p, g, d) = (path.clone(), git_bin.clone(), dl());
assert!(
returns_within(move || list_all_objects_with_type(&p, &g, d)).is_err(),
"list_all_objects_with_type must error out on a hung cat-file"
);
}
}
+49 -731
View File
@@ -11,12 +11,9 @@
use std::collections::HashSet;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::time::Duration;
use anyhow::{Context, Result};
use sqlx::pool::PoolConnection;
use sqlx::postgres::PgPoolOptions;
use sqlx::{PgPool, Postgres};
use sqlx::PgPool;
use tokio::sync::Mutex;
use tracing::{debug, info, warn};
@@ -28,79 +25,30 @@ use super::tigris::TigrisClient;
pub struct RepoStore {
repos_dir: PathBuf,
tigris: Option<TigrisClient>,
/// Dedicated Postgres pool for repo write advisory locks, built by
/// `build_lock_pool` (see there for why it is separate and why it carries an
/// `after_release` hook). Never use this for ordinary queries.
lock_pool: PgPool,
/// Shared Postgres pool for advisory locks.
pool: PgPool,
/// Tracks repos already confirmed to exist in Tigris — avoids redundant
/// HEAD checks and background uploads for repos we've already migrated.
migrated: Arc<Mutex<HashSet<String>>>,
/// Test-only stall injected at the head of `acquire_write`'s Tigris phase,
/// i.e. AFTER the advisory lock is taken and BEFORE the guard exists. That
/// window is exactly where the outer `tokio::time::timeout` in
/// `api/repos.rs` can drop the future (#173). `TigrisClient` takes its
/// endpoint from process-wide AWS env vars and has no injectable seam, so
/// this flag is the smallest way to hold a real `acquire_write` open in that
/// window and cancel it there.
#[cfg(test)]
tigris_stall: Option<std::time::Duration>,
/// Test-only counter of how many times a write guard from this store REACHED the
/// Tigris upload site in `release` (the point past the `success` check, where a
/// configured client would be uploaded to). It counts the decision, not a network
/// call: `TigrisClient` takes its endpoint from process-wide AWS env vars and has no
/// injectable seam, so every test runs with `tigris: None` and a counter inside the
/// `Some` arm could never move. Reaching the site is the property under test anyway:
/// an interrupted push must not publish a half-applied repo, and the disconnect path
/// must therefore never get here (#173 F2).
///
/// Per store rather than a process global, so cases running in parallel do not see
/// each other's uploads, and an `Arc` rather than a `thread_local` because the guard
/// is released from a detached task on another worker thread. Same test-only counter
/// idiom as `ipfs_pin::note_legacy_repair_read`.
#[cfg(test)]
upload_site_reached: Arc<std::sync::atomic::AtomicUsize>,
}
impl RepoStore {
/// Derives its own lock pool from `pool`, so callers that only have the main
/// pool (tests, `for_testing` sites in other modules) still get the
/// `after_release` semantics `acquire_write` depends on.
#[cfg(test)]
pub fn for_testing(repos_dir: PathBuf, pool: PgPool) -> Self {
Self::new(
Self {
repos_dir,
None,
build_lock_pool(&pool, 8, Duration::from_secs(5)),
)
tigris: None,
pool,
migrated: Arc::new(tokio::sync::Mutex::new(std::collections::HashSet::new())),
}
}
/// Test-only: see `tigris_stall`.
#[cfg(test)]
pub fn with_tigris_stall(mut self, stall: Duration) -> Self {
self.tigris_stall = Some(stall);
self
}
/// Test-only: how many write guards from this store have reached the Tigris upload
/// site. See [`RepoStore::upload_site_reached`].
#[cfg(test)]
pub fn tigris_upload_site_reached(&self) -> usize {
self.upload_site_reached
.load(std::sync::atomic::Ordering::SeqCst)
}
/// `lock_pool` must come from `build_lock_pool`; a plain pool leaks advisory
/// locks on cancellation.
pub fn new(repos_dir: PathBuf, tigris: Option<TigrisClient>, lock_pool: PgPool) -> Self {
pub fn new(repos_dir: PathBuf, tigris: Option<TigrisClient>, pool: PgPool) -> Self {
Self {
repos_dir,
tigris,
lock_pool,
pool,
migrated: Arc::new(Mutex::new(HashSet::new())),
#[cfg(test)]
tigris_stall: None,
#[cfg(test)]
upload_site_reached: Arc::new(std::sync::atomic::AtomicUsize::new(0)),
}
}
@@ -209,64 +157,25 @@ impl RepoStore {
let (owner_slug, local_path) = self.local_path(owner_did, repo_name)?;
let lock_key = advisory_lock_key(&owner_slug, repo_name);
// Acquire the Postgres advisory lock with retry, using pg_try_advisory_lock so a
// stale lock from a crashed connection can't block us indefinitely.
//
// The connection is checked out INSIDE the loop and RETURNED before each sleep.
// Only the connection that actually took the lock is retained. Two constraints
// pull in opposite directions here, and this is what satisfies both:
//
// * Session ownership. A session-level advisory lock belongs to the CONNECTION
// that took it, so the lock and its `pg_advisory_unlock` must run on the same
// one. Running them through the pool (`fetch_one(&self.pool)`) lets them land
// on different connections: the unlock silently returns false and the lock
// leaks, while a competing acquire that happens to draw the holding
// connection re-enters the lock and two pushes to one repo run concurrently.
// Hence: keep the connection that WON.
// * Occupancy. Holding a connection across the ~60 one-second sleeps would let
// one spinning acquire park a lock-pool connection for a minute. That is not
// just a push-path concern: `api/issues.rs` and `api/pulls.rs` reach
// acquire_write holding no concurrency permit at all, so a caller could park
// the whole pool and starve authenticated pushes on every repo (#173 F1).
// Hence: return the connection when we LOSE, before sleeping.
//
// Returning a losing connection is safe with respect to the cancellation design:
// `after_release` runs `pg_advisory_unlock_all()`, a no-op on a connection that
// took nothing, so it cannot disturb a lock held by any other connection
// (proven by `returning_an_unlocked_connection_does_not_clear_another_connections_lock`).
//
// Cancellation safety is unchanged: the future can only be dropped while a
// connection is checked out, and dropping it runs the same `after_release` hook,
// which clears whatever lock it had just taken (#173 U1).
let mut lock_conn = None;
// Acquire Postgres advisory lock with retry using pg_try_advisory_lock
// to avoid blocking indefinitely on stale locks from crashed connections.
let mut acquired = false;
for attempt in 0..60 {
let mut conn = self.lock_pool.acquire().await.map_err(|e| {
anyhow::Error::new(LockPoolBusy)
.context(format!("checking out a lock-pool connection: {e}"))
})?;
let row: (bool,) = sqlx::query_as("SELECT pg_try_advisory_lock($1)")
.bind(lock_key)
.fetch_one(&mut *conn)
.fetch_one(&self.pool)
.await
.context("trying advisory lock")?;
if row.0 {
lock_conn = Some(conn);
acquired = true;
break;
}
// Lost the race: give the connection back so a spinning acquire occupies
// nothing while it waits.
drop(conn);
if attempt < 59 {
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
}
}
let Some(lock_conn) = lock_conn else {
if !acquired {
anyhow::bail!("could not acquire advisory lock after 60s — possible stale lock for {owner_slug}/{repo_name}");
};
#[cfg(test)]
if let Some(stall) = self.tigris_stall {
tokio::time::sleep(stall).await;
}
// Always download the latest from Tigris before writing.
@@ -293,10 +202,8 @@ impl RepoStore {
repo_name: repo_name.to_string(),
local_path,
lock_key,
lock_conn,
pool: self.pool.clone(),
tigris: self.tigris.clone(),
#[cfg(test)]
upload_site_reached: Arc::clone(&self.upload_site_reached),
})
}
@@ -350,61 +257,40 @@ impl RepoStore {
/// segment is rejected. This is the CodeQL-recognised barrier
/// pattern for `rust/path-injection`.
fn local_path(&self, owner_did: &str, repo_name: &str) -> Result<(String, PathBuf)> {
validate_path_components(owner_did, repo_name)?;
let owner_slug = owner_did.replace([':', '/'], "_");
let local_path = validated_repo_disk_path(&self.repos_dir, owner_did, repo_name)?;
Ok((owner_slug, local_path))
}
}
let local_path = self
.repos_dir
.join(&owner_slug)
.join(format!("{repo_name}.git"));
/// The three-layer validated form of `store::repo_disk_path`, with NO Tigris fetch and
/// no `RepoStore` (#173 round 11, F3). Extracted from `RepoStore::local_path` so a
/// second caller that must not pull a cold repo, the U4 legacy provider-CID sweep, gets
/// the same barrier instead of the raw join. `local_path` is now a thin wrapper over
/// this, so the two cannot drift.
///
/// Three-layer defence against path traversal:
/// 1. Strict allowlist on `owner_did` and `repo_name` (no `..`, slashes,
/// null bytes, leading dots; length-bounded).
/// 2. The joined path must remain rooted at `repos_dir`.
/// 3. Every component of the joined path must be `Component::Normal`
/// (or the prefix/root from `repos_dir`); any `ParentDir`/`CurDir`
/// segment is rejected. This is the CodeQL-recognised barrier
/// pattern for `rust/path-injection`.
pub(crate) fn validated_repo_disk_path(
repos_dir: &Path,
owner_did: &str,
repo_name: &str,
) -> Result<PathBuf> {
validate_path_components(owner_did, repo_name)?;
if !local_path.starts_with(&self.repos_dir) {
anyhow::bail!(
"computed repo path escaped repos_dir: {}",
local_path.display()
);
}
let owner_slug = owner_did.replace([':', '/'], "_");
let local_path = repos_dir.join(&owner_slug).join(format!("{repo_name}.git"));
if !local_path.starts_with(repos_dir) {
anyhow::bail!(
"computed repo path escaped repos_dir: {}",
local_path.display()
);
}
// Explicit component walk — sanitisation barrier that static analysers
// (CodeQL `rust/path-injection`) recognise. The path must be composed
// entirely of Normal segments after the root prefix; any ParentDir or
// CurDir component is a traversal attempt.
for component in local_path.components() {
use std::path::Component;
match component {
Component::Prefix(_) | Component::RootDir | Component::Normal(_) => {}
Component::ParentDir => {
anyhow::bail!("path contains parent-directory component");
}
Component::CurDir => {
anyhow::bail!("path contains current-directory component");
// Explicit component walk — sanitisation barrier that static analysers
// (CodeQL `rust/path-injection`) recognise. The path must be composed
// entirely of Normal segments after the root prefix; any ParentDir or
// CurDir component is a traversal attempt.
for component in local_path.components() {
use std::path::Component;
match component {
Component::Prefix(_) | Component::RootDir | Component::Normal(_) => {}
Component::ParentDir => {
anyhow::bail!("path contains parent-directory component");
}
Component::CurDir => {
anyhow::bail!("path contains current-directory component");
}
}
}
}
Ok(local_path)
Ok((owner_slug, local_path))
}
}
/// Strict allowlist validator for `owner_did` and `repo_name`.
@@ -461,19 +347,6 @@ fn validate_repo_name(repo_name: &str) -> Result<()> {
Ok(())
}
/// Error marker for "no lock-pool connection was available in time".
///
/// Carried through the `anyhow` chain (like [`smart_http::GitServiceTimeout`]) so the
/// HTTP handler can `downcast_ref` it and shed a 503 + Retry-After instead of the
/// generic 500 a git error maps to: an exhausted lock pool is a CAPACITY signal, and
/// telling the client to retry shortly is the same shed semantics the surrounding
/// admission code already uses (#173 F1).
///
/// [`smart_http::GitServiceTimeout`]: crate::git::smart_http::GitServiceTimeout
#[derive(Debug, thiserror::Error)]
#[error("no lock-pool connection available")]
pub struct LockPoolBusy;
/// Guard returned by `acquire_write()`. Holds the Postgres advisory lock and
/// uploads to Tigris + releases the lock on `release()`.
pub struct RepoWriteGuard {
@@ -481,17 +354,8 @@ pub struct RepoWriteGuard {
repo_name: String,
pub local_path: PathBuf,
lock_key: i64,
/// The lock-pool connection that TOOK the advisory lock. It must be the one
/// that releases it (session locks are owned by their connection), and
/// holding it here is also what makes a guard dropped without `release`
/// safe: the drop returns the connection through `after_release`, which
/// clears the lock.
lock_conn: PoolConnection<Postgres>,
pool: PgPool,
tigris: Option<TigrisClient>,
/// Shared with the store that handed this guard out; see
/// [`RepoStore::upload_site_reached`].
#[cfg(test)]
upload_site_reached: Arc<std::sync::atomic::AtomicUsize>,
}
impl RepoWriteGuard {
@@ -505,16 +369,9 @@ impl RepoWriteGuard {
/// half-applied or otherwise inconsistent repo would propagate corruption to
/// Tigris (and to every node that later downloads it). The lock is always
/// released regardless, to avoid stale locks blocking future writes.
pub async fn release(mut self, success: bool) {
pub async fn release(self, success: bool) {
// Upload to Tigris only on success.
if success {
// The upload site, recorded for tests before the client is consulted: with
// no injectable seam on `TigrisClient` a counter inside the arm below could
// never move, and it is reaching this point at all that an interrupted push
// must not do (#173 F2).
#[cfg(test)]
self.upload_site_reached
.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
if let Some(ref tigris) = self.tigris {
if let Err(e) = tigris
.upload(&self.owner_slug, &self.repo_name, &self.local_path)
@@ -527,56 +384,14 @@ impl RepoWriteGuard {
warn!(repo = %self.repo_name, "write failed — skipping tigris upload to avoid propagating an inconsistent repo");
}
// Release the advisory lock on the connection that took it. Anything else
// (a fresh `&pool` checkout) is a no-op that returns false: Postgres
// scopes a session lock to its owning connection.
// Release advisory lock
let _ = sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(self.lock_key)
.execute(&mut *self.lock_conn)
.execute(&self.pool)
.await;
// Dropping `self` returns the connection to the lock pool, where
// `after_release` sweeps anything the unlock above missed.
}
}
/// Build the dedicated advisory-lock pool a `RepoStore` runs its write locks on.
/// Connect options are cloned off an existing pool so callers need not re-parse
/// the database URL; the pool is lazy, so no connection is opened here.
///
/// Two properties, both load-bearing:
///
/// * The `after_release` hook runs `pg_advisory_unlock_all()` before a
/// connection goes back into the pool. sqlx's `PoolConnection::drop` spawns
/// `return_to_pool()`, which invokes this hook, so a connection dropped by
/// CANCELLATION still clears its locks. That is what keeps an `acquire_write`
/// killed mid-Tigris by the caller's `tokio::time::timeout` from leaking a
/// lock and wedging every later push to that repo (#173). Note the hook runs
/// from that spawned task, so the unlock is asynchronous with respect to the
/// drop: the lock clears shortly after the connection goes away, not
/// synchronously with it.
/// * It is a SEPARATE pool from the main query pool, not a slice of it. A push
/// holds its lock connection for the whole receive-pack, and
/// `db_max_connections` (default 20) is well below
/// `max_concurrent_git_pushes` (default 32), so drawing these from the main
/// pool would starve every other query during a push burst.
///
/// `acquire_timeout` bounds the wait when every lock-pool connection is busy, so
/// exhaustion surfaces as a clean error rather than an unbounded hang.
pub fn build_lock_pool(source: &PgPool, max_connections: u32, acquire_timeout: Duration) -> PgPool {
PgPoolOptions::new()
.max_connections(max_connections)
.acquire_timeout(acquire_timeout)
.after_release(|conn, _meta| {
Box::pin(async move {
sqlx::query("SELECT pg_advisory_unlock_all()")
.execute(&mut *conn)
.await?;
Ok(true)
})
})
.connect_lazy_with((*source.connect_options()).clone())
}
/// Compute a stable i64 hash for a Postgres advisory lock key.
fn advisory_lock_key(owner_slug: &str, repo_name: &str) -> i64 {
use std::hash::{Hash, Hasher};
@@ -589,503 +404,6 @@ fn advisory_lock_key(owner_slug: &str, repo_name: &str) -> i64 {
#[cfg(test)]
mod tests {
use super::*;
use std::time::Duration;
// ── advisory-lock test helpers (#173 U1) ───────────────────────────────
/// Postgres advisory locks live in a CLUSTER-wide space, not a per-database
/// one, so two `#[sqlx::test]` cases running against their own temporary
/// databases still share the key space. Every lock test therefore mints its
/// own key instead of reusing a fixed constant.
fn unique_lock_key() -> i64 {
use std::sync::atomic::{AtomicI64, Ordering};
static NEXT: AtomicI64 = AtomicI64::new(0);
let n = NEXT.fetch_add(1, Ordering::Relaxed);
((std::process::id() as i64) << 24) | (n & 0xff_ffff)
}
/// A plain pool (no `after_release` hook, no idle timeout) at the same
/// database as the `#[sqlx::test]` pool. Two separate reasons these tests
/// cannot just use the pool the harness hands them:
///
/// 1. Observing lock state has to happen from a session that is definitely
/// not the one under test. Session advisory locks are re-entrant, so
/// `pg_try_advisory_lock` on the very connection that already holds the key
/// returns true, and a same-pool probe silently reports a leaked lock free.
/// 2. The harness pool sets `idle_timeout(1s)`, so a connection returned to it
/// is closed about a second later and Postgres drops every lock that
/// session held. That would mask exactly the leak these tests exist to
/// catch, so the store under test runs on one of these too.
fn sibling_pool(pool: &PgPool, max_connections: u32) -> PgPool {
sqlx::postgres::PgPoolOptions::new()
.max_connections(max_connections)
.connect_lazy_with((*pool.connect_options()).clone())
}
/// Probe the lock from a connection that is NOT the one under test. Session
/// advisory locks are re-entrant within their own session, so a check from the
/// holding connection would pass vacuously and prove nothing.
async fn lock_is_free_elsewhere(pool: &PgPool, key: i64) -> bool {
let mut probe = pool.acquire().await.expect("probe connection");
let taken: (bool,) = sqlx::query_as("SELECT pg_try_advisory_lock($1)")
.bind(key)
.fetch_one(&mut *probe)
.await
.expect("probe try-lock");
if taken.0 {
sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(key)
.execute(&mut *probe)
.await
.expect("probe unlock");
}
taken.0
}
/// `after_release` runs from the task sqlx spawns in `PoolConnection::drop`,
/// so the unlock is ASYNCHRONOUS with respect to the drop. Callers must poll
/// rather than assume the lock is gone the instant the connection goes away.
async fn wait_until_free(pool: &PgPool, key: i64, within: Duration) -> bool {
let deadline = std::time::Instant::now() + within;
loop {
if lock_is_free_elsewhere(pool, key).await {
return true;
}
if std::time::Instant::now() >= deadline {
return false;
}
tokio::time::sleep(Duration::from_millis(25)).await;
}
}
// ── DESIGN GATE ────────────────────────────────────────────────────────
// The whole cancellation-safety design rests on one sqlx behaviour:
// `PoolConnection::drop` spawns `return_to_pool()`, which invokes the pool's
// `after_release` hook before the connection is reused. If that holds, a
// connection dropped by cancellation still runs `pg_advisory_unlock_all()`
// and the lock cannot leak. This test proves it by execution, through the
// production `build_lock_pool` so that stripping the hook there turns it red.
#[sqlx::test]
async fn dropped_pool_connection_runs_after_release_and_clears_locks(pool: PgPool) {
let key = unique_lock_key();
let lock_pool = build_lock_pool(&pool, 4, Duration::from_secs(5));
{
let mut conn = lock_pool.acquire().await.expect("lock-pool connection");
let taken: (bool,) = sqlx::query_as("SELECT pg_try_advisory_lock($1)")
.bind(key)
.fetch_one(&mut *conn)
.await
.expect("try-lock");
assert!(taken.0, "first try-lock must succeed");
assert!(
!lock_is_free_elsewhere(&pool, key).await,
"lock must be observably HELD from another session while the connection lives"
);
// Drop WITHOUT calling pg_advisory_unlock: this models cancellation.
}
assert!(
wait_until_free(&pool, key, Duration::from_secs(5)).await,
"after_release must clear the advisory lock of a dropped connection"
);
}
// ── acquire_write cancellation safety (#173 U1) ────────────────────────
/// The reviewer's named regression. `api/repos.rs` wraps `acquire_write` in a
/// `tokio::time::timeout`; when that fires during the Tigris phase the future
/// is dropped after the advisory lock was taken and before `RepoWriteGuard`
/// (the only thing that unlocks) exists. The lock then leaks and every later
/// push to the same repo spins the 60-attempt / 60s ceiling and fails.
#[sqlx::test]
async fn cancelled_acquire_write_mid_tigris_does_not_leak_the_lock(pool: PgPool) {
let repos_dir = PathBuf::from("/tmp/gitlawb-test-repos");
let owner = "did:key:z6MkCancelMidTigris";
let repo = "cancel-mid-tigris";
let store_pool = sibling_pool(&pool, 8);
let stalling = RepoStore::for_testing(repos_dir.clone(), store_pool.clone())
.with_tigris_stall(Duration::from_secs(30));
let cancelled = tokio::time::timeout(
Duration::from_millis(500),
stalling.acquire_write(owner, repo),
)
.await;
assert!(
cancelled.is_err(),
"the acquire must still be inside the Tigris phase when the timeout fires"
);
// Observed from an independent session, so the check cannot be satisfied
// by re-entrancy on whichever pooled connection happens to be handed back.
let probe = sibling_pool(&pool, 2);
let key = advisory_lock_key(&owner.replace([':', '/'], "_"), repo);
assert!(
wait_until_free(&probe, key, Duration::from_secs(5)).await,
"a cancelled acquire_write must leave no advisory lock held"
);
// A subsequent acquire for the SAME repo must succeed promptly. Before the
// fix it blocks on the leaked lock until the 60-attempt ceiling.
let store = RepoStore::for_testing(repos_dir, store_pool);
let guard = tokio::time::timeout(Duration::from_secs(5), store.acquire_write(owner, repo))
.await
.expect("second acquire_write must not block on a leaked lock")
.expect("second acquire_write must succeed");
guard.release(false).await;
}
/// Cancellation BEFORE the lock is taken must leave nothing behind: no lock,
/// and no lock-pool connection stranded. The lock pool here holds exactly one
/// connection, so a stranded one would make the follow-up acquire time out
/// waiting for a checkout.
#[sqlx::test]
async fn cancelled_acquire_write_before_the_lock_leaves_nothing_held(pool: PgPool) {
let probe = sibling_pool(&pool, 2);
let owner = "did:key:z6MkCancelEarly";
let repo = "cancel-early";
let key = advisory_lock_key(&owner.replace([':', '/'], "_"), repo);
let store = RepoStore::new(
PathBuf::from("/tmp/gitlawb-test-repos"),
None,
build_lock_pool(&pool, 1, Duration::from_secs(3)),
);
// A zero deadline polls the future once, which gets it no further than the
// first await (the pool checkout / the first try-lock round trip), so it is
// cancelled before any lock can be taken.
let cancelled =
tokio::time::timeout(Duration::ZERO, store.acquire_write(owner, repo)).await;
assert!(cancelled.is_err(), "the acquire must be cancelled");
assert!(
lock_is_free_elsewhere(&probe, key).await,
"no lock may be held when the acquire never got that far"
);
// The single lock-pool connection must be back: if cancellation stranded
// it, this checkout blocks until the 3s acquire timeout and fails.
let guard = tokio::time::timeout(Duration::from_secs(2), store.acquire_write(owner, repo))
.await
.expect("the lock-pool connection must have been returned")
.expect("acquire after cancellation");
guard.release(false).await;
}
/// Lock-pool exhaustion is a bounded wait and a clean error, never a panic and
/// never an unbounded hang.
#[sqlx::test]
async fn lock_pool_exhaustion_is_a_bounded_error(pool: PgPool) {
let owner = "did:key:z6MkExhaustion";
let store = RepoStore::new(
PathBuf::from("/tmp/gitlawb-test-repos"),
None,
build_lock_pool(&pool, 1, Duration::from_secs(2)),
);
let held = store
.acquire_write(owner, "exhaust-a")
.await
.expect("first acquire");
// Different repo, so this is not the advisory lock queueing: the only
// connection in the lock pool is checked out by `held`.
let started = std::time::Instant::now();
let err = tokio::time::timeout(
Duration::from_secs(10),
store.acquire_write(owner, "exhaust-b"),
)
.await
.expect("the wait must be bounded by the pool acquire timeout");
let err = match err {
Ok(_) => panic!("an exhausted lock pool must surface an error, not a guard"),
Err(e) => e,
};
assert!(
started.elapsed() < Duration::from_secs(6),
"the error must arrive on the acquire timeout, not after a long hang"
);
assert!(
err.to_string().contains("lock-pool connection"),
"the error must name the lock-pool checkout, got: {err}"
);
held.release(false).await;
}
/// #173 F1 (RED-before/GREEN-after). A contended `acquire_write` spins for up to
/// 60 one-second attempts. It must not OCCUPY a lock-pool connection for that whole
/// spin: `acquire_write` has non-push callers (`api/issues.rs`, `api/pulls.rs`) that
/// hold no concurrency permit, so any self-minted did:key could otherwise park a
/// connection per call and starve authenticated pushes on EVERY repo.
///
/// Lock pool of exactly 2, two spinners. Pre-fix (checkout hoisted above the retry
/// loop) they pin both connections for the full spin and an UNCONTENDED acquire on a
/// third repo dies on the pool acquire timeout. Post-fix each spinner returns its
/// connection before sleeping, so it occupies ~0 and the uncontended acquire sails
/// through.
#[sqlx::test]
async fn a_spinning_acquire_write_does_not_occupy_a_lock_pool_connection(pool: PgPool) {
let owner = "did:key:z6MkSpinOccupancy";
let owner_slug = owner.replace([':', '/'], "_");
let store = RepoStore::new(
PathBuf::from("/tmp/gitlawb-test-repos"),
None,
build_lock_pool(&pool, 2, Duration::from_secs(2)),
);
// An independent session holds both contended keys, so the spinners' try-locks
// return false on every iteration and they stay in the retry loop.
let holder = sibling_pool(&pool, 2);
let mut held_conn = holder.acquire().await.expect("holder connection");
for repo in ["spin-a", "spin-b"] {
let taken: (bool,) = sqlx::query_as("SELECT pg_try_advisory_lock($1)")
.bind(advisory_lock_key(&owner_slug, repo))
.fetch_one(&mut *held_conn)
.await
.expect("holder try-lock");
assert!(taken.0, "the holder must own {repo}'s key");
}
let mut spinners = Vec::new();
for repo in ["spin-a", "spin-b"] {
let store = store.clone();
spinners.push(tokio::spawn(async move {
store.acquire_write(owner, repo).await
}));
}
// Let both reach the spin (each has done at least one failed try-lock by now).
tokio::time::sleep(Duration::from_millis(500)).await;
let started = std::time::Instant::now();
let uncontended = tokio::time::timeout(
Duration::from_secs(10),
store.acquire_write(owner, "spin-free"),
)
.await
.expect("the uncontended acquire must return, not hang");
let elapsed = started.elapsed();
let free_guard = uncontended.unwrap_or_else(|e| {
panic!(
"an UNCONTENDED acquire_write on a DIFFERENT repo must not be starved by \
spinners holding the lock pool; got: {e}"
)
});
assert!(
elapsed < Duration::from_secs(2),
"the uncontended acquire must not queue behind the spinners for the pool \
acquire timeout; took {elapsed:?}"
);
free_guard.release(false).await;
// The drop-and-retake cycle must still END in a real, exclusive lock: free
// spin-a's key and the spinner that was cycling connections must take it.
sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(advisory_lock_key(&owner_slug, "spin-a"))
.execute(&mut *held_conn)
.await
.expect("release spin-a");
let winner = tokio::time::timeout(Duration::from_secs(15), spinners.remove(0))
.await
.expect("the spinner must finish once its key frees")
.expect("spinner task")
.expect("the spinner must acquire once the key frees");
let probe = sibling_pool(&pool, 2);
assert!(
!lock_is_free_elsewhere(&probe, advisory_lock_key(&owner_slug, "spin-a")).await,
"the lock a spinner finally took must be observably held from another session"
);
winner.release(false).await;
for s in spinners {
s.abort();
}
sqlx::query("SELECT pg_advisory_unlock_all()")
.execute(&mut *held_conn)
.await
.expect("release the remaining holder lock");
}
/// #173 F1, the property the fix rests on: returning a lock-pool connection that
/// holds NOTHING runs `after_release`'s `pg_advisory_unlock_all()`, which is a no-op
/// and must not disturb a lock held on a DIFFERENT connection of the same pool.
/// Session advisory locks are per connection, so this is by construction, but the
/// spin fix depends on it, so it is proven by execution rather than assumed.
#[sqlx::test]
async fn returning_an_unlocked_connection_does_not_clear_another_connections_lock(
pool: PgPool,
) {
let owner = "did:key:z6MkNoOpUnlockAll";
let repo = "noop-unlock";
let key = advisory_lock_key(&owner.replace([':', '/'], "_"), repo);
let probe = sibling_pool(&pool, 2);
let lock_pool = build_lock_pool(&pool, 4, Duration::from_secs(5));
let store = RepoStore::new(
PathBuf::from("/tmp/gitlawb-test-repos"),
None,
lock_pool.clone(),
);
let guard = store.acquire_write(owner, repo).await.expect("acquire");
// Churn the pool: check out and drop connections that hold no lock, exactly what
// a spinning acquire now does between attempts. Each return fires
// pg_advisory_unlock_all() on that connection.
for _ in 0..10 {
let mut conn = lock_pool.acquire().await.expect("churn checkout");
let _: (i32,) = sqlx::query_as("SELECT 1")
.fetch_one(&mut *conn)
.await
.expect("churn query");
drop(conn);
tokio::time::sleep(Duration::from_millis(10)).await;
}
assert!(
!lock_is_free_elsewhere(&probe, key).await,
"a held write lock must survive other lock-pool connections being returned"
);
guard.release(true).await;
assert!(
lock_is_free_elsewhere(&probe, key).await,
"release must still free the lock after the churn"
);
}
/// #173 F1: lock-pool exhaustion is a DISTINCT error the handler can shed as a 503,
/// not a generic git 500. Both directions: an exhausted pool downcasts to
/// [`LockPoolBusy`], and an unrelated failure (a rejected repo name) does not.
#[sqlx::test]
async fn lock_pool_exhaustion_is_a_distinct_downcastable_error(pool: PgPool) {
let owner = "did:key:z6MkBusyDowncast";
let store = RepoStore::new(
PathBuf::from("/tmp/gitlawb-test-repos"),
None,
build_lock_pool(&pool, 1, Duration::from_secs(1)),
);
let held = store
.acquire_write(owner, "busy-a")
.await
.expect("first acquire");
let err = match store.acquire_write(owner, "busy-b").await {
Ok(_) => panic!("an exhausted lock pool must error, not hand back a guard"),
Err(e) => e,
};
assert!(
err.downcast_ref::<LockPoolBusy>().is_some(),
"lock-pool exhaustion must be downcastable so the handler sheds 503, got: {err}"
);
// MUST-NOT: an ordinary rejection is not a capacity signal.
let other = match store.acquire_write(owner, "../escape").await {
Ok(_) => panic!("a traversal repo name must be rejected"),
Err(e) => e,
};
assert!(
other.downcast_ref::<LockPoolBusy>().is_none(),
"a validation failure must not masquerade as lock-pool capacity, got: {other}"
);
held.release(false).await;
}
/// Round trip: the lock is observably HELD between acquire and release, and
/// observably FREE after. Both checks run from an independent session; from
/// the holding session they would pass vacuously (session locks are
/// re-entrant) and would not notice an unlock that landed on the wrong
/// connection.
#[sqlx::test]
async fn acquire_write_holds_the_lock_until_release(pool: PgPool) {
let probe = sibling_pool(&pool, 2);
let owner = "did:key:z6MkRoundTrip";
let repo = "round-trip";
let key = advisory_lock_key(&owner.replace([':', '/'], "_"), repo);
let store = RepoStore::for_testing(PathBuf::from("/tmp/gitlawb-test-repos"), pool.clone());
let guard = store.acquire_write(owner, repo).await.expect("acquire");
assert!(
!lock_is_free_elsewhere(&probe, key).await,
"the lock must be held while the guard is alive"
);
// No polling here, deliberately. `release` must free the lock SYNCHRONOUSLY,
// which it can only do by unlocking on the connection that took it; a
// `pg_advisory_unlock` sent through the pool would land on some other
// session and return false. The `after_release` hook is a net for the
// cancellation path and fires from a spawned task well after this point, so
// it must not be what makes this assertion pass.
guard.release(true).await;
assert!(
lock_is_free_elsewhere(&probe, key).await,
"release must free the lock as seen from another session"
);
}
/// `release(false)` skips the Tigris upload but must still free the lock; a
/// failed write that kept the lock would wedge the repo.
#[sqlx::test]
async fn release_after_failed_write_still_frees_the_lock(pool: PgPool) {
let probe = sibling_pool(&pool, 2);
let owner = "did:key:z6MkFailedWrite";
let repo = "failed-write";
let key = advisory_lock_key(&owner.replace([':', '/'], "_"), repo);
let store = RepoStore::for_testing(PathBuf::from("/tmp/gitlawb-test-repos"), pool.clone());
let guard = store.acquire_write(owner, repo).await.expect("acquire");
guard.release(false).await;
assert!(
wait_until_free(&probe, key, Duration::from_secs(5)).await,
"release(success = false) must still free the lock"
);
}
/// The lock is per repo: a second acquire for the SAME repo waits for the
/// first to release, while a different repo proceeds straight through.
#[sqlx::test]
async fn same_repo_acquires_serialize_and_different_repos_do_not(pool: PgPool) {
let repos_dir = PathBuf::from("/tmp/gitlawb-test-repos");
let owner = "did:key:z6MkSerialize";
let store = RepoStore::for_testing(repos_dir, pool.clone());
let first = store
.acquire_write(owner, "serialize-a")
.await
.expect("first acquire");
// Different repo: unaffected by the held lock.
let other = tokio::time::timeout(
Duration::from_secs(2),
store.acquire_write(owner, "serialize-b"),
)
.await
.expect("a different repo must not wait on this lock")
.expect("acquire other repo");
other.release(false).await;
// Same repo: must not acquire while `first` is alive.
let contender = tokio::spawn({
let store = store.clone();
async move { store.acquire_write(owner, "serialize-a").await }
});
tokio::time::sleep(Duration::from_millis(1500)).await;
assert!(
!contender.is_finished(),
"a second acquire for the same repo must block while the first guard lives"
);
first.release(false).await;
let second = tokio::time::timeout(Duration::from_secs(10), contender)
.await
.expect("contender must finish once the lock is free")
.expect("contender task")
.expect("contender acquire");
second.release(false).await;
}
// ── repo_name validation ───────────────────────────────────────────────
File diff suppressed because it is too large. Load diff
-200
View File
@@ -306,90 +306,6 @@ pub fn read_object_content(repo_path: &Path, sha256_hex: &str, obj_type: &str) -
Ok(content_output.stdout)
}
/// Bounded twin of [`object_type`] for the `GET /ipfs/{cid}` serve path (#173
/// round-10, R1/KTD2). Runs `git cat-file -t` under
/// [`run_bounded_git`](crate::git::visibility_pack::run_bounded_git) so the child runs
/// in its own process group and a watchdog reaps it (SIGTERM -> grace -> SIGKILL) at
/// `timeout`. The bare [`object_type`] is a `spawn_blocking` `Command::output` that an
/// async timeout cannot cancel, so a wedged `cat-file` there pins the caller's held
/// /ipfs walk admission for the whole hang; this twin cannot. Semantics mirror
/// [`object_type`]: `Ok(Some(t))` for an existing object, `Ok(None)` when git exits
/// non-zero without timing out (the object is absent), and
/// `Err(`[`GitServiceTimeout`](crate::git::smart_http::GitServiceTimeout)`)` on the
/// deadline so the handler can mark the search truncated (retryable 503) rather than a
/// false not-found. Callers off the /ipfs path keep the bare helper.
pub fn object_type_bounded(
git_bin: &str,
repo_path: &Path,
sha256_hex: &str,
timeout: std::time::Duration,
) -> Result<Option<String>> {
let deadline = std::time::Instant::now() + timeout;
match crate::git::visibility_pack::run_bounded_git(
git_bin,
&["cat-file", "-t", sha256_hex],
repo_path,
b"",
deadline,
) {
Ok(out) => Ok(Some(String::from_utf8_lossy(&out).trim().to_string())),
Err(e) if e.is::<crate::git::smart_http::GitServiceTimeout>() => Err(e),
// A non-timeout failure is git reporting no such object (a non-zero exit),
// matching `object_type`'s `Ok(None)`.
Err(_) => Ok(None),
}
}
/// Bounded `git cat-file -s` size read for the `GET /ipfs/{cid}` serve path (#173
/// round-10, R1/KTD2): reads the object size WITHOUT its content (so an oversized object
/// is rejected before it is buffered, #173 F6), under
/// [`run_bounded_git`](crate::git::visibility_pack::run_bounded_git) so a wedged size
/// read is reaped at `timeout` instead of pinning the held /ipfs walk admission.
/// `Ok(Some(n))` on success, `Ok(None)` when the object is absent (a non-timeout
/// non-zero exit), `Err(GitServiceTimeout)` on the deadline.
pub fn object_size_bounded(
git_bin: &str,
repo_path: &Path,
sha256_hex: &str,
timeout: std::time::Duration,
) -> Result<Option<u64>> {
let deadline = std::time::Instant::now() + timeout;
match crate::git::visibility_pack::run_bounded_git(
git_bin,
&["cat-file", "-s", sha256_hex],
repo_path,
b"",
deadline,
) {
Ok(out) => Ok(String::from_utf8_lossy(&out).trim().parse::<u64>().ok()),
Err(e) if e.is::<crate::git::smart_http::GitServiceTimeout>() => Err(e),
Err(_) => Ok(None),
}
}
/// Bounded twin of [`read_object_content`] for the `GET /ipfs/{cid}` serve path (#173
/// round-10, R1/KTD2): `git cat-file <type>` under
/// [`run_bounded_git`](crate::git::visibility_pack::run_bounded_git) so a wedged content
/// read is reaped at `timeout` instead of pinning the held /ipfs walk admission. Returns
/// the raw object bytes on success and an error (including `GitServiceTimeout` on the
/// deadline) otherwise, mirroring [`read_object_content`].
pub fn read_object_content_bounded(
git_bin: &str,
repo_path: &Path,
sha256_hex: &str,
obj_type: &str,
timeout: std::time::Duration,
) -> Result<Vec<u8>> {
let deadline = std::time::Instant::now() + timeout;
crate::git::visibility_pack::run_bounded_git(
git_bin,
&["cat-file", obj_type, sha256_hex],
repo_path,
b"",
deadline,
)
}
/// Read a git object by its SHA-256 hex object ID.
///
/// Returns `(object_type, content_bytes)` where `content_bytes` is the raw
@@ -407,42 +323,6 @@ pub fn read_object(repo_path: &Path, sha256_hex: &str) -> Result<Option<(String,
Ok(Some((obj_type, content)))
}
/// Bounded twin of [`read_object`] for write-side callers that must not hang on a
/// wedged `git cat-file` (the post-push pin path, #173): composes
/// [`object_type_bounded`] and [`read_object_content_bounded`] under ONE shared
/// deadline (mirroring `get_by_cid`'s read stage and `build_filtered_pack`), so a
/// single object read holds for at most `timeout` total, not one full timeout per
/// stage. `Ok(Some((type, bytes)))` on success, `Ok(None)` when the object is absent,
/// and `Err(`[`GitServiceTimeout`](crate::git::smart_http::GitServiceTimeout)`)` when
/// either stage overruns the deadline. The bare [`read_object`] is an unbounded
/// `Command::output` an async timeout cannot cancel, so a wedged `cat-file` there pins
/// the post-push coalescing key until process death; this twin cannot.
pub fn read_object_bounded(
git_bin: &str,
repo_path: &Path,
sha256_hex: &str,
timeout: std::time::Duration,
) -> Result<Option<(String, Vec<u8>)>> {
let deadline = std::time::Instant::now() + timeout;
let obj_type = match object_type_bounded(
git_bin,
repo_path,
sha256_hex,
deadline.saturating_duration_since(std::time::Instant::now()),
)? {
Some(t) => t,
None => return Ok(None),
};
let content = read_object_content_bounded(
git_bin,
repo_path,
sha256_hex,
&obj_type,
deadline.saturating_duration_since(std::time::Instant::now()),
)?;
Ok(Some((obj_type, content)))
}
/// Get the diff between two branches: changes on source_branch not in target_branch.
pub fn branch_diff(repo_path: &Path, target_branch: &str, source_branch: &str) -> Result<String> {
let output = Command::new("git")
@@ -621,84 +501,4 @@ mod tests {
"unchanged file must not appear: {names:?}"
);
}
/// #173 round-10 (KTD2): `object_type_bounded` reaps a wedged `cat-file` child at its
/// deadline instead of blocking on it to natural exit, so a hung probe cannot pin the
/// /ipfs walk admission the owning task holds. A fake `git` records its pid and sleeps
/// far past the 1s deadline; the `run_bounded_git` watchdog (SIGTERM -> grace ->
/// SIGKILL of the process group) must kill it well before that natural exit, and the
/// call must surface `GitServiceTimeout`. REVERT PROOF (RED): swap the twin's
/// `run_bounded_git` for the bare `Command::output()` and the wedged child stays alive
/// past the deadline — the mid-flight liveness poll below reads it still running.
#[cfg(unix)]
#[test]
fn object_type_bounded_reaps_wedged_child_at_deadline() {
use std::time::Duration;
let tmp = tempfile::TempDir::new().unwrap();
let pidfile = tmp.path().join("catfile.pid");
// `cat-file` records its own pid then sleeps 8s (>> the 1s deadline) so the probe
// is genuinely wedged; the watchdog is what must end it.
let body = format!(
"#!/bin/sh\n\
case \"$1\" in\n\
cat-file) echo $$ > \"{}\"; sleep 8 ;;\n\
*) : ;;\n\
esac\n\
exit 0\n",
pidfile.display()
);
let git_path = tmp.path().join("fakegit");
std::fs::write(&git_path, &body).unwrap();
{
use std::os::unix::fs::PermissionsExt;
let mut perm = std::fs::metadata(&git_path).unwrap().permissions();
perm.set_mode(0o755);
std::fs::set_permissions(&git_path, perm).unwrap();
}
let repo = tmp.path().to_path_buf();
let git = git_path.to_str().unwrap().to_string();
let alive = |pid: i32| unsafe { libc::kill(pid, 0) == 0 };
// The bounded probe blocks until the watchdog tears the child down, so run it on
// a worker thread and poll for the reap from here.
let handle = std::thread::spawn(move || {
super::object_type_bounded(&git, &repo, "deadbeef", Duration::from_secs(1))
});
let mut pid = None;
for _ in 0..500 {
if let Some(p) = std::fs::read_to_string(&pidfile)
.ok()
.and_then(|s| s.trim().parse::<i32>().ok())
{
pid = Some(p);
break;
}
std::thread::sleep(Duration::from_millis(10));
}
let pid = pid.expect("the fake cat-file must have spawned and recorded its pid");
// Past the 1s deadline + SIGTERM grace but well before the 8s natural exit: the
// watchdog must already have reaped the wedged group. A bare, unbounded read would
// leave it running here — the load-bearing RED.
std::thread::sleep(Duration::from_secs(3));
let reaped = !alive(pid);
// Defensive reap so a RED run leaks no orphan.
unsafe {
libc::kill(pid, libc::SIGKILL);
}
assert!(
reaped,
"object_type_bounded must reap the wedged cat-file child at the deadline, \
not leave it running to its natural exit"
);
let res = handle.join().expect("probe thread joins");
let err = res.expect_err("a deadline overrun must be an error, not a value");
assert!(
err.is::<crate::git::smart_http::GitServiceTimeout>(),
"a deadline overrun must surface GitServiceTimeout, got: {err:?}"
);
}
}
File diff suppressed because it is too large. Load diff
+29 -5
View File
@@ -73,9 +73,25 @@ impl QueryRoot {
.await
.map_err(|e| async_graphql::Error::new(e.to_string()))?;
Ok(updates
// Resolve the trusted display owner_did per row, identical to the REST
// feed: the stored wire value is untrusted, so it is echoed only when it
// matches the canonical owner of the local repo the slug names (#P1);
// legacy None rows are attributed via an exact unique local match (#P3).
// The batch resolver issues at most one query per distinct local repo
// rather than one per event row (#P2).
let pairs: Vec<(&str, Option<&str>)> = updates
.iter()
.map(|u| (u.repo.as_str(), u.owner_did.as_deref()))
.collect();
let owner_dids = db
.resolve_ref_update_owner_dids(&pairs)
.await
.map_err(|e| async_graphql::Error::new(e.to_string()))?;
let resolved: Vec<RefUpdateType> = updates
.into_iter()
.map(|u| RefUpdateType {
.zip(owner_dids)
.map(|(u, owner_did)| RefUpdateType {
repo: u.repo,
ref_name: u.ref_name,
old_sha: u.old_sha,
@@ -83,8 +99,10 @@ impl QueryRoot {
pusher_did: u.pusher_did,
node_did: u.node_did,
timestamp: u.timestamp,
owner_did,
})
.collect())
.collect();
Ok(resolved)
}
async fn tasks(
@@ -162,6 +180,7 @@ mod tests {
cert_id: None,
received_at: Utc::now().to_rfc3339(),
from_peer: "peer1".into(),
owner_did: None,
}
}
@@ -245,10 +264,10 @@ mod tests {
.await
.unwrap();
let schema = schema(db);
let q = r#"{ refUpdates { repo refName } }"#;
let q = r#"{ refUpdates { repo refName ownerDid } }"#;
let resp = anon(&schema, q).await;
assert_eq!(count(&resp), 1);
// The one row returned must be the public repo's.
// The one row returned must be the public repo's with owner_did echoed.
let async_graphql::Value::Object(obj) = &resp.data else {
unreachable!()
};
@@ -262,6 +281,11 @@ mod tests {
row.get("repo").unwrap(),
&async_graphql::Value::from("z6MkOwner/openrepo")
);
assert_eq!(
row.get("ownerDid").unwrap(),
&async_graphql::Value::from(OWNER),
"ownerDid must fall back to the local record's owner for legacy rows"
);
}
// Scenario 4 — alias fail-closed: private repo's row stored full-DID form.
@@ -39,6 +39,7 @@ impl SubscriptionRoot {
pusher_did: ev.pusher_did,
node_did: ev.node_did,
timestamp: ev.timestamp,
owner_did: Some(ev.owner_did),
}),
_ => None,
}
+1
View File
@@ -57,6 +57,7 @@ pub struct RefUpdateType {
pub pusher_did: String,
pub node_did: String,
pub timestamp: String,
pub owner_did: Option<String>,
}
#[derive(SimpleObject, Clone)]
+13 -544
View File
@@ -9,372 +9,6 @@
use anyhow::Result;
use gitlawb_core::cid::Cid;
use std::time::Duration;
/// Attempts (including the first) for a transient DB-record retry.
const PIN_RECORD_ATTEMPTS: u32 = 3;
/// Backoff between DB-record retry attempts.
const PIN_RECORD_BACKOFF: Duration = Duration::from_millis(50);
/// Run an idempotent DB-record operation with a bounded retry so a sub-second
/// transient error does not silently leave the pin-source set permanently
/// incomplete. The resolver treats a nonempty below-cap source set as complete,
/// so a dropped `record_pin_source`/`record_pinned_cid` makes `GET /ipfs/{cid}`
/// 404 a valid public copy. Every wrapped insert is idempotent (`ON CONFLICT DO
/// NOTHING` / provenance-preserving upsert), so re-running is safe. On exhausted
/// attempts the last error is returned and the caller records the durable
/// `pin_sources_incomplete` marker (U3, #173), which is what keeps the resolver's
/// bounded scan fallback available for that object instead of 404ing a public copy.
/// Shared with the `pinata.rs` twin so both pin paths retry identically. Runs
/// inside the already-detached post-push task, so the backoff adds no push latency.
pub(crate) async fn retry_db_record<F, Fut>(mut op: F) -> Result<()>
where
F: FnMut() -> Fut,
Fut: std::future::Future<Output = Result<()>>,
{
let mut attempt = 1;
loop {
match op().await {
Ok(()) => return Ok(()),
Err(e) => {
if attempt >= PIN_RECORD_ATTEMPTS {
return Err(e);
}
tokio::time::sleep(PIN_RECORD_BACKOFF).await;
attempt += 1;
}
}
}
}
/// Opportunistically repair a legacy provider-CID row on the already-pinned skip
/// path (#173 R8, KTD8). Releases before this branch stored the PROVIDER CID
/// (Kubo dag-pb / Pinata CIDv0) in `pinned_cids.cid`; the `/ipfs` resolver
/// recomputes the raw CID from object bytes and 404s any row whose key does not
/// match, yet `list_pinned_cids` still advertises the stored key — so a client
/// gets a CID the resolver deliberately withholds. When a re-push carries the
/// object again, rewrite the key to the raw CID and stash the old provider value
/// in `legacy_provider_cid`.
///
/// COST GATE: candidacy is decided from the stored key's codec alone — a
/// CIDv1/raw key is already the resolver key and reads NO bytes, keeping the
/// steady-state skip cost DB-only. Only a legacy-codec row reads the object to
/// recompute. A row whose bytes are gone stays withheld (no destructive rewrite).
async fn repair_legacy_provider_cid(
repo_path: &std::path::Path,
git_bin: &str,
git_timeout: Duration,
sha: &str,
db: &crate::db::Db,
) -> Result<RepairOutcome> {
let stored = match db.cid_for_oid(sha).await? {
Some(c) => c,
None => return Ok(RepairOutcome::Settled),
};
// Cost gate: a canonical raw CIDv1 key is already correct — never read bytes.
if gitlawb_core::cid::is_raw_cidv1(&stored) {
return Ok(RepairOutcome::Settled);
}
// Legacy-codec row: read the object to recompute. Counted so a test can prove
// the gate above spares non-legacy rows this read.
#[cfg(test)]
note_legacy_repair_read();
// `read_object_bounded` is SYNCHRONOUS `git cat-file`, and its budget is
// `git_service_timeout_secs` (600 by default), so running it inline parks a tokio
// worker for as long as git takes: one wedged read on the sweep's first pass at boot
// holds a worker for ten minutes, per legacy row. Push it to the blocking pool, the
// same shape `replication_withheld_set` uses in api/repos.rs (#173 round 11, F4).
// Both callers of this function are async, so neither changes shape. The read-counter
// increment above stays on THIS thread so the thread_local cost-gate assertion holds.
let read = {
let repo_path = repo_path.to_path_buf();
let git_bin = git_bin.to_string();
let sha = sha.to_string();
tokio::task::spawn_blocking(move || {
crate::git::store::read_object_bounded(&git_bin, &repo_path, &sha, git_timeout)
})
.await
};
let data = match read {
Ok(Ok(Some((_ty, bytes)))) => bytes,
// Bytes gone: the row stays withheld, never destructively rewritten. Nothing a
// later pass changes, so this is a TERMINAL outcome for the sweep's re-walk gate.
Ok(Ok(None)) => return Ok(RepairOutcome::Settled),
// A wedged/D-state `git cat-file` (timeout/infra): the repair is opportunistic
// and best-effort, so skip it and return Ok so the pin task PROCEEDS to
// requeue_or_release rather than hanging the coalescing key until process death
// (grok F2, #173). A later re-push or the deferred sweep retries the repair.
Ok(Err(e)) => {
tracing::warn!(sha = %sha, err = %e, "skipping legacy provider-CID repair: bounded object read failed");
return Ok(RepairOutcome::Retryable);
}
// The blocking task panicked or was cancelled: same best-effort treatment, and
// worth another walk because it says nothing about the row itself.
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "skipping legacy provider-CID repair: object read task failed");
return Ok(RepairOutcome::Retryable);
}
};
let raw = Cid::from_git_object_bytes(&data).to_string();
if raw == stored {
return Ok(RepairOutcome::Settled);
}
db.repair_legacy_provider_cid(sha, &raw, &stored).await?;
Ok(RepairOutcome::Repaired)
}
/// What one opportunistic repair did with a row, so the sweep can tell a skip a later
/// run could fix from one nothing will (U4 re-walk, #173 round 11). The push skip path
/// ignores the value: it repairs whatever the push happens to carry and a failure there
/// is already warn-only.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum RepairOutcome {
/// Nothing to do, or nothing a re-walk would change: the stored key is already the
/// raw resolver key, the recomputed key matches it, or the object's bytes are gone.
Settled,
/// The bounded object read failed (a wedged `git cat-file`, an unreadable repo).
/// The bytes may be readable later, so the row is worth walking again.
Retryable,
/// The row's key was rewritten to the raw-content CID.
Repaired,
}
/// What one sweep pass (or a whole sweep run) did. `scanned` counts `pinned_cids`
/// rows READ, which is the quantity the batch size bounds; `repaired` counts rows
/// whose key was actually rewritten to the raw CID.
#[derive(Debug, Default, PartialEq, Eq)]
pub(crate) struct SweepStats {
pub scanned: usize,
pub repaired: usize,
pub passes: usize,
/// Rows left unrepaired for a reason a LATER run could fix (the source repo is not
/// on this node's local disk, a DB read failed, a bounded object read failed). A
/// nonzero count is what makes the run rewind its cursor instead of parking it at
/// the end of the table forever. Rows that are unrepairable in principle (no
/// provenance, the repo row is gone, the bytes are gone) are NOT counted here.
pub retryable_skips: usize,
}
/// One bounded pass of the U4 sweep: read at most `batch` `pinned_cids` rows after
/// the persisted cursor, repair the legacy ones, and persist the new cursor.
///
/// The batch is what bounds the pass. It caps rows READ, not rows repaired, because
/// the legacy predicate is a codec decode SQL cannot express; a table of raw rows
/// therefore costs one indexed range scan per pass and nothing else.
///
/// The cursor advances to the LAST row read whatever happened to each row, including
/// rows that were skipped as unrepairable. A cursor that only advanced on success
/// would re-read the same unrepairable row on every pass and the sweep would never
/// reach the rows behind it.
async fn sweep_pass(
repos_dir: &std::path::Path,
git_bin: &str,
git_timeout: Duration,
batch: i64,
db: &crate::db::Db,
) -> Result<SweepStats> {
let cursor = db.pin_repair_cursor().await?;
let rows = db.pinned_cids_after(&cursor, batch).await?;
let scanned = rows.len();
let mut repaired = 0usize;
let mut retryable_skips = 0usize;
let mut last = cursor;
for (sha, stored) in rows {
// Advance FIRST: every path below this line may skip the row, and none of them
// may wedge the walk (scenario 7).
last = sha.clone();
// Same cost gate as the skip-path repair: a canonical raw CIDv1 key is already
// the resolver key, so it reads no bytes and resolves no repo.
if gitlawb_core::cid::is_raw_cidv1(&stored) {
continue;
}
// Resolve the row's repo from its recorded provenance (first-pinner plus the
// bounded additional source set). An empty set is a pin recorded before
// provenance existed: nothing to read the bytes from, so skip it.
let sources = match db.pin_sources_for_oid(&sha).await {
Ok(s) => s,
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "sweep: failed to read pin sources");
// A DB read error says nothing about the row, so a later run retries it.
retryable_skips += 1;
continue;
}
};
// Whether this row ended the source walk repaired, and whether anything it hit
// along the way was a transient obstacle rather than a permanent one.
let mut row_repaired = false;
let mut row_retryable = false;
for repo_id in sources {
let repo = match db.get_repo_by_id(&repo_id).await {
Ok(Some(r)) => r,
// The repo row is gone: a later source may still hold the bytes. A
// deleted repo does not come back, so this is not a retryable skip.
Ok(None) => continue,
Err(e) => {
tracing::warn!(repo_id = %repo_id, err = %e, "sweep: failed to read repo");
row_retryable = true;
continue;
}
};
// Derive the LOCAL disk path rather than going through `repo_store.acquire`.
// The sweep is opportunistic background maintenance over every pinned row on
// the node, so it must never pull a cold repo back from remote storage: that
// would turn a repair pass into a bulk restore. A repo that is not on local
// disk simply reads no bytes here and stays withheld, but it IS a retryable
// skip: on a Tigris-backed node the repo is cold now and warm later, and
// without the re-walk that row would never be repaired by anything.
// The path goes through the repo store's VALIDATED resolver (allowlisted
// components, rooted at repos_dir, no ParentDir/CurDir segment), not the raw
// join: the sweep is a second caller of that path logic and gets the same
// barrier the acquire path has (#173 round 11, F3). It is the non-fetching
// variant, so the no-cold-pull property above is untouched.
let repo_path = match crate::git::repo_store::validated_repo_disk_path(
repos_dir,
&repo.owner_did,
&repo.name,
) {
Ok(p) => p,
// An unsafe name is not something a later run fixes, so it is terminal.
Err(e) => {
tracing::warn!(repo_id = %repo_id, err = %e, "sweep: rejected unsafe repo path");
continue;
}
};
if !repo_path.is_dir() {
row_retryable = true;
continue;
}
match repair_legacy_provider_cid(&repo_path, git_bin, git_timeout, &sha, db).await {
Ok(RepairOutcome::Repaired) => {
repaired += 1;
row_repaired = true;
break;
}
// The bytes could not be read from this source right now: try the next
// source, and if none of them works, walk the row again on a later run.
Ok(RepairOutcome::Retryable) => row_retryable = true,
// Nothing to repair from this source and nothing a re-walk changes.
Ok(RepairOutcome::Settled) => {}
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "sweep: legacy provider-CID repair failed");
row_retryable = true;
}
}
}
if !row_repaired && row_retryable {
retryable_skips += 1;
}
}
db.set_pin_repair_cursor(&last).await?;
Ok(SweepStats {
scanned,
repaired,
passes: 1,
retryable_skips,
})
}
/// Test seam for a single bounded pass (scenarios 4 and 5 drive passes by hand to
/// observe the batch bound and the restart-resumes-from-cursor behavior).
#[cfg_attr(not(test), allow(dead_code))]
pub(crate) async fn sweep_legacy_provider_cids_once(
repos_dir: &std::path::Path,
git_bin: &str,
git_timeout: Duration,
batch: i64,
db: &crate::db::Db,
) -> Result<SweepStats> {
sweep_pass(repos_dir, git_bin, git_timeout, batch, db).await
}
/// U4 (#173): the one-shot legacy provider-CID migration sweep.
///
/// Releases before this branch stored the PROVIDER CID (Kubo dag-pb / Pinata CIDv0)
/// in `pinned_cids.cid`. This branch's `/ipfs/{cid}` resolver recomputes the raw
/// content CID and withholds any row whose key does not match, so those rows are
/// unresolvable. The opportunistic repair on the already-pinned skip path only fires
/// when a later push re-carries the object, and normal git negotiation omits objects
/// the node already has, so on an upgraded node that push generally never comes. This
/// walks the table instead.
///
/// Runs until a pass comes back short of a full batch, which is the end of the table.
/// Sleeps `delay` between full batches so it cannot monopolize the DB, and persists
/// its cursor every pass so a restart continues instead of rewinding. Errors reading
/// or repairing an individual row are warn-and-skip; only a failure of the batch query
/// or the cursor write ends the run, and a later run picks up from the stored cursor.
///
/// A run that skipped at least one RETRYABLE row rewinds the cursor to the start of the
/// table on its way out (#173 round 11). Without that the cursor parked at the maximum
/// `sha256_hex` for good: every later boot read zero rows, so a row skipped for a
/// transient reason (its repo cold on a Tigris-backed node, a DB or object read error)
/// was skipped permanently, unadvertised and unresolvable with nothing left to fix it.
/// The rewind is a per-RUN decision made after the walk has already finished, never
/// mid-walk, so it cannot spin: the cost is one extra ordered scan on the next run, and
/// a row that is unrepairable in principle (bytes gone, provenance gone) does not count
/// as retryable, so a node holding one does not re-walk on every boot forever.
pub(crate) async fn sweep_legacy_provider_cids(
repos_dir: &std::path::Path,
git_bin: &str,
git_timeout: Duration,
batch: i64,
delay: Duration,
db: &crate::db::Db,
) -> SweepStats {
let mut totals = SweepStats::default();
loop {
let pass = match sweep_pass(repos_dir, git_bin, git_timeout, batch, db).await {
Ok(p) => p,
Err(e) => {
tracing::warn!(err = %e, "legacy provider-CID sweep pass failed; stopping");
break;
}
};
totals.scanned += pass.scanned;
totals.repaired += pass.repaired;
totals.retryable_skips += pass.retryable_skips;
totals.passes += 1;
// A short batch means the ordered walk reached the end of the table. Stop here
// rather than after an extra empty pass, and do NOT sleep on the way out.
if (pass.scanned as i64) < batch {
break;
}
tokio::time::sleep(delay).await;
}
if totals.retryable_skips > 0 {
if let Err(e) = db.set_pin_repair_cursor("").await {
tracing::warn!(err = %e, "failed to rewind the legacy provider-CID sweep cursor");
}
}
totals
}
// Test-only cost-gate counter (R8, U7): how many times the opportunistic repair
// read an object's bytes on the skip path. The codec gate must spare a CIDv1/raw
// row this read; the counter is the both-ways guard (removing the gate reads the
// raw row and increments it). Same thread_local discipline as the serve-path
// oversize counter — the pin tests await `pin_new_objects` on a current-thread
// runtime, so the increment and the assertion share one thread.
#[cfg(test)]
thread_local! {
static LEGACY_REPAIR_READS: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
}
#[cfg(test)]
pub(crate) fn reset_legacy_repair_reads() {
LEGACY_REPAIR_READS.with(|c| c.set(0));
}
#[cfg(test)]
pub(crate) fn legacy_repair_reads() -> usize {
LEGACY_REPAIR_READS.with(|c| c.get())
}
#[cfg(test)]
fn note_legacy_repair_read() {
LEGACY_REPAIR_READS.with(|c| c.set(c.get() + 1));
}
/// Pin a single git object to the local IPFS/Kubo node.
///
@@ -457,18 +91,14 @@ pub async fn cat(ipfs_api: &str, cid: &str) -> Result<Vec<u8>> {
/// `..._fail_closed` filter on the full-scan path before calling, so this
/// function never sees a withheld blob. `repo_path` is still needed to read each
/// object's bytes. The twin in `pinata.rs` mirrors this shape — change both in
/// lockstep. `repo_id` records the pin's provenance so `GET /ipfs/{cid}` resolves
/// straight to this repo instead of scanning every repo (#173).
/// lockstep.
///
/// Returns a list of `(sha256_hex, cid)` pairs for objects pinned this call.
pub async fn pin_new_objects(
ipfs_api: &str,
repo_path: &std::path::Path,
git_bin: &str,
git_timeout: Duration,
object_list: Vec<String>,
db: &crate::db::Db,
repo_id: &str,
) -> Vec<(String, String)> {
if ipfs_api.is_empty() {
return vec![];
@@ -477,54 +107,9 @@ pub async fn pin_new_objects(
let mut pinned = Vec::new();
for sha in object_list {
// Skip if already pinned — but first backfill provenance if the existing
// pin has none. A legacy pin (recorded before repo_id existed, #173, jatmn)
// is skipped here before record_pinned_cid ever runs, so its NULL provenance
// would never resolve to one repo and known CIDs keep hitting the scan. The
// backfill only sets repo_id (AND repo_id IS NULL guard preserves
// first-pinner-owns) and never re-pins the bytes — the object is already on IPFS.
// Skip if already pinned
match db.is_pinned(&sha).await {
Ok(true) => {
match db.provenance_for_oid(&sha).await {
Ok(None) => {
if let Err(e) = db.backfill_pin_provenance(&sha, repo_id).await {
tracing::warn!(sha = %sha, err = %e, "failed to backfill pin provenance");
}
}
Ok(Some(_)) => {}
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "DB error reading pin provenance");
}
}
// F1 (#173 round 8): record this repo as an ADDITIONAL source for the
// already-pinned object. This is the load-bearing skip-branch insert —
// a later repo pushing a shared object hits this path (already pinned),
// and without it `GET /ipfs/{cid}` only ever knows the first pinner, so a
// shared object first pinned from a private/quarantined repo 404s even
// when this repo would serve it. Bounded per object (MAX_PIN_SOURCES).
if let Err(e) = retry_db_record(|| db.record_pin_source(&sha, repo_id)).await {
tracing::warn!(sha = %sha, err = %e, "failed to record pin source");
// U3 (#173): the retries are spent and this repo is NOT in the source
// set, so the set is known incomplete. Persist that, or the resolver
// reads a non-empty below-cap set as COMPLETE and 404s an object this
// repo would serve. Warn-only in turn: if the marker write also fails
// the object degrades to the pre-U3 behavior, never worse.
if let Err(e) = db.mark_pin_sources_incomplete(&sha).await {
tracing::warn!(sha = %sha, err = %e, "failed to mark pin sources incomplete");
}
}
// R8 (#173 round 10): opportunistically repair a legacy provider-CID
// row (Kubo dag-pb / Pinata) to the raw-content resolver key on this
// re-push. Cost-gated on the stored key's codec — a non-legacy row
// reads no bytes. Warn-only: a failure leaves the row as-is for a
// later re-push or the deferred one-shot sweep.
if let Err(e) =
repair_legacy_provider_cid(repo_path, git_bin, git_timeout, &sha, db).await
{
tracing::warn!(sha = %sha, err = %e, "failed to repair legacy provider CID");
}
continue;
}
Ok(true) => continue,
Ok(false) => {}
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "DB error checking pinned status");
@@ -532,49 +117,22 @@ pub async fn pin_new_objects(
}
}
// Read raw object content under a bounded read so a wedged/D-state `git
// cat-file` (stuck NFS/Tigris backend) is reaped at `git_timeout` instead of
// hanging pin_new_objects forever — which would pin the post-push coalescing
// key until process death (grok F2, #173). On Err the object is simply not
// pinned this pass; a later pass/push retries.
let data =
match crate::git::store::read_object_bounded(git_bin, repo_path, &sha, git_timeout) {
Ok(Some((_obj_type, bytes))) => bytes,
Ok(None) => continue,
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "failed to read git object for pinning");
continue;
}
};
// Read raw object content
let data = match crate::git::store::read_object(repo_path, &sha) {
Ok(Some((_obj_type, bytes))) => bytes,
Ok(None) => continue,
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "failed to read git object for pinning");
continue;
}
};
// Pin to IPFS
match pin_git_object(ipfs_api, &sha, &data).await {
Ok(cid) if !cid.is_empty() => {
// The resolver key (`pinned_cids.cid`) must be the locally-computed
// raw-content CID, never the provider Hash: Kubo returns a dag-pb/UnixFS
// root for objects above its block size, which does not hash the raw
// content, so `GET /ipfs/{provider_cid}` would resolve then fail the F2
// integrity check (list-then-404). The serve path reads bytes from git and
// verifies them against the requested CID, so the raw CID is the correct
// key. Mirrors the pinata twin, which already records the raw CID.
let raw_cid = gitlawb_core::cid::Cid::from_git_object_bytes(&data).to_string();
// F1 (#173 round 8): the first pinner is recorded in pin_repo_sources too,
// so every source (first and subsequent) is tried uniformly by the
// resolver. U3 (#173): the pin and its source go down in ONE transaction.
// As two independent best-effort calls this path could land the pin while
// dropping its own source, producing a source set silently missing its
// first pinner; atomically there is no such window, and a total failure
// leaves the object unpinned so the next push retries the whole thing.
if let Err(e) =
retry_db_record(|| db.record_pinned_cid_with_source(&sha, &raw_cid, repo_id))
.await
{
if let Err(e) = db.record_pinned_cid(&sha, &cid).await {
tracing::warn!(sha = %sha, err = %e, "failed to record pinned CID in DB");
}
// Return the provider Hash (not the resolver key), mirroring the pinata
// twin's contract: the DB `cid` is the raw resolver key (recorded above),
// the returned value is the provider CID. Here the return is consumed only
// for logging, but keeping the twins structurally identical avoids drift.
pinned.push((sha, cid));
}
Ok(_) => {}
@@ -586,92 +144,3 @@ pub async fn pin_new_objects(
pinned
}
#[cfg(test)]
mod tests {
use super::*;
use std::cell::Cell;
// The retry helper is the load-bearing unit: it converts a sub-second
// transient DB error at the three warn-only record sites into a landed row,
// instead of a permanently incomplete pin-source set. These drive the helper
// directly against a controlled closure (the record sites take a concrete
// `&Db` over a `PgPool`, so a failing-first wrapper cannot slot in without
// changing signatures — see U6 seam note).
#[tokio::test]
async fn retry_lands_after_transient_failures() {
let calls = Cell::new(0u32);
let result = retry_db_record(|| {
let n = calls.get() + 1;
calls.set(n);
async move {
if n < PIN_RECORD_ATTEMPTS {
Err(anyhow::anyhow!("transient failure on attempt {n}"))
} else {
Ok(())
}
}
})
.await;
assert!(
result.is_ok(),
"retry lands the row after transient failures"
);
assert_eq!(
calls.get(),
PIN_RECORD_ATTEMPTS,
"op is retried until it succeeds"
);
}
#[tokio::test]
async fn retry_returns_last_err_after_exhaustion() {
let calls = Cell::new(0u32);
let result = retry_db_record(|| {
let n = calls.get() + 1;
calls.set(n);
async move { Err::<(), _>(anyhow::anyhow!("attempt {n} failed")) }
})
.await;
let err = result.expect_err("all attempts fail so the last error surfaces");
assert_eq!(
calls.get(),
PIN_RECORD_ATTEMPTS,
"attempts are bounded to the cap"
);
assert_eq!(
err.to_string(),
"attempt 3 failed",
"the LAST error is returned, not the first"
);
}
// Happy path against a real DB: a single-attempt success lands the row, and a
// redundant call is idempotent (`ON CONFLICT DO NOTHING`), so the source set
// holds exactly one row.
#[sqlx::test]
async fn retry_records_pin_source_once(pool: sqlx::PgPool) {
let db = crate::db::Db::for_testing(pool);
db.run_migrations().await.unwrap();
let sha = "a".repeat(64);
let repo_id = "repo-retry-1";
retry_db_record(|| db.record_pin_source(&sha, repo_id))
.await
.expect("happy-path record succeeds in one attempt");
retry_db_record(|| db.record_pin_source(&sha, repo_id))
.await
.expect("a redundant record is idempotent");
let sources = db.pin_sources_for_oid(&sha).await.unwrap();
assert_eq!(
sources,
vec![repo_id.to_string()],
"exactly one source row lands under ON CONFLICT DO NOTHING"
);
}
}
+12 -292
View File
@@ -58,32 +58,6 @@ struct DbStartupStatus {
next_retry_secs: AtomicU64,
}
/// Hard ceiling on the advisory-lock pool's `max_connections`.
///
/// `max_concurrent_git_pushes` is validated all the way up to 1_048_576, and the lock
/// pool used to derive its size straight from that knob, so raising the push cap
/// silently raised the node's Postgres connection ceiling with no CLI error and no
/// relation to the server's own `max_connections` (#173 F4). The node's total budget is
/// now bounded: `db_max_connections` (default 20) + at most this.
const LOCK_POOL_MAX_CONNECTIONS: u32 = 64;
/// Connections the lock pool keeps above the push cap. Covers the three non-push
/// `acquire_write` callers (`api/issues.rs` x2, `api/pulls.rs`), which hold no
/// concurrency permit, so a push never queues here for a connection where it did not
/// before.
const LOCK_POOL_PUSH_HEADROOM: u8 = 8;
/// Size the advisory-lock pool for a given push cap: the cap plus
/// [`LOCK_POOL_PUSH_HEADROOM`], clamped to [`LOCK_POOL_MAX_CONNECTIONS`]. Past the
/// clamp a push may wait for a lock-pool connection, which is a bounded wait that sheds
/// a clean 503 (see `LockPoolBusy`), not an unbounded hang.
fn lock_pool_size(max_concurrent_git_pushes: usize) -> u32 {
u32::try_from(max_concurrent_git_pushes)
.unwrap_or(u32::MAX)
.saturating_add(u32::from(LOCK_POOL_PUSH_HEADROOM))
.min(LOCK_POOL_MAX_CONNECTIONS)
}
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt()
@@ -305,17 +279,8 @@ async fn main() -> Result<()> {
None
};
// Repo write locks run on their own pool, never the main query pool: each
// push holds its connection for the whole receive-pack, and
// db_max_connections (20) is below max_concurrent_git_pushes (32), so sharing
// would starve every other query under a push burst. See build_lock_pool for
// the cancellation semantics (#173).
let lock_pool = git::repo_store::build_lock_pool(
db.pool(),
lock_pool_size(config.max_concurrent_git_pushes),
std::time::Duration::from_secs(config.db_acquire_timeout_secs),
);
let repo_store = git::repo_store::RepoStore::new(config.repos_dir.clone(), tigris, lock_pool);
let repo_store =
git::repo_store::RepoStore::new(config.repos_dir.clone(), tigris, db.pool().clone());
// Per-DID limiter for the creation endpoints. Keyed on the authenticated
// DID (attacker-varied), so bound its key set to cap memory.
@@ -409,83 +374,11 @@ async fn main() -> Result<()> {
rate_limiter,
create_ip_rate_limiter,
push_rate_limiter,
ipfs_max_history_walks: crate::api::ipfs::MAX_HISTORY_WALKS_PER_REQUEST,
// The legacy-probe budget is operator-tunable via GITLAWB_IPFS_MAX_REPOS_WALKED
// (R5); the history-walk ceiling above stays constant (a smaller value false-503s
// a provenanced request). Default 256 preserves the shipped behaviour.
ipfs_max_legacy_probes: AppState::ipfs_legacy_probe_budget(&config),
ipfs_max_served_object_bytes: crate::api::ipfs::MAX_SERVED_OBJECT_BYTES,
push_limiter_trust,
sync_trigger_rate_limiter,
peer_write_rate_limiter,
shutdown_tx: shutdown_tx.clone(),
git_read_semaphore: Arc::new(tokio::sync::Semaphore::new(config.max_concurrent_git_ops)),
git_write_semaphore: Arc::new(tokio::sync::Semaphore::new(
config.max_concurrent_git_pushes,
)),
// Anon receive-pack advertisements get their OWN pool, same size as the
// write pool but disjoint, so filling it (which takes many source IPs, each
// capped by git_push_advert_per_caller) never occupies a permit the
// authenticated POST needs (#174).
git_push_advert_semaphore: Arc::new(tokio::sync::Semaphore::new(
config.max_concurrent_git_pushes,
)),
// Bounds concurrent detached post-push encryption walks, sized from the push
// pool (no separate knob — Q1): completed pushes cannot outnumber active
// encryption walks past this (#174 P1-e).
git_encrypt_semaphore: Arc::new(tokio::sync::Semaphore::new(
config.max_concurrent_git_pushes,
)),
// Coalesces the DETACHED post-push encryption tasks per repo so a rapid pusher
// cannot grow the outstanding parked-waiter set past one task per repo (#174
// P2-2). No knob: it is a natural cap (one entry per distinct repo), not a
// sized pool.
encrypt_inflight: crate::state::EncryptInflight::new(),
git_read_per_caller: rate_limit::PerCallerConcurrency::with_default_max_keys(
config.max_concurrent_reads_per_caller,
),
// Per-source cap on the receive-pack advertisement, sized to an eighth of the
// write pool (min 1): a single source can hold at most this many write-pool
// slots via the anon advertisement, so saturating the pool takes ~8 distinct
// source IPs, each also rate-limited (#174).
git_push_advert_per_caller: rate_limit::PerCallerConcurrency::with_default_max_keys(
(config.max_concurrent_git_pushes / 8).max(1),
),
// Per-source cap on the authenticated receive-pack POST, sized like the advert
// cap: one source IP can hold at most this many write-pool slots, so
// monopolizing the pool takes ~8 distinct source IPs, each also rate-limited
// (#174 P1-d).
git_write_per_caller: rate_limit::PerCallerConcurrency::with_default_max_keys(
(config.max_concurrent_git_pushes / 8).max(1),
),
// Bounds concurrent /ipfs visibility walks — a distinct public cost center, so
// its own pool + per-source sub-cap + per-IP rate limiter, never a git pool
// (#174 P1-3). The per-source map is bounded (reject-before-insert, INV-15).
git_ipfs_walk_semaphore: Arc::new(tokio::sync::Semaphore::new(
config.max_concurrent_ipfs_walks,
)),
git_ipfs_walk_per_caller: rate_limit::PerCallerConcurrency::with_default_max_keys(
config.ipfs_walk_per_source,
),
ipfs_rate_limiter: rate_limit::RateLimiter::new_bounded(
config.ipfs_rate_limit,
std::time::Duration::from_secs(3600),
200_000,
),
// Separate WORK-budget bucket for the resolver's per-probe/per-walk charges (R6).
// Its capacity is DERIVED from the route limit (no new knob) and floored at the
// legacy-probe budget, so one full default-config legacy scan never self-throttles
// mid-request while the route brake above stays the pure once-per-request cap.
ipfs_work_rate_limiter: rate_limit::RateLimiter::new_bounded(
AppState::ipfs_work_budget(&config),
std::time::Duration::from_secs(3600),
200_000,
),
git_bin: "git".to_string(),
};
if config.ipfs_rate_limit == 0 {
tracing::warn!("GITLAWB_IPFS_RATE_LIMIT=0 — per-IP /ipfs rate limiting disabled");
}
// Periodic peer-count poll for the metrics gauge. If p2p is disabled
// we still set the gauge to 0 so dashboards don't show "no data".
@@ -515,16 +408,22 @@ async fn main() -> Result<()> {
// Periodic cleanup of expired rate limit entries + consumed-proof ledger
{
let cleanup_state = state.clone();
let rl = state.rate_limiter.clone();
let create_ip_rl = state.create_ip_rate_limiter.clone();
let push_rl = state.push_rate_limiter.clone();
let sync_trigger_rl = state.sync_trigger_rate_limiter.clone();
let peer_write_rl = state.peer_write_rate_limiter.clone();
let db = state.db.clone();
let mut shutdown_rx = state.subscribe_shutdown();
tokio::spawn(async move {
loop {
tokio::select! {
_ = tokio::time::sleep(std::time::Duration::from_secs(300)) => {
// Sweep every per-IP/DID limiter (incl. the ipfs walk brake)
// so bounded maps shed stale keys instead of sitting at cap.
cleanup_state.sweep_rate_limiters().await;
rl.cleanup().await;
create_ip_rl.cleanup().await;
push_rl.cleanup().await;
sync_trigger_rl.cleanup().await;
peer_write_rl.cleanup().await;
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs() as i64)
@@ -543,8 +442,6 @@ async fn main() -> Result<()> {
});
}
let _legacy_cid_sweep = spawn_legacy_cid_sweep(&state, &config);
let router = server::build_router(state.clone());
// Re-register the socket bound at startup — same fd, so there was never a
// moment with the port closed between the degraded and full servers.
@@ -667,49 +564,6 @@ async fn main() -> Result<()> {
Ok(())
}
/// U4 (#173): spawn the one-shot legacy provider-CID repair sweep. Releases before this
/// version stored the PROVIDER CID (Kubo dag-pb / Pinata CIDv0) in `pinned_cids.cid`,
/// and this version's `/ipfs/{cid}` resolver withholds any row whose stored key is not
/// the raw-content CID. The opportunistic repair on the pin path only fires when a push
/// re-carries the object, which normal git negotiation makes it not do, so those rows
/// need a walk. DETACHED, never on the boot path: the caller keeps serving while this
/// runs, and the sweep's own batch bound plus inter-batch delay keep it off the DB's
/// critical path. Its cursor is durable, so a restart mid-walk resumes instead of
/// rewinding.
///
/// A named function rather than an inline block in `main` so the WIRING has a seam a
/// test can call: that the task is spawned at all, that it reads its batch and delay
/// from the config knobs rather than some other field, that the caller is not blocked
/// on it, and that the shutdown watcher actually ends it mid-walk. The sweep's own
/// behavior is covered elsewhere; this is the boot-path half.
fn spawn_legacy_cid_sweep(state: &AppState, config: &Config) -> tokio::task::JoinHandle<()> {
let db = state.db.clone();
let repos_dir = config.repos_dir.clone();
let git_bin = state.git_bin.clone();
let git_timeout = std::time::Duration::from_secs(config.git_service_timeout_secs);
let batch = config.pin_repair_sweep_batch;
let delay = std::time::Duration::from_secs(config.pin_repair_sweep_delay_secs);
let mut shutdown_rx = state.subscribe_shutdown();
tokio::spawn(async move {
tokio::select! {
stats = ipfs_pin::sweep_legacy_provider_cids(
&repos_dir, &git_bin, git_timeout, batch, delay, &db,
) => {
if stats.repaired > 0 {
tracing::info!(
scanned = stats.scanned,
repaired = stats.repaired,
"legacy provider-CID sweep finished"
);
}
}
// Shutdown mid-walk simply drops the run; the persisted cursor means the
// next boot picks up where this one stopped.
_ = shutdown_rx.changed() => {}
}
})
}
fn spawn_shutdown_signal(tx: watch::Sender<bool>) {
tokio::spawn(async move {
#[cfg(unix)]
@@ -1170,140 +1024,6 @@ fn load_or_create_keypair(config: &Config) -> Result<Keypair> {
}
}
#[cfg(test)]
mod legacy_cid_sweep_wiring_tests {
use super::spawn_legacy_cid_sweep;
use sqlx::PgPool;
use std::time::Duration;
/// Seed `count` `pinned_cids` rows whose keys are already canonical raw CIDv1, in a
/// known `sha256_hex` order. The sweep's own cost gate skips a raw-CIDv1 row without
/// reading bytes or resolving a repo, so each row is SCANNED (it advances the cursor)
/// and nothing else. That is what makes the cursor a clean readout of how far the
/// walk got, with no dependency on repos on disk.
async fn seed_scannable_rows(pool: &PgPool, count: usize) -> Vec<String> {
let mut shas = Vec::new();
for i in 1..=count {
let sha = format!("wire{i:02}");
let cid = gitlawb_core::cid::Cid::from_git_object_bytes(sha.as_bytes()).to_string();
assert!(
gitlawb_core::cid::is_raw_cidv1(&cid),
"the seeded key must hit the sweep's raw-CIDv1 skip, not a repair attempt"
);
sqlx::query("INSERT INTO pinned_cids (sha256_hex, cid, pinned_at) VALUES ($1, $2, $3)")
.bind(&sha)
.bind(&cid)
.bind("2020-01-01T00:00:00Z")
.execute(pool)
.await
.unwrap();
shas.push(sha);
}
shas
}
/// Poll the persisted sweep cursor until it reaches `want`, or give up.
async fn cursor_reaches(db: &crate::db::Db, want: &str, within: Duration) -> String {
let deadline = std::time::Instant::now() + within;
loop {
let c = db.pin_repair_cursor().await.unwrap();
if c == want || std::time::Instant::now() >= deadline {
return c;
}
tokio::time::sleep(Duration::from_millis(20)).await;
}
}
/// #173 U4, the BOOT-PATH half. The sweep's own logic (batching, cursor resumption,
/// terminal vs retryable skips) is covered in `test_support`; what this covers is the
/// wiring `main` performs, which nothing else executes: the task is spawned at all,
/// it takes its batch and delay from the two `pin_repair_sweep_*` knobs rather than
/// some other config field, the caller is not blocked on the walk, and the shutdown
/// watcher ends the run mid-walk.
///
/// Six scannable rows, batch 2, delay 30s. One pass must land the cursor on exactly
/// the second row and the task must then still be alive in its inter-batch sleep,
/// which pins both knobs at once: a different batch stops at a different row, and a
/// delay that did not come from the knob either finishes the table or leaves the task
/// gone. Shutdown must then end it while four rows are still unwalked.
#[sqlx::test]
async fn the_boot_path_spawns_the_sweep_detached_with_its_configured_knobs(pool: PgPool) {
let state = crate::test_support::test_state(pool.clone()).await;
let shas = seed_scannable_rows(&pool, 6).await;
let repos_dir = tempfile::TempDir::new().unwrap();
let mut config = (*state.config).clone();
config.repos_dir = repos_dir.path().to_path_buf();
config.pin_repair_sweep_batch = 2;
// Far longer than this test runs, so a task still alive after the first pass can
// only be one that is honoring the configured inter-batch delay.
config.pin_repair_sweep_delay_secs = 30;
let started = std::time::Instant::now();
let handle = spawn_legacy_cid_sweep(&state, &config);
let spawn_cost = started.elapsed();
let cursor = cursor_reaches(&state.db, &shas[1], Duration::from_secs(10)).await;
assert_eq!(
cursor, shas[1],
"the spawned sweep must run and stop its first pass at the CONFIGURED batch \
bound (2), leaving the cursor on the second row"
);
assert!(
spawn_cost < Duration::from_secs(1),
"the sweep must be detached, not awaited on the boot path; the spawn took \
{spawn_cost:?}"
);
assert!(
!handle.is_finished(),
"with a 30s inter-batch delay the task must still be sleeping between passes, \
not finished: a finished task means the delay was not the configured one"
);
state.shutdown();
tokio::time::timeout(Duration::from_secs(10), handle)
.await
.expect("the shutdown watcher must end the sweep, and not after its 30s delay")
.expect("the sweep task must not panic");
assert_eq!(
state.db.pin_repair_cursor().await.unwrap(),
shas[1],
"shutdown must have ended the run MID-walk, with the remaining rows unwalked"
);
}
}
#[cfg(test)]
mod lock_pool_sizing_tests {
use super::{lock_pool_size, LOCK_POOL_MAX_CONNECTIONS, LOCK_POOL_PUSH_HEADROOM};
/// The default push cap gets its cap plus headroom, so no push ever queues for a
/// lock-pool connection where it did not before.
#[test]
fn default_push_cap_gets_headroom_over_the_cap() {
assert_eq!(lock_pool_size(32), 32 + u32::from(LOCK_POOL_PUSH_HEADROOM));
assert_eq!(lock_pool_size(1), 1 + u32::from(LOCK_POOL_PUSH_HEADROOM));
}
/// #173 F4: `max_concurrent_git_pushes` is validated all the way to 1_048_576, so an
/// operator raising it used to raise the node's Postgres connection ceiling with it,
/// silently and without bound. The lock pool is CLAMPED instead.
#[test]
fn an_oversized_push_cap_is_clamped_not_propagated() {
assert_eq!(lock_pool_size(1_048_576), LOCK_POOL_MAX_CONNECTIONS);
assert_eq!(lock_pool_size(usize::MAX), LOCK_POOL_MAX_CONNECTIONS);
// The largest cap that still fits under the clamp keeps its full headroom.
let widest = (LOCK_POOL_MAX_CONNECTIONS - u32::from(LOCK_POOL_PUSH_HEADROOM)) as usize;
assert_eq!(lock_pool_size(widest), LOCK_POOL_MAX_CONNECTIONS);
assert_eq!(
lock_pool_size(widest - 1),
LOCK_POOL_MAX_CONNECTIONS - 1,
"values below the clamp must not be rounded up to it"
);
}
}
#[cfg(test)]
mod gossip_ssrf_tests {
use super::ping_peer_health;
+70
View File
@@ -39,6 +39,11 @@ pub struct RefUpdateEvent {
pub pusher_did: String,
/// Repository identifier (owner/name)
pub repo: String,
/// Full owner DID — added in #144 for display and storage; not yet
/// wired into the feed gate matcher. Optional for backward compat with
/// older peers that don't include it.
#[serde(default)]
pub owner_did: Option<String>,
/// Git ref that changed (e.g., "refs/heads/main")
pub ref_name: String,
/// SHA before the push (all-zeros for new ref)
@@ -307,6 +312,7 @@ pub async fn start(
node_did: event.node_did.clone(),
pusher_did: event.pusher_did.clone(),
repo: event.repo.clone(),
owner_did: event.owner_did.clone(),
ref_name: event.ref_name.clone(),
old_sha: event.old_sha.clone(),
new_sha: event.new_sha.clone(),
@@ -432,3 +438,67 @@ pub async fn start(
Ok(handle)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn ref_update_event_round_trip_with_owner_did() {
let event = RefUpdateEvent {
node_did: "did:key:zNode".into(),
pusher_did: "did:key:zPusher".into(),
repo: "zOwner/myrepo".into(),
owner_did: Some("did:key:zOwner".into()),
ref_name: "refs/heads/main".into(),
old_sha: "0000000000000000000000000000000000000000".into(),
new_sha: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa".into(),
timestamp: "2026-07-02T12:00:00Z".into(),
cert_id: None,
cid: None,
};
let json = serde_json::to_value(&event).unwrap();
// owner_did must be present in the serialized output
assert_eq!(json["owner_did"], "did:key:zOwner");
assert_eq!(json["repo"], "zOwner/myrepo");
let deserialized: RefUpdateEvent = serde_json::from_value(json).unwrap();
assert_eq!(deserialized.owner_did, Some("did:key:zOwner".into()));
}
#[test]
fn ref_update_event_backward_compat_no_owner_did() {
let old_json = serde_json::json!({
"node_did": "did:key:zNode",
"pusher_did": "did:key:zPusher",
"repo": "zOwner/myrepo",
"ref_name": "refs/heads/main",
"old_sha": "0000000000000000000000000000000000000000",
"new_sha": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"timestamp": "2026-07-02T12:00:00Z",
"cert_id": null,
"cid": null
});
let deserialized: RefUpdateEvent = serde_json::from_value(old_json).unwrap();
assert_eq!(deserialized.owner_did, None);
assert_eq!(deserialized.repo, "zOwner/myrepo");
}
#[test]
fn ref_update_event_backward_compat_null_owner_did() {
let with_null = serde_json::json!({
"node_did": "did:key:zNode",
"pusher_did": "did:key:zPusher",
"repo": "zOwner/myrepo",
"owner_did": null,
"ref_name": "refs/heads/main",
"old_sha": "0000000000000000000000000000000000000000",
"new_sha": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"timestamp": "2026-07-02T12:00:00Z",
"cert_id": null,
"cid": null
});
let deserialized: RefUpdateEvent = serde_json::from_value(with_null).unwrap();
assert_eq!(deserialized.owner_did, None);
}
}
+4 -62
View File
@@ -74,10 +74,8 @@ pub async fn pin_object(
/// `..._fail_closed` filter on the full-scan path before calling. `repo_path` is
/// still needed to read each object's bytes. The twin in `ipfs_pin.rs` mirrors
/// this shape — change both in lockstep. Objects already recorded with a
/// `pinata_cid` are skipped. `repo_id` records the pin's provenance (#173).
/// Returns `(sha_hex, provider_cid)` pairs for each newly pinned object: the
/// provider CID is the Pinata gateway CID (used for branch→CID recording and
/// ref-update gossip), NOT the raw resolver-key CID stored in `pinned_cids.cid`.
/// `pinata_cid` are skipped. Returns `(sha_hex, cid)` pairs for each newly
/// pinned object.
pub async fn pin_new_objects(
client: &reqwest::Client,
upload_url: &str,
@@ -85,7 +83,6 @@ pub async fn pin_new_objects(
repo_path: &std::path::Path,
object_list: Vec<String>,
db: &crate::db::Db,
repo_id: &str,
) -> Vec<(String, String)> {
if jwt.is_empty() {
return vec![];
@@ -95,39 +92,7 @@ pub async fn pin_new_objects(
for sha in object_list {
match db.has_pinata_cid(&sha).await {
Ok(true) => {
// Backfill NULL first-pinner provenance from a known source, in lockstep
// with the ipfs_pin skip branch: a pinata-only node otherwise leaves
// pre-provenance rows' `pinned_cids.repo_id` NULL forever (grok P2-D). The
// resolver still finds the object via the pin_repo_sources union below, so
// this is a consistency backfill, not a correctness fix.
match db.provenance_for_oid(&sha).await {
Ok(None) => {
if let Err(e) = db.backfill_pin_provenance(&sha, repo_id).await {
tracing::warn!(sha = %sha, err = %e, "failed to backfill pin provenance");
}
}
Ok(Some(_)) => {}
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "DB error reading pin provenance");
}
}
// F1 (#173 round 8): record this repo as an additional source for the
// already-pinned object (mirrors the ipfs_pin skip-branch insert) so the
// resolver can serve a shared object from any pin-path source. U3 (#173):
// retried through the SHARED helper (this was a bare call, so a single
// transient error dropped the source outright) and, on exhaustion, marked
// durably so the resolver keeps the bounded scan fallback for the object.
if let Err(e) =
crate::ipfs_pin::retry_db_record(|| db.record_pin_source(&sha, repo_id)).await
{
tracing::warn!(sha = %sha, err = %e, "failed to record pin source");
if let Err(e) = db.mark_pin_sources_incomplete(&sha).await {
tracing::warn!(sha = %sha, err = %e, "failed to mark pin sources incomplete");
}
}
continue;
}
Ok(true) => continue,
Ok(false) => {}
Err(e) => {
tracing::warn!(sha = %sha, err = %e, "DB error checking pinata_cid");
@@ -146,32 +111,9 @@ pub async fn pin_new_objects(
match pin_object(client, upload_url, jwt, &sha, &data).await {
Ok(cid) if !cid.is_empty() => {
// The resolver key (`pinned_cids.cid`) must be the locally-computed
// raw-content CID, never the provider CID: Pinata wraps the bytes in
// dag-pb/UnixFS, so its returned CID does not hash the raw content and
// must not become an alias `/ipfs/{cid}` serves raw git bytes for (#173).
let raw_cid = gitlawb_core::cid::Cid::from_git_object_bytes(&data).to_string();
// U3 (#173): both records go through the shared retry helper, at parity
// with the ipfs_pin twin. These were bare calls, so one transient DB error
// permanently dropped a pin source.
if let Err(e) = crate::ipfs_pin::retry_db_record(|| {
db.record_pinata_cid(&sha, &raw_cid, &cid, Some(repo_id))
})
.await
{
if let Err(e) = db.record_pinata_cid(&sha, &cid).await {
tracing::warn!(sha = %sha, err = %e, "failed to record pinata_cid in DB");
}
// F1 (#173 round 8): also record the first pinner in pin_repo_sources.
// U3: an exhausted retry marks the set incomplete so the resolver keeps
// the scan fallback rather than 404ing a copy it could serve.
if let Err(e) =
crate::ipfs_pin::retry_db_record(|| db.record_pin_source(&sha, repo_id)).await
{
tracing::warn!(sha = %sha, err = %e, "failed to record pin source");
if let Err(e) = db.mark_pin_sources_incomplete(&sha).await {
tracing::warn!(sha = %sha, err = %e, "failed to mark pin sources incomplete");
}
}
pinned.push((sha, cid));
}
Ok(_) => {}
-176
View File
@@ -121,28 +121,6 @@ impl RateLimiter {
true
}
/// Non-consuming check: is this key ALREADY at its limit for the current window?
/// Unlike [`check`], it records nothing and never inserts a new key — used to shed
/// expensive preparatory work (e.g. the `/ipfs/{cid}` legacy scan's O(repos) DB
/// preload) BEFORE it runs, without perturbing the per-unit budget the consuming
/// `check` maintains (#173, F3). An unknown key or a disabled limiter is not
/// throttled. Prunes the key's expired timestamps as a side effect (keeps state
/// tidy) but adds none, so it cannot itself fill or grow the map.
pub(crate) async fn is_throttled(&self, key: &str) -> bool {
if self.max_requests == 0 {
return false;
}
let now = Instant::now();
let mut state = self.state.lock().await;
if let Some(window) = state.get_mut(key) {
window
.timestamps
.retain(|t| now.duration_since(*t) < self.window);
return window.timestamps.len() >= self.max_requests;
}
false
}
pub async fn cleanup(&self) {
let now = Instant::now();
let mut state = self.state.lock().await;
@@ -152,103 +130,6 @@ impl RateLimiter {
!w.timestamps.is_empty()
});
}
/// Number of distinct keys currently tracked. Test-only introspection so a
/// cross-module test can assert that a sweep actually evicted expired entries.
#[cfg(test)]
pub(crate) async fn tracked_keys(&self) -> usize {
self.state.lock().await.len()
}
}
/// A bounded per-caller CONCURRENCY limiter — distinct from [`RateLimiter`], which
/// caps request RATE. Each caller key may hold at most `per_caller` in-flight
/// permits at once; beyond that [`try_acquire`](Self::try_acquire) returns `None`
/// and the caller sheds. Used to stop one caller (a single anonymous source-IP or
/// DID) monopolizing the served-git read pool (#174).
///
/// The key map is self-bounding: a key is removed the instant its in-flight count
/// reaches zero, so it never holds more keys than there are concurrently-active
/// callers (itself bounded by the read semaphore). A `max_keys` reject-before-insert
/// backstop guarantees a key farm can never grow the map even if that invariant
/// weakened — a NEW key at the cap is rejected WITHOUT allocating an entry (INV-15).
///
/// Uses a `std::sync::Mutex` (not the file's `tokio::sync::Mutex`) because the
/// permit's `Drop` must release synchronously; the critical section holds no await.
#[derive(Clone)]
pub struct PerCallerConcurrency {
state: Arc<std::sync::Mutex<HashMap<String, usize>>>,
per_caller: usize,
max_keys: usize,
}
/// RAII permit from [`PerCallerConcurrency::try_acquire`]. On drop it decrements
/// the caller's in-flight count and removes the key when it reaches zero.
pub struct PerCallerPermit {
state: Arc<std::sync::Mutex<HashMap<String, usize>>>,
key: String,
}
impl PerCallerConcurrency {
pub fn new(per_caller: usize, max_keys: usize) -> Self {
Self {
state: Arc::new(std::sync::Mutex::new(HashMap::new())),
per_caller: per_caller.max(1),
max_keys: max_keys.max(1),
}
}
/// Convenience constructor with the default key bound.
pub fn with_default_max_keys(per_caller: usize) -> Self {
Self::new(per_caller, DEFAULT_MAX_KEYS)
}
/// `Some(permit)` when the caller is under its cap and the map has room;
/// `None` (shed) otherwise. Reject-before-insert: a new key at `max_keys` is
/// rejected without allocating.
pub fn try_acquire(&self, key: &str) -> Option<PerCallerPermit> {
// Recover from a poisoned lock rather than panicking: the critical section
// is pure counter arithmetic and cannot itself panic, so a poisoned mutex
// would only ever come from an unrelated abort, and a slightly-off count
// self-heals as permits drop. A panic here would instead brick the limiter
// for every caller (each subsequent lock re-panics).
let mut map = self.state.lock().unwrap_or_else(|e| e.into_inner());
match map.get_mut(key) {
Some(count) => {
if *count >= self.per_caller {
return None;
}
*count += 1;
}
None => {
if map.len() >= self.max_keys {
return None;
}
map.insert(key.to_string(), 1);
}
}
Some(PerCallerPermit {
state: self.state.clone(),
key: key.to_string(),
})
}
#[cfg(test)]
pub fn tracked_keys(&self) -> usize {
self.state.lock().unwrap().len()
}
}
impl Drop for PerCallerPermit {
fn drop(&mut self) {
let mut map = self.state.lock().unwrap_or_else(|e| e.into_inner());
if let Some(count) = map.get_mut(&self.key) {
*count -= 1;
if *count == 0 {
map.remove(&self.key);
}
}
}
}
pub async fn rate_limit_by_did(request: Request, next: Next) -> Response {
@@ -407,63 +288,6 @@ pub async fn rate_limit_by_ip(request: Request, next: Next) -> Response {
mod tests {
use super::*;
#[test]
fn per_caller_concurrency_caps_one_caller_and_frees_on_drop() {
let lim = PerCallerConcurrency::new(2, 100);
let p1 = lim.try_acquire("did:key:zA").expect("first under cap");
let p2 = lim.try_acquire("did:key:zA").expect("second under cap");
assert!(
lim.try_acquire("did:key:zA").is_none(),
"a third in-flight op for the same caller sheds (over the per-caller cap)"
);
// A DIFFERENT caller is unaffected — the cap is per-caller, not global.
let _other = lim
.try_acquire("did:key:zB")
.expect("a different caller has its own budget");
drop(p1);
assert!(
lim.try_acquire("did:key:zA").is_some(),
"freeing one in-flight slot lets the same caller back in"
);
drop(p2);
}
#[test]
fn per_caller_concurrency_map_is_self_bounding_and_reject_before_insert() {
// Self-bounding: acquire+drop many distinct keys — the map never grows
// because a key is removed the instant its in-flight count hits zero.
let lim = PerCallerConcurrency::new(4, 3);
for i in 0..50 {
let _p = lim.try_acquire(&format!("k{i}"));
}
assert_eq!(
lim.tracked_keys(),
0,
"keys with zero in-flight ops are removed, so an acquire+drop flood leaves the map empty"
);
// Reject-before-insert: HOLD max_keys distinct keys, then a new key sheds
// WITHOUT growing the map past the cap (INV-15 — a rejected request never
// allocates an entry).
let held: Vec<_> = (0..3)
.map(|i| lim.try_acquire(&format!("h{i}")).unwrap())
.collect();
assert_eq!(
lim.tracked_keys(),
3,
"three distinct callers held concurrently"
);
assert!(
lim.try_acquire("h3").is_none(),
"a new key at max_keys is rejected"
);
assert_eq!(
lim.tracked_keys(),
3,
"the rejected new key did not allocate an entry (reject-before-insert)"
);
drop(held);
}
#[tokio::test]
async fn allows_within_limit() {
let limiter = RateLimiter::new(3, Duration::from_secs(60));
-11
View File
@@ -214,20 +214,9 @@ pub fn build_router(state: AppState) -> Router {
// identity and can apply per-repo visibility (#110); anonymous callers stay
// anonymous and still read genuinely public content. `/api/v1/ipfs/pins`
// stays unsigned — gating the pin index is tracked separately (#121).
// `/ipfs/{cid}` also carries a per-IP flood brake: it is anon-reachable and each
// request can drive a full-history git walk, so the per-IP rate limiter is the
// outermost layer (rejects a flood before the walk-admission work), mirroring the
// push/create routers. The extension MUST be attached or rate_limit_by_ip is a
// silent no-op. `/api/v1/ipfs/pins` (no walk) is merged in unbraked, as before.
let ipfs_limiter = rate_limit::IpRateLimiter {
limiter: state.ipfs_rate_limiter.clone(),
trust: state.push_limiter_trust,
};
let ipfs_routes = Router::new()
.route("/ipfs/{cid}", get(ipfs::get_by_cid))
.layer(middleware::from_fn(auth::optional_signature))
.layer(middleware::from_fn(rate_limit::rate_limit_by_ip))
.layer(axum::Extension(ipfs_limiter))
.merge(Router::new().route("/api/v1/ipfs/pins", get(ipfs::list_pins)));
// ── Arweave permanent anchors ──────────────────────────────────────────
+1 -311
View File
@@ -17,6 +17,7 @@ pub struct RefUpdateBroadcast {
pub pusher_did: String,
pub node_did: String,
pub timestamp: String,
pub owner_did: String,
}
#[derive(Clone, Debug)]
@@ -65,45 +66,6 @@ pub struct AppState {
/// brake a push flood from a DID farm (one throwaway DID per repo), so the
/// push path throttles on the resolved client IP instead.
pub push_rate_limiter: RateLimiter,
/// Per-client-IP ROUTE brake for `GET /ipfs/{cid}`: charged ONCE per request by the
/// `rate_limit_by_ip` middleware (server.rs), never inside the handler. It bounds
/// request RATE (the "requests per hour" contract of `GITLAWB_IPFS_RATE_LIMIT`) on
/// the non-farmable source IP, so an anonymous flood of the public route is capped.
/// The per-probe/per-walk WORK accounting the resolver does WITHIN a request draws
/// from the SEPARATE `ipfs_work_rate_limiter` below — the two cannot share one bucket
/// or a single request that spends a route token and then its own probe token off the
/// same bucket is admitted at the route and falsely shed mid-request (#173 round-10,
/// R6). Keyed by `push_limiter_trust`.
pub ipfs_rate_limiter: RateLimiter,
/// Per-client-IP WORK-budget limiter for the `GET /ipfs/{cid}` resolver's internal
/// fan-out: charged per legacy (NULL-provenance) PROBE (`acquire` + `cat-file`) and
/// per provenance-path WALK, and peeked non-consuming before the O(repos) legacy
/// preload. A legacy CID from the public pins index otherwise lets one request drive
/// O(repos) subprocess spawns and cold Tigris fetches, and repeat requests amplify
/// that across requests with zero limiter contact (INV-10, F3). Charging the work to
/// the non-farmable source IP bounds it. A bucket DISTINCT from the route brake above:
/// one request legitimately spends many work tokens (a full legacy scan is up to
/// `ipfs_max_legacy_probes` probes), so it must not double as the once-per-request
/// route bucket. Capacity is DERIVED from the route limit (`AppState::ipfs_work_budget`,
/// no separate operator knob), floored at the legacy-probe budget so a single default-
/// config deep search never self-throttles mid-scan. Keyed by `push_limiter_trust`.
pub ipfs_work_rate_limiter: RateLimiter,
/// Per-request ceiling on full-history reachability walks the CID resolver
/// may spawn (default `api::ipfs::MAX_HISTORY_WALKS_PER_REQUEST`). A field,
/// not a bare const, so tests can shrink it to exercise the cap cheaply;
/// production keeps the const default.
pub ipfs_max_history_walks: u32,
/// Per-request ceiling on legacy (NULL-provenance) repo probes in the CID
/// resolver's scan fallback (default `api::ipfs::MAX_LEGACY_PROBES_PER_REQUEST`).
/// Bounds the anonymous `acquire` + `cat-file` fan-out across the node (#173,
/// INV-10); a field for the same test-seam reason as `ipfs_max_history_walks`.
pub ipfs_max_legacy_probes: u32,
/// Hard ceiling on the byte size of an object `GET /ipfs/{cid}` will buffer and
/// serve (default `api::ipfs::MAX_SERVED_OBJECT_BYTES`). The serve reads via a
/// blocking `git cat-file` and buffers the whole object; without a bound a large
/// public blob could exhaust memory or block a runtime worker (#173, F6, INV-10).
/// A field for the same test-seam reason as the sibling caps.
pub ipfs_max_served_object_bytes: u64,
/// Which forwarded header (if any) the edge is trusted to set, for
/// resolving the push limiter's client-IP key. See `GITLAWB_TRUSTED_PROXY`.
/// Node-wide; also keys the two peer-sync limiters below.
@@ -127,94 +89,6 @@ pub struct AppState {
/// * the libp2p swarm task
/// * the gossip, sync, operator heartbeat, and rate-limit cleanup loops
pub shutdown_tx: tokio::sync::watch::Sender<bool>,
/// Bounds concurrent served git READ operations (upload-pack + both info/refs
/// advertisements). A read handler acquires a permit before spawning git and
/// holds it for the op; when none are free the request is shed with a 503.
/// Writes draw from `git_write_semaphore` so a read flood cannot shed an
/// authenticated push at admission (#174).
pub git_read_semaphore: Arc<tokio::sync::Semaphore>,
/// Bounds concurrent `git-receive-pack` (push) operations, a pool separate
/// from `git_read_semaphore` so an anonymous READ flood can never shed an
/// authenticated push (#174). Sized by `max_concurrent_git_pushes`. Drawn from
/// by the `git-receive-pack` POST (owner-gated) ONLY. The anon-reachable
/// receive-pack `info/refs` advertisement draws from the SEPARATE
/// `git_push_advert_semaphore` below, never this pool, so a multi-source flood
/// of push-handshake advertisements can never occupy a permit an authenticated
/// POST needs at admission (#174).
pub git_write_semaphore: Arc<tokio::sync::Semaphore>,
/// Bounds concurrent anon-reachable `git-receive-pack` `info/refs`
/// advertisements — a pool SEPARATE from `git_write_semaphore` so adverts (which
/// hold a permit across `acquire_fresh` + `info/refs`) can never consume a slot
/// the authenticated POST relies on. A per-source flood can at worst exhaust this
/// advert pool (each source also capped by `git_push_advert_per_caller` and the
/// per-IP push rate limiter), and the reserved POST pool is untouched (#174).
pub git_push_advert_semaphore: Arc<tokio::sync::Semaphore>,
/// Bounds concurrent post-push encrypt-then-pin history walks. Each successful
/// path-scoped push releases its handler write permit and then runs a DETACHED
/// full-history walk (`withheld_blob_recipients_bounded`) to seal withheld blobs;
/// without a cap, N fast pushes spawn N concurrent full-history git walks past
/// `max_concurrent_git_pushes` (which only bounds the in-handler phase). The walk
/// acquires a permit here and DEFERS (blocks) when the pool is full rather than
/// shedding — the work is background and dropping it would lose the recovery copy
/// (#174 P1-e). A pool of its own, not `git_write_semaphore`: a long background
/// walk must not hold a foreground write slot, and a handler already holding a
/// write permit that needed a second would self-deadlock at pool size 1.
pub git_encrypt_semaphore: Arc<tokio::sync::Semaphore>,
/// Bounds the OUTSTANDING post-push encryption-task set by per-repo coalescing
/// (#174 P2-2). `git_encrypt_semaphore` caps *active* walks; this caps how many
/// detached tasks *spawn and park* on that semaphore's `acquire_owned().await`.
/// Before spawning a per-push encryption task, the receive-pack handler consults
/// this set: if the repo already has a task in flight it coalesces (skips the
/// duplicate spawn) rather than parking a new unbounded waiter. Coalescing only
/// delays a duplicate walk — it never drops the withheld-blob recovery copy, which
/// `2a54c15` deliberately kept fail-closed (there is no reconciliation sweep to
/// re-derive a dropped copy). See [`EncryptInflight`].
pub encrypt_inflight: EncryptInflight,
/// Per-caller concurrency sub-cap on the read pool: each caller (keyed on the
/// resolved source IP, #174 U1) may hold at most `max_concurrent_reads_per_caller`
/// in-flight read ops, so one caller cannot monopolize `git_read_semaphore`
/// (#174). Applied by `git_upload_pack` and the upload-pack `info/refs`
/// advertisement.
pub git_read_per_caller: crate::rate_limit::PerCallerConcurrency,
/// Per-source concurrency sub-cap on the anon-reachable receive-pack `info/refs`
/// advertisement: each source IP may hold at most a small share of the write
/// pool, so a multi-source flood of push-handshake advertisements cannot
/// saturate `git_write_semaphore` and shed authenticated pushes (#174). Sized as
/// a fraction of `max_concurrent_git_pushes`, so filling the write pool takes many
/// distinct source IPs (each also braked by the per-IP push rate limiter).
pub git_push_advert_per_caller: crate::rate_limit::PerCallerConcurrency,
/// Per-source concurrency sub-cap on the authenticated `git-receive-pack` POST:
/// each source IP may hold at most a small share of `git_write_semaphore`, so one
/// host minting disposable `did:key` identities cannot open enough slow pushes to
/// monopolize the write pool and 503 every other source's push (#174 P1-d). Keyed
/// on the resolved source IP (never the DID — a DID farm defeats a DID key). Sized
/// like `git_push_advert_per_caller`, a fraction of `max_concurrent_git_pushes`.
pub git_write_per_caller: crate::rate_limit::PerCallerConcurrency,
/// Bounds concurrent `GET /ipfs/{cid}` visibility-walk requests. The public
/// `/ipfs/{cid}` route runs `allowed_blob_set_for_caller_bounded` in
/// `spawn_blocking` (a full-history git walk) with NO served-git admission of its
/// own; without this a permissionless caller fans out concurrent walks past every
/// git pool, exhausting the blocking pool + PIDs (#174 P1-3). A request acquires a
/// permit before the repo loop and holds it for the whole request (across every
/// `spawn_blocking` walk), so the slot reflects real thread occupancy — a tokio
/// walk-timeout cannot free it while the blocking work still runs. A pool of its
/// own (`max_concurrent_ipfs_walks`), NOT a git pool: distinct cost center + public
/// surface, so anonymous /ipfs traffic can never shed an authenticated git op.
pub git_ipfs_walk_semaphore: Arc<tokio::sync::Semaphore>,
/// Per-source concurrency sub-cap on the `/ipfs/{cid}` walk pool: each source
/// (keyed on the resolved source IP, never the DID — `/ipfs` admits any `did:key`
/// unthrottled, so a DID key would be free to mint around) may hold at most
/// `ipfs_walk_per_source` in-flight walk slots, so one source cannot monopolize
/// `git_ipfs_walk_semaphore` (#174 P1-3). A request with no resolvable key is
/// bounded by the global pool only, never this sub-cap. The key map is bounded
/// (`with_default_max_keys`, reject-before-insert) so a source-key farm cannot grow
/// it (INV-15).
pub git_ipfs_walk_per_caller: crate::rate_limit::PerCallerConcurrency,
/// The `git` executable the served-git withheld-blob walk spawns. Production is
/// `"git"` (resolved via PATH); injectable so a fake `git` can drive the walk's
/// process-group teardown in handler tests without mutating the process-global
/// PATH (#174).
pub git_bin: String,
}
impl AppState {
@@ -224,21 +98,6 @@ impl AppState {
self.shutdown_tx.subscribe()
}
/// Sweep expired entries from every per-IP/DID rate limiter. Driven by the
/// periodic cleanup task so a bounded limiter's key map sheds stale entries
/// instead of sitting near its cap until an inline capacity sweep reclaims
/// them. Every limiter on the state is swept here; adding a new limiter means
/// adding it to this list.
pub(crate) async fn sweep_rate_limiters(&self) {
self.rate_limiter.cleanup().await;
self.create_ip_rate_limiter.cleanup().await;
self.push_rate_limiter.cleanup().await;
self.ipfs_rate_limiter.cleanup().await;
self.ipfs_work_rate_limiter.cleanup().await;
self.sync_trigger_rate_limiter.cleanup().await;
self.peer_write_rate_limiter.cleanup().await;
}
/// Trigger graceful shutdown. Idempotent — calling more than once
/// has no effect. Returns `true` if this call was the one that
/// flipped the signal.
@@ -259,173 +118,4 @@ impl AppState {
pub fn is_shutting_down(&self) -> bool {
*self.shutdown_tx.borrow()
}
/// Legacy-probe budget wired from the `GITLAWB_IPFS_MAX_REPOS_WALKED` operator
/// knob (R5, KTD5). The knob seeds `ipfs_max_legacy_probes` at construction so it
/// controls the per-request legacy (NULL-provenance) probe fan-out it advertises.
/// It deliberately does NOT feed `ipfs_max_history_walks`: that ceiling must stay
/// at `MAX_HISTORY_WALKS_PER_REQUEST` (`MAX_PIN_SOURCES + 1`) or a provenanced
/// request with a full source set is truncated into a false 503, and the knob's
/// range starts at 1. The knob is `usize`, the field `u32`; the range cap
/// (1_048_576) keeps the cast lossless.
pub(crate) fn ipfs_legacy_probe_budget(config: &crate::config::Config) -> u32 {
config.ipfs_max_repos_walked as u32
}
/// Work-budget capacity for [`ipfs_work_rate_limiter`](Self#structfield.ipfs_work_rate_limiter)
/// (R6, KTD6), DERIVED from the route limit rather than a new operator knob. The route
/// limiter (`ipfs_rate_limiter`) charges once per request; this separate bucket absorbs
/// the resolver's per-probe/per-walk work charges so both the route "requests per hour"
/// contract and the amplification bound hold. Floor: at least one full legacy search per
/// window — the effective `ipfs_max_legacy_probes` (the `GITLAWB_IPFS_MAX_REPOS_WALKED`
/// knob, U4) — so a single default-config deep search cannot self-throttle mid-scan and
/// recreate the F6 admit-then-429 for a legitimate caller. `GITLAWB_IPFS_RATE_LIMIT=0`
/// disables the route brake and this derived bucket alike (a 0-capacity limiter admits
/// everything).
pub(crate) fn ipfs_work_budget(config: &crate::config::Config) -> usize {
if config.ipfs_rate_limit == 0 {
return 0;
}
config.ipfs_rate_limit.max(config.ipfs_max_repos_walked)
}
}
/// Bounds the OUTSTANDING post-push encryption-task set by per-repo coalescing
/// (#174 P2-2). Each successful path-scoped push `tokio::spawn`s a DETACHED task that
/// parks on `git_encrypt_semaphore.acquire_owned().await` (which DEFERS when the pool
/// is full rather than shedding — `2a54c15` kept it fail-closed so the withheld-blob
/// recovery copy is never dropped). The semaphore caps *active* walks, but nothing
/// capped how many detached tasks *spawn and park* on that await: N rapid pushes to a
/// repo spawn N parked tasks, each holding cloned object lists/rules/paths/keys — an
/// unbounded outstanding set.
///
/// This tracks the repo keys with an in-flight encryption task, each carrying a
/// DIRTY flag. Before spawning, the handler calls [`try_begin`](Self::try_begin): if
/// a task for the repo is already in-flight it MARKS THE REPO DIRTY and returns `None`
/// so the handler SKIPS spawning a duplicate (coalesce). The in-flight task consults
/// the flag at its tail via [`requeue_or_release`](EncryptInflightGuard::requeue_or_release):
/// a set flag makes it run ONE MORE pass (re-reading repo state) before it exits, so a
/// push coalesced during the in-flight window is REQUEUED, never dropped. This bounds
/// the outstanding set to <=1 task per repo without losing work: there is no
/// reconciliation sweep, so a *dropped* job would be lost forever.
///
/// The returned [`EncryptInflightGuard`] is moved into the detached task. The normal
/// exit path is `requeue_or_release`, which removes the repo key ATOMICALLY with the
/// "clean" decision — no push can land in the gap between "checked clean" and "task
/// exits". `Drop` is a panic backstop only: if the task panics (or returns without
/// calling `requeue_or_release`) it still removes the key so a crashed walk can never
/// permanently lock a repo out of future recovery copies.
#[derive(Clone, Default)]
pub struct EncryptInflight {
// std::sync::Mutex: only ever held for O(1) HashMap ops in a sync context (before
// `tokio::spawn`, at the task tail, and in Drop) — never across an await, so a std
// Mutex is correct and cheaper than a tokio one. The value is the DIRTY flag: a
// coalesced push flips it true so the in-flight task requeues one more pass.
repos: Arc<std::sync::Mutex<std::collections::HashMap<String, bool>>>,
}
impl EncryptInflight {
pub fn new() -> Self {
Self::default()
}
/// Try to begin an encryption task for `repo_id`. Returns `Some(guard)` if no task
/// for the repo was in-flight (the caller should spawn). If one already is, MARKS
/// the repo dirty and returns `None` (the caller COALESCES — skips the duplicate
/// spawn; the in-flight task requeues one more pass to cover this push).
pub fn try_begin(&self, repo_id: &str) -> Option<EncryptInflightGuard> {
let mut map = self.repos.lock().expect("encrypt_inflight mutex poisoned");
match map.get_mut(repo_id) {
Some(dirty) => {
*dirty = true;
None
}
None => {
map.insert(repo_id.to_string(), false);
Some(EncryptInflightGuard {
repos: Arc::clone(&self.repos),
repo_id: repo_id.to_string(),
released: false,
})
}
}
}
/// Number of repos with an in-flight encryption task. Test/metrics observability;
/// the bound under saturation is `len() <= number of distinct repos`, i.e. at most
/// one task per repo.
#[allow(dead_code)]
pub fn len(&self) -> usize {
self.repos
.lock()
.expect("encrypt_inflight mutex poisoned")
.len()
}
#[allow(dead_code)]
pub fn is_empty(&self) -> bool {
self.len() == 0
}
/// Test-only: read the dirty flag for `repo_id` (`None` if no task is in-flight).
#[cfg(test)]
pub fn dirty(&self, repo_id: &str) -> Option<bool> {
self.repos
.lock()
.expect("encrypt_inflight mutex poisoned")
.get(repo_id)
.copied()
}
}
/// RAII guard for one in-flight encryption task's repo key. The task drives it at its
/// tail with [`requeue_or_release`](Self::requeue_or_release); `Drop` is a panic
/// backstop. Move-only — cloning would double-release.
pub struct EncryptInflightGuard {
repos: Arc<std::sync::Mutex<std::collections::HashMap<String, bool>>>,
repo_id: String,
released: bool,
}
impl EncryptInflightGuard {
/// TASK-TAIL check-and-clear, atomic with the release decision. If a push
/// coalesced since the last pass (dirty), clear the flag and return `true` (the
/// task loops one more pass). Otherwise remove the key and return `false` (the task
/// exits). Atomic under the mutex: a concurrent push either sets the flag BEFORE
/// this reads it (-> requeue covers it) or arrives AFTER the key is removed (-> a
/// fresh `try_begin` spawns a new task) — there is no window where the key is
/// present-but-clean while the task exits.
pub fn requeue_or_release(&mut self) -> bool {
let mut map = self.repos.lock().expect("encrypt_inflight mutex poisoned");
match map.get_mut(&self.repo_id) {
Some(dirty) if *dirty => {
*dirty = false;
true
}
_ => {
map.remove(&self.repo_id);
self.released = true;
false
}
}
}
}
impl Drop for EncryptInflightGuard {
fn drop(&mut self) {
// Panic backstop ONLY. The normal exit is `requeue_or_release` (which removed
// the key atomically and set `released`), so this does nothing then. If the
// task panicked or returned without releasing, remove the key so one crashed
// walk never permanently locks the repo out.
//
// Accepted residual: a panic with the dirty flag still set drops the requeued
// work — Drop cannot loop — the same loss class as the on-panic behavior before
// this change. No reconciliation sweep re-derives it.
if self.released {
return;
}
if let Ok(mut map) = self.repos.lock() {
map.remove(&self.repo_id);
}
}
}
File diff suppressed because it is too large. Load diff
-25
View File
@@ -437,31 +437,6 @@ mod tests {
);
}
// #135 T1: a Mode-B rule on `/secret/**` must DENY the withheld directory's
// OWN path `/secret` (the `path == prefix` arm), not just strict descendants —
// otherwise get_by_cid's tree gate would serve the /secret tree object and leak
// its children. Pins parity with get_tree, which denies the /secret path.
#[test]
fn subtree_rule_denies_the_withheld_directory_itself() {
let reader = "did:key:z6MkReader";
let rules = [rule("/secret/**", VisibilityMode::B, &[reader])];
assert_eq!(
visibility_check(&rules, true, OWNER, None, "/secret"),
Decision::Deny,
"anon denied at the withheld directory's OWN path /secret"
);
assert_eq!(
visibility_check(&rules, true, OWNER, None, "/public"),
Decision::Allow,
"anon allowed at a sibling path outside the withheld subtree"
);
assert_eq!(
visibility_check(&rules, true, OWNER, Some(reader), "/secret"),
Decision::Allow,
"listed reader allowed at the withheld directory (caller-aware)"
);
}
// #153 regression: cross-method DID must still be denied even when the
// trailing segment collides with a bare owner key.
#[test]
-139
View File
@@ -1,139 +0,0 @@
//! #174 U6 — INV-22 completeness guard (rung-raising).
//!
//! INV-22: a permit held per op recovers only if every path that holds it is also
//! duration-bounded and reaps the process group before releasing admission, and every
//! detached git/blocking task carries its own admission. PR #174 fixed five paths
//! (U1-U5) that violated this. Each fix has a per-unit RED/GREEN regression; together
//! those form the five-revert matrix. This guard adds the missing piece: a source-scan
//! tripwire that fails when a NEW site reintroduces the class, or when one of the five
//! gates is removed.
//!
//! It lives in `tests/` (a separate crate) on purpose: a guard that scanned the same
//! file it lives in would match its own identifier literals and pass vacuously. Here
//! the scanned `src/` files never contain this file's literals, so each check is
//! load-bearing — reverting the named gate turns the assertion red.
//!
//! These are deliberately coarse structural checks, not a parser. They cannot prove a
//! gate is *correct* (the per-unit tests do that); they prove a gate is *present and
//! not bypassed*, which is what stops the class from silently regressing.
use std::path::Path;
fn src(rel: &str) -> String {
let p = Path::new(env!("CARGO_MANIFEST_DIR")).join("src").join(rel);
std::fs::read_to_string(&p).unwrap_or_else(|e| panic!("read {}: {e}", p.display()))
}
#[test]
fn inv22_concurrency_gates_present_and_not_bypassed() {
let repos = src("api/repos.rs");
let smart_http = src("git/smart_http.rs");
let vis = src("git/visibility_pack.rs");
let ipfs = src("api/ipfs.rs");
// U1 / P1-a: run_bounded_git stands the watchdog down only after confirming the
// child actually terminated (WNOWAIT), not on the raw stdout-drain EOF — otherwise
// a child that closes stdout then hangs pins the permit past the deadline. The
// probe is defined and called, so >= 2 occurrences; reverting the fix removes both.
assert!(
vis.matches("child_terminated_without_reaping").count() >= 2,
"U1/P1-a gate missing: run_bounded_git must confirm child exit via \
child_terminated_without_reaping before signalling the watchdog"
);
// U2 / P1-c: on client disconnect KillGroupOnDrop must launch a detached reaper
// that runs the full TERM/grace/KILL/reap, not a lone SIGTERM. The reaper is spawned
// via a runtime handle in Drop; `Handle::try_current` is unique to that launch (the
// timeout path already has an async context and never calls it).
assert!(
smart_http.contains("Handle::try_current"),
"U2/P1-c gate missing: KillGroupOnDrop::drop must launch the full \
TERM/grace/KILL reaper on disconnect (Handle::try_current), not a lone SIGTERM"
);
// U4 / P1-d: git_receive_pack must acquire the per-source write sub-cap before the
// global write permit. The acquire reads `state.git_write_per_caller`; comments name
// the field without the `state.` prefix, so this targets the real acquire site.
assert!(
repos.contains("state.git_write_per_caller"),
"U4/P1-d gate missing: git_receive_pack must acquire the per-source write cap \
(state.git_write_per_caller) before the global write permit"
);
// U5 / P1-e: the detached post-push encryption walk must run through the
// admission-gated helper, which is wired to the shared encrypt pool.
assert!(
repos.contains("fn withheld_recipients_gated")
&& repos.contains("state.git_encrypt_semaphore"),
"U5/P1-e gate missing: the encryption walk must run through \
withheld_recipients_gated, which acquires git_encrypt_semaphore"
);
// U1 / R2 (#173 round-10): the path-scoped filtered-pack serve must thread the
// caller's AdmissionGuard through BOTH git stages so read + per-caller admission is
// held until the pack-objects group is reaped on disconnect, closing the cap bypass
// the plain upload_pack path already fixed. Two load-bearing markers: rev-list hands
// the disarmed guard back (its tuple return type), and build_filtered_pack forwards
// that guard into the pack-objects stage (the `admission` arg after the
// "pack-objects" label). Reverting either — dropping the guard between stages, or
// passing `None` to pack-objects — trips this.
assert!(
smart_http.contains("Result<(Vec<String>, Option<AdmissionGuard>)>")
&& smart_http.contains("\"pack-objects\",\n admission,"),
"U1/R2 gate missing: build_filtered_pack must thread the AdmissionGuard through \
rev-list -> pack-objects so the permits are held until the pack-objects group \
is reaped on disconnect (the path-scoped half of #174 P1-a)"
);
// U2 / R1 (#173 round-10): the `GET /ipfs/{cid}` serve pipeline must run in a
// DETACHED tokio task that OWNS the AdmissionGuard built from BOTH walk permits, so a
// cancelled or timed-out request releases admission only after the spawned (bounded)
// work completes — not the instant the handler future drops (the /ipfs half of #174
// P1-a). Two load-bearing markers: the guard is constructed from the two named
// permits, and the whole pipeline is moved into a `tokio::spawn`. Reverting to
// handler-local permits (dropping the guard/spawn) trips this.
assert!(
ipfs.contains("AdmissionGuard::new(ipfs_walk_permit, ipfs_caller_permit)")
&& ipfs.contains(
"let serve: tokio::task::JoinHandle<Result<Response>> = tokio::spawn(async move {"
),
"U2/R1 gate missing: get_by_cid must move both /ipfs admission permits into an \
AdmissionGuard owned by a detached tokio::spawn task so admission is released \
only after the spawned serve work completes (the /ipfs half of #174 P1-a)"
);
// U2 / KTD2 (#173 round-10): the probe/read children on the /ipfs path must be the
// duration-bounded twins (process-group teardown via run_bounded_git), not the bare
// `store::object_type` / `read_object_content` (or an unbounded `cat-file -s`) a tokio
// timeout cannot cancel — otherwise a wedged cat-file lingers and pins the held
// admission past the deadline. Reverting any twin call site back to a bare read trips
// this.
assert!(
ipfs.contains("object_type_bounded(")
&& ipfs.contains("object_size_bounded(")
&& ipfs.contains("read_object_content_bounded("),
"U2/KTD2 gate missing: the /ipfs probe+read must call the run_bounded_git-backed \
*_bounded twins so a wedged cat-file is reaped at the deadline, not left to pin \
the held /ipfs walk admission"
);
// P1-e non-bypass tripwire: the bounded recipients walk is spawn_blocking'd nowhere
// but inside withheld_recipients_gated. A second call site (count > 1) is a new
// detached git walk that skips the admission gate — exactly the class U5 closed.
assert_eq!(
repos.matches("withheld_blob_recipients_bounded").count(),
1,
"P1-e bypass: the bounded recipients walk must be invoked only inside \
withheld_recipients_gated; a new call site bypasses the encrypt-walk admission cap"
);
// U4 / P2-2: the detached post-push encryption task must be gated by the per-repo
// coalescing set (`encrypt_inflight.try_begin`) so the OUTSTANDING parked-task set is
// bounded to <=1 per repo. Removing the gate lets N rapid pushes spawn N parked
// waiters (the unbounded set U4 closed); the semaphore only caps active walks.
assert!(
repos.contains("state.encrypt_inflight.try_begin"),
"U4/P2-2 gate missing: the detached post-push encryption spawn must consult \
encrypt_inflight.try_begin to coalesce per repo (bound the outstanding-task set)"
);
}
+2 -1
View File
@@ -1,7 +1,7 @@
[package]
name = "gl"
description = "The gitlawb CLI — identity management, node control, MCP server"
version = "0.5.1" # x-release-please-version
version = "0.7.0" # x-release-please-version
edition.workspace = true
rust-version.workspace = true
license.workspace = true
@@ -13,6 +13,7 @@ path = "src/main.rs"
[dependencies]
gitlawb-core = { path = "../gitlawb-core" }
icaptcha-client = { path = "../icaptcha-client" }
base64 = { workspace = true }
tokio = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
+216 -27
View File
@@ -41,6 +41,16 @@ pub enum CertCmd {
node: String,
#[arg(long)]
dir: Option<PathBuf>,
/// Exit non-zero unless the Ed25519 signature verifies AND the
/// issuing node matches the queried node (or --expect-node)
#[arg(long)]
verify: bool,
/// Expected issuing node DID for --verify. A valid signature alone
/// only proves the cert is internally consistent — signed by whatever
/// key it names — so --verify also anchors the issuer to a DID you
/// trust: this value when given, else the queried node's DID.
#[arg(long, requires = "verify")]
expect_node: Option<String>,
},
}
@@ -52,7 +62,9 @@ pub async fn run(args: CertArgs) -> Result<()> {
id,
node,
dir,
} => cmd_show(repo, id, node, dir).await,
verify,
expect_node,
} => cmd_show(repo, id, node, dir, verify, expect_node).await,
}
}
@@ -114,7 +126,14 @@ async fn cmd_list(repo: String, node: String, dir: Option<PathBuf>) -> Result<()
Ok(())
}
async fn cmd_show(repo: String, id: String, node: String, dir: Option<PathBuf>) -> Result<()> {
async fn cmd_show(
repo: String,
id: String,
node: String,
dir: Option<PathBuf>,
require_valid: bool,
expect_node: Option<String>,
) -> Result<()> {
let (owner, name) = resolve_repo(&repo, &node, dir.as_deref()).await?;
let client = signed_client(&node, dir.as_deref());
@@ -148,39 +167,124 @@ async fn cmd_show(repo: String, id: String, node: String, dir: Option<PathBuf>)
println!(" Signature: {signature}");
println!();
// Reconstruct the signing payload and verify
// Fetch the node's current public key to verify
let info: Value = client
.get("/")
.await?
.json()
.await
.context("failed to fetch node info")?;
let current_node_did = info["did"].as_str().unwrap_or("");
// Verify the Ed25519 signature: rebuild the exact canonical payload the
// node signed (see gitlawb-node/src/cert.rs::issue_ref_certificate) and
// check it against the public key embedded in the certificate's node DID.
// This proves the cert is internally authentic — signed by the key it
// names; the node-DID comparison below covers *which* node that is.
let repo_id = cert["repo_id"].as_str().unwrap_or("");
let verdict = verify_signature(
repo_id, ref_name, old_sha, new_sha, pusher, node_did, issued_at, signature,
);
println!("Signature verification:");
println!(" Signing payload would be:");
println!(" {{\"repo_id\": ..., \"ref\": \"{ref_name}\", \"old\": \"{old_sha}\",");
println!(" \"new\": \"{new_sha}\", \"pusher\": \"{pusher}\",");
println!(" \"node\": \"{node_did}\", \"ts\": \"{issued_at}\"}}");
println!();
if current_node_did == node_did {
println!(" Node DID matches current node. Signature is an Ed25519/base64url value.");
println!(" To verify offline, use the node's Ed25519 public key derived from:");
println!(" did:key → {node_did}");
} else {
println!(" WARNING: Certificate node DID ({node_did}) does not match");
println!(" current node DID ({current_node_did}).");
println!(" This certificate was issued by a different node.");
match &verdict {
Ok(()) => {
println!(
" VALID — Ed25519 signature verified against the key the certificate names ({node_did})"
);
}
Err(reason) => {
println!(" INVALID — {reason}");
}
}
println!();
println!(" Signature (base64url): {signature}");
// Contextual only — the verdict above stands on its own, so a node-info
// hiccup here must not turn a successfully displayed certificate into an
// error exit.
let current_node_did = match client.get("/").await {
Ok(resp) => resp
.json::<Value>()
.await
.ok()
.and_then(|info| info["did"].as_str().map(str::to_string)),
Err(_) => None,
};
match current_node_did.as_deref() {
Some(current) if current == node_did => {
println!(" Issuing node DID matches the node being queried.");
}
Some(current) => {
println!(" WARNING: Certificate node DID ({node_did}) does not match");
println!(" current node DID ({current}).");
println!(" This certificate was issued by a different node.");
}
None => {
println!(" NOTE: could not fetch current node info — skipping node-DID comparison.");
}
}
if require_valid {
if let Err(reason) = verdict {
anyhow::bail!("certificate signature did not verify: {reason}");
}
// A valid signature proves internal consistency only: the payload was
// signed by whatever key the certificate itself names. A hostile
// source can mint a keypair, put its DID in node_did, and self-sign.
// --verify therefore also anchors the issuer to a trusted DID:
// --expect-node when given, else the DID of the node being queried.
let expected = expect_node.as_deref().or(current_node_did.as_deref());
match expected {
Some(expected) if expected == node_did => {}
Some(expected) => anyhow::bail!(
"certificate is signed by {node_did}, but the expected issuer is {expected} — \
a valid signature alone proves internal consistency, not a trusted issuer"
),
None => anyhow::bail!(
"cannot anchor the issuer: node info is unreachable and no --expect-node was given"
),
}
}
Ok(())
}
/// Rebuild the node's canonical signing payload (field order must match
/// gitlawb-node/src/cert.rs::issue_ref_certificate exactly) and verify the
/// certificate's Ed25519 signature against the key embedded in `node_did`.
#[allow(clippy::too_many_arguments)]
fn verify_signature(
repo_id: &str,
ref_name: &str,
old_sha: &str,
new_sha: &str,
pusher: &str,
node_did: &str,
issued_at: &str,
signature_b64: &str,
) -> std::result::Result<(), String> {
use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine};
use std::str::FromStr;
let payload = serde_json::json!({
"repo_id": repo_id,
"ref": ref_name,
"old": old_sha,
"new": new_sha,
"pusher": pusher,
"node": node_did,
"ts": issued_at,
});
let payload_bytes =
serde_json::to_vec(&payload).map_err(|e| format!("could not serialize payload: {e}"))?;
let did =
gitlawb_core::did::Did::from_str(node_did).map_err(|e| format!("bad node DID: {e}"))?;
let verifying_key = did
.to_verifying_key()
.map_err(|e| format!("cannot derive a public key from {node_did}: {e}"))?;
let sig_vec = URL_SAFE_NO_PAD
.decode(signature_b64)
.map_err(|e| format!("signature is not valid base64url: {e}"))?;
let sig_bytes: [u8; 64] = sig_vec
.try_into()
.map_err(|_| "signature is not 64 bytes".to_string())?;
gitlawb_core::identity::verify(&verifying_key, &payload_bytes, &sig_bytes)
.map_err(|_| "Ed25519 signature does not match the signed payload".to_string())
}
async fn resolve_cert_id(client: &NodeClient, owner: &str, name: &str, id: &str) -> Result<String> {
if id.len() >= 36 {
return Ok(id.to_string());
@@ -209,3 +313,88 @@ async fn resolve_cert_id(client: &NodeClient, owner: &str, name: &str, id: &str)
_ => anyhow::bail!("certificate prefix {id} matches multiple certificates"),
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Pins gl's payload reconstruction to the frozen canonical byte form the
/// node signs (default serde_json maps = alphabetically ordered keys). If
/// serialization drifts — a field added, or a preserve_order feature
/// landing anywhere in the workspace (feature unification flips every
/// crate at once) — this literal stops matching and the test fails,
/// instead of every real certificate silently rendering INVALID.
#[test]
fn payload_serialization_matches_frozen_canonical_form() {
let payload = serde_json::json!({
"repo_id": "repo-1",
"ref": "refs/heads/main",
"old": "oldsha",
"new": "newsha",
"pusher": "did:key:z6MkPusher",
"node": "did:key:z6MkNode",
"ts": "2026-07-22T00:00:00+00:00",
});
let frozen = concat!(
r#"{"new":"newsha","node":"did:key:z6MkNode","old":"oldsha","#,
r#""pusher":"did:key:z6MkPusher","ref":"refs/heads/main","#,
r#""repo_id":"repo-1","ts":"2026-07-22T00:00:00+00:00"}"#,
);
assert_eq!(serde_json::to_string(&payload).unwrap(), frozen);
}
/// Signing exactly as the node does must round-trip through
/// verify_signature; any field tampering must fail it.
#[test]
fn verify_signature_round_trip_and_tamper() {
let kp = gitlawb_core::identity::Keypair::generate();
let node_did = kp.did().as_str().to_string();
let payload = serde_json::json!({
"repo_id": "repo-1",
"ref": "refs/heads/main",
"old": "0".repeat(40),
"new": "a".repeat(40),
"pusher": "did:key:z6MkPusher",
"node": node_did,
"ts": "2026-07-22T00:00:00+00:00",
});
let sig = kp.sign_b64(&serde_json::to_vec(&payload).unwrap());
let ok = verify_signature(
"repo-1",
"refs/heads/main",
&"0".repeat(40),
&"a".repeat(40),
"did:key:z6MkPusher",
&node_did,
"2026-07-22T00:00:00+00:00",
&sig,
);
assert!(ok.is_ok(), "expected valid signature, got: {ok:?}");
let tampered = verify_signature(
"repo-1",
"refs/heads/main",
&"0".repeat(40),
&"b".repeat(40), // new_sha changed after signing
"did:key:z6MkPusher",
&node_did,
"2026-07-22T00:00:00+00:00",
&sig,
);
assert!(tampered.is_err(), "tampered payload must not verify");
let garbage = verify_signature(
"repo-1",
"refs/heads/main",
&"0".repeat(40),
&"a".repeat(40),
"did:key:z6MkPusher",
&node_did,
"2026-07-22T00:00:00+00:00",
"not-base64url!!!",
);
assert!(garbage.is_err(), "malformed signature must not verify");
}
}
+238 -11
View File
@@ -16,6 +16,7 @@ use std::path::PathBuf;
use crate::http::NodeClient;
const PUBLIC_NODE: &str = "https://node.gitlawb.com";
const GITHUB_API_BASE: &str = "https://api.github.com";
#[derive(Args)]
pub struct DoctorArgs {
@@ -141,18 +142,20 @@ pub async fn run(args: DoctorArgs) -> Result<()> {
// ── 3. GITLAWB_NODE env var ───────────────────────────────────────────
match std::env::var("GITLAWB_NODE") {
Ok(v) if !v.is_empty() && !v.contains("127.0.0.1") && !v.contains("localhost") => {
checks.push(Check::pass("GITLAWB_NODE", v.to_string()));
}
Ok(v) if v.contains("127.0.0.1") || v.contains("localhost") => {
checks.push(Check::fail(
// A loopback host is a legitimate setup (self-hosted node, dev
// harness) — the connectivity check below fails loudly if it is not
// actually reachable, so don't red-flag the configuration itself.
Ok(v) if is_loopback_url(&v) => {
checks.push(Check::pass(
"GITLAWB_NODE",
format!(
"set to local address ({v}) — git push/clone will fail against remote nodes"
"{v} (local node — intentional for self-hosting/dev; unset to target the public network)"
),
"export GITLAWB_NODE=https://node.gitlawb.com",
));
}
Ok(v) if !v.is_empty() => {
checks.push(Check::pass("GITLAWB_NODE", v.to_string()));
}
_ => {
checks.push(Check::fail(
"GITLAWB_NODE",
@@ -242,6 +245,15 @@ pub async fn run(args: DoctorArgs) -> Result<()> {
));
}
// ── 5b. shell alias shadowing ─────────────────────────────────────────
// oh-my-zsh's default `git` plugin aliases gl='git pull', which silently
// shadows this binary in every interactive zsh — the classic symptom is
// `gl` printing "fatal: not a git repository". Aliases beat PATH, so no
// install method can fix it; the user's rc file has to unalias.
if let Some(check) = check_shell_alias_shadowing(dirs::home_dir()) {
checks.push(check);
}
// ── 6. git ────────────────────────────────────────────────────────────
match std::process::Command::new("git")
.arg("--version")
@@ -263,7 +275,7 @@ pub async fn run(args: DoctorArgs) -> Result<()> {
// ── 7. Version up to date ─────────────────────────────────────────────
let current = env!("CARGO_PKG_VERSION");
checks.push(check_version(current).await);
checks.push(check_version(current, GITHUB_API_BASE).await);
// ── Render ────────────────────────────────────────────────────────────
for check in &checks {
@@ -308,6 +320,83 @@ pub async fn run(args: DoctorArgs) -> Result<()> {
}
/// Check if a binary name exists anywhere on PATH.
/// True when the rc file contains a real `unalias` command naming `gl` —
/// not a comment, and not a longer word like `unalias global`. Ordering
/// relative to oh-my-zsh loading is not modeled (an unalias placed before the
/// plugin loads would be ineffective); acceptable slack for a warning check.
fn rc_unaliases_gl(rc: &str) -> bool {
rc.lines().any(|line| {
let line = line.trim_start();
if line.starts_with('#') {
return false;
}
line.split([';', '&', '|']).any(|cmd| {
// A trailing comment must not count: `unalias foo # gl`
let cmd = cmd.split('#').next().unwrap_or("");
let mut tokens = cmd.split_whitespace();
tokens.next() == Some("unalias") && tokens.any(|t| t == "gl")
})
})
}
/// True when the URL's host is exactly a loopback address. Substring checks
/// misclassify hosts like `localhost.example` or URLs with "localhost" in the
/// path, so parse and compare the actual host.
fn is_loopback_url(value: &str) -> bool {
reqwest::Url::parse(value)
.ok()
.and_then(|u| {
u.host_str()
.map(|h| h == "localhost" || h == "127.0.0.1" || h == "[::1]" || h == "::1")
})
.unwrap_or(false)
}
/// Detect interactive-shell setups where an alias shadows `gl` (aliases beat
/// PATH, so no install method can fix this). Best-effort heuristic over
/// ~/.zshrc: an explicit `alias gl=`, or oh-my-zsh's `git` plugin (which
/// ships gl='git pull'). Returns None when nothing suspicious is found or the
/// rc file already contains an `unalias gl`.
fn check_shell_alias_shadowing(home: Option<PathBuf>) -> Option<Check> {
let home = home?;
let rc = std::fs::read_to_string(home.join(".zshrc")).ok()?;
if rc_unaliases_gl(&rc) {
return None;
}
let explicit_alias = rc.lines().any(|l| l.trim_start().starts_with("alias gl="));
// Single-line `plugins=(git ...)` is the overwhelmingly common form; a
// multi-line plugins array slips past this heuristic, which is acceptable
// for a warning-level check.
let omz_git_plugin = home.join(".oh-my-zsh/plugins/git").exists()
&& rc.lines().any(|l| {
let l = l.trim_start();
l.starts_with("plugins=")
&& l.trim_start_matches("plugins=(")
.trim_end_matches(')')
.split_whitespace()
.any(|p| p == "git")
});
if explicit_alias || omz_git_plugin {
let source = if explicit_alias {
"~/.zshrc defines `alias gl=`"
} else {
"oh-my-zsh's git plugin aliases gl='git pull'"
};
Some(Check::warn(
"shell alias",
format!(
"{source} — interactive shells run that instead of this binary \
(symptom: `gl` prints \"fatal: not a git repository\")"
),
"echo 'unalias gl 2>/dev/null' >> ~/.zshrc && source ~/.zshrc (must come after oh-my-zsh loads)",
))
} else {
None
}
}
fn which_in_path(name: &str) -> bool {
std::env::var_os("PATH")
.map(|paths| {
@@ -319,8 +408,9 @@ fn which_in_path(name: &str) -> bool {
.unwrap_or(false)
}
/// Fetch the latest release tag from gitlawb/releases and compare to current version.
async fn check_version(current: &'static str) -> Check {
/// Fetch the latest release tag from Gitlawb/node (the actual release repo — see install.sh's
/// `GITLAWB_RELEASE_REPO` default) and compare to current version.
async fn check_version(current: &'static str, github_api_base: &str) -> Check {
let client = match reqwest::Client::builder()
.user_agent(format!("gl/{current}"))
.timeout(std::time::Duration::from_secs(5))
@@ -331,7 +421,9 @@ async fn check_version(current: &'static str) -> Check {
};
let resp = match client
.get("https://api.github.com/repos/gitlawb/releases/releases/latest")
.get(format!(
"{github_api_base}/repos/Gitlawb/node/releases/latest"
))
.send()
.await
{
@@ -339,6 +431,13 @@ async fn check_version(current: &'static str) -> Check {
Err(_) => return Check::pass("version", format!("v{current} (offline — could not check)")),
};
if !resp.status().is_success() {
return Check::pass(
"version",
format!("v{current} (GitHub API returned HTTP {})", resp.status()),
);
}
let body: serde_json::Value = match resp.json().await {
Ok(v) => v,
Err(_) => return Check::pass("version", format!("v{current} (invalid response)")),
@@ -382,6 +481,88 @@ fn is_newer(latest: &str, current: &str) -> bool {
mod tests {
use super::*;
fn fake_home(zshrc: Option<&str>, with_omz_git: bool) -> tempfile::TempDir {
let home = tempfile::TempDir::new().unwrap();
if let Some(rc) = zshrc {
std::fs::write(home.path().join(".zshrc"), rc).unwrap();
}
if with_omz_git {
std::fs::create_dir_all(home.path().join(".oh-my-zsh/plugins/git")).unwrap();
}
home
}
#[test]
fn alias_shadowing_flags_omz_git_plugin() {
let home = fake_home(Some("plugins=(git z)\nsource $ZSH/oh-my-zsh.sh\n"), true);
let check = check_shell_alias_shadowing(Some(home.path().to_path_buf()));
assert!(check.is_some(), "omz git plugin without unalias must warn");
}
#[test]
fn alias_shadowing_flags_explicit_alias() {
let home = fake_home(Some("alias gl='git pull'\n"), false);
assert!(check_shell_alias_shadowing(Some(home.path().to_path_buf())).is_some());
}
#[test]
fn alias_shadowing_silent_when_unaliased() {
let home = fake_home(
Some("plugins=(git z)\nsource $ZSH/oh-my-zsh.sh\nunalias gl 2>/dev/null\n"),
true,
);
assert!(check_shell_alias_shadowing(Some(home.path().to_path_buf())).is_none());
}
#[test]
fn alias_shadowing_ignores_fake_unaliases() {
// None of these actually free `gl` — the warning must survive them.
for rc in [
"plugins=(git)\n# unalias gl\n", // commented out
"plugins=(git)\nunalias global\n", // longer word
"plugins=(git)\nunalias foo # gl\n", // gl only in a comment
"plugins=(git)\necho unalias-gl-later\n", // not an unalias command
] {
let home = fake_home(Some(rc), true);
assert!(
check_shell_alias_shadowing(Some(home.path().to_path_buf())).is_some(),
"must still warn for rc: {rc:?}"
);
}
// Real unalias forms that do free it — all must silence the warning.
for rc in [
"plugins=(git)\nunalias gl\n",
"plugins=(git)\nunalias gl 2>/dev/null\n",
"plugins=(git)\ntrue; unalias gl\n",
"plugins=(git)\nunalias glog gl gst\n",
] {
let home = fake_home(Some(rc), true);
assert!(
check_shell_alias_shadowing(Some(home.path().to_path_buf())).is_none(),
"must be silent for rc: {rc:?}"
);
}
}
#[test]
fn loopback_url_detection_is_host_exact() {
assert!(is_loopback_url("http://127.0.0.1:7545"));
assert!(is_loopback_url("http://localhost:7545"));
assert!(is_loopback_url("http://[::1]:7545"));
assert!(!is_loopback_url("https://localhost.example"));
assert!(!is_loopback_url("https://node.gitlawb.com/localhost"));
assert!(!is_loopback_url("not a url"));
}
#[test]
fn alias_shadowing_silent_without_signals() {
let home = fake_home(Some("plugins=(z)\nexport EDITOR=vim\n"), false);
assert!(check_shell_alias_shadowing(Some(home.path().to_path_buf())).is_none());
let no_rc = fake_home(None, false);
assert!(check_shell_alias_shadowing(Some(no_rc.path().to_path_buf())).is_none());
assert!(check_shell_alias_shadowing(None).is_none());
}
#[test]
fn test_is_newer_minor_bump() {
assert!(is_newer("0.2.0", "0.1.0"));
@@ -417,4 +598,50 @@ mod tests {
fn test_which_in_path_missing_binary() {
assert!(!which_in_path("this-binary-does-not-exist-gl-test-12345"));
}
#[tokio::test]
async fn test_check_version_queries_gitlawb_node_releases() {
let mut server = mockito::Server::new_async().await;
let _m = server
.mock("GET", "/repos/Gitlawb/node/releases/latest")
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"tag_name":"v9.9.9"}"#)
.create_async()
.await;
let check = check_version("0.1.0", &server.url()).await;
assert!(matches!(check.state, CheckState::Warn));
assert!(check.detail.contains("v9.9.9"));
}
#[tokio::test]
async fn test_check_version_up_to_date() {
let mut server = mockito::Server::new_async().await;
let _m = server
.mock("GET", "/repos/Gitlawb/node/releases/latest")
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"tag_name":"v0.1.0"}"#)
.create_async()
.await;
let check = check_version("0.1.0", &server.url()).await;
assert!(matches!(check.state, CheckState::Ok));
assert!(check.detail.contains("up to date"));
}
#[tokio::test]
async fn test_check_version_http_error() {
let mut server = mockito::Server::new_async().await;
let _m = server
.mock("GET", "/repos/Gitlawb/node/releases/latest")
.with_status(403)
.create_async()
.await;
let check = check_version("0.1.0", &server.url()).await;
assert!(matches!(check.state, CheckState::Ok));
assert!(check.detail.contains("GitHub API returned HTTP 403"));
}
}
+17
View File
@@ -72,6 +72,23 @@ impl NodeClient {
req.send().await.with_context(|| format!("GET {url}"))
}
/// GET that signs when an identity keypair is present and falls back to an
/// anonymous GET otherwise — for read-visibility endpoints, where a public
/// repo is readable anonymously but a private repo requires the owner/reader
/// to be authenticated. Mirrors the conditional signing of post/put/delete.
pub async fn get_maybe_signed(&self, path: &str) -> Result<reqwest::Response> {
let url = format!("{}{}", self.node_url, path);
let mut req = self.inner.get(&url);
if let Some(kp) = &self.keypair {
let signed = sign_request(kp, "GET", path, b"");
req = req
.header("Content-Digest", signed.content_digest)
.header("Signature-Input", signed.signature_input)
.header("Signature", signed.signature);
}
req.send().await.with_context(|| format!("GET {url}"))
}
/// POST with JSON body + RFC 9421 signing + transparent iCaptcha solve/retry.
pub async fn post(&self, path: &str, body: &[u8]) -> Result<reqwest::Response> {
self.send_signed("POST", path, body).await
+52 -4
View File
@@ -38,13 +38,23 @@ pub async fn run(args: InitArgs) -> Result<()> {
let git_dir = cwd.join(".git");
if !git_dir.exists() {
println!("Initializing git repository...");
// -b main: the push flow targets `main`; without it a fresh repo uses
// the user's init.defaultBranch (often `master`). Older git (<2.28)
// lacks the flag, so fall back to a plain init.
let status = std::process::Command::new("git")
.args(["init"])
.args(["init", "-b", "main"])
.current_dir(&cwd)
.status()
.context("failed to run git init")?;
if !status.success() {
anyhow::bail!("git init failed");
let status = std::process::Command::new("git")
.args(["init"])
.current_dir(&cwd)
.status()
.context("failed to run git init")?;
if !status.success() {
anyhow::bail!("git init failed");
}
}
} else {
println!("Git repository detected.");
@@ -165,9 +175,47 @@ pub async fn run(args: InitArgs) -> Result<()> {
}
}
// The hint must match the repo's actual state: with no commits yet, a bare
// `git push` fails with "src refspec ... does not match any" — exactly the
// trap a zero-to-push command must not set.
let has_commits = std::process::Command::new("git")
.args(["rev-parse", "--verify", "HEAD"])
.current_dir(&cwd)
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.map(|s| s.success())
.unwrap_or(false);
// symbolic-ref succeeds on a branch (including an unborn one) and fails
// on a detached HEAD — where guessing "main" could push the wrong ref.
let branch = std::process::Command::new("git")
.args(["symbolic-ref", "--short", "HEAD"])
.current_dir(&cwd)
.stderr(std::process::Stdio::null())
.output()
.ok()
.filter(|o| o.status.success())
.and_then(|o| String::from_utf8(o.stdout).ok())
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty());
println!();
println!("Ready! Push with:");
println!(" git push gitlawb main");
match (branch, has_commits) {
(Some(branch), true) => {
println!("Ready! Push with:");
println!(" git push gitlawb {branch}");
}
(Some(branch), false) => {
println!("Ready! Nothing is committed yet — commit, then push:");
println!(" git add -A && git commit -m \"initial commit\"");
println!(" git push gitlawb {branch}");
}
(None, _) => {
println!("Ready! HEAD is detached — create or switch to a branch, then push:");
println!(" git switch -c main");
println!(" git push gitlawb main");
}
}
Ok(())
}
+5 -197
View File
@@ -33,16 +33,13 @@ pub enum IpfsCmd {
cid: String,
#[arg(long, default_value = "https://node.gitlawb.com", env = "GITLAWB_NODE")]
node: String,
/// Identity directory (default: ~/.gitlawb)
#[arg(long)]
dir: Option<PathBuf>,
},
}
pub async fn run(args: IpfsArgs) -> Result<()> {
match args.cmd {
IpfsCmd::List { node, dir } => cmd_list(node, dir).await,
IpfsCmd::Get { cid, node, dir } => cmd_get(cid, node, dir).await,
IpfsCmd::Get { cid, node } => cmd_get(cid, node).await,
}
}
@@ -89,32 +86,11 @@ async fn cmd_list(node: String, dir: Option<PathBuf>) -> Result<()> {
Ok(())
}
async fn cmd_get(cid: String, node: String, dir: Option<PathBuf>) -> Result<()> {
// #173 (F5): the resolver now serves path-scoped objects to authorized readers,
// so sign with an available identity like `gl ipfs list` — otherwise an owner or
// listed reader gets the opaque anonymous 404 for content they can read.
// `get_authed` signs when a keypair is present and falls back to unsigned.
//
// An explicit `--dir` is a request to use THAT identity: propagate a
// missing/corrupt-keystore error (like `list`) instead of silently sending an
// anonymous request the authorized reader would see as the node's opaque 404
// (#173 review). Only the default (no `--dir`) keeps the best-effort unsigned
// fallback, so `get` stays usable for genuinely public content.
let keypair = match dir.as_deref() {
Some(dir) => Some(crate::identity::load_keypair_from_dir(Some(dir))?),
None => crate::identity::load_keypair_from_dir(None).ok(),
};
let client = NodeClient::new(&node, keypair);
// #173 review (F1): the node now accepts equivalent multibase spellings,
// including base64 CIDs (prefix 'm'), whose alphabet contains '/', '+', '='.
// Interpolating the CID raw would make the client request (and sign)
// `/ipfs/<prefix>/<suffix>`, which neither matches the single-segment Axum
// route nor points at the intended target. Percent-encode the CID as exactly
// one path segment so the signed and sent target agree and the server's
// `Path` extractor decodes it back to the original CID.
let path = format!("/ipfs/{}", encode_cid_segment(&cid));
async fn cmd_get(cid: String, node: String) -> Result<()> {
let client = NodeClient::new(&node, None);
let path = format!("/ipfs/{cid}");
let resp = client
.get_authed(&path)
.get(&path)
.await
.with_context(|| format!("failed to fetch CID {cid} from {node}"))?;
@@ -143,16 +119,6 @@ async fn cmd_get(cid: String, node: String, dir: Option<PathBuf>) -> Result<()>
Ok(())
}
/// Percent-encode a CID so it occupies exactly one path segment of `/ipfs/<cid>`.
/// `urlencoding::encode` escapes every byte outside the RFC 3986 unreserved set
/// (ALPHA / DIGIT / `-._~`), so the base64-CID characters that would otherwise
/// break the single-segment route — `/`, `+`, `=` — are all escaped, and the
/// server's `Path` extractor decodes the result back to the original CID (#173
/// review, F1).
fn encode_cid_segment(cid: &str) -> String {
urlencoding::encode(cid).into_owned()
}
#[cfg(test)]
mod tests {
use super::*;
@@ -269,162 +235,4 @@ mod tests {
m.assert_async().await;
}
/// #173 (F5): `gl ipfs get` must SIGN with an available identity, like
/// `gl ipfs list`, so an owner/reader can retrieve a path-scoped object the node
/// now resolves by CID. RED before the fix: cmd_get ignores the identity dir and
/// sends an unsigned request, so the signature-matching mock is never hit
/// (cmd_get errors on the unmatched 501, and m.assert fails). GREEN after: the
/// signed request carries the RFC 9421 headers and is served 200.
#[tokio::test]
async fn test_cmd_get_signs_when_identity_present() {
let mut server = mockito::Server::new_async().await;
let keystore = seed_keystore();
let m = server
.mock("GET", "/ipfs/bafkreitestcid")
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/octet-stream")
.with_header("x-git-hash", "abc123")
.with_body("object bytes")
.create_async()
.await;
cmd_get(
"bafkreitestcid".to_string(),
server.url(),
Some(keystore.path().to_path_buf()),
)
.await
.expect("signed get of a resolvable object should succeed");
m.assert_async().await;
}
/// #173 (F5) must-not: a genuine anonymous denial must surface as an error, not
/// be masked as success. With no identity dir the request is unsigned; a 404
/// from the node must produce an Err mentioning the status.
#[tokio::test]
async fn test_cmd_get_anonymous_denial_is_error() {
let mut server = mockito::Server::new_async().await;
let m = server
.mock("GET", "/ipfs/bafkreidenied")
.with_status(404)
.with_header("content-type", "text/plain")
.with_body("no git object found")
.create_async()
.await;
let err = cmd_get("bafkreidenied".to_string(), server.url(), None)
.await
.expect_err("a 404 denial must be an error, not masked success");
assert!(
err.to_string().contains("404"),
"error should mention the status, got: {err}"
);
m.assert_async().await;
}
/// #173 (INV-8) must-not: the node's new 503 "search incomplete" (the legacy CID
/// scan hit its bound and could not prove absence) must surface as an actionable
/// Err naming the status, NOT be rendered as an empty/"not found" success — a
/// retryable outcome the caller has to see. Mirrors the 404 denial case for the
/// bounded-search response the resolver now emits.
#[tokio::test]
async fn test_cmd_get_search_incomplete_503_is_error() {
let mut server = mockito::Server::new_async().await;
let m = server
.mock("GET", "/ipfs/bafkreiincomplete")
.with_status(503)
.with_header("content-type", "application/json")
.with_body(r#"{"error":"search_incomplete","message":"CID search incomplete — retry"}"#)
.create_async()
.await;
let err = cmd_get("bafkreiincomplete".to_string(), server.url(), None)
.await
.expect_err("a 503 incomplete-search must be an error, not masked as not-found");
assert!(
err.to_string().contains("503"),
"error should mention the status, got: {err}"
);
m.assert_async().await;
}
/// #173 review (F1): a base64 CID (multibase prefix 'm') can contain '/', '+',
/// and '='. The client must percent-encode it into ONE path segment before
/// building and signing `/ipfs/<cid>`; otherwise the '/' splits the target so
/// it misses the single-segment Axum route and the signature covers the wrong
/// path. Assert the encoded segment carries no raw '/', '+', or '=', and that
/// it decodes back to the original CID (the server's `Path` extractor performs
/// that same decode). RED with the old raw `format!("/ipfs/{cid}")`: the
/// segment still contains '/'.
#[test]
fn test_encode_cid_segment_escapes_base64_alphabet() {
let cid = "mFoo/Bar+baz==";
let encoded = encode_cid_segment(cid);
assert!(
!encoded.contains('/'),
"encoded CID must be a single path segment (no raw '/'), got: {encoded}"
);
assert!(
!encoded.contains('+'),
"encoded CID must escape '+', got: {encoded}"
);
assert!(
!encoded.contains('='),
"encoded CID must escape '=', got: {encoded}"
);
let decoded = urlencoding::decode(&encoded).expect("encoded CID must decode");
assert_eq!(
decoded, cid,
"encoding must round-trip back to the original CID"
);
}
/// #173 review: `gl ipfs get --dir <path>` must PROPAGATE a missing/corrupt
/// identity-load error like `gl ipfs list`, not silently fall back to an anonymous
/// request — otherwise an authorized reader pointing `--dir` at a broken keystore
/// gets the node's opaque 404 instead of the actionable key-load error. The
/// unsigned fallback is preserved only when NO `--dir` is given (covered by
/// `test_cmd_get_anonymous_denial_is_error`). RED before the fix (`.ok()` swallows
/// the error, an anonymous request is sent, and the `.expect(0)` mock is hit),
/// GREEN after.
#[tokio::test]
async fn test_cmd_get_explicit_dir_no_identity_errors_without_request() {
let mut server = mockito::Server::new_async().await;
// Empty keystore dir passed explicitly via --dir: no identity.pem present.
let empty = tempfile::TempDir::new().unwrap();
// The endpoint must never be hit when an explicit --dir fails to load.
let m = server
.mock("GET", "/ipfs/bafkreitestcid")
.expect(0)
.create_async()
.await;
let err = cmd_get(
"bafkreitestcid".to_string(),
server.url(),
Some(empty.path().to_path_buf()),
)
.await
.expect_err("an explicit --dir that fails to load must be an error");
assert!(
err.to_string().contains("gl identity new")
|| err.to_string().contains("no identity found")
|| err.to_string().contains("failed to load keypair"),
"error should name the key-load failure, got: {err}"
);
m.assert_async().await;
}
}
+207 -6
View File
@@ -932,13 +932,33 @@ async fn call_tool(
"webhook_list" => {
let repo = args["repo"].as_str().context("missing 'repo'")?;
let owner = resolve_owner(&args, &client).await?;
let resp: Value = client
.get(&format!("/api/v1/repos/{owner}/{repo}/hooks"))
.await?
.json()
// Owner-gated route: an explicit `owner` arg wins, otherwise default to
// the signing keypair's short DID (NOT the node DID). The request is
// signed by this keypair, so the path owner must match it or the node
// returns 403/404.
let owner = if let Some(o) = args.get("owner").and_then(|v| v.as_str()) {
o.to_string()
} else {
let kp = keypair
.as_ref()
.context("no identity found — run `gl identity new` first")?;
let did = kp.did().to_string();
did.split(':').next_back().unwrap_or(&did).to_string()
};
// Owner-gated route: must be signed (get_signed), not a plain get().
let resp = client
.get_signed(&format!("/api/v1/repos/{owner}/{repo}/hooks"))
.await?;
Ok(serde_json::to_string_pretty(&resp)?)
// Check the HTTP status before deserializing: a 401/403/404 JSON error
// body (missing identity, wrong owner, private/deleted repo) must fail
// the tool call, not be returned as a successful result.
let status = resp.status();
let body: Value = resp.json().await?;
if !status.is_success() {
let msg = body["message"].as_str().unwrap_or("unknown error");
anyhow::bail!("webhook_list failed ({status}): {msg}");
}
Ok(serde_json::to_string_pretty(&body)?)
}
"webhook_delete" => {
@@ -1825,6 +1845,187 @@ mod tests {
assert_eq!(parsed["body"], "looks good");
}
#[tokio::test]
async fn test_webhook_list_via_mcp_signs_the_request() {
let mut server = mockito::Server::new_async().await;
let dir = tempfile::TempDir::new().unwrap();
let kp = gitlawb_core::identity::Keypair::generate();
std::fs::write(
dir.path().join("identity.pem"),
kp.to_pem().unwrap().as_bytes(),
)
.unwrap();
// /hooks is owner-gated, so the MCP webhook_list tool must send a SIGNED
// request (get_signed). Requiring the header catches a regression to get().
// Passing `owner` in args makes resolve_owner skip the node-root lookup.
let _m = server
.mock("GET", mockito::Matcher::Regex(r"/hooks$".to_string()))
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"webhooks":[],"count":0}"#)
.create_async()
.await;
let result = call_tool(
"webhook_list",
json!({"owner": "alice", "repo": "myrepo"}),
&server.url(),
Some(dir.path()),
)
.await
.unwrap();
assert!(result.contains("webhooks"), "got: {result}");
}
#[tokio::test]
async fn test_webhook_list_default_owner_is_keypair_not_node_did() {
// When `owner` is omitted, the owner segment must come from the signing
// keypair's short DID, NOT the node root DID. The "/" mock returns a
// different DID; the /hooks mock only matches the keypair's short DID,
// so a regression to the node DID would leave the request unmatched.
let mut server = mockito::Server::new_async().await;
let dir = tempfile::TempDir::new().unwrap();
let kp = gitlawb_core::identity::Keypair::generate();
std::fs::write(
dir.path().join("identity.pem"),
kp.to_pem().unwrap().as_bytes(),
)
.unwrap();
let _root = server
.mock("GET", "/")
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"did":"did:key:z6MkNodeRootDid"}"#)
.create_async()
.await;
let did = kp.did().to_string();
let short = did.split(':').next_back().unwrap_or(&did).to_string();
let _m = server
.mock(
"GET",
mockito::Matcher::Regex(format!(r"/api/v1/repos/{short}/myrepo/hooks$")),
)
.match_header("signature", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"webhooks":[],"count":0}"#)
.expect(1)
.create_async()
.await;
let result = call_tool(
"webhook_list",
json!({"repo": "myrepo"}),
&server.url(),
Some(dir.path()),
)
.await
.unwrap();
assert!(result.contains("webhooks"), "got: {result}");
_m.assert_async().await;
}
#[tokio::test]
async fn test_webhook_list_explicit_owner_overrides_keypair() {
// An explicit `owner` arg must win over the keypair default and the node
// DID. The path must use the supplied owner; "/" must not even be hit.
let mut server = mockito::Server::new_async().await;
let dir = tempfile::TempDir::new().unwrap();
let kp = gitlawb_core::identity::Keypair::generate();
std::fs::write(
dir.path().join("identity.pem"),
kp.to_pem().unwrap().as_bytes(),
)
.unwrap();
let _m = server
.mock(
"GET",
mockito::Matcher::Regex(r"/api/v1/repos/someoneelse/myrepo/hooks$".to_string()),
)
.match_header("signature", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"webhooks":[],"count":0}"#)
.expect(1)
.create_async()
.await;
let result = call_tool(
"webhook_list",
json!({"repo": "myrepo", "owner": "someoneelse"}),
&server.url(),
Some(dir.path()),
)
.await
.unwrap();
assert!(result.contains("webhooks"), "got: {result}");
_m.assert_async().await;
}
#[tokio::test]
async fn test_webhook_list_non_success_status_errors() {
// A 403 (or any non-2xx) JSON error body for a missing identity, wrong
// owner, or private/deleted repo must fail the tool call, NOT be returned
// as a successful result. Mirrors the `gl webhook list` status check.
let mut server = mockito::Server::new_async().await;
let dir = tempfile::TempDir::new().unwrap();
let kp = gitlawb_core::identity::Keypair::generate();
std::fs::write(
dir.path().join("identity.pem"),
kp.to_pem().unwrap().as_bytes(),
)
.unwrap();
let _m = server
.mock("GET", mockito::Matcher::Regex(r"/hooks$".to_string()))
.match_header("signature", mockito::Matcher::Any)
.with_status(403)
.with_header("content-type", "application/json")
.with_body(r#"{"message":"only the repo owner can perform this action"}"#)
.create_async()
.await;
let err = call_tool(
"webhook_list",
json!({"owner": "alice", "repo": "myrepo"}),
&server.url(),
Some(dir.path()),
)
.await
.expect_err("non-2xx must error, not return the error body as success");
let msg = err.to_string();
assert!(msg.contains("webhook_list failed (403"), "got: {msg}");
assert!(
msg.contains("only the repo owner"),
"must surface the node message, got: {msg}"
);
}
#[tokio::test]
async fn test_webhook_list_without_keypair_errors() {
// No identity loaded and no explicit owner → the tool must error rather
// than fall back to the node DID and issue an unsigned, mis-scoped GET.
let server = mockito::Server::new_async().await;
let dir = tempfile::TempDir::new().unwrap(); // empty: no identity.pem
let err = call_tool(
"webhook_list",
json!({"repo": "myrepo"}),
&server.url(),
Some(dir.path()),
)
.await
.expect_err("must error without an identity");
assert!(err.to_string().contains("no identity found"), "got: {err}");
}
#[test]
fn test_tool_count_is_42() {
let tools = tool_definitions();
+9 -2
View File
@@ -154,10 +154,12 @@ async fn cmd_remove(
async fn cmd_list(repo: String, node: String, dir: Option<PathBuf>) -> Result<()> {
let (owner, name) = resolve_owner_repo(&repo, &node, dir.as_deref()).await?;
let client = NodeClient::new(&node, None);
// Read-visibility-gated on the node: public repos list anonymously, private
// repos need the owner's signature. Sign when an identity is available.
let client = NodeClient::new(&node, load_keypair_from_dir(dir.as_deref()).ok());
let resp = client
.get(&format!("/api/v1/repos/{owner}/{name}/branches/protected"))
.get_maybe_signed(&format!("/api/v1/repos/{owner}/{name}/branches/protected"))
.await
.context("failed to connect to node")?;
@@ -307,6 +309,9 @@ mod tests {
"GET",
mockito::Matcher::Regex(r"branches/protected".to_string()),
)
// An identity is supplied, so get_maybe_signed must sign the request.
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"protected_branches":[],"count":0}"#)
@@ -338,6 +343,8 @@ mod tests {
"GET",
mockito::Matcher::Regex(r"branches/protected".to_string()),
)
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"protected_branches":["main","release"],"count":2}"#)
+431 -34
View File
@@ -146,6 +146,8 @@ pub enum RepoCmd {
repo: String,
#[arg(long, default_value = "https://node.gitlawb.com", env = "GITLAWB_NODE")]
node: String,
#[arg(long)]
dir: Option<PathBuf>,
},
}
@@ -203,7 +205,7 @@ pub async fn run(args: RepoArgs) -> Result<()> {
RepoCmd::ReplicaUnregister { repo, node, dir } => {
cmd_replica_unregister(repo, node, dir).await
}
RepoCmd::Replicas { repo, node } => cmd_replicas(repo, node).await,
RepoCmd::Replicas { repo, node, dir } => cmd_replicas(repo, node, dir).await,
}
}
@@ -329,7 +331,10 @@ async fn cmd_clone(name: String, node: String, dir: Option<PathBuf>) -> Result<(
}
async fn cmd_info(repo: String, node: String, dir: Option<PathBuf>) -> Result<()> {
let client = NodeClient::new(&node, None);
// Sign when an identity is available so the read-visibility-gated replica
// sub-fetch below resolves for a private repo's owner (public repos still
// work anonymously).
let client = NodeClient::new(&node, load_keypair_from_dir(dir.as_deref()).ok());
let (owner, name) = if repo.contains('/') {
let (o, n) = repo.split_once('/').unwrap();
@@ -339,7 +344,10 @@ async fn cmd_info(repo: String, node: String, dir: Option<PathBuf>) -> Result<()
(short, repo)
};
let resp = client.get(&format!("/api/v1/repos/{owner}/{name}")).await?;
let resp = client
.get_maybe_signed(&format!("/api/v1/repos/{owner}/{name}"))
.await
.context("failed to connect to node")?;
// A non-existent (or unreadable/quarantined) repo is a real 404 from the
// node — surface it plainly instead of printing a stub card with `?` fields
// and a placeholder owner DID.
@@ -379,7 +387,7 @@ async fn cmd_info(repo: String, node: String, dir: Option<PathBuf>) -> Result<()
// Replica count — failure to fetch is non-fatal (older nodes don't expose this).
if let Ok(resp) = client
.get(&format!("/api/v1/repos/{owner}/{name}/replicas"))
.get_maybe_signed(&format!("/api/v1/repos/{owner}/{name}/replicas"))
.await
{
if resp.status().is_success() {
@@ -459,15 +467,17 @@ async fn cmd_replica_unregister(repo: String, node: String, dir: Option<PathBuf>
Ok(())
}
async fn cmd_replicas(repo: String, node: String) -> Result<()> {
async fn cmd_replicas(repo: String, node: String, dir: Option<PathBuf>) -> Result<()> {
let (owner, name) = repo
.split_once('/')
.map(|(o, n)| (o.to_string(), n.to_string()))
.context("use owner/repo format")?;
let client = NodeClient::new(&node, None);
// Read-visibility-gated: public repos list anonymously, private repos need
// the owner's signature. Sign when an identity is available.
let client = NodeClient::new(&node, load_keypair_from_dir(dir.as_deref()).ok());
let resp = client
.get(&format!("/api/v1/repos/{owner}/{name}/replicas"))
.get_maybe_signed(&format!("/api/v1/repos/{owner}/{name}/replicas"))
.await
.context("failed to connect to node")?;
let status = resp.status();
@@ -585,7 +595,7 @@ async fn cmd_fork(
}
/// Returns (owner, repo_name) — if repo contains '/', splits on it; otherwise uses caller's DID.
async fn resolve_owner_repo_pair(
pub(crate) async fn resolve_owner_repo_pair(
repo: &str,
node: &str,
dir: Option<&std::path::Path>,
@@ -658,16 +668,23 @@ async fn cmd_label_remove(
async fn cmd_label_list(repo: String, node: String, dir: Option<PathBuf>) -> Result<()> {
let (owner, name) = resolve_owner_repo_pair(&repo, &node, dir.as_deref()).await?;
// Read-visibility-gated like the sibling read surfaces (replicas, protected
// branches): sign when an identity is available so a private-repo owner can
// read their own labels, while public repos stay anonymously listable.
let client = NodeClient::new(&node, load_keypair_from_dir(dir.as_deref()).ok());
let resp: Value = client
.get_authed(&format!("/api/v1/repos/{owner}/{name}/labels"))
.await?
.json()
let resp = client
.get_maybe_signed(&format!("/api/v1/repos/{owner}/{name}/labels"))
.await
.context("invalid JSON")?;
.context("failed to connect to node")?;
let status = resp.status();
let body: Value = resp.json().await.unwrap_or_default();
if !status.is_success() {
let msg = body["message"].as_str().unwrap_or("unknown error");
anyhow::bail!("list labels failed ({status}): {msg}");
}
let labels = resp["labels"].as_array().cloned().unwrap_or_default();
let labels = body["labels"].as_array().cloned().unwrap_or_default();
if labels.is_empty() {
println!("No labels on {owner}/{name}");
} else {
@@ -679,17 +696,39 @@ async fn cmd_label_list(repo: String, node: String, dir: Option<PathBuf>) -> Res
Ok(())
}
/// Parse the protected-branch list from the node's `GET /branches/protected`
/// response. The endpoint returns `{"protected_branches": [...], "count": N}`
/// (see `crates/gitlawb-node/src/api/protect.rs`); each entry may be a plain
/// branch-name string or an object carrying a `name` field.
fn parse_protected_branches(val: &Value) -> Vec<String> {
val["protected_branches"]
.as_array()
.map(|arr| {
arr.iter()
.filter_map(|b| {
b.as_str()
.map(String::from)
.or_else(|| b["name"].as_str().map(String::from))
})
.collect()
})
.unwrap_or_default()
}
async fn cmd_owner(repo: String, node: String, dir: Option<PathBuf>, json_out: bool) -> Result<()> {
let keypair = load_keypair_from_dir(dir.as_deref())?;
let my_did = keypair.did().to_string();
let my_short = my_did.split(':').next_back().unwrap_or(&my_did).to_string();
let (owner, name) = resolve_owner_repo_pair(&repo, &node, dir.as_deref()).await?;
let client = NodeClient::new(&node, None);
// Sign with the loaded identity so the read-visibility-gated protected-branch
// fetch below works on the owner's own private repos.
let client = NodeClient::new(&node, Some(keypair));
// Fetch repo info
// Fetch repo info (get_repo is read-visibility-gated; sign so the owner can
// inspect their own private repo).
let resp = client
.get(&format!("/api/v1/repos/{owner}/{name}"))
.get_maybe_signed(&format!("/api/v1/repos/{owner}/{name}"))
.await
.context("failed to connect to node")?;
if !resp.status().is_success() {
@@ -704,25 +743,14 @@ async fn cmd_owner(repo: String, node: String, dir: Option<PathBuf>, json_out: b
.to_string();
let is_owner = my_did == owner_did || my_short == owner_short;
// Fetch protected branches
// Fetch protected branches (read-visibility-gated; signed via the client).
let protected: Vec<String> = match client
.get(&format!("/api/v1/repos/{owner}/{name}/branches/protected"))
.get_maybe_signed(&format!("/api/v1/repos/{owner}/{name}/branches/protected"))
.await
{
Ok(resp) if resp.status().is_success() => {
let val: Value = resp.json().await.unwrap_or_default();
val.as_array()
.or_else(|| val["branches"].as_array())
.map(|arr| {
arr.iter()
.filter_map(|b| {
b.as_str()
.map(String::from)
.or_else(|| b["name"].as_str().map(String::from))
})
.collect()
})
.unwrap_or_default()
parse_protected_branches(&val)
}
_ => Vec::new(),
};
@@ -760,6 +788,20 @@ mod tests {
use super::*;
use tempfile::TempDir;
#[test]
fn parse_protected_branches_reads_node_shape() {
use serde_json::json;
assert_eq!(
parse_protected_branches(&json!({"protected_branches":["main","release"],"count":2})),
vec!["main".to_string(), "release".to_string()]
);
assert!(parse_protected_branches(&json!({"count":0})).is_empty());
assert_eq!(
parse_protected_branches(&json!({"protected_branches":[{"name":"main"}]})),
vec!["main".to_string()]
);
}
fn write_identity(dir: &TempDir) {
let kp = gitlawb_core::identity::Keypair::generate();
let pem = kp.to_pem().unwrap();
@@ -1097,6 +1139,10 @@ mod tests {
"GET",
mockito::Matcher::Regex(r"^/api/v1/repos/[^/]+/myrepo/labels$".to_string()),
)
// Identity is loaded, so get_maybe_signed must sign — requiring the
// header guards against a regression back to bare get().
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"labels":[]}"#)
@@ -1123,6 +1169,10 @@ mod tests {
"GET",
mockito::Matcher::Regex(r"^/api/v1/repos/[^/]+/myrepo/labels$".to_string()),
)
// Identity is loaded, so get_maybe_signed must sign — requiring the
// header guards against a regression back to bare get().
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"labels":["language:rust","topic:defi"]}"#)
@@ -1138,6 +1188,97 @@ mod tests {
.unwrap();
}
#[tokio::test]
async fn test_cmd_label_list_anonymous_when_no_identity() {
// Empty dir → no identity → get_maybe_signed must fall back to an
// anonymous GET (the public-repo path). Assert NO signature header.
let dir = TempDir::new().unwrap();
let mut server = mockito::Server::new_async().await;
let _m = server
.mock("GET", mockito::Matcher::Regex(r"/labels$".to_string()))
.match_header("signature", mockito::Matcher::Missing)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"labels":[]}"#)
.create_async()
.await;
cmd_label_list(
"owner/pubrepo".to_string(),
server.url(),
Some(dir.path().to_path_buf()),
)
.await
.unwrap();
}
#[tokio::test]
async fn test_cmd_label_list_404_errors() {
// A non-reader of a private repo gets 404 from the now-gated endpoint.
// The client must surface that as an error, not print "No labels" and
// exit 0 (the error body has no `labels` key).
let dir = TempDir::new().unwrap();
write_identity(&dir);
let mut server = mockito::Server::new_async().await;
let _m = server
.mock(
"GET",
mockito::Matcher::Regex(r"^/api/v1/repos/[^/]+/myrepo/labels$".to_string()),
)
.with_status(404)
.with_header("content-type", "application/json")
.with_body(r#"{"message":"repo not found"}"#)
.create_async()
.await;
let err = cmd_label_list(
"myrepo".to_string(),
server.url(),
Some(dir.path().to_path_buf()),
)
.await
.unwrap_err();
assert!(
err.to_string().contains("list labels failed (404")
&& err.to_string().contains("repo not found"),
"got: {err}"
);
}
#[tokio::test]
async fn test_cmd_label_list_non_json_error_surfaces_status() {
// A non-2xx response with a non-JSON body (e.g. a proxy 502) must still
// surface the HTTP status, not collapse to a bare JSON-parse error.
let dir = TempDir::new().unwrap();
write_identity(&dir);
let mut server = mockito::Server::new_async().await;
let _m = server
.mock(
"GET",
mockito::Matcher::Regex(r"^/api/v1/repos/[^/]+/myrepo/labels$".to_string()),
)
.with_status(502)
.with_header("content-type", "text/html")
.with_body("<html>502 Bad Gateway</html>")
.create_async()
.await;
let err = cmd_label_list(
"myrepo".to_string(),
server.url(),
Some(dir.path().to_path_buf()),
)
.await
.unwrap_err();
assert!(
err.to_string().contains("list labels failed (502"),
"got: {err}"
);
}
#[tokio::test]
async fn test_cmd_owner_is_owner() {
let dir = TempDir::new().unwrap();
@@ -1148,11 +1289,15 @@ mod tests {
let short = did.split(':').next_back().unwrap().to_string();
let mut server = mockito::Server::new_async().await;
// Both fetches are read-visibility-gated; with an identity loaded they
// must be signed. Requiring the header guards a regression to get().
let _repo = server
.mock(
"GET",
mockito::Matcher::Regex(format!(r"^/api/v1/repos/{short}/myrepo$")),
)
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(format!(
@@ -1167,9 +1312,11 @@ mod tests {
r"^/api/v1/repos/{short}/myrepo/branches/protected$"
)),
)
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"["main"]"#)
.with_body(r#"{"protected_branches":["main"],"count":1}"#)
.create_async()
.await;
@@ -1210,7 +1357,7 @@ mod tests {
)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"["main","release"]"#)
.with_body(r#"{"protected_branches":["main","release"],"count":2}"#)
.create_async()
.await;
@@ -1251,7 +1398,7 @@ mod tests {
)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"["main"]"#)
.with_body(r#"{"protected_branches":["main"],"count":1}"#)
.create_async()
.await;
@@ -1292,4 +1439,254 @@ mod tests {
.unwrap_err();
assert!(err.to_string().contains("not found"), "got: {err}");
}
#[tokio::test]
async fn test_cmd_replicas_signs_when_identity_present() {
let dir = TempDir::new().unwrap();
write_identity(&dir);
let mut server = mockito::Server::new_async().await;
// Read-visibility-gated: with an identity the request must be signed.
let _m = server
.mock("GET", mockito::Matcher::Regex(r"/replicas$".to_string()))
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"repo":"owner/myrepo","replica_count":0,"replicas":[]}"#)
.create_async()
.await;
cmd_replicas(
"owner/myrepo".to_string(),
server.url(),
Some(dir.path().to_path_buf()),
)
.await
.unwrap();
}
#[tokio::test]
async fn test_cmd_replicas_anonymous_when_no_identity() {
// Empty dir → no identity → get_maybe_signed must fall back to an
// anonymous GET (the public-repo path). Assert NO signature header.
let dir = TempDir::new().unwrap();
let mut server = mockito::Server::new_async().await;
let _m = server
.mock("GET", mockito::Matcher::Regex(r"/replicas$".to_string()))
.match_header("signature", mockito::Matcher::Missing)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"repo":"owner/pubrepo","replica_count":0,"replicas":[]}"#)
.create_async()
.await;
cmd_replicas(
"owner/pubrepo".to_string(),
server.url(),
Some(dir.path().to_path_buf()),
)
.await
.unwrap();
}
#[tokio::test]
async fn test_cmd_info_signs_repo_and_replica_fetches() {
let dir = TempDir::new().unwrap();
write_identity(&dir);
let mut server = mockito::Server::new_async().await;
// Both the primary repo fetch and the replica sub-fetch are
// read-visibility-gated; with an identity loaded both must be signed.
let _repo = server
.mock(
"GET",
mockito::Matcher::Regex(r"^/api/v1/repos/owner/myrepo$".to_string()),
)
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(
r#"{"id":"r1","name":"myrepo","owner_did":"did:key:zOwner","is_public":false,"default_branch":"main"}"#,
)
.create_async()
.await;
let _replicas = server
.mock("GET", mockito::Matcher::Regex(r"/replicas$".to_string()))
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"replica_count":0,"replicas":[]}"#)
.create_async()
.await;
cmd_info(
"owner/myrepo".to_string(),
server.url(),
Some(dir.path().to_path_buf()),
)
.await
.unwrap();
}
#[tokio::test]
async fn test_cmd_info_404_surfaces_error_not_fake_repo() {
// A non-reader of a private repo gets a JSON 404 from the now-gated
// GET /api/v1/repos/{owner}/{name}. The command must surface that as an
// error, not parse the error body and print a fabricated repo summary
// (owner / "?" / public=true / main) before exiting 0.
let dir = TempDir::new().unwrap();
write_identity(&dir);
let mut server = mockito::Server::new_async().await;
let _repo = server
.mock(
"GET",
mockito::Matcher::Regex(r"^/api/v1/repos/owner/myrepo$".to_string()),
)
.with_status(404)
.with_header("content-type", "application/json")
.with_body(r#"{"message":"repo not found"}"#)
.create_async()
.await;
let err = cmd_info(
"owner/myrepo".to_string(),
server.url(),
Some(dir.path().to_path_buf()),
)
.await
.unwrap_err();
// The merged cmd_info maps a 404 to a generic "not found" (it does not
// echo the server's message body on 404), which still surfaces the error
// rather than fabricating a repo card, and avoids leaking any
// server-provided detail on a private-repo denial.
assert!(
err.to_string()
.contains("repository 'owner/myrepo' not found"),
"got: {err}"
);
}
#[tokio::test]
async fn test_cmd_info_non_json_error_surfaces_status() {
// A non-2xx response with a non-JSON body (e.g. a proxy 502) must still
// surface the HTTP status, not collapse to a bare JSON-parse error.
let dir = TempDir::new().unwrap();
write_identity(&dir);
let mut server = mockito::Server::new_async().await;
let _repo = server
.mock(
"GET",
mockito::Matcher::Regex(r"^/api/v1/repos/owner/myrepo$".to_string()),
)
.with_status(502)
.with_header("content-type", "text/html")
.with_body("<html>502 Bad Gateway</html>")
.create_async()
.await;
let err = cmd_info(
"owner/myrepo".to_string(),
server.url(),
Some(dir.path().to_path_buf()),
)
.await
.unwrap_err();
assert!(
err.to_string().contains("repo info failed (502"),
"got: {err}"
);
}
}
// Drives the three deny/negative arms that the merge reconciliation changed but
// that no existing test executed (they were only reasoned): resolve_owner_did
// with no identity, cmd_info on an authenticated-but-denied 403, and
// get_maybe_signed with no keypair. Each asserts the must-not case.
#[cfg(test)]
mod vet_merge_deny_arms {
use super::*;
use tempfile::TempDir;
fn write_identity(dir: &TempDir) {
let kp = gitlawb_core::identity::Keypair::generate();
let pem = kp.to_pem().unwrap();
std::fs::write(dir.path().join("identity.pem"), pem.as_bytes()).unwrap();
}
// No local identity must hard-error, never fall back to the node's own DID.
// (Load-bearing: #113's pre-merge resolve_owner_did returned Ok via a node
// fetch here; a reintroduced fallback makes this go red.)
#[tokio::test]
async fn resolve_owner_did_no_identity_errors_not_fallback() {
let empty = TempDir::new().unwrap(); // no identity.pem written
let err = resolve_owner_did("http://unused.invalid", Some(empty.path()))
.await
.unwrap_err();
assert!(
err.to_string()
.contains("no local identity to resolve the repo owner"),
"got: {err}"
);
}
// An authenticated caller DENIED with a 403 must surface an error, never
// render a repo card (deny-as-success is the INV-8 client-contract break).
#[tokio::test]
async fn cmd_info_403_denied_surfaces_error_not_card() {
let dir = TempDir::new().unwrap();
write_identity(&dir);
let mut server = mockito::Server::new_async().await;
let _m = server
.mock(
"GET",
mockito::Matcher::Regex(r"^/api/v1/repos/owner/myrepo$".to_string()),
)
.with_status(403)
.with_header("content-type", "application/json")
.with_body(r#"{"message":"forbidden"}"#)
.create_async()
.await;
let err = cmd_info(
"owner/myrepo".to_string(),
server.url(),
Some(dir.path().to_path_buf()),
)
.await
.unwrap_err();
assert!(
err.to_string().contains("repo info failed (403"),
"got: {err}"
);
}
// No keypair must send an unsigned GET; the mock only matches when the
// Signature header is absent, so a wrongly-signed request fails to match.
#[tokio::test]
async fn get_maybe_signed_no_keypair_sends_unsigned() {
let mut server = mockito::Server::new_async().await;
let _m = server
.mock("GET", "/api/v1/repos/owner/pub")
.match_header("signature", mockito::Matcher::Missing)
.with_status(200)
.with_header("content-type", "application/json")
.with_body("{}")
.create_async()
.await;
let client = NodeClient::new(server.url(), None);
let resp = client
.get_maybe_signed("/api/v1/repos/owner/pub")
.await
.unwrap();
assert!(
resp.status().is_success(),
"unsigned GET should match the Missing-signature mock; got {}",
resp.status()
);
}
}
+52 -13
View File
@@ -59,7 +59,7 @@ pub async fn run(args: StatusArgs) -> Result<()> {
println!(" repo {short_did}/{repo}");
}
None => {
println!(" repo (not in a gitlawb repo — no gitlawb:// origin)");
println!(" repo (not in a gitlawb repo — no gitlawb:// remote)");
}
}
@@ -143,29 +143,68 @@ fn trust_bar(score: f64) -> String {
format!("{}{}", "█".repeat(filled), "░".repeat(empty))
}
/// Parse `gitlawb://<did>/<repo>` from the git origin remote.
/// Find a `gitlawb://<did>/<repo>` remote in the current repo.
///
/// Checks `origin` first (clones), then `gitlawb` (the name `gl init` adds),
/// then any other remote with a gitlawb:// URL.
fn detect_gitlawb_remote() -> Option<(String, String)> {
let out = std::process::Command::new("git")
.args(["remote", "get-url", "origin"])
.args(["remote"])
.stderr(std::process::Stdio::null())
.output()
.ok()?;
if !out.status.success() {
return None;
}
let url = String::from_utf8(out.stdout).ok()?;
let rest = url.trim().strip_prefix("gitlawb://")?;
let slash = rest.rfind('/')?;
let did = rest[..slash].to_string();
let repo = rest[slash + 1..].to_string();
if did.is_empty() || repo.is_empty() {
return None;
let names: Vec<String> = String::from_utf8(out.stdout)
.ok()?
.lines()
.map(str::trim)
.filter(|l| !l.is_empty())
.map(str::to_string)
.collect();
let preferred = ["origin", "gitlawb"];
let ordered = preferred
.iter()
.filter(|p| names.iter().any(|n| n == *p))
.map(|p| p.to_string())
.chain(
names
.iter()
.filter(|n| !preferred.contains(&n.as_str()))
.cloned(),
);
for name in ordered {
// A remote can carry several URLs, and the gitlawb:// one may be a
// push URL behind an https fetch URL — check every fetch and push URL.
for extra in [&["--all"][..], &["--push", "--all"][..]] {
let mut args = vec!["remote", "get-url"];
args.extend_from_slice(extra);
args.push(&name);
let Ok(out) = std::process::Command::new("git")
.args(&args)
.stderr(std::process::Stdio::null())
.output()
else {
continue;
};
if !out.status.success() {
continue;
}
let Ok(urls) = String::from_utf8(out.stdout) else {
continue;
};
if let Some(parsed) = urls.lines().find_map(parse_gitlawb_url) {
return Some(parsed);
}
}
}
Some((did, repo))
None
}
/// Parse a gitlawb:// URL string into (did, repo) — extracted for testing.
#[cfg(test)]
/// Parse a gitlawb:// URL string into (did, repo).
fn parse_gitlawb_url(url: &str) -> Option<(String, String)> {
let rest = url.trim().strip_prefix("gitlawb://")?;
let slash = rest.rfind('/')?;
+11 -13
View File
@@ -132,23 +132,11 @@ async fn read_body_capped(mut resp: reqwest::Response, cap: usize) -> String {
/// `char::is_control` does not cover — they can reorder the displayed line).
fn sanitize_node_msg(s: &str) -> String {
s.chars()
.filter(|c| !c.is_control() && !is_bidi_format(*c))
.filter(|c| !c.is_control() && !gitlawb_core::sanitize::is_bidi_format(*c))
.take(200)
.collect()
}
/// Unicode bidirectional and directional-isolate format characters (category
/// `Cf`). These are not `char::is_control()` (that is category `Cc` only), but a
/// right-to-left override or isolate can visually reorder a terminal line to
/// spoof the error text, so they are stripped alongside the control bytes.
fn is_bidi_format(c: char) -> bool {
matches!(c,
'\u{200E}' | '\u{200F}' | '\u{061C}' // LRM, RLM, ALM
| '\u{202A}'..='\u{202E}' // LRE, RLE, PDF, LRO, RLO
| '\u{2066}'..='\u{2069}' // LRI, RLI, FSI, PDI
)
}
#[cfg(test)]
mod tests {
use super::*;
@@ -293,6 +281,16 @@ mod tests {
assert_eq!(sanitize_node_msg(&long).chars().count(), 200);
}
#[test]
fn sanitize_preserves_legitimate_and_rtl_text() {
// Must not over-strip: a plain word, a genuine RTL SCRIPT letter (Arabic
// U+0627, category Lo — NOT a format char), and ZWJ (U+200D, a legitimate
// Cf char, e.g. emoji sequences) all survive. Guards the shared predicate
// against being widened into a blanket Cf stripper.
let out = sanitize_node_msg("ok \u{0627}\u{200D}b");
assert_eq!(out, "ok \u{0627}\u{200D}b");
}
#[tokio::test]
async fn trigger_handles_oversized_error_body_without_unbounded_output() {
// A hostile/broken node returns a 2 MB error body. The command must still
+169 -26
View File
@@ -42,6 +42,8 @@ pub enum WebhookCmd {
repo: String,
#[arg(long, default_value = "https://node.gitlawb.com", env = "GITLAWB_NODE")]
node: String,
#[arg(long)]
dir: Option<PathBuf>,
},
/// Delete a webhook
Delete {
@@ -66,7 +68,7 @@ pub async fn run(args: WebhookArgs) -> Result<()> {
node,
dir,
} => cmd_create(repo, url, events, secret, node, dir).await,
WebhookCmd::List { repo, node } => cmd_list(repo, node).await,
WebhookCmd::List { repo, node, dir } => cmd_list(repo, node, dir).await,
WebhookCmd::Delete {
repo,
id,
@@ -76,15 +78,6 @@ pub async fn run(args: WebhookArgs) -> Result<()> {
}
}
async fn resolve_owner(client: &NodeClient) -> Result<String> {
let info: Value = client.get("/").await?.json().await?;
let did = info["did"]
.as_str()
.context("node missing DID")?
.to_string();
Ok(did.split(':').next_back().unwrap_or(&did).to_string())
}
async fn cmd_create(
repo: String,
url: String,
@@ -94,8 +87,8 @@ async fn cmd_create(
dir: Option<PathBuf>,
) -> Result<()> {
let keypair = load_keypair_from_dir(dir.as_deref())?;
let (owner, name) = crate::repo::resolve_owner_repo_pair(&repo, &node, dir.as_deref()).await?;
let client = NodeClient::new(&node, Some(keypair));
let owner = resolve_owner(&client).await?;
let event_list: Vec<&str> = events.split(',').map(str::trim).collect();
@@ -106,7 +99,7 @@ async fn cmd_create(
}))?;
let resp = client
.post(&format!("/api/v1/repos/{owner}/{repo}/hooks"), &payload)
.post(&format!("/api/v1/repos/{owner}/{name}/hooks"), &payload)
.await
.context("failed to connect to node")?;
let status = resp.status();
@@ -143,18 +136,31 @@ async fn cmd_create(
Ok(())
}
async fn cmd_list(repo: String, node: String) -> Result<()> {
let client = NodeClient::new(&node, None);
let owner = resolve_owner(&client).await?;
async fn cmd_list(repo: String, node: String, dir: Option<PathBuf>) -> Result<()> {
// The node owner-gates GET /hooks (callback URLs are owner-secret), so the
// list request must be signed — anonymous callers get 401.
let keypair = load_keypair_from_dir(dir.as_deref())?;
let (owner, name) = crate::repo::resolve_owner_repo_pair(&repo, &node, dir.as_deref()).await?;
let client = NodeClient::new(&node, Some(keypair));
let resp: Value = client
.get(&format!("/api/v1/repos/{owner}/{repo}/hooks"))
.await?
.json()
.await
.context("invalid JSON")?;
// get_signed (not get) attaches the RFC 9421 signature — plain get() never
// signs, and the node owner-gates this route, so an unsigned GET 401s.
let resp = client
.get_signed(&format!("/api/v1/repos/{owner}/{name}/hooks"))
.await?;
let status = resp.status();
let body: Value = resp.json().await.context("invalid JSON")?;
let hooks = resp["webhooks"].as_array().cloned().unwrap_or_default();
if !status.is_success() {
let msg = body["message"].as_str().unwrap_or("unknown error");
anyhow::bail!("list webhooks failed ({status}): {msg}");
}
let hooks = body
.get("webhooks")
.and_then(Value::as_array)
.cloned()
.context("node response missing webhooks array")?;
if hooks.is_empty() {
println!("No webhooks for {repo}");
return Ok(());
@@ -185,13 +191,13 @@ async fn cmd_list(repo: String, node: String) -> Result<()> {
async fn cmd_delete(repo: String, id: String, node: String, dir: Option<PathBuf>) -> Result<()> {
let keypair = load_keypair_from_dir(dir.as_deref())?;
let (owner, name) = crate::repo::resolve_owner_repo_pair(&repo, &node, dir.as_deref()).await?;
let client = NodeClient::new(&node, Some(keypair));
let owner = resolve_owner(&client).await?;
let payload = serde_json::to_vec(&serde_json::json!({}))?;
let resp = client
.delete(
&format!("/api/v1/repos/{owner}/{repo}/hooks/{id}"),
&format!("/api/v1/repos/{owner}/{name}/hooks/{id}"),
&payload,
)
.await
@@ -337,33 +343,170 @@ mod tests {
#[tokio::test]
async fn test_list_webhooks_empty() {
let mut server = mockito::Server::new_async().await;
let (dir, _kp) = tmp_identity();
let _root = mock_root(&mut server).await;
let _m = server
.mock("GET", mockito::Matcher::Regex(r"/hooks$".to_string()))
// The route is owner-gated, so the list request must be signed.
// Requiring the header here is what catches a regression to plain get().
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"webhooks":[]}"#)
.create_async()
.await;
cmd_list("my-repo".to_string(), server.url()).await.unwrap();
cmd_list(
"my-repo".to_string(),
server.url(),
Some(dir.path().to_path_buf()),
)
.await
.unwrap();
}
#[tokio::test]
async fn test_list_webhooks_with_items() {
let mut server = mockito::Server::new_async().await;
let (dir, _kp) = tmp_identity();
let _root = mock_root(&mut server).await;
let _m = server
.mock("GET", mockito::Matcher::Regex(r"/hooks$".to_string()))
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"webhooks":[{"id":"h1","url":"https://a.com","events":["push"],"active":true},{"id":"h2","url":"https://b.com","events":["*"],"active":false}]}"#)
.create_async()
.await;
cmd_list("my-repo".to_string(), server.url()).await.unwrap();
cmd_list(
"my-repo".to_string(),
server.url(),
Some(dir.path().to_path_buf()),
)
.await
.unwrap();
}
#[tokio::test]
async fn test_list_owner_comes_from_keypair_not_node_did() {
// The owner segment must be derived from the caller's keypair (loaded
// from --dir), NOT the node root DID. mock_root returns a different DID;
// if the code regressed to using it, this path mock would not match.
let mut server = mockito::Server::new_async().await;
let (dir, kp) = tmp_identity();
let _root = mock_root(&mut server).await;
let did = kp.did().to_string();
let short = did.split(':').next_back().unwrap_or(&did).to_string();
let _m = server
.mock(
"GET",
mockito::Matcher::Regex(format!(r"/api/v1/repos/{short}/my-repo/hooks$")),
)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"webhooks":[]}"#)
.expect(1)
.create_async()
.await;
cmd_list(
"my-repo".to_string(),
server.url(),
Some(dir.path().to_path_buf()),
)
.await
.unwrap();
_m.assert_async().await;
}
#[tokio::test]
async fn test_list_honors_owner_repo_arg() {
// An explicit `owner/repo` argument is split and used verbatim.
let mut server = mockito::Server::new_async().await;
let (dir, _kp) = tmp_identity();
let _root = mock_root(&mut server).await;
let _m = server
.mock(
"GET",
mockito::Matcher::Regex(r"/api/v1/repos/someoneelse/their-repo/hooks$".to_string()),
)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"webhooks":[]}"#)
.expect(1)
.create_async()
.await;
cmd_list(
"someoneelse/their-repo".to_string(),
server.url(),
Some(dir.path().to_path_buf()),
)
.await
.unwrap();
_m.assert_async().await;
}
#[tokio::test]
async fn test_list_webhooks_non_success_status_errors() {
// A 403 (or any non-2xx) JSON body has no `webhooks` field; cmd_list must
// surface the failure, not print "No webhooks" and exit 0.
let mut server = mockito::Server::new_async().await;
let (dir, _kp) = tmp_identity();
let _root = mock_root(&mut server).await;
let _m = server
.mock("GET", mockito::Matcher::Regex(r"/hooks$".to_string()))
.with_status(403)
.with_header("content-type", "application/json")
.with_body(r#"{"message":"not the owner"}"#)
.create_async()
.await;
let err = cmd_list(
"my-repo".to_string(),
server.url(),
Some(dir.path().to_path_buf()),
)
.await
.unwrap_err();
assert!(err.to_string().contains("not the owner"));
}
#[tokio::test]
async fn test_list_webhooks_2xx_missing_array_errors() {
// A 2xx body without a `webhooks` array signals a node/API contract
// regression. cmd_list must error rather than print "No webhooks".
let mut server = mockito::Server::new_async().await;
let (dir, _kp) = tmp_identity();
let _root = mock_root(&mut server).await;
let _m = server
.mock("GET", mockito::Matcher::Regex(r"/hooks$".to_string()))
.match_header("signature", mockito::Matcher::Any)
.match_header("signature-input", mockito::Matcher::Any)
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"ok":true}"#)
.create_async()
.await;
let err = cmd_list(
"my-repo".to_string(),
server.url(),
Some(dir.path().to_path_buf()),
)
.await
.unwrap_err();
assert!(err.to_string().contains("missing webhooks array"));
}
// ── delete ───────────────────────────────────────────────────────
+3
View File
@@ -16,3 +16,6 @@ anyhow = { workspace = true }
thiserror = { workspace = true }
tracing = { workspace = true }
sha2 = { workspace = true }
[dev-dependencies]
mockito = "1"
+46
View File
@@ -64,6 +64,12 @@ fn sanitize_excerpt(s: &str) -> String {
}
continue;
}
if gitlawb_core::sanitize::is_bidi_format(ch) {
// Drop Cf bidi/format controls (INV-6, #183) without setting
// pending_space: they are not whitespace, so stripping leaves no
// phantom gap.
continue;
}
if ch.is_whitespace() {
pending_space = true;
continue;
@@ -421,6 +427,34 @@ mod tests {
assert!(clean.contains("owned") && clean.contains("done"));
}
#[test]
fn sanitize_strips_bidi_format_controls() {
// The Cc-only test above is vacuous for Cf: it passes whether or not bidi
// controls survive, which is why this gap shipped (#183). Here every Cf
// bidi class is stripped — override (RLO), embedding + pop (LRE/PDF),
// isolate (LRI/PDI), marks (LRM/ALM) — asserted per code point, and the
// visible words JOIN with no phantom space (a bidi char is dropped like a
// control byte, but unlike a newline/tab it is not whitespace). The leading
// bidi char (index 0) also exercises the empty-out path.
let out = sanitize_excerpt("\u{202e}err\u{202a}\u{202c}\u{2066}\u{2069}\u{200e}\u{061c}ok");
for c in [
'\u{202e}', '\u{202a}', '\u{202c}', '\u{2066}', '\u{2069}', '\u{200e}', '\u{061c}',
] {
assert!(!out.contains(c), "bidi U+{:04X} leaked: {out:?}", c as u32);
}
assert_eq!(out, "errok");
}
#[test]
fn sanitize_preserves_legitimate_and_rtl_text() {
// Must not over-strip: plain text, a genuine RTL SCRIPT letter (Arabic
// U+0627, category Lo), and ZWJ (U+200D, a legitimate Cf char) survive.
assert_eq!(
sanitize_excerpt("ok \u{0627}\u{200D}b"),
"ok \u{0627}\u{200D}b"
);
}
#[test]
fn sanitize_collapses_whitespace_and_caps_length() {
let collapsed = sanitize_excerpt("a\n\n\t b");
@@ -429,6 +463,18 @@ mod tests {
assert_eq!(sanitize_excerpt(&long).chars().count(), MAX_ERR_CHARS);
}
#[test]
fn sanitize_bidi_chars_do_not_consume_the_length_cap() {
// A dropped bidi char must not count toward MAX_ERR_CHARS: it is dropped
// before `kept` increments, so interleaving bidi controls cannot truncate
// the visible output early. MAX_ERR_CHARS + 100 visible 'x', each trailed by
// a bidi override, must still yield exactly MAX_ERR_CHARS visible 'x' with
// no phantom spaces. (Would regress to fewer 'x' if bidi consumed budget, or
// to spaces if the drop branch wrongly set pending_space.)
let body = "x\u{202e}".repeat(MAX_ERR_CHARS + 100);
assert_eq!(sanitize_excerpt(&body), "x".repeat(MAX_ERR_CHARS));
}
// ── P2: never talk to (or hand a key to) an untrusted node-chosen URL ──
#[test]
@@ -0,0 +1,94 @@
//! INV-19 guard: the icaptcha flow must actually call the service (consume the
//! mock), not short-circuit or make a live call. If a future change let
//! `obtain_proof` skip the network (e.g. a stray `cfg(test)` relaxation, which
//! INV-19/INV-20 warn is inert across crates), the `.assert()` calls below go
//! RED because the mocked endpoints were never hit.
//!
//! The mock-consumed assertions alone only prove the mock was hit; they cannot
//! see a request that ALSO leaks the DID / answer / API key to `DEFAULT_URL` or
//! any other origin (a different host the mock never observes). To make the
//! "no live call" half of the guard load-bearing, the test blackholes every
//! non-loopback destination: `NO_PROXY` lets the loopback mock through while a
//! closed proxy port routes any other host nowhere, so the client contacting
//! anything but the injected mock fails the request instead of silently
//! succeeding. Point `cfg.url` at `DEFAULT_URL` and this test goes RED.
//!
//! `ALL_PROXY` alone is NOT enough: reqwest honors the scheme-specific
//! `HTTPS_PROXY` / `https_proxy` (and the http equivalents) OVER `ALL_PROXY`, so
//! a runner with a working `HTTPS_PROXY` in its environment would route an https
//! leak to `DEFAULT_URL` (which is https) through that proxy while the loopback
//! mock is still consumed and this test stayed green. So we blackhole the
//! scheme-specific variables too. No restore is needed: this file has a single
//! `#[test]`, so it runs in its own process and the override never races a
//! sibling test.
use icaptcha_client::{obtain_proof, Challenge, IcaptchaCfg};
#[test]
fn obtain_proof_consumes_the_mocked_service_and_makes_no_live_call() {
// Blackhole every non-loopback origin: the loopback mock bypasses the proxy
// (NO_PROXY); any other host is forced through a closed port and fails.
// reqwest reads these at client-build time, and obtain_proof builds its
// client fresh, so this bounds the transport for the whole flow.
//
// Set the scheme-specific vars too (both cases), not just ALL_PROXY: reqwest
// gives HTTPS_PROXY/https_proxy precedence over ALL_PROXY for https URLs, so
// an ALL_PROXY-only blackhole leaves an https leak to DEFAULT_URL open on a
// runner whose environment already has a working HTTPS_PROXY.
let blackhole = "http://127.0.0.1:1";
std::env::set_var("NO_PROXY", "127.0.0.1,localhost");
std::env::set_var("no_proxy", "127.0.0.1,localhost");
for var in [
"ALL_PROXY",
"all_proxy",
"HTTPS_PROXY",
"https_proxy",
"HTTP_PROXY",
"http_proxy",
] {
std::env::set_var(var, blackhole);
}
let mut server = mockito::Server::new();
// The two endpoints the flow hits, each expected exactly once.
let challenge = server
.mock("POST", "/v1/challenge")
.with_status(200)
.with_header("content-type", "application/json")
.with_body(
r#"{"challengeId":"c1","type":"anagram","difficulty":1,"prompt":"listen","token":"tok-1"}"#,
)
.expect(1)
.create();
let answer = server
.mock("POST", "/v1/answer")
.with_status(200)
.with_header("content-type", "application/json")
.with_body(r#"{"status":"passed","proof":"PROOF-XYZ"}"#)
.expect(1)
.create();
let cfg = IcaptchaCfg {
url: server.url(),
did: "did:key:zTEST".to_string(),
level: 1,
api_key: None,
};
// "anagram" is not a built-in solvable type, so the flow consults this
// solver callback. The answer body itself does not matter: the mocked
// /v1/answer returns "passed" regardless of what is submitted.
let solve = |_c: &Challenge| Some("silent".to_string());
let solver: &dyn Fn(&Challenge) -> Option<String> = &solve;
let proof = obtain_proof(&cfg, Some(solver))
.expect("obtain_proof should complete against the mocked service");
assert_eq!(proof, "PROOF-XYZ");
// INV-19: prove the client actually called the mocked endpoints. Had it made
// a live call or skipped the network, these would fail (mock never hit).
challenge.assert();
answer.assert();
}
+14
View File
@@ -193,5 +193,19 @@ else
echo " gl quickstart"
fi
# oh-my-zsh's default git plugin aliases gl='git pull', which silently shadows
# the gl binary in every interactive zsh (aliases beat PATH). Detect the common
# setup and say so now, at install time, instead of letting the first `gl`
# print git's baffling "fatal: not a git repository".
if [ -f "$HOME/.zshrc" ] && [ -d "$HOME/.oh-my-zsh/plugins/git" ] \
&& grep -Eq '^[[:space:]]*plugins=\(([^)]*[[:space:]])?git([[:space:]][^)]*)?\)' "$HOME/.zshrc" \
&& ! grep -Eq '^([^#]*[[:space:];&|])?unalias[[:space:]]+([^;&|#]*[[:space:]])?gl([[:space:]]|$)' "$HOME/.zshrc"; then
echo ""
echo "NOTE: oh-my-zsh's git plugin aliases gl='git pull', which will shadow"
echo "the gl command in interactive shells. To use gl by name, run:"
echo ""
echo " echo 'unalias gl 2>/dev/null' >> ~/.zshrc && source ~/.zshrc"
fi
echo ""
echo "Docs: https://docs.gitlawb.com"
+71
View File
@@ -0,0 +1,71 @@
#!/usr/bin/env bash
#
# Dependency-purity gate for the `gitlawb-core` crate.
#
# gitlawb-core is embedded by every consumer (the gl CLI, git-remote-gitlawb,
# the node daemon), so it must stay lean. This script recomputes gitlawb-core's
# NORMAL (non-dev, non-build) transitive dependency set and fails if it contains
# any crate not present in ci/gitlawb-core-allowed-deps.txt.
#
# Hard-fail direction: a crate present now but NOT allowlisted. That is the case
# the gate exists to catch (core silently gaining a heavy dependency).
# Informational only: an allowlisted crate no longer present (stale entry) — a
# legitimate dependency removal should not red CI, so it is reported, not failed.
#
# Runnable from anywhere in the repo.
set -euo pipefail
ROOT="$(git rev-parse --show-toplevel)"
ALLOW="$ROOT/ci/gitlawb-core-allowed-deps.txt"
if [ ! -f "$ALLOW" ]; then
echo "ERROR: allowlist not found at $ALLOW" >&2
exit 1
fi
# `cargo tree` resolves like the build/test jobs (no --locked): the committed
# Cargo.lock can lag the manifests, and --locked would red this gate for
# lock-staleness reasons unrelated to gitlawb-core's dependencies. Resolving can
# refresh Cargo.lock as a side effect, so snapshot and restore it — this check
# must never leave the working tree dirty when run locally.
lock_backup="$(mktemp)"
cp "$ROOT/Cargo.lock" "$lock_backup"
restore_lock() { cp "$lock_backup" "$ROOT/Cargo.lock"; rm -f "$lock_backup"; }
trap restore_lock EXIT
# Current normal-dependency closure of gitlawb-core, one crate name per line.
# Must match the regen command documented in the allowlist header exactly.
current="$(
cargo tree -p gitlawb-core --edges normal --prefix none --manifest-path "$ROOT/Cargo.toml" \
| sed -E 's/ v[0-9].*$//' \
| grep -v '^gitlawb-core$' \
| sort -u
)"
# Allowlist with comments and blank lines stripped.
allowed="$(grep -vE '^[[:space:]]*(#|$)' "$ALLOW" | sort -u)"
# comm needs sorted input; both sides are sorted above.
offenders="$(comm -23 <(printf '%s\n' "$current") <(printf '%s\n' "$allowed"))"
stale="$(comm -13 <(printf '%s\n' "$current") <(printf '%s\n' "$allowed"))"
if [ -n "$stale" ]; then
echo "NOTE: allowlisted crates no longer in gitlawb-core's dependency tree"
echo " (safe to prune from ci/gitlawb-core-allowed-deps.txt):"
printf ' %s\n' $stale
echo
fi
if [ -n "$offenders" ]; then
{
echo "ERROR: gitlawb-core gained dependencies not on the allowlist:"
printf ' %s\n' $offenders
echo
echo "gitlawb-core must stay embeddable and lean. If a new dependency is"
echo "intentional, add it to ci/gitlawb-core-allowed-deps.txt (regen command"
echo "is in that file's header). Otherwise, drop the dependency."
} >&2
exit 1
fi
echo "gitlawb-core dependency purity: OK ($(printf '%s\n' "$current" | wc -l | tr -d ' ') normal deps, all allowlisted)."