* Keep node alive during database outages
* Fix startup-resilience gaps: reachable 503s, bounded retries, seamless handoff
- Route sqlx errors buried in anyhow chains back to AppError::Db and widen
the unavailable class (Io/Tls/pool), so DB outages return 503 instead of
500 — the previous 503 arm was unreachable dead code
- Bound each connect+migrate attempt (GITLAWB_DB_CONNECT_TIMEOUT_SECS) so a
held migration advisory lock cannot wedge the startup retry loop forever
- Share one bound socket between the degraded and full servers and stop
awaiting the degraded drain: no closed-port window during the handoff and
no slow client can block the full server from starting
- Probe the database from /ready (2s bound) and gate Fly routing/deploys on
it, so mid-life outages drain traffic and a bad DATABASE_URL fails the
deploy visibly instead of going green; classify auth/catalog errors as
likely-permanent (error-level log, straight to max backoff)
- Start the metrics listener before the DB connects so the degraded window
is observable; degraded /health now returns 503 so peer pings and uptime
monitors stop counting a DB-less node as alive
- Move the five GITLAWB_DB_* knobs onto clap Config (validated, in --help),
drop the redundant outer connect timeout, default the pool to 20 conns,
fix the backoff exponent cap that defeated GITLAWB_DB_RETRY_MAX_SECS, and
strip at most one .git suffix in smart-HTTP repo names
* fix: apply CodeRabbit auto-fixes
Fixed 1 file(s) based on 1 unresolved review comment.
Co-authored-by: CodeRabbit <[email protected]>
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: CodeRabbit <[email protected]>
* fix(node): close two spam-vector root causes (trust upsert + ungated push)
The June 2026 spam wave kept re-materializing after cleanup because two
paths let an unauthenticated-by-captcha actor keep operating:
1. `update_trust_score` was an upsert. Any authenticated push/issue/PR
from a DID with no agent row silently INSERTed one with a fresh
`registered_at`, bypassing the iCaptcha gate on /api/register — so
deregistered spam DIDs came back the moment they pushed. Make it an
UPDATE-only no-op for unregistered DIDs; registration is the sole way
into the agents table. Regression test added.
2. `git-receive-pack` had no rate limit. The per-DID limiter on the
creation routes can't brake a push flood from a DID farm (one
throwaway identity per repo), so the node absorbed several pushes/sec
per IP, each a Tigris round-trip + peer-notify fan-out. Add a per-
client-IP limiter (Fly-Client-IP / X-Forwarded-For) on the push path,
default 600/h, `GITLAWB_PUSH_RATE_LIMIT` override, 0 disables. Fails
open without a proxy header so self-hosted nodes aren't broken.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
* fix(node): address PR review — advertisement throttle, trusted-proxy IP key, bounded limiter
Resolves the review findings on the push rate limiter:
P1 — advertisement path bypassed the limiter. A push hits
GET info/refs?service=git-receive-pack first, which calls acquire_fresh()
(a Tigris download); it sat in git_read_routes under only optional_signature,
so the flood was reachable via the cheap GET without ever sending the throttled
POST. Now git_info_refs applies the same per-IP brake before the fresh acquire.
P1 — client-supplied forwarded headers were trusted. The key came from
Fly-Client-IP / the first X-Forwarded-For hop, both client-settable, so a
flooder could rotate the header and never fill a bucket. Replaced with a
TrustedProxy policy (GITLAWB_TRUSTED_PROXY = fly | x-forwarded-for | unset):
trust only the operator's edge header, fall back to the real socket peer
(ConnectInfo) otherwise. Fly config trusts Fly-Client-IP; the AWS/Caddy
compose trusts the rightmost X-Forwarded-For hop.
P2 — unbounded limiter key set / insert-before-reject. Added a max_keys cap
(reject-before-insert, inline eviction when full) so a varied key cannot grow
the map, and a rejected request never allocates. Applied to both the per-IP
and per-DID limiters.
P2 — malformed forwarded header disabled the brake. Empty leading XFF hop
returned None and skipped the limiter; empty Fly-Client-IP returned Some("").
client_key now takes the rightmost XFF hop and always falls back to the peer,
so a malformed header can never turn the limiter off.
P2 — missing tests. Added: middleware 429 path, per-peer isolation, key-cap
eviction/rejection, trusted-proxy header resolution, and an integration test
asserting the receive-pack advertisement is throttled before the acquire.
Also serve with into_make_service_with_connect_info so the peer address is
available. New env: GITLAWB_PUSH_RATE_LIMIT, GITLAWB_TRUSTED_PROXY (documented
in .env.example).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
* Update infra/aws/compose.yaml.tftpl
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* fix(node): amortize the rate limiter's capacity eviction scan
`RateLimiter::check` ran a full O(max_keys) `retain` sweep on every new-key
miss once the map was at capacity — which is exactly the distinct-key-flood
state the cap defends against, so the limiter serialized all traffic behind a
200k-entry scan per request under the mutex.
Gate the inline sweep to run at most once per `sweep_interval` (min of the
window and 1s), tracked by `last_sweep`. Between sweeps a new key is rejected
without scanning; the background `cleanup()` loop still reclaims independently.
The fast path for existing keys is unchanged, the key cap invariant holds
(map never exceeds max_keys, rejected keys never allocate), and post-flood
self-healing stays bounded to one interval. Added a test asserting a burst of
capacity misses does not trigger repeated eviction scans.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
---------
Co-authored-by: Claude Opus 4.8 <[email protected]>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
The AWS EC2 node (manila) was upgraded to v0.4.0 and enabled iCaptcha
enforce by hand-editing its running /opt/gitlawb/compose.yaml — which would
be lost on instance replacement or a user-data re-run. Codify it so rebuilds
stay enforced:
- variables.tf: add icaptcha_mode (default "enforce"), icaptcha_pubkey
(default the pinned live icaptcha.gitlawb.com Ed25519 key), and
icaptcha_required_level (default 3).
- main.tf: pass the three vars into the compose templatefile render.
- compose.yaml.tftpl: set ICAPTCHA_MODE / ICAPTCHA_PUBKEY /
ICAPTCHA_REQUIRED_LEVEL on the node service environment.
Operators can override per-deployment via terraform.tfvars.
* feat(infra): optional private ECR repo for the node image
create_ecr_repo = true provisions an ECR repository in-account and
switches the instance to pull from it: the instance role gets
AmazonEC2ContainerRegistryReadOnly and user-data configures
amazon-ecr-credential-helper, so first-boot pulls and SSM upgrades
authenticate via the role (no expiring login tokens). README documents
the local build-and-push flow.
Motivation: ghcr.io/gitlawb/node is not anonymously pullable (403),
which left first boot unable to start the stack.
Co-Authored-By: OpenClaude <[email protected]>
* feat(infra): HTTPS via Caddy + production tier (RDS, alarms, S3 state)
- domain_name: Caddy sidecar terminates TLS on 80/443 with auto-renewed
Let's Encrypt certs (persisted on the data volume); public_url
defaults to https://<domain>; SG opens 80/443 — matches the Fly nodes
- use_rds: managed Postgres (db.t4g.micro, encrypted, 7-day PITR,
deletion protection) replaces the in-container postgres; DB security
group only admits the node instance
- alert_email: SNS email alerts wired to CloudWatch alarms — external
HTTPS /health via Route53 health check, EC2 status checks, CPU,
RDS free storage
- user-data grows the filesystem when the EBS volume is enlarged
- terraform state moved to S3 (versioned, KMS-encrypted, public-blocked
bucket) with lockfile locking (requires terraform >= 1.10)
Co-Authored-By: OpenClaude <[email protected]>
---------
Co-authored-by: OpenClaude <[email protected]>
600s idle windows let hung clients pin Fly proxy connection slots up to
hard_limit, which amplified the 2026-06-12 production outage. 120s was
deployed there as the fix; carry it into the reference config.
Co-authored-by: Claude Fable 5 <[email protected]>
* refactor: move fly.toml into infra/fly/
Deployment configs now live under infra/, organized per target.
Dockerfiles and docker-compose.yml intentionally stay at the repo
root (shared by CI, build scripts, and the macOS app).
Deploy with: fly deploy -c infra/fly/fly.toml
Co-Authored-By: OpenClaude <[email protected]>
* feat(infra): Terraform for single-EC2 AWS deployment
infra/aws/ provisions a production-leaning single-node deployment:
- t4g.small (AL2023 arm64) running the published ghcr.io/gitlawb/node
image + postgres:16 via docker compose
- separate encrypted EBS data volume (prevent_destroy) with daily DLM
snapshots; survives instance replacement
- postgres password via random_password -> SSM SecureString, fetched at
boot by instance profile (never in user-data); optional secrets
(operator key, Pinata JWT, S3 secret) follow the same path
- SSM Session Manager access (no SSH by default), IMDSv2 required,
metrics port closed unless explicitly opened
- SSM command document for image upgrades (user-data runs once)
Verified: terraform fmt/init/validate clean; rendered compose passes
docker compose config; rendered user-data passes bash -n.
Co-Authored-By: OpenClaude <[email protected]>
* fix(infra): address CodeRabbit review feedback
- api_url output now honors the public_url override (local.public_url)
- disable xtrace around SSM secret fetch + .env write so secret values
never reach the bootstrap log
- add retry/timeout flags to the docker compose download
- validate snapshot_retain_count (1-1000) at plan time
- add language tags to fenced tree blocks (MD040)
Co-Authored-By: OpenClaude <[email protected]>
* fix(infra): address PR #27 review feedback
- security group now derives its VPC from the selected subnet, so
subnet_id overrides into non-default VPCs work
- s3_access_key_id moves to SSM SecureString (same path as the secret
key) instead of being embedded in user-data and state
- ignore user_data drift on the instance: it only runs at first boot,
so re-rendering caused pointless stop/starts; README documents the
-replace workflow for config changes
- DLM snapshot targeting is stack-specific (Snapshot + Name tags), not
any Snapshot=true volume in the account
- comment in fly.toml that it targets the shared test instance
- user-data compose up gains --remove-orphans (consistent with the
upgrade SSM document)
- node service gets an HTTP /health healthcheck in compose
Co-Authored-By: OpenClaude <[email protected]>
* feat(infra): optional customer-managed KMS key for SSM secrets
Add ssm_kms_key_id (default null = AWS-managed aws/ssm key). When set,
all five SecureString parameters are encrypted with the CMK and the
instance role gains kms:Decrypt scoped to that key — without it the
boot-time secret fetch would fail.
Co-Authored-By: OpenClaude <[email protected]>
* fix(infra): scope CMK kms:Decrypt to this stack's SSM parameters
The grant on the customer-managed key was unconditioned, allowing the
instance role to decrypt any ciphertext under the same CMK. Restrict it
with kms:ViaService = ssm.<region>.amazonaws.com and
kms:EncryptionContext:PARAMETER_ARN limited to the stack's parameter
ARNs.
Co-Authored-By: OpenClaude <[email protected]>
---------
Co-authored-by: OpenClaude <[email protected]>
