node/infra/aws

Twigpine node on AWS (Terraform)

Single EC2 instance running the published node image + Postgres via Docker compose, with a persistent encrypted EBS volume, Elastic IP, SSM access, and daily snapshots.

Elastic IP ──► EC2 t4g.small (Amazon Linux 2023, arm64)
  7545/tcp       docker compose:
  7546/udp        ├─ node      (ghcr.io/twigpine/node, pulled — not built)
                  └─ postgres:16-alpine
                 EBS gp3 volume mounted at /mnt/data
                  ├─ node/   → container /data (repos + identity key)
                  └─ postgres/ → postgres data dir

Prerequisites

• Terraform ≥ 1.6
• AWS credentials configured (aws sts get-caller-identity works)
• AWS CLI + Session Manager plugin (for shell access)
• A default VPC in the target region (or pass subnet_id)

Quick start

cd infra/aws
cp terraform.tfvars.example terraform.tfvars   # edit: public_url at minimum
terraform init
terraform plan
terraform apply    # ⚠ creates billable resources (~$25/mo: EC2 + EBS + snapshots)

After apply (~3-5 min for first boot to pull images and start):

curl "$(terraform output -raw api_url)/health"

⚠ First boot: back up the identity key

The node generates /data/keys/identity.pem on first start — it defines the node's DID. Losing it permanently changes the node's identity. Back it up immediately:

$(terraform output -raw ssm_session_command)
# in the session:
sudo cat /mnt/data/node/keys/identity.pem

Store the key somewhere safe (password manager / offline). The volume's prevent_destroy guard and daily DLM snapshots protect against accidents, but are not a substitute for an offline backup.

Shell access

SSM Session Manager — no SSH port, no keys to manage:

$(terraform output -raw ssm_session_command)

Bootstrap log: /var/log/gitlawb-bootstrap.log. Stack lives in /opt/gitlawb (docker compose ps, docker compose logs node).

SSH is off by default; set ssh_ingress_cidr + ssh_key_name if you need it.

Building and pushing the image (ECR mode)

With create_ecr_repo = true the instance pulls from a private ECR repo in this account instead of an external registry (the instance role gets pull rights via the amazon-ecr-credential-helper). Build from the repo root — arm64 to match t4g (native on Apple Silicon):

ECR_URL=$(terraform -chdir=infra/aws output -raw ecr_repository_url)
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin "${ECR_URL%%/*}"
docker build --platform linux/arm64 -t "$ECR_URL:latest" .
docker push "$ECR_URL:latest"

Then run the upgrade command (below) to roll it out.

Upgrading the node

User-data only runs at first boot, so upgrades go through SSM:

$(terraform output -raw upgrade_command)

This reinstalls the rendered /opt/gitlawb/compose.yaml, then runs docker compose pull && docker compose up -d on the instance.

Run terraform apply before the upgrade command. The command is an SSM document managed by Terraform, and it embeds the compose file rendered from your current variables. An instance that has not had a fresh apply still holds the previous document and will reinstall the previous compose file.

• With image_tag = "latest" (default) that picks up the newest release.
• With a pinned tag, set image_tag in terraform.tfvars and apply — the rendered compose carries it, so the upgrade installs the right version.
• compose.yaml is Terraform-owned and is overwritten on every upgrade. Per-instance settings belong in /opt/gitlawb/.env, which is never touched.
• The upgrade reconciles the service set, not just the image. The rendering is conditional — postgres exists only when db_host is empty, caddy only when domain_name is set — and the command runs --remove-orphans. So setting use_rds, or clearing domain_name, and then upgrading will stop and remove the local postgres or the TLS terminator. That is the intended outcome of those variables, but it happens on the upgrade rather than on apply. Data survives either way: both bind-mount under /mnt/data.

Why the upgrade rewrites the compose file

aws_instance deliberately ignores user_data drift, so an instance keeps the compose file it was created with. Compose passes only the variables named in a service's environment: block into the container, so a node setting added to the template never reaches an existing instance — writing it to /opt/gitlawb/.env and restarting leaves the node on its built-in default, with no error to indicate it. pull && up -d alone cannot fix that, because the file on disk is the authoritative one.

Reinstalling the rendering first makes the upgrade a real migration. It is idempotent: on an already-current instance it writes identical bytes.

Replace the instance itself (OS/AMI/instance-type changes) with terraform apply -replace=aws_instance.node — the data volume reattaches and /data (including the identity key) survives.

Changing configuration

terraform apply alone never changes a running instance: user-data runs once and the instance ignores user_data drift. How a change rolls out depends on which of the two files carries it.

Rendered into compose.yaml — image_tag, gitlawb_port, metrics_port, domain_name, db_host, the icaptcha_* values. apply, then run upgrade_command: it installs the new rendering and restarts.

terraform apply
$(terraform output -raw upgrade_command)

Written into /opt/gitlawb/.env at first boot — public_url, bootstrap_peers, auto_sync, max_pack_bytes, and the integration secrets. Editing these in terraform.tfvars does not reach a running instance, because nothing rewrites .env after the first boot. Either edit /opt/gitlawb/.env on the instance and docker compose up -d, or replace it:

terraform apply -replace=aws_instance.node

The data volume reattaches; repos, postgres data, and the identity key survive.

A node setting the installed compose.yaml does not already name — GITLAWB_ENFORCE_OWNER_PUSH on an instance created before it was added, for example. Editing .env is not enough: compose only passes through the variables its environment: block lists, so the value is read from .env and then dropped. Run apply plus upgrade_command first, which installs a compose file that names the key; after that, .env governs it like any other.

That last case is the one worth remembering: .env reaches the container only for keys the installed compose interpolates, and the upgrade is what makes a newly added key one of them.

Remote state (optional)

Local state is the default. To move state to S3: create a versioned bucket, uncomment the backend "s3" block in versions.tf, then:

terraform init -migrate-state

Teardown

terraform destroy will fail on the data volume by design (prevent_destroy). To tear everything down:

1. Back up the identity key (above) and take a final snapshot if you may return.
2. Remove the prevent_destroy line from aws_ebs_volume.data in main.tf.
3. terraform destroy.

Note: DLM snapshots created by the policy are not deleted by destroy — clean them up in the EC2 console if unwanted. The Elastic IP is released on destroy.

Security notes

• Postgres password: generated by Terraform, stored as an SSM SecureString, fetched at boot via the instance profile — never in user-data or state-free files on disk (only in /opt/gitlawb/.env, mode 600). It IS in Terraform state — treat state as sensitive (another reason for the S3 backend).
• Sensitive optional vars (operator_private_key, pinata_jwt, s3_access_key_id, s3_secret_access_key) follow the same SSM path.
• SSM secrets use the AWS-managed aws/ssm key by default; set ssm_kms_key_id to encrypt with a customer-managed KMS key instead (the instance role is granted kms:Decrypt on that key automatically).
• IMDSv2 is required; metrics port is closed unless metrics_ingress_cidr is set.
• The node itself serves plain HTTP on 7545. Set domain_name (with DNS pointing at the Elastic IP) to run a Caddy sidecar with automatic Let's Encrypt TLS on 443 (+ http→https redirect on 80) — matching the Fly nodes. Certs persist on the data volume (/mnt/data/caddy).