Nicholas TindleandClaude Opus 5.5
20cc3bfc47
**Breaking change:** Docker Compose doesn't start until `make init-env`
or an installer has generated `POSTGRES_PASSWORD` in
`autogpt_platform/.env`. A `backend/.env` with no
`GRAPHITI_FALKORDB_PASSWORD` or `RABBITMQ_DEFAULT_PASS` line needs `make
init-env` too: without it FalkorDB doesn't start, and the backend can't
log in to RabbitMQ. PostgreSQL, RabbitMQ, Redis, FalkorDB and ClamAV
answer on `127.0.0.1` only. `make run-frontend` needs `DATABASE_URL` in
`frontend/.env`. An existing install upgrades by "Upgrading: service
passwords are generated per install" in
`docs/platform/getting-started.md`.
### Why / What / How
**Why:** The Docker Compose stack ran its database, RabbitMQ and
FalkorDB on passwords written into the repository's public
`.env.default` and Compose files, ran Redis without one, and published
all of them on every interface of the host. This gives each install its
own passwords and publishes those services on `127.0.0.1` only.
**What:**
- `make init-env` and both installers generate the passwords of the
bundled PostgreSQL, RabbitMQ, Redis and FalkorDB, with the existing
`runtime_config.py fill-env`. It now fills all three `.env` files in one
call, so they hold the same database password.
- The `.env.default` files ship those passwords, and the frontend's
`DATABASE_URL`, blank. `check-env-defaults` keeps them blank.
- Compose takes the database password from `autogpt_platform/.env` for
the `db` service and every connection string, and stops with a message
when it isn't set. The Redis shards require `REDIS_PASSWORD` once it is
set.
- PostgreSQL, RabbitMQ, Redis, FalkorDB and ClamAV publish their ports
on `127.0.0.1` only, in the development stack and in the backend's test
stack.
- An install set up before this keeps running on the passwords its
`.env` files hold, on the new ports. A password a file has no line for
came from `.env.default`, so it is blank now, and `make init-env` adds
it. The setup guide's new section, "Upgrading: service passwords are
generated per install", moves an install to generated passwords:
- it empties RabbitMQ's queues first, since they don't survive the
broker's container being recreated;
- then it runs `git pull`, `make init-env` and `docker compose up -d
--build`.
That `make init-env` replaces a service password still holding a value
this repository used to ship, and adds the ones an older file lacks. It
never changes a key that encrypts stored data.
- Until then, the backend logs a warning at startup for each service
password that still holds such a value, and for a Compose Redis cluster,
FalkorDB or RabbitMQ without a password.
**How:**
- `fill-env` reads `.env` files by Compose's own rules (compose-go's
dotenv parser and its `${...}` templates), checked against `docker
compose config`, and changes values in place, keeping quotes, comments
and line endings.
- It stops without changing anything on a file Compose would refuse, and
on a service password that Compose resolves only when the stack starts
(a `$` reference, an unclosed quote) or that a connection string can't
carry.
- It leaves alone the password of a service that `backend/.env` points
at another host: a database, Redis, broker or FalkorDB of the operator's
own keeps the password it has, or none. The host that counts is the one
the backend reads: `DB_HOST`; `REDIS_CLUSTER_HOST`, or else
`REDIS_HOST`; `RABBITMQ_CLUSTER_HOST`, or else `RABBITMQ_HOST`; and
`GRAPHITI_FALKORDB_HOST`.
- It points out a `POSTGRES_PASSWORD` set in the shell, which Compose
would use instead of the file's.
- `db/entrypoint.sh` wraps the image's entrypoint. On an existing data
directory it applies `POSTGRES_PASSWORD` at every start:
- through a temporary server that listens on a socket only, with psql's
`\password`, so only a SCRAM verifier reaches the server and nothing is
logged;
- if it can't apply the password, it doesn't start the database, and
says how to start it once with `PGPASSWORD`.
Started as root, it switches to the postgres user with the script text
it has already read, so the checkout's file modes don't matter.
- The Redis shards read their password as config on stdin, which keeps
it out of the process list, and get 60 seconds to write their data out
when they stop. `redis-init` reintroduces shards that Compose recreated
on new addresses. It starts a cluster that won't form over only when
none of its shards holds a key, because the backend keeps work in
progress in Redis, not only cache.
- The frontend's auth database pool has no fallback connection string.
`make init-env` writes `DATABASE_URL` into `frontend/.env`.
- CI derives its broker and FalkorDB passwords from the run.
### Changes 🏗️
- `.env.default` (in `autogpt_platform/`, `backend/` and `frontend/`):
the service passwords ship blank. The root file's `POSTGRES_PASSWORD` is
the one Compose reads.
- `single-container/runtime_config.py`:
- `fill-env` takes several `--path`s and fills the service passwords,
reading the files by Compose's rules;
- `check-env-defaults` covers the root `.env.default` too.
The single-container image's own runtime config is unchanged.
- `docker-compose.yml`, `docker-compose.platform.yml` and
`backend/docker-compose.test.yaml`: the password from `.env`, the ports
on `127.0.0.1`, the db entrypoint, Redis's password and stop grace,
`redis-init`, and FalkorDB's exit message for a missing password, which
now says to run `make init-env`.
- `db/entrypoint.sh` (new). `.gitattributes` keeps it LF in a Windows
checkout, since Compose mounts it into the container.
- `backend/util/secrets_guard.py`: the startup warnings.
- `backend/scripts/run_tests.py`: the test stack's readiness probe
authenticates to Redis.
- `frontend/src/lib/auth/auth.ts`: no fallback connection string.
- `Makefile` (`init-env`) and `installer/setup-autogpt.sh`/`.bat`: one
`fill-env` call over all three files. The installers pass a
`POSTGRES_PASSWORD` from the shell into the container that runs it, so
the warning about it still shows.
- `.github/workflows/platform-backend-ci.yml`: passwords derived from
the run, for the services and the tests alike. `.secrets.baseline` drops
the old passwords' entries, and `.pre-commit-config.yaml` runs
`check-env-defaults` on the root `.env.default` too.
- Tests:
- `single-container/tests/test_compose_credentials.py` (new);
- `single-container/tests/test_runtime_config.py`;
- `backend/util/secrets_guard_test.py`;
- `backend/scripts/test_test_stack.py`.
- Docs: `getting-started.md` (the upgrade section), `installer.md`,
`advanced_setup.md` and the platform README.
**Configuration changes:**
- `make init-env` and the installers generate:
- `POSTGRES_PASSWORD` in `autogpt_platform/.env`;
- `DB_PASS`, `RABBITMQ_DEFAULT_PASS`, `REDIS_PASSWORD` and
`GRAPHITI_FALKORDB_PASSWORD` in `backend/.env`;
- `DATABASE_URL` in `frontend/.env`.
- Docker Compose doesn't start without `POSTGRES_PASSWORD` in
`autogpt_platform/.env`.
- Ports 5432, 5672, 17000–17002, 6380, 3001 and 3310 are published on
`127.0.0.1` only.
- The `db` service starts through `db/entrypoint.sh`.
### Agents and large language models used
- Claude Code with Claude Opus 5.5: implementation and tests.
- Codex with Daybreak Blue (gpt-daybreak-blue-latest): read-only
security review.
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan, on this
branch merged with current `dev`:
- [x] `test_compose_credentials.py` and `test_runtime_config.py`, on
Linux and on Windows. They include:
- `fill-env` against `docker compose config`: no difference in any of
the values, whole files and templates compared, and it reads every
version of the `.env.default` and `.env.example` files in the
repository's history;
- the db entrypoint and `redis-init` run with stub `postgres` and
`redis-cli`, including an entrypoint started as root on a script the
postgres user can't read (Linux, as a non-root user).
- [x] Mutation testing: 88 mutants, one per control, each failing its
test. That's 50 in `fill-env` and the Compose files, 15 in the rule that
leaves an external service's password alone, 8 in the db entrypoint, 5
in the check that CI's test job gets the passwords its services require,
6 in the startup warnings, and 4 in `redis-init`.
- [x] RabbitMQ 4.1.4, the image Compose runs, with a blank
`RABBITMQ_DEFAULT_PASS`: it starts and creates the user, but refuses
that user's login.
- [x] The db entrypoint on a real PostgreSQL:
- it applies the new password, and the old one is refused;
- setting queries (`postgres -C`) start no server;
- with local connections that need a password, it stops with its
message, and the `PGPASSWORD` route works;
- no password reaches the server log in plain text.
- [x] An install of `dev` on the repo's own Compose stack, upgraded by
the guide:
- the passwords from `.env.default` work before and are refused after;
- every data port is published on `127.0.0.1`;
- the backend's startup warnings go from 5 to 0;
- a row in PostgreSQL and the keys in Redis survive, also when Redis's
containers are recreated on new addresses;
- the guide's queue check reads every RabbitMQ virtual host, whatever
its name.
- [x] A fresh install, and the backend's test stack.
- [x] The guide's PowerShell commands in Windows PowerShell 5.1 and
PowerShell 7, with a stub `docker`.
- [x] `check-env-defaults`, `secrets_guard_test.py` and
`test_test_stack.py`.
#### For configuration changes:
- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
5.3 KiB
Advanced Setup
The advanced steps below are intended for people with sysadmin experience. If you are not comfortable with these steps, please refer to the basic setup guide.
Introduction
For the advanced setup, first follow the manual setup to get the server up and running. Once you have the server running, you can follow the steps below to configure the server for your specific needs. These steps apply to the Docker Compose setup; the single-container image is configured with the environment variables you pass to docker run.
Configuration
Setting config via environment variables
The server uses environment variables to store configs. Each part of the platform reads its own .env file inside autogpt_platform/:
backend/.envfor the backend servicesfrontend/.envfor the frontend.envfor Docker Compose itself
The valid options are listed, with a comment for each, in the .env.default file next to each .env. make init-env creates any missing .env from its .env.default and generates the secrets that .env.default leaves blank:
cd autogpt_platform
make init-env
Then edit the .env files to change what you need. Do not copy .env.default over an existing .env: that erases your generated ENCRYPTION_KEY, and the integration credentials stored under it can no longer be read.
With Docker Compose, the environment: entries in docker-compose.platform.yml take precedence over the .env files. They connect the services to each other by their Compose names and set the database URLs, so change those settings in docker-compose.platform.yml. When you run the backend outside Docker, you can also set the environment variables directly in your shell. Refer to your operating system's documentation on how to set environment variables in the current session.
Database selection
PostgreSQL
We use PostgreSQL 15 with the pgvector extension as the database. Docker Compose runs it as the db service, keeps its data in autogpt_platform/data/db/data, and applies the Prisma migrations through the migrate service every time you run docker compose up.
To use your own PostgreSQL server instead:
- Make sure the
vectorandpg_trgmextensions are available on it; the migrations create them. - Run
autogpt_platform/db/init/00-init.sqlagainst the database once. It creates theplatformschema and an emptyauth.userstable that older migrations reference. - Point
DATABASE_URLandDIRECT_URLat it, keepingschema=platform(indocker-compose.platform.ymlunder Docker, or inbackend/.envwhen you run the backend directly), and point the frontend'sDATABASE_URLat the same database.
The migrate service, or make migrate outside Docker, then applies the migrations.
Redis
The platform uses Redis for caching, distributed locks, rate limits, and the streams that carry agent output to the browser.
The Docker Compose stack runs its own Redis Cluster: three redis:7 shards on ports 17000 to 17002 of 127.0.0.1, joined by a one-shot redis-init service. They require the REDIS_PASSWORD that make init-env generates in backend/.env. make start-core starts it along with the other dependencies.
Using an external Redis cluster
The backend connects with a cluster client only, so the deployment must have cluster mode enabled. A standalone server, with cluster mode disabled, does not work.
The cluster also needs:
- Redis 7.0 or later, or a compatible engine such as Valkey. The backend uses sharded pub/sub (
SPUBLISH,SSUBSCRIBE), which Redis 7.0 introduced, along with Streams and Lua scripts. It uses no Redis modules. - Shard addresses the backend can reach. With
REDIS_USE_ANNOUNCED_ADDRESS=true, the backend connects to each shard at the address the cluster announces, so those addresses must resolve from where the backend runs. Without it, the backend connects to every shard atREDIS_HOSTand keeps only the announced port, which works only when every shard is reachable on that one host. - A private network between the backend and every shard. The backend connects without TLS and does not send a username. It sends
REDIS_PASSWORDwhen one is set, and a cluster without a password works too. Nothing on the connection is encrypted, so don't route it over a network you don't trust.
Point REDIS_HOST and REDIS_PORT at any node of the cluster, and set REDIS_PASSWORD if the cluster has one.
In the Compose stack, REDIS_HOST, REDIS_PORT and REDIS_USE_ANNOUNCED_ADDRESS are set in the x-backend-env block of autogpt_platform/docker-compose.platform.yml, which takes precedence over backend/.env, so change them there. REDIS_PASSWORD is read from backend/.env. Point backend/.env at the cluster too, with REDIS_HOST or with REDIS_CLUSTER_HOST, which the backend reads first: while the one it reads names the bundled cluster (localhost, as .env.default has it), make init-env gives a blank REDIS_PASSWORD a generated value, which a cluster without a password refuses. The bundled shards still start, because the backend services wait for redis-0 to be healthy.
AutoGPT Agent Server Advanced set up
To run the backend server outside Docker with Poetry, follow Backend Development.