Significant-Gravitas/AutoGPT
1
0 Fork 0
mirror of https://github.com/Significant-Gravitas/AutoGPT.git synced 2026-11-09 04:49:50 -10:00
Code Issues Packages Projects Releases 1 Wiki Activity
10,525 Commits 1,321 Branches 131 Tags
dev
Commit Graph
39 Commits
Author SHA1 Message Date
Reinier van der LeerandClaude Opus 5.5 0b6912505b ci(platform): cache the Prisma CLI and retry its download in backend and full-stack CI (#16070) 
Backend CI and full-stack CI now cache the Prisma CLI between runs and
retry its download, so a dropped connection while fetching it no longer
fails a run. Resolves #16066

### Why / What / How

**Why:** every backend job downloads the Prisma CLI and its engines
before it can generate the Prisma client, and a run fails when that
download loses its connection. "Generate Prisma Client" was the first
failed step in 7 of 201 failed Backend CI runs from 16 to 23 Oct: two
merge-queue runs (39670924275, 39821302302), whose failure ejected a PR,
and a `dev` push (40025378565). Each failed in prisma-python's `npm
install [email protected].0`, where `@prisma/engines`' postinstall lost its
connection to Prisma's engine host (`Error: aborted`, `ECONNRESET`).
Nothing cached the CLI and nothing retried it.

**What:** the backend type-check and test jobs and full-stack CI's
API-type check now generate the client through one composite action. It
restores the CLI directory from the Actions cache, so a warm job
downloads nothing. On a cold job, the CLI install is tried up to three
times, 15 s and then 30 s apart, before the step fails.

**How:** `prisma py version --json` reports the directory prisma-python
installs the CLI and engines into, along with the CLI version, engine
hash and platform. The cache's path and key come from that output, so no
Prisma version is written in the workflows and a prisma-python bump
changes the key by itself. The retried command is `prisma --version`,
which installs the CLI and fetches every engine. Once it has succeeded,
`prisma generate` and `prisma migrate` make no network calls, so a
schema error fails at once rather than three times.

### Changes 🏗️

- `.github/actions/generate-prisma-client/action.yml`: resolves the
cache path and key, restores and saves the cache, installs the CLI with
the retry, then runs `prisma generate` and `gen-prisma-stub`.
- `.github/scripts/install-prisma-cli-with-retry.sh`: the retry, in the
shape of `docker-pull-with-retry.sh` from #15978.
- `.github/scripts/test_install_prisma_cli_retry.py`: success after 0, 1
or 2 failures, and a terminal failure that exits 1. Like
`test_docker_pull_retry.py`, no workflow runs it.
- `platform-backend-ci.yml` (two jobs) and `platform-fullstack-ci.yml`
use the action, and their path filters include it and the script.
- `.github/dependabot.yml`: the github-actions entry also scans the new
action's folder. `"/"` covers only `.github/workflows`, so the action's
`actions/cache` pin would otherwise never be bumped.

Settled choices:
- **A composite action** rather than three inline copies, so the key is
defined in one place. `.github/actions/start-minio` is the precedent.
- **Named `generate-prisma-client`**, because the root `.gitignore`'s
`pri*` ignores any path starting `prisma-`.
- **Three attempts**, because the failures were single dropped
connections rather than outages. The Docker pull retry uses five for
Docker Hub's 20 s refusals.
- **Not covered:** `docs-block-sync.yml`, `copilot-setup-steps.yml`,
`platform-preview-seed-fixture.yml` and both deploy workflows still run
a bare `prisma generate`.
- **Merge-queue runs:** GitHub documents no cache scoping for them.
Whether they can read `dev`'s entry or only `master`'s is unknown, and
the retry covers them either way.

The CI workflows are mostly Nick's area, so he is the natural reviewer.

**Verified:** I ran the action's resolve step verbatim, and it printed
the cache directory and the key
`prisma-cli-Linux-X64-<platform>-5.17.0-393aa359…`. I ran the real retry
script against a cold CLI directory twice. With the `@prisma/engines`
postinstall failing once, CI's failure, it retried after 15 s and
installed. With the postinstall always failing, it exited 1 after three
attempts. Locally I showed that once `prisma --version` has run, `prisma
generate` succeeds with both the npm registry and the engine host
pointed at a dead port. The script's two tests pass, and each of three
mutations turns them red (two attempts, exit 0 on the last failure, a
different delay). In this PR's first CI run, all 16 jobs using the
action missed the cache and installed on the first attempt, and the
action saved the entry. A re-run of `type-check (3.12)` restored it and
installed nothing, and on the next head all 16 jobs restored it. Per-job
evidence is in a comment. The Dependabot entry is only reasoned about:
Dependabot reads it after the change reaches `dev`.

### Agents and large language models used

- Claude Code with Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
  - [x] The resolve step prints the cache directory and key locally
- [x] The retry script recovers from one failed CLI install and exits 1
after three
  - [x] `prisma generate` needs no network once the CLI is installed
- [x] This PR's CI: every job generates the client, and the first run
saves the cache
- [x] A re-run of one job restores the cache and logs no "Installing
Prisma CLI"

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
 2026-11-04 04:27:12 +00:00
Nicholas TindleandClaude Opus 5.5 e27ff40367 fix(frontend): bump better-auth to 1.7.16 for the Nov 3 security release (#16577) 
### Why / What / How

**Why:** Better Auth's [2026-11-03 security
release](https://better-auth.com/blog/security-release-2026-11-03) ships
its fixes in better-auth 1.7.16. Dev runs 1.7.12, and the 1.6 line gets
no fix. It publishes twelve advisories, ten at release time and two more
an hour later:

| Advisory | Severity | What it takes |
|---|---|---|
|
[GHSA-4wq4-2ppv-q3hg](https://github.com/better-auth/better-auth/security/advisories/GHSA-4wq4-2ppv-q3hg)
| critical | the `expo()` plugin's authorization proxy |
|
[GHSA-q747-mqgr-4gh7](https://github.com/better-auth/better-auth/security/advisories/GHSA-q747-mqgr-4gh7)
| high | the phone number plugin |
|
[GHSA-9p5w-h4pr-fcrc](https://github.com/better-auth/better-auth/security/advisories/GHSA-9p5w-h4pr-fcrc)
| high | the `electron()` plugin |
|
[GHSA-2f9c-7xj7-gh55](https://github.com/better-auth/better-auth/security/advisories/GHSA-2f9c-7xj7-gh55)
| medium | the `emailOTP()` plugin |
|
[GHSA-48c2-w96j-mv3m](https://github.com/better-auth/better-auth/security/advisories/GHSA-48c2-w96j-mv3m)
| medium | MySQL with `generateId: "serial"` |
|
[GHSA-9hm7-xhfp-cx58](https://github.com/better-auth/better-auth/security/advisories/GHSA-9hm7-xhfp-cx58)
| medium | OAuth providers that send `email_verified` as a string
(Cognito, Microsoft Entra ID), `genericOAuth()` with a JWKS endpoint, or
Zoom |
|
[GHSA-x8j7-xh58-rv6x](https://github.com/better-auth/better-auth/security/advisories/GHSA-x8j7-xh58-rv6x)
| medium | the `stripe()` plugin |
|
[GHSA-c85h-5mrr-mgx4](https://github.com/better-auth/better-auth/security/advisories/GHSA-c85h-5mrr-mgx4)
| medium | the organization plugin |
|
[GHSA-fvxx-76fc-893x](https://github.com/better-auth/better-auth/security/advisories/GHSA-fvxx-76fc-893x)
| medium | `secondaryStorage`, or the session cookie cache together with
passkeys, device authorization or passwordless two-factor |
|
[GHSA-hwq2-v29r-p69r](https://github.com/better-auth/better-auth/security/advisories/GHSA-hwq2-v29r-p69r)
| medium | `user.changeEmail` with verification emails, where a released
address can be registered again |
|
[GHSA-g95c-wqvr-8c3j](https://github.com/better-auth/better-auth/security/advisories/GHSA-g95c-wqvr-8c3j)
| medium | the admin and phone number plugins, with a non-admin role
holding `user:update` |
|
[GHSA-gv75-j3v8-3crm](https://github.com/better-auth/better-auth/security/advisories/GHSA-gv75-j3v8-3crm)
| low | the passkey plugin |

Most of these need a plugin or setup AutoGPT doesn't use. Every route of
the affected plugins answers 404 on our config, both before and after
the bump.

Microsoft Entra ID is one of the providers covered by the string
`email_verified` fix, so #16269 (Continue with Microsoft) should land on
1.7.16 or later.

**What:** moves `better-auth` and `@better-auth/core` from 1.7.12 to
1.7.16. In the lockfile, the `@better-auth/*` adapters and telemetry
follow at 1.7.16, along with `better-call` 1.4.5, `@better-auth/utils`
0.5.1 and `@better-fetch/fetch` 1.3.3.

**How:** `frontend/.npmrc` sets `minimum-release-age=10080`, so pnpm
refuses releases younger than seven days. `pnpm-workspace.yaml`
therefore lists the exact new versions under `minimumReleaseAgeExclude`,
and the entries can go after 2026-11-10. CI and Docker install with
`--frozen-lockfile`, which doesn't check release age.

Other changes from 1.7.13 to 1.7.16 that touch our setup, per the
[release
notes](https://github.com/better-auth/better-auth/releases/tag/v1.7.16):
- Headers and cookies set by a `hooks.before` hook that returns nothing
are now kept (1.7.15).
- The admin plugin refuses banned users on sessions created outside an
endpoint (1.7.15).
- Built-in social providers stop following redirects on token exchange
and JWKS requests (1.7.15).
- Email verification and email change links only work for the account
they were issued to (1.7.16).
- Linking a social account and two-factor routes read the session store
even with the cookie cache on (1.7.16).
- Type-only exports are no longer declared as values (1.7.13).

### Changes 🏗️

- `autogpt_platform/frontend/package.json`: `better-auth` and
`@better-auth/core` 1.7.12 → 1.7.16.
- `autogpt_platform/frontend/pnpm-lock.yaml`: the packages above. pnpm
also rewrote the peer keys of three eslint import packages, with no
version changes.
- `autogpt_platform/frontend/pnpm-workspace.yaml` (configuration):
`minimumReleaseAgeExclude` for `better-auth`, `@better-auth/core`,
`@better-auth/{drizzle,kysely,memory,mongo,prisma}-adapter` and
`@better-auth/telemetry` at 1.7.16, plus `[email protected]`. It carries
a comment saying it can go after 2026-11-10.
- No schema change: `auth-schema.test.ts` finds no column 1.7.16 needs
that we lack.
- `.github/scripts/install-trivy.sh` (new) and
`.github/workflows/platform-single-container-docker.yml` (CI, outside
the bump's scope): the image scans now install Trivy themselves.
- **What broke:** trivy-action installs Trivy through setup-trivy, whose
script downloads from `get.trivy.dev`. Since 2026-11-03 21:34 UTC that
host returns NXDOMAIN, because `trivy.dev` moved to new nameservers
without the record (`aquasecurity/setup-trivy#42`). Every PR's "Build,
smoke, and scan" has failed since, and so would the release's image
publish.
- **The fix:** both scanning jobs install Trivy 0.70.0 from its GitHub
release and check the archive against the SHA-256 that release
publishes. The trivy-action steps then run with `skip-setup-trivy:
true`. The new script joins the workflow's path filter.

### Agents and large language models used

- Claude Code with Claude Opus 5.5: the bump, the checks and this PR.
- Claude Code with Claude Mythos 5.1: checking each advisory against our
auth config.

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [ ] I have tested my changes according to the test plan:
  - [x] `pnpm generate:api` and `pnpm types` pass on 1.7.16
- [x] `pnpm format` changes nothing else; `next lint` (existing warnings
only), `pnpm lint:shadcn` and `prettier --check .` pass
- [x] `TZ=UTC vitest run src/lib/auth src/app/api/auth
'src/app/(no-navbar)/login' 'src/app/(no-navbar)/signup'
'src/app/(platform)/reset-password' 'src/app/(platform)/auth'
src/app/api/admin/ban-accounts src/app/api/proxy`: 67 files, 842 tests
pass, `auth-schema.test.ts` included
- [x] Each advisory checked against our real auth config on Postgres, on
1.7.12 and on 1.7.16. That includes confirming that on 1.7.16, email
change and verification links work only for the account they were issued
to.
- [x] `install-trivy.sh` passes shellcheck. Both release archives match
their published SHA-256 and hold `trivy` at the top level. The workflow
parses, and actionlint reports the same four findings on it as on dev's
copy, all about the background-step syntax it doesn't know.
- [ ] "Build, smoke, and scan" passes on amd64 and arm64 with the new
install step

#### For frontend changes (design system, see
`docs/engineering/design-system.md`):

- [x] `pnpm lint` passes, including the `design-system/*` rules; I did
not raise any count in
`scripts/eslint-rules/design-system-baseline.json` except for a moved,
renamed or split file (explained under **Changes**)
- [x] No `dark:`/`light:` classes, raw palette classes (`bg-zinc-200`)
or hex/`rgb()` colours (`bg-[#fff]`); colours use role tokens
(`bg-surface`, `text-foreground-secondary`)
- [x] No imports from `src/components/__legacy__`, `legacy/` paths or
`@/components/ui`, and no `LegacyButton` / `legacy-*` Button variants
- [x] Copy uses the `Text` atom with a `variant`/`tone`; primitives get
a `variant`/`size`. `className` holds layout, plus on `Text` only
truncation, line clamping, wrapping, breaking,
`text-balance`/`text-pretty` and alignment
- [x] Before/after screenshots at 1440 and 390 px are attached for
changes to `src/components/**`, `globals.css` or `tailwind*.ts`, or I
wrote "no pixel change" and how I checked: no pixel change, since only
dependency files changed
- [x] User-facing copy says workflow and Otto, never agent or graph
(root `AGENTS.md`, "User-facing copy")

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
 2026-11-04 02:09:19 +00:00
Reinier van der LeerandClaude Opus 5.5 7e2eaf267a feat(platform): payment enforcement is the ENABLE_PLATFORM_PAYMENT setting (#16459) 
**Breaking change:** payment enforcement is now the backend setting
`ENABLE_PLATFORM_PAYMENT` (default off). An install that leaves it unset
runs without a paywall and without plan checkout, whatever its flag
vendor serves for `enable-platform-payment`;
`FORCE_FLAG_ENABLE_PLATFORM_PAYMENT` and
`NEXT_PUBLIC_FORCE_FLAG_ENABLE_PLATFORM_PAYMENT` no longer do anything.
The hosted platform sets it in
Significant-Gravitas/AutoGPT_cloud_infrastructure#475, which must be
live before this deploys.

Payment enforcement becomes a deployment setting: the backend reads
`ENABLE_PLATFORM_PAYMENT` from its environment instead of evaluating the
`enable-platform-payment` feature flag per user, and the browser reads
the same value from the backend.

Resolves [SECRT-3269](https://linear.app/autogpt/issue/SECRT-3269)

### Why / What / How

**Why:** whether the platform is paid for is a property of the
deployment, not of a user: the hosted platform charges, a self-hosted
install does not. It was a feature flag nonetheless, so turning payment
on for a self-hosted install took two flag overrides (`FORCE_FLAG_…` for
the backend, a build-time `NEXT_PUBLIC_FORCE_FLAG_…` for the frontend)
that could disagree, and the hosted platform's billing switch lived in
the flag vendor instead of beside `ENABLE_CREDIT` in its deployment
config. The flag served the same value to every user on prod and Dev, so
nothing targeted per user is lost.

**What:** every backend reader of the flag (the paywall check
`is_user_paywalled`, the cost-limit multiplier for accounts without a
plan, plan checkout and the upgrade preview, the onboarding
personalization budget, Otto's `get_platform_info` subscription answer,
and the card-required trial offer) reads
`Settings.config.enable_platform_payment` instead. The frontend's
readers (the paywall modal, onboarding's step layout, billing pages,
wallet, top-up prompt, usage experience, and `useIsBillingEnabled`) read
it from a new `GET /api/platform-config`. The flag is removed from the
backend `Flag` enum and the frontend flag list.

**How:** one module, `backend/util/platform_payment.py`, reads the
setting, so every caller and every test goes through one place; tests
turn it on with the `set_platform_payment` fixture and an autouse
fixture keeps it off otherwise, whatever a developer's `.env` says. `GET
/api/platform-config` needs no sign-in and is open while paywalled,
since the paywall modal itself needs the answer; the browser reads it
once per session (`usePlatformPayment`), retrying twice a second apart
so it settles within about three seconds; a read that still fails
reports `isError` rather than payment off. The Playwright journeys leg
gets `ENABLE_PLATFORM_PAYMENT=true` from `hosted_journeys_env.py`
instead of a flag in `hosted-flags.json`.

### Changes 🏗️

- `backend/util/settings.py`: `enable_platform_payment` (env
`ENABLE_PLATFORM_PAYMENT`, default `False`).
- `backend/util/platform_payment.py`: `is_platform_payment_enabled()`,
the single reader.
- Readers moved: `copilot/rate_limit.py`,
`api/features/billing/subscriptions/routes.py`,
`api/features/onboarding_dump/rate_limit.py`,
`copilot/tools/platform_info.py`, `data/subscription_trial_config.py`.
- The subscription route's guard against cancelling on an unreadable
flag is removed: a setting is always readable. A plan cancel (`NO_TIER`)
now goes through `cancel_stripe_subscription` whether payment is on or
off, so an account still holding a Stripe subscription with payment off
has it cancelled at period end instead of only losing its plan in the
database while Stripe keeps billing. An account with no Stripe customer,
which is every account on an install that never had payment on, makes no
Stripe call and has its plan cleared as before.
- `api/features/platform_config/routes.py`: `GET /api/platform-config`
returning `{payment_enabled}`; listed in `paywall_routes_test.py` as
open.
- `Flag.ENABLE_PLATFORM_PAYMENT` and
`FrontendFlag.ENABLE_PLATFORM_PAYMENT` removed; `openapi.json` and the
generated client regenerated.
- Frontend: `hooks/usePlatformPayment.ts` (two quick retries,
`isError`); the eight direct readers and `useIsBillingEnabled` use it;
the default MSW handlers answer payment off.
- CI: `.github/scripts/hosted_journeys_env.py` adds
`ENABLE_PLATFORM_PAYMENT=true`; the flag leaves `hosted-flags.json`.

Configuration: new backend env `ENABLE_PLATFORM_PAYMENT` (bool, default
`false`). `.env.default` and `docker-compose.yml` need nothing: the
default is the self-hosted behaviour.

Borderline choices, settled:
- Onboarding waited at most 5 s for the flag vendor before laying out
its steps without the paywall. It now waits for `/api/platform-config`
to answer or fail, like its other backend reads, so that fallback and
its test are gone. A read that still fails after its retries lays the
steps out without the subscription step but is not frozen for the
account, so a later successful read adds it.
- The setting defaults to off for every install, cloud ones included, as
asked; the hosted platform turns it on in its deployment config.

### Verified

I ran every backend test file that reads, stubs or asserts the paywall
(46 files plus the two trial-route files that import their helpers,
2,013 passed), `architecture_test.py` with `test_block.py` (3,754
passed), the journeys env script's tests, `pnpm types`, the 23 touched
frontend test files plus the new hook test, and the full frontend unit
suite: 1,155 of 1,162 files passed, and the seven that had stubbed every
flag on, payment included, now stub payment on and pass (97 tests).
Seven mutations each turned their test red and were restored:
`is_user_paywalled` reading the flag instead of the setting, the setting
defaulting on, the route answering a constant, `is_user_paywalled`
ignoring the setting, and three in `usePlatformPayment` (ignoring the
response, resolving before it, reading an error as on). The hook's tests
use a client with the app's default retries: a transient failure ends
payment-on and a hard failure settles as an error within the old 5 s,
and both fail with retries turned off; the layout test lays out a failed
read as off and adds the step once a read succeeds, and fails when a
failed read is snapshotted. The upgrade preview's payment-off case
answers 422 and fails with that guard removed. The plan-cancel tests run
with payment on and off; with the route's old payment check restored,
the three payment-off cases fail (Stripe not called, or the plan cleared
in the database while a subscription exists). I did not exercise a
deployed environment; that waits on the infra change.

### Agents and large language models used

- Claude Code with Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] A NO_TIER account is paywalled with the setting on and no flag
client configured, and is not with it off
- [x] An unset `ENABLE_PLATFORM_PAYMENT` reads as off; `true` reads as
on
  - [x] `GET /api/platform-config` answers the setting both ways
- [x] The browser hook serves the backend's answer, and resolves as off
when the request fails
- [ ] After deploy: Dev's paywall still shows for an account without a
plan

#### For frontend changes (design system, see
`docs/engineering/design-system.md`):

- [x] `pnpm lint` passes, including the `design-system/*` rules; I did
not raise any count in
`scripts/eslint-rules/design-system-baseline.json`
- [x] No `dark:`/`light:` classes, raw palette classes or hex/`rgb()`
colours
- [x] No imports from `src/components/__legacy__`, `legacy/` paths or
`@/components/ui`
- [x] No copy changes
- [x] No pixel change: the components under `src/components/` read the
same boolean from a different hook, and their render tests pass with
unchanged assertions
- [x] User-facing copy says workflow and Otto, never agent or graph

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
 2026-11-03 10:50:12 +00:00
Reinier van der LeerandClaude Opus 5.5 f31259c702 ci(platform): start the e2e backend services after the frontend image is built (#16423) 
Full-stack CI's e2e jobs now start the backend services only once the
frontend image is built, and log the runner's memory while the stack
builds and starts. Resolves
[SECRT-3254](https://linear.app/autogpt/issue/SECRT-3254).

### Why / What / How

**Why.** The e2e jobs keep losing their hosted runner while `next build`
runs. On 29 Oct 15 jobs died that way: 6 that GitHub reported as "The
hosted runner lost communication with the server" about 53 minutes in,
with no log, and 9 whose frontend build ended with exit code 143. Those
15 include all three of `dev`'s Full-stack reds that day and four merge
groups the queue ejected (#16399, #16402, and #16340 twice).
- Every one of the 25 such deaths since 25 Oct came after "Start (all
other services)" had brought up the backend stack (seven Python
services, ClamAV, FalkorDB, three Redis nodes, RabbitMQ, Postgres),
while the frontend image was still building.
- A cold `next build` of the frontend peaks near 8 GiB on its own:
#15440 measured 7.68 GiB in its builder, and a local cold build of `dev`
reached 7.87 GiB, still rising. The runner has 15.61 GiB.
- The rate rose on 29 Oct at about 08:00Z, from 1 death in 212 e2e jobs
over the 20 hours before to 17 in 454 over the 18 hours after.

**What.** Only the database, its migrations and a cached seed's load
overlap the frontend build now. The backend services, and the cached
seed's credential check that needs them, start after "Wait for frontend
image build". Each e2e job also logs used and available memory and swap
every 10 s, and each container's usage once a minute, from before the
image builds until the frontend answers, so a runner that dies again
leaves readings.

**How.**
- "Start (all other services)" and "Verify cached seed credentials with
fresh encryption key" move below "Wait for frontend image build". The
fresh seed already waited there since #16198 (OPEN-3734, the same
failure with the seed beside the build).
- "Log runner memory" is a background step that runs for at most 30
minutes. "Stop runner memory log" runs with `if: always()`, so a failed
job never waits on it, and "Wait for runner memory log" collects it on
the way to the tests.
- The contract test now asserts that the services start, the cached
check and the seed all come after the frontend wait.
- "Pull service images and start database" now starts `db` with
`--no-deps`: `db` depends on RabbitMQ for startup ordering only
(`docker-compose.yml`), so without the flag that step would start
RabbitMQ too, overlapping the build. RabbitMQ now comes up with the rest
of the stack, in "Start (all other services)".

Jobs take no longer: the backend's start now waits for the build, but
the build runs about two minutes faster without the backend competing
for memory. Known limit: why the rate rose on 29 Oct is not established.
The first deaths that day were on Dependabot branches built on `dev`
from before any of that day's frontend dependency bumps. GitHub's new
runner image `20261025.353.1` (kernel 7.0.0-1017-azure) ran 87 jobs in
the 16 hours before the rise without a death. This PR removes the
overlap that all 25 deaths share, whatever pushed the total over.

### Changes 🏗️

- `.github/workflows/platform-fullstack-ci.yml`: the backend services
and the cached credential check start after the frontend build; `db`
starts with `--no-deps` so it doesn't pull RabbitMQ in early; a
background memory log, sized to the job's timeout, with a stop step and
a wait step.
- `.github/scripts/test_e2e_seed_cache_contract.py`: the order test
covers the services' start and the cached check.

The CI workflows are mostly Nick's area, so he is the natural reviewer.

**Verified:** the full-stack contract tests pass (17), and restoring
`dev`'s step order turns the order test red on both moved steps. On the
runner, I ran Full-stack CI twice on the memory-log-only commit and
twice on this head (8 e2e jobs, all green). With `dev`'s order every job
fell to 0.94–1.35 GiB available with the 3 GiB swap full, the builder
holding 8.4 GiB and the started stack 5.8 GiB. With this order every job
kept at least 5.5 GiB available and used at most 39 MB of swap. The
frontend build took 476–503 s against 525–657 s, and job times were
781–1,031 s against 841–1,062 s. The readings are in a comment.

### Agents and large language models used

- Claude Code with Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Contract tests pass, and `dev`'s step order turns the order test
red
- [x] This PR's e2e jobs start the services after the frontend build,
log memory, and pass
- [x] Runs on the memory-log-only commit and on this head compare the
runner's lowest available memory

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
 2026-11-02 15:30:07 +00:00
Reinier van der LeerandClaude Opus 5.5 08a2557a97 ci(platform): seed E2E data after the frontend image build, and time-box it (#16198) 
The smoke E2E job now seeds its database only after the frontend image
has built. The seed step also gets a 20-minute timeout, and each
embedding request the seed makes gets a 30 s timeout, so seeding can no
longer freeze the CI runner beside the Next.js build or hold a
merge-queue slot for 45 minutes. Resolves #16196

### Why / What / How

**Why:** a PR that touches what the cached seed dump depends on
(migrations, `schema.prisma`, the seed script or the backend modules it
writes through) gets no cache hit, so its smoke job seeds from scratch
while the frontend image is still building in the background. On 3 of
the 23 such seeds in the 20 hours to 00:52Z 26 Oct, the seed and the
Next.js build both went silent and the runner froze. Merge-group run
40245560978 went silent at 23:47:10Z and got a runner shutdown signal at
23:54:47Z, which ejected #12206 from the merge queue. Two pull-request
attempts (40224897876 and 40238798212) sat in the seed step for 48 and
35 minutes, uploading no log, until they were cancelled. Re-runs of the
same code seeded in 2–4 minutes. In the 335 cache-hit runs of the same
window, which skip the seed, the frontend build never stalled. Nothing
bounded the step: it had no `timeout-minutes`, and the seed's OpenAI
requests used the client's defaults of 600 s with 2 retries.

**What:**
- The seed step waits for the frontend image build, so the two no longer
run at the same time.
- The seed step times out after 20 minutes.
- Each embedding request the seed makes times out after 30 s.
- A restored dump's credential check moves into its own step, which
still runs before the wait, so cache-hit runs keep overlapping it with
the build.

**How:** a `wait:` step can't take an `if:`, so it can't apply to
seeding runs alone. It now sits just before "Create E2E test data". The
credential check used to be one step after the seed, with `if:
matrix.seed`. It is now two steps with the same command. The cached-dump
check runs before the wait, after a restore. The fresh-seed check runs
after the seed, on the seed step's own condition. Each seeding run still
verifies its credentials exactly once, and the dump push still requires
the fresh check.

The 20-minute timeout is 5× the slowest passing seed in that window (236
s). It is above the seed's own 900 s embedding deadline, so a slow but
working seed gives up by itself first: it dumps, doesn't publish the
dump, and lets the suite run. The timeout only stops a seed that is
stuck.

The per-request timeout is set on the process-cached OpenAI client that
`generate_embedding` uses, from the seed script's own entry point
(`main()`). The admin test-data route runs the seed inside a local
server, which shares that client with every other caller, so it never
sees the change. The slowest of 13,509 requests in two passing CI seeds
took 4.4 s.

### Changes 🏗️

What changes for everyone running Full-stack CI:
- **Cache-hit runs (335 of 408 smoke jobs in that window):** no new
waiting. The credential check is a step with a new name, "Verify cached
seed credentials with fresh encryption key", and it still overlaps the
build.
- **Seeding runs:** the seed starts once the frontend image is built. In
that window the seed overlapped the build by 0–242 s (median about 110
s), and that time is now added to these jobs.
- **Once, after merge:** the dump's cache key hashes the seed script and
the seed step's text, and this PR changes both. Until the first `dev`
push or merge-queue run publishes a dump under the new key, every smoke
job seeds.
- **A stuck seed** fails the job after 20 minutes instead of running
until the runner dies or someone cancels it.
- `E2E_EMBEDDING_REQUEST_TIMEOUT_SECONDS` (default 30) sits beside the
seed's two existing `E2E_EMBEDDING_*` overrides.

Settled choices:
- **Wait for the build rather than lower the embedding batch size.** The
seed process holds about 860 MB (measured on a dev machine) before it
does any work, for the backend imports and the 631-block registry, and
the batch size doesn't change that. A smaller batch means more rounds,
and each round re-reads every doc file and recounts coverage, so the
seed would overlap the build for longer. Batch size 100 also isn't
rate-limited from one job: the two passing seeds read logged no
`RateLimitError`.
- **The cause is inferred, not measured.** No memory readings exist for
these runs. Two things point to memory: the seed and the build froze
together, and the runner was measured swapping (2.6 GB in swap, CPUs 0%
idle) during E2E while #15759 was investigated on 20 Oct. `next build`
also runs with `--max-old-space-size=16384` on a 16 GB runner.
- **No job-level timeout.** The two attempts that uploaded no log may
have lost their runner. A step timeout may not fire on a lost runner,
and a job timeout would. Successful smoke jobs took at most 19.1 minutes
in that window, so one would be easy to size, but it's left for a
separate change.

The CI workflows are mostly Nick's area, so he is the natural reviewer.

**Verified:** I ran the full-stack contract tests as CI runs them, and
17 pass. Each of six mutations turns one red: moving the wait back after
the seed, dropping the step timeout, setting it to 15 minutes, raising
the seed's embedding deadline to 1,800 s, keying the cached check on
`matrix.seed`, and moving the cached check after the wait. I imported
the seed module, and confirmed that the cached client
`generate_embedding` resolves is the one the seed sets, and that a
request it builds carries the 30 s timeout. The seed's own tests pass
(19). Each of three mutations turns them red: dropping the bound,
setting it inside the backfill (the admin route's path), and setting it
on the refresh-only path. `architecture_test.py` (16 passed) and
`test_block.py` (3,714 passed) pass. The 5 embedding seeds that happened
to start after the build had finished took 125–225 s and none stalled,
against 3 stalls in 18 that overlapped it. This PR's own smoke job
seeded because of the key change, after the build, in 171 s with 100%
embedding coverage. Its slowest of 6,755 requests took 3.1 s, and
Playwright passed. Details are in a comment. The cache-hit path, where
the renamed check runs, was not exercised in CI, because a pull request
never publishes a dump.

### Agents and large language models used

- Claude Code with Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
  - [x] Contract tests pass, and each of six mutations turns one red
- [x] The seed's per-request timeout reaches the client
`generate_embedding` uses
- [x] This PR's smoke job seeds after the frontend build, within the
timeout, and publishes nothing (pull requests never publish)

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
 2026-10-26 09:02:49 +00:00
Bentlyandgpt-6-astra ee41e2822c ci(platform): cache the e2e seed dump in GHCR instead of the Actions cache (#15983) 
Resolves SECRT-2699

### Why / What / How

**Why:** the smoke e2e's "Create E2E test data" step keeps running 50+ minutes or timing out, which ejects PRs from the merge queue. The seed dump that should skip it lives in the Actions cache, and the cache almost never hits: 9 hits in 206 smoke jobs since 10-21. The repo's Actions cache is at its cap, so the ~36 MB dump is evicted within minutes; the key also changes on any `backend/**` edit; and a failed job never saves it. On a miss the seed embeds about 6,600 items through OpenAI (rate-limited); on a hit the step is skipped (~45 s).

**What:** the dump now lives in GHCR, as a single-file image at `ghcr.io/significant-gravitas/autogpt-platform-e2e-seed:<key>`, next to the e2e build cache the job already reads and writes. The key is narrowed to what changes the dump's content.

**How:** plain `docker` on the runner, with the GHCR login the job already does. No new tool, action or secret.

- **Key** (`Compute E2E test data cache key`, right after checkout): `v4-<emb|noemb>-` + sha256 of the git blob hashes of:
  - `backend/test/e2e_test_data.py`: the seed script itself
  - `backend/agents/calculator-agent.json`: the graph template it loads
  - `backend/schema.prisma` and `backend/migrations/`: the tables the data-only dump is inserted into
  - the text of the `Create E2E test data` step: the seed command and the `pg_dump` flags (not the whole workflow file, so other CI edits keep the key)
  - the embeddings on/off flag, in the tag prefix

  Other backend code (graph model, blocks, store/library code) no longer changes the key. That is the deliberate trade-off: the old `backend/**` key was why it almost never hit. A dump that no longer fits the schema can't break a run (see below), and `v4` can be bumped to force a fresh seed.
- **Restore:** after the GHCR login, `docker pull` the tag and `docker cp` the dump out (`continue-on-error`, 3-minute timeout). A missing tag, a pull error or no access (fork PRs) just means seeding as today. The load into Postgres now runs in one transaction (`--single-transaction`, still `ON_ERROR_STOP`), so a stale dump that fails leaves nothing behind, logs an `e2e-seed-cache-stale` warning and the seed step runs instead.
- **Save:** only where `WRITE_BUILD_CACHE` is true (smoke leg on dev pushes, merge_group, and `workflow_dispatch` with `publish_build_cache`); pull requests only read. It pushes only after a fresh seed has passed the credential check, the Playwright suite and its report validator. The seed script logs and skips some failed inserts, so a dump the suite didn't pass on is never published. Trade-off: a red or flaky run doesn't publish. With a key that rarely changes and no eviction, one green dev or merge-queue run per key is enough. `continue-on-error`, 5-minute timeout. The image carries `org.opencontainers.image.source` so the package links to this repo.
- The old Actions-cache step is removed rather than kept as a second fallback: it was evicted within minutes, so it adds a step for almost no hits.

### Changes 🏗️

- `.github/workflows/platform-fullstack-ci.yml`: the Actions-cache step for `/tmp/e2e_test_data.sql` is replaced by a key step, a GHCR pull step and a GHCR push step; the cached load is transactional and falls back to seeding. `WRITE_BUILD_CACHE` moves from the Prepare step's `env` to the job's `env` (same expression) so the push step can read it.
- `.github/scripts/test_e2e_seed_cache_contract.py`: new contract. The dump is in no Actions cache step; the key hashes exactly the files above and gives a stable, valid tag (the test runs the key script); the pull can't fail the job and runs after the GHCR login; the restore is one transaction and the seed runs when it didn't load; the push runs only under `WRITE_BUILD_CACHE`, after a successful seed, credential check, Playwright run and report validation. `test_seed_cache_covers_transitive_inputs` (which pinned `backend/**` in the key) is replaced by `test_seed_cache_key_covers_only_seed_inputs`, as described above.

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
  - [x] Contract tests written first: 6 of 14 failed against the unchanged workflow, 14 of 14 pass after the change (`python3 -m unittest test_e2e_seed_cache_contract`)
  - [x] actionlint 1.7.7 (checksum-verified): the same findings before and after (only the `background:` / `wait:` step keys it doesn't know yet), nothing new
  - [x] Round-trip on a tester VM, running the push, pull and load step scripts copied verbatim from this workflow, against a local registry and a throwaway Postgres (9/9 checks passed):
    - missing tag: pull exits 0, no hit, so the job seeds
    - unreachable registry: pull exits 0, no hit, so the job seeds
    - push: `docker import` + `docker push` works, and the image carries the `org.opencontainers.image.source` label
    - hit: pull gives a byte-identical dump, the load sets `restored=true`, and every row is back
    - stale dump (schema dropped a column the dump still inserts): the load fails, logs the `e2e-seed-cache-stale` warning, exits 0 without `restored=true`, and leaves 0 rows behind (rolled back), so the seed step runs
  - [ ] This PR's smoke leg runs read-only: it exercises the miss → seed path (the tag doesn't exist yet) and must stay green
  - [ ] After merge: the first dev push or merge_group run pushes the image, and the next smoke run restores it. The save path can't run on a PR, so it is only proven after merge.

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my changes
- [x] I have included a list of my configuration changes in the PR description (under **Changes**)

Co-authored-by: Bentlybro <[email protected]>
Co-authored-by: gpt-6-astra (Codex) <[email protected]>
 2026-10-22 18:13:45 -05:00
Reinier van der LeerClaude Opus 5.5gpt-6-astra
 		f47204a61e ci(platform): pull full-stack CI's service images with a retry that outlasts a Docker Hub refusal (#15978) 
Pre-pull the full-stack CI service images through docker-pull-with-retry.sh, excluding images produced by the build. Allow five attempts with 15, 30, 45 and 60 second waits so short registry refusals can recover.

Resolves #15977

Co-authored-by: Reinier van der Leer <[email protected]>
Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
Co-authored-by: gpt-6-astra (Codex) <[email protected]>
 2026-10-22 15:30:06 -05:00
Nicholas TindleandClaude Opus 5.5 7fafc428b0 feat(platform): require a shared key on calls between the backend's internal services (#15949) 
### Why / What / How

**Why:** The backend's internal services (database manager, scheduler,
notifications, platform linking, copilot chat bridge, swap service) call
each other over HTTP. This makes every one of those calls carry a shared
key and every service require it, so being able to reach a service's
port is no longer enough to call it.

**What:**
- Every `AppService` refuses any request without `Authorization: Bearer
<key>`. Only `/health_check`, `/health_check_async` and `/metrics` stay
open.
- Every RPC client sends the key: `get_service_client`, and the swap
proxy's own client.
- Setup generates the key along with the other secrets: `make init-env`,
both installers, and the single-container image.
- Docker Compose no longer publishes the internal services' ports on the
host.

**How:**
- `backend/util/service_auth.py` adds an ASGI middleware, outermost on
every `AppService` app:
- it reads the bearer token from the raw headers before anything reads
the body, and compares it in constant time;
- a refusal is a 401 that also closes the connection, and a websocket is
closed with 1008;
  - a service with no key at all refuses every call.
- The key is `INTERNAL_API_KEY`. `backend/.env.default` ships it blank
and `SECRET_GENERATORS` fills it, like `ENCRYPTION_KEY`.
- An install set up before the key existed has no line for it. There the
key is derived from `ENCRYPTION_KEY` (HMAC-SHA256 over the key's bytes
with a fixed label), which every service of a deployment already shares,
so it enforces too with nothing new to configure.
- The single-container image keeps the key in its runtime config, which
moves to version 2. A version 1 config gets a key on the new image's
first boot. It is saved only once it passes every check, so a boot that
is refused leaves the volume as it was.
- Every process checks the key at start, and stops on a malformed
`INTERNAL_API_KEY`. When the key is derived, it also stops on an
`ENCRYPTION_KEY` that Fernet would refuse.
- The RPC clients ignore proxy settings in the environment
(`trust_env=False`), so these plain-HTTP calls and their key never go
through an outbound proxy.

### Changes 🏗️

- `backend/util/service_auth.py` (new): the key and the middleware.
- `backend/util/service.py`: installs the middleware on every
`AppService`. The RPC clients send the key and use `trust_env=False`.
- `backend/util/process.py`: the key check at process start.
- `backend/util/settings.py`: `INTERNAL_API_KEY`.
- `backend/.env.default`: a blank `INTERNAL_API_KEY=` for setup to fill.
- `single-container/runtime_config.py`:
- the generator, and the key added to the settings `.env.default` must
ship blank;
  - the runtime config's version 2, and the upgrade from version 1.
- `single-container/common.sh`: the runtime config loader accepts the
key.
- `swap_proxy`: sends the key, refuses to start without one, and uses
`trust_env=False`. Its README lists the setting.
- `docker-compose.platform.yml`: `ports:` becomes `expose:` for the
executor (8002), scheduler (8003), database manager (8005),
notifications (8007), copilot executor (8008) and platform linking
(8009).
- CI:
- `.github/workflows/platform-backend-ci.yml` generates a key per run,
like the other secrets.
  - `.oss-scanner/Dockerfile` does the same per build.
- `.github/workflows/platform-fullstack-ci.yml`: the scheduler readiness
probe now runs inside its container.
- Tests:
  - `backend/util/service_auth_test.py` (new);
- `single-container/tests/test_compose_internal_ports.py` (new): no
Compose file publishes an internal port, in any syntax;
- `single-container/tests/test_runtime_config.py`: the generated key and
the version 1 upgrade;
- the frontend environment test and the single-container smoke test
check that the frontend never gets the key;
  - the swap service and service tests send the key.
- Docs: the README, `getting-started.md` and `installer.md` list the key
among the generated secrets, as do the installers' comments.
`single-container.md` covers it too.
- Port notes updated in `.claude/skills/pr-test/SKILL.md` and
`frontend/src/playwright/journeys/README.md`.

**Configuration changes:**
- New `INTERNAL_API_KEY`.
- `make init-env` and the installers generate it into `backend/.env`.
Every service of one deployment must have the same value.
- Left unset (an existing `.env` has no line for it), the key is derived
from `ENCRYPTION_KEY`, which must then be a Fernet key, as `make
init-env` generates.
- A deployment that runs the swap proxy sets it explicitly, on the
backend and the proxy.
- The single-container runtime config moves to version 2 and gains
`INTERNAL_API_KEY`. As the docs already say, don't start an older image
on a volume a newer one has started on.
- Compose no longer publishes ports 8002, 8003, 8005, 8007, 8008 and
8009 on the host. The containers still reach each other over the Compose
network.
- A deploy that replaces services running an earlier release fails the
calls those still make. The services on this release refuse them, since
they carry no key.

### Agents and large language models used

- Claude Code with Claude Opus 5.5: implementation and tests.
- Codex with Daybreak Blue (gpt-daybreak-blue-latest): read-only
security review.

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan, on this
branch merged with current `dev`:
  - [x] `service_auth_test.py`, including:
    - real uvicorn sockets: the sync and async clients send the key;
    - a client with another key is refused without retries;
    - an outbound proxy in the environment never sees a call;
    - a refused request's body is never read.
- [x] The backend tests on Linux with real Redis, with a derived key and
with an explicit one (109 each).
- [x] The single-container suite (187) and the Compose check. The
Compose check fails closed: a `ports:` entry or key spelling it can't
read fails it.
- [x] The swap proxy's own CI job on Linux (Python 3.13): isort, black,
ruff, pyright and all 373 tests.
- [x] Every control is mutation-tested: one mutant per control, 62 in
all, each failing its test. That's 21 backend, 7 swap proxy, 18 Compose
check, and 16 single-container (the key's generation and the version 1
upgrade).
  - [x] On the repo's own Compose stack:
    - the internal port is not reachable from the host;
- inside the network, a call without the key or with a wrong one is
refused, and one with the key is served;
    - the backend's own RPC clients are authorized;
- an `ENCRYPTION_KEY` that is not a Fernet key stops each service at
start;
- with `backend/.env` made the way `make init-env` makes it, the
services require the generated key, and an install with no
`INTERNAL_API_KEY` line works on the derived one.

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
 2026-10-22 12:51:27 +00:00
Nicholas TindleandClaude Opus 5.5 ec6ab8b03e feat(backend/copilot): send the browser tools' traffic through an egress proxy (#15859) 
### Why / What / How

**Why:** The Copilot browser tools (`browser_navigate`, `browser_act`,
`browser_screenshot`) drive headless Chromium through the
`agent-browser` CLI. Until now only the URL handed to a tool was checked
against the backend's outbound address rules. Everything Chromium
fetched after that (redirects, links, scripts, page resources) went out
directly, without those rules.

**What:** Every connection Chromium opens now goes through the same
address rules the backend's HTTP client applies, so the browser can
reach public sites and nothing else.

**How:** A small in-process SOCKS5 proxy sits between Chromium and the
network. For each connection it resolves the host, accepts it only when
every address is public, and connects to the address it checked. The
browser is configured so it has no route that skips the proxy.

### Changes 🏗️

- **`backend/util/egress_proxy.py` (new):** the loopback SOCKS5 proxy
described above.
- It reuses `resolve_and_check_blocked` from `util/request.py`, then
also requires `is_global`, so IPv6 forms that carry an IPv4 address are
refused.
  - It connects over IPv4 only.
  - It runs one per process, on its own thread and event loop.
- **`copilot/tools/agent_browser.py`:** every `agent-browser` call
starts Chromium:
- with the proxy, including for `localhost` (Chromium otherwise bypasses
a proxy for loopback);
  - with QUIC off;
- with an empty `agent-browser` config file, so no config is picked up
from the home or working directory;
- with a minimal environment, so `AGENT_BROWSER_*` and proxy variables
from the host don't change how the browser is launched.

Arguments that look like `agent-browser` options are refused, because
the CLI reads its options from anywhere on the command line.
- **`backend/Dockerfile`:** a managed Chromium policy keeps WebRTC on
proxied transports. The equivalent command-line switch isn't honoured.
- **Tests:**
  - Unit tests for the proxy and the launch options.
- Real-browser integration tests: they run where `agent-browser` is
installed and are skipped in CI like the existing ones, so their names
are added to `backend-allowed-skips.json`.
- **Configuration changes:** none. No new settings or environment
variables. The proxy binds an ephemeral port on 127.0.0.1.

### Agents and large language models used

- Claude Code with Claude Opus 5.5: implementation and tests.
- Codex with Daybreak Blue (`gpt-daybreak-blue-latest`): independent
security review; its three findings are fixed in this PR.

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Unit tests: 166 pass for `agent_browser_test.py` and
`egress_proxy_test.py`; lint and pyright are clean.
- [x] Real-browser integration tests in the backend image (agent-browser
0.37.1, Chromium 155, Python 3.13):
    - 8/8 egress tests pass on this head;
- with the launch options, config pin, environment cleanup and Chromium
policy removed, all 8 fail;
    - removing any single control fails only the test written for it.
- [x] Existing browser behaviour still works through the proxy: the
tool-level integration tests navigate a public site, scroll, and fill a
form (19/19 pass on this head, including the existing URL-check tests).
  - [ ] GitHub Actions on the PR head

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
 2026-10-21 19:36:54 +00:00
Nicholas Tindleandgpt-6-astra 37899b0645 refactor(platform): retire mature product flags (#15799) 
### Why / What / How

Retires nine mature product rollout flags so files, experts, skills,
onboarding, memory and community maintenance, spend approvals, and
execution summaries are available independently of flag-provider
configuration. Removes the obsolete variants and disabled branches while
preserving authentication, ownership, user preferences, payment
configuration, and the independent dream, skill-learning, voice, and
payment controls.

Supersedes #15766. Sidebar and chat-search retirement already landed in
#15106 and is included in the `dev` base.

### Changes 🏗️

- Retire `artifacts-page`, `hire-experts`, `skills-hub`,
`onboarding-brain-dump`, `onboarding-expert-team`, `graphiti-memory`,
`graphiti-communities-enabled`, `expert-spend-approval`, and
`ai-agent-execution-summary`.
- Remove the old onboarding and marketplace variants; update wizard
handoff, intro/dialog coordination, browser-test helpers, stale rollout
comments, and dead test mocks.
- Preserve memory scope/lifecycle safeguards. Register community
maintenance without a provider decision while retaining optional-job
intent when dream/learning evaluation is unavailable.
`ENABLE_CREDIT=false` bypasses spend approval.
- Read skill-write configuration once per batch now that hiring always
installs bundled skills. Authorization, current quota, scanning, and
publication checks still run per package; a later batch reads fresh
configuration.
- Cover setup-turn replies with the real chat SDK: a pending-message 409
waits for the existing stream to settle, including a delayed resume
handshake. This adds regression coverage; it does not claim to fix the
reported live race.
- Regenerate OpenAPI, remove retired environment/CI overrides, document
memory service/model requirements, and retain the Tuesday audit prompt.
Provider definitions remain for older releases. No migrations or
dependency changes relative to `dev`.

### Agents and large language models used

Codex with gpt-6-astra, including implementation and independent review
subagents.

### Checklist 📋

#### For code changes:

- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [ ] I have tested my changes according to the test plan:
- [x] Current merge `3992fc46b5` includes `dev` at `379862d327` (#15850
Postmark/PostHog analytics and #15851 in-app DataFast removal). The sole
conflict was the onboarding tracking test: it now preserves PostHog
assertions with the simplified layout API. All other incoming code
remains intact; user lookup and webhook attribution/authentication
checks are preserved.
- [x] Current merge validation: 203 backend tests and 186 frontend tests
across 20 files passed. Full frontend types, scoped
Prettier/ESLint/oxlint, and 15-file backend Black/isort/Pyright passed.
Backend Ruff findings are in files identical to dev. The Windows `pnpm
format` script still fails on `--fix;`; repository-wide frontend lint
reaches its existing Prettier/CRLF failures. No broad formatting,
dependency installation, live-provider or shared-service changes were
made.
- [x] Helper follow-up `59f1b1ec27` guards intro-dialog button lookup
when login navigation replaces its document. The prior smoke run passed
30 tests and failed the settings test on Playwright's cross-document
adoption error in this helper. Two lookup regressions failed before the
fix; all 14 onboarding/signup helper tests now pass, along with full
frontend types and scoped Prettier/ESLint/oxlint. Unexpected errors
still propagate and acquired handles are released. This is a test-helper
fix; no application behavior changed. Fresh browser CI remains pending.
- [x] Dev merge `6392cc1282` includes `dev` at `3fcf47bb2e` and security
fix #15864: certifi 2026.10.19, Pillow 13.0.0, the pinned MoviePy
revision, and Python >=3.11. Dependency files and image-scan wiring
match dev.
- [x] Validation of dev merge `6392cc1282`: 433 focused backend tests
passed, 2 skipped; 264 frontend tests passed across 18 files. Full
frontend types and scoped Prettier/ESLint/oxlint passed. All 17 changed
Python files passed Black/isort/Pyright; Ruff introduced no findings.
Poetry lock validation passed.
- [ ] Local validation limits: cloud storage had 39 passes and 15
Windows path-normalization failures in source/tests identical to dev;
full billing/subscription groups stopped on Redis retries. Local Python
tests used the existing private dependencies, so fresh CI validates the
upgraded image stack. No shared services or dependency installations
were used.
- [x] Earlier merge `fd88aaf2d6` includes `dev` at `a689834cd3`,
including the related PostHog cache fix #15765 and Sentry URL redaction.
Backend integration: 326 passed, 2 skipped. Frontend: 136 passed in the
seven-file run; one first-import timeout passed on an isolated 28-test
file rerun. Full TypeScript and scoped Prettier/ESLint/oxlint pass. No
additional runtime edits were needed for the merge.
- [x] Earlier follow-up head `4426611a57` included `dev` at
`6c07c09d4c`; its validation follows below.
- [x] Backend flag/memory-tool regressions: 261 passed, 2 skipped.
Warm-context/retry regressions: 47 passed.
- [x] Skill/workspace/storage regressions: 213 passed; six Windows
path/chmod failures reproduced with unchanged implementation and
excluded. The new real-write-path regression fails before the fix (six
configuration loads for three skills) and passes afterward (one load).
Quota changes during a batch and configuration changes between batches
remain effective.
- [x] Frontend follow-ups: 114 passed across 12 files; full TypeScript
check passes. Focused Prettier, ESLint, and oxlint pass. Whole-tree
formatting still reports existing CRLF noise; no broad formatting
changes included.
- [x] New setup-turn tests fail when the existing settle wait is removed
and reproduce the reported SDK `undefined.state` exception. Both pass
with the guard intact; production chat code remains unchanged. The
original live report remains unconfirmed.
- [x] Changed backend formatting/type checks pass; changed-file Ruff
comparison introduces no findings. No live providers or shared services
used for these follow-ups.
- [x] CI on `4426611a57` confirmed the exact roster/hire acceptance test
passes: Python 3.11 56.164 s, 3.12 48.425 s, 3.13 27.535 s. These are
not controlled comparisons with the reviewer's VM timings.
- [x] Both complete-image smoke tests on `4426611a57` passed: ARM64
406.84 s and AMD64 612.84 s. All 14 Codecov checks passed with 99.21%
overall patch coverage.
- [ ] Fresh complete-image security validation on `3992fc46b5`. The
inherited certifi/Pillow failures are addressed by merged #15864, whose
AMD64 and ARM64 build/smoke/scan checks passed. This PR now includes
that exact dependency update; its fresh checks are pending.
- [ ] Fresh CI on `3992fc46b5`; live PostgreSQL/FalkorDB checks were not
run locally for these follow-ups. On previous head `fd88aaf2d6`,
test/coverage checks passed with 99.21% patch coverage; both image smoke
tests passed, while security validation failed on the now-updated
dependencies.

<details>
<summary>Earlier validation before this follow-up</summary>

Memory/dream: 1,211 passed, 4 optional PostgreSQL skips, 102 integration
exclusions, 1 existing xfail. Product-backend unit collection: 836
passed across the original run and fixture rerun; four additional
real-SDK thread-billing cases passed with a fake provider.
Hosted-journey configuration, flag-sync, and documentation: 15 passed.
Frontend flag, welcome, marketplace, sidebar, and layout groups passed
their focused reruns, including 244 tests after the earlier dev merge.
These groups overlap and were run on earlier heads; the current-head
checks are listed above.

</details>

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

---------

Co-authored-by: gpt-6-astra (Codex) <[email protected]>
 2026-10-21 17:08:56 +00:00
Reinier van der LeerandClaude Opus 5.5 e0217702d0 fix(platform/copilot): a burst of approval answers runs in one turn that is never dropped (#15062) 
### Why / What / How

When several approval cards wait in a chat, answering them now starts
one turn rather than one per answer, and the chat's approval box keeps
the answers, with Close and continue, across a reload. That turn is
never dropped and left running, and what an approved call returns is
kept even when the turn is stopped before it is saved.

**Why:** two defects in the turn that runs a chat's answered approvals.
- On an idle chat, every answer started that turn at once, carrying only
what was answered so far. Three cards answered one by one could start
three turns: each billed, each taking a running slot, the chat
restarting between clicks.
- On prod on 2026-09-30, a user answered three cards while a turn ran.
The follow-up turn was started from inside the ending turn, while it
still sat in the executor's `active_tasks`. The queue handed it to the
same pod, which dropped it as "already running locally, rejecting
duplicate" without closing it. It stayed `running` in Redis: the chat
showed Stop, typed messages went into the pending buffer, every reload
waited on an empty stream until "Load failed", and nothing the user
approved ran. The same Dev session also showed a follow-up turn starting
over an open question and swallowing its answer, and an approved call
running for 15 s with only "Preparing conversation context…" on screen.

**What:**
- **One turn for a burst of answers.** In the chat's own "Waiting for
you" box, an answer starts no turn while another card in the chat still
waits. /home, the Expert page and the run page cannot continue a chat,
so an answer there starts the turn at once, as before. The last answer
starts one turn that carries them all. The "Waiting for you" box gains
two controls:
- "Review all" opens the review dialog that /home and the Expert page
use, over this chat's cards.
- "Close and continue" starts the turn with the cards answered so far,
and so does closing that dialog part-way. The unanswered cards stay
open, the turn is told how many still wait, and the box folds to its
header. Closing with nothing answered sends nothing.
- After a reload, or on coming back to the chat, the box still shows the
cards answered there as answered, with Close and continue, until their
turn runs them.
- **Channels wait 3 s.** A linked Slack, Discord, Telegram or Teams chat
has no Continue. A channel answer starts the turn 3 s after the most
recent answer, or at once on the chat's last card. Answers during a
running turn are unchanged: that turn's end starts the next one.
- **Wake after the turn leaves.** Work a turn wants to run after it ends
waits for the executor's done-callback, where the engine-switch
continuation already ran. The held-call wake now goes the same way. The
continuation runs first, and the wake then finds the chat running and
leaves the cards to that turn.
- **A dropped turn is closed.** When the executor rejects a delivery as
a local duplicate of a *different* turn, it closes that turn with an
error. So does a message it cannot parse, when the message still names
its session and turn. A message naming no turn closes nothing: without
the id the close has no guard and would end the turn the chat is
running. A redelivery of the running turn is left alone. A turn's end
also frees its cluster and stream locks before its turn slot, so a turn
admitted in between cannot meet the old lock on another pod.
- **No wake over an open question.** The wake does nothing while the
chat has a `pending_question`; the answer's turn runs the answered cards
first.
- **The approved call is visible.** Just before an approved call runs,
both engines stream "Running the action you approved: <the card's
headline>". Declined, expired and lost cards say nothing, nor does a
call held without its usage origin (#15208), which asks for a resend.
- **A stopped turn keeps what already ran.** A call's approval is spent
when it runs. A turn closed or stopped after one card's call ran, but
before the turn saved its result, now saves the result on the way out,
in both engines. The card it stopped at stays held for the next turn.
The baseline engine also ends the counted turn on its sandbox there.

**How:**
- `held.wake(…, until_last=True)` is the wake of an answer from the
chat's own box, which sends `wait_for_other_cards` on the review
request, and of the channel button. Other callers send nothing and wake
without it. It counts the chat's waiting held-call cards, read from
Postgres. A workflow step's card (#15564) is not counted: its answer
resumes its run rather than the chat, and it can leave by the run page
or a stopped run, neither of which wakes the chat. On an idle chat it
returns False while any remain, and the route returns that as
`awaiting_other_cards`, so the page follows a turn only when one is
coming. A running turn's end calls `wake` without it. `POST
/api/review/session/{id}/continue` does the same for Close and continue,
behind the payment paywall as `/action` is. Either way the wake row
reads "… N more still wait for my answer."
- `held.answered_cards` lists the chat's answered cards whose held calls
have not run, and `GET /api/review/session/{id}/answered` returns them
as review rows. The chat page fetches that list fresh on every mount,
never from a cache, and the box seeds its receipts from it once, so a
remount draws the same receipts and Close and continue as before.
- No timer on the web: the trigger is state. Every answer, every turn
end and Continue re-evaluates it, and the existing `uuid5` wake id makes
a repeat a no-op. A wake lost between the last answer and its dispatch
starts on the next evaluation.
- The channel delay is `Scheduler.schedule_held_wake`: an APScheduler
date job `held_wake_<chat>` in the Postgres `EXECUTION` jobstore with
`replace_existing`. Each answer moves the one job, and it runs after a
scheduler restart however late, since the scheduler's job defaults set
no misfire grace. If the scheduler cannot be reached, the channel wakes
at once.
- Frontend: `useChatReview` adapts the chat's own queue to /home's
`HeldReviewDialog` and `useReviewCarousel`, so the box and the dialog
share one set of answers. Those two now take only the fields they read
(`Pick<…>`), and `chatLink={false}` hides the dialog's "Open chat".
While the dialog is open the box does not fade out once its last card is
answered (#15051), so the dialog's "All N reviewed" keeps its answers;
it leaves a second after the dialog closes. /home and the Expert page
are unchanged.
- `copilot/after_turn.py` is a process-local registry keyed by turn id:
the manager registers each turn before submitting it, its done-callback
runs the deferred work, and `run_after_turn` runs it at once outside a
registered turn (the API's stale-session path). The engine-switch
dispatch and the deferred work share one `asyncio.run` on the after-turn
thread, because the thread-cached queue client is bound to one loop.
- The manager records each running session's turn id, so the duplicate
branch can tell a redelivery from a different turn, and closes the
dropped one through `mark_session_completed(turn_id=…)`.
- `resolve_answered` is an async generator: a pure `_due` decides from
the card's row and the call's usage origin whether its call runs, the
status is yielded before the HDEL claims the card (a turn closed at that
moment leaves it for the next turn), and nothing suspends between the
claim and the run.
- A delivered result waits in memory until the turn folds it in, which
the SDK engine does after it has wrapped the user's message. Each engine
keeps the results it has not saved yet, and every exit saves them with
`persist_unfolded_rows`, which empties the list so only the first exit
saves.

**Coordination:** the browser side of the same chained turn is #15044;
its files are not touched here.

### Screenshots

**BEFORE: one of three cards answered (on dev this answer has already
started a turn)**

![before, one
answered](https://github.com/user-attachments/assets/5fbd4d30-3326-4469-9995-d0fd2e46d6b9)

**AFTER: one of three answered. No turn yet; "Review all" and "Close and
continue" are new**

![after, one
answered](https://github.com/user-attachments/assets/90d3e1bc-4ec2-4a9b-874d-a6949377af7b)

**AFTER: "Review all" opens /home's dialog over this chat's cards**

![after, review
dialog](https://github.com/user-attachments/assets/793c7460-c197-42e0-a253-e1d60fbb3a03)

**AFTER: after "Close and continue", the box folds to its header**

![after,
folded](https://github.com/user-attachments/assets/43cddf62-3a9d-4629-be59-37396c29befd)

### Changes 🏗️

- `copilot/gate/held.py`: `wake(until_last=…)` waits for the chat's last
card and reports it; the wake row names how many still wait; `wake`
skips while a question is open; `resolve_answered` yields a status
before each approved run; `answered_cards` lists the answered cards
still to run.
- `copilot/gate/channel.py`: a channel answer schedules the delayed
wake; `executor/scheduler.py`: `schedule_held_wake` and
`execute_held_wake`.
- `graph_executions/review/routes.py` and `model.py`:
`wait_for_other_cards` on the review request and `awaiting_other_cards`
on its response; `POST /review/session/{id}/continue`; `GET
/review/session/{id}/answered`.
- Frontend: `useProcessReviews` takes `waitForOtherCards`, which only
`useApprovalQueue` sets; `ApprovalQueue` (Review all, Close and
continue, folded header; `RejectAllFooter` became `QueueFooter`),
`useApprovalQueue` (seeds its receipts from the answered list), new
`useChatReview`, `CopilotPendingReviews` and `useCopilotPendingReviews`
(follow a turn only when one comes; fetch the answered list on mount);
`HeldReviewDialog` and `useReviewCarousel` types and `chatLink`. The
test mocks answer the new route with no rows unless a test sets some.
- `copilot/after_turn.py` (new), `executor/manager.py`,
`stream_registry.py`, `sdk/service.py`, `baseline/service.py`: the
after-turn wake, the dropped-turn close, the lock order and the streamed
status.
- `pending_message_helpers.py`: `persist_unfolded_rows`, which both
engines call on a turn's exit for results it ran but did not save.
`pending_messages.py`: a held result that a rolled-back save re-queues
drains as a `HeldResult`, keeping its larger cap.
- Tests: `executor/turn_handoff_test.py` and
`CopilotPendingReviews.batch.test.tsx` (new); new cases in
`gate/held_db_test.py`, `gate/held_engines_test.py`,
`bot/card_buttons_db_test.py`, `review_routes_test.py`,
`executor/jobstore_test.py`, `stream_registry_test.py`,
`executor/manager_test.py` and
`ApprovalQueue/__tests__/leaving.test.tsx`; the stubs moved to the
generator API. #15060's strict xfail on
`drift_causes_test.py::test_a_turn_the_executor_drops_is_not_left_running`
is removed with its `backend-allowed-skips.json` entry, because this PR
fixes what it reproduces.

### Agents and large language models used

- Claude Code with Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Three answers on an idle chat start one turn, at the third,
carrying all three; Close and continue starts it with one answered and
the other still waiting; an answer during a running turn is left to that
turn's end; a wake lost after the last answer starts on the next one
- [x] The answer route passes `until_last` and returns
`awaiting_other_cards`; Continue wakes without it
- [x] Channel clicks: a burst schedules one moved wake and starts
nothing; the last card starts its turn at once; an unreachable scheduler
wakes at once
- [x] The delayed wake is one job per chat, and a fresh scheduler on the
same store runs it after the first was lost
- [x] Page level: no turn is followed until the last answer; Close and
continue sends once and folds the box; the dialog's last verdict follows
the turn and Done sends nothing more; closing it part-way continues;
closing it with nothing answered sends nothing
- [x] Reload: an answered card waiting on the chat's others is listed
until its call runs, and a remounted box shows it as answered, with
Close and continue
- [x] The hand-off, dropped-turn, open-question, status and lock-order
cases listed under Verified

**Verified:** ran locally against Postgres and Redis. The batching is
covered by new tests in `gate/held_db_test.py`,
`bot/card_buttons_db_test.py`, `review_routes_test.py` and
`executor/jobstore_test.py`, and on the frontend by
`CopilotPendingReviews.batch.test.tsx`. 21 mutations each turned one of
them red: 15 on the backend (the wait, the running-turn exemption, the
still-waiting note, a fired-marker, the route flag and its kwarg,
Continue, the channel's per-click wake, its delay, its fallback and its
3 s, the job store, the job id, the reschedule and the job's wake) and 6
on the frontend. A stopped turn was checked in each engine, driven
through its real entry point: closed at the next card's status, the
first call's result is saved; cancelled before the fold, both are. A
burst of three answers sent through the review route against Postgres,
while a workflow step's card waited in the chat, starts one turn. An
answer from /home with another card waiting starts its turn at once,
through the route against Postgres. Answering one of two cards through
the route lists it as answered, and the waiting one as waiting, until
its call runs; a box remounted after that answer shows the receipt with
Close and continue, and Continue reaches the server. Each of those two
tests goes red with its half removed. On the final head, the gate,
card-button, hand-off and review-route suites, the paywall route table,
`util/architecture_test.py` and `blocks/test/test_block.py` gave 4,700
passed, and the `copilot`, `home` and `hooks` frontend suites 4,307
passed; all of `backend/copilot` with those gave 14,668 passed on the
previous head, `8f5bdef8e4`. The hand-off fix was verified as before:
one real `CopilotExecutor` hands its queued turns back to itself, both
hand-off tests fail on dev's `stream_registry.py` and `manager.py`, and
ten mutations each turn one red. Mutation lists and screenshots are in
comments. I did not run it against a deployed environment, a real
channel or the scheduler service.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
 2026-10-20 15:27:50 +00:00
Nicholas Tindleandgpt-6-astra ec951d81c6 fix(backend): preserve concrete memory details in Luna extraction (#15677) 
### Why / What / How

Luna's Graphiti extraction can drop concrete details embedded in
requests, leaving recent episode text as their only retrievable copy.
The pinned upstream system prompt also forbids abstract concepts while
our custom instructions require specific Concept, Preference, and Rule
entities.

Replace that conflicting system policy for Luna's four known entity/edge
extraction contracts. Retain concrete values, qualifiers, request
status, and subject-qualified relationship anchors while preserving the
upstream source context, schema, language handling, and retries. Model
routing and request parameters are unchanged.

Follow-up to #15656; related to #15622. The recall issue #15655 and the
shared fact-only forgetting failure documented in the report remain
separate.

The live AgentProbe memory-component comparison found:

| Configuration | Normal memory checks | Fact-only checks | Complete
normal probes |
| --- | ---: | ---: | ---: |
| Mini | 132/137 | 104/137 | 31/34 |
| Luna, original prompts | 129/137 | 79/137 | 30/34 |
| Luna, migrated prompts | 131/137 | 125/137 | 31/34 |

No normal/fact-only check passed with original Luna and failed with the
candidate in this run. Ten additional controls scored 61/61 with both
normal and fact-only memory for the candidate. Normal answer quality
does **not** establish strict parity with mini: pricing abstention and
shipping arithmetic differences remain despite the necessary facts being
retrieved. The full report retains raw scores, exclusions, sensitivity
checks, latency/cost tradeoffs, and provider-limit failures. A later
unvalidated wording experiment was reverted.

[Evaluation report and per-probe
evidence](https://github.com/Significant-Gravitas/AutoGPT/blob/codex/graphiti-luna-prompts-15622/docs/engineering/graphiti-luna-prompt-evaluation.md).
The evidence supports improvement over the Luna configuration on dev,
not a blanket no-regression certification or a full-chat benchmark.

Migration references: [prompt auditing and model-specific
evaluation](https://developers.openai.com/api/docs/guides/latest-model#prompting-best-practices),
[request
compatibility](https://developers.openai.com/api/docs/guides/latest-model#migration-quickstart).

### Changes 🏗️

- Add a scoped extraction policy for message, JSON, text, and edge
extraction on Luna, including Flex.
- Preserve explicit other models, non-extraction tasks, attribute
extraction, retry feedback, and source messages.
- Add dispatcher/client regressions, a guard exercising upstream
extraction entry points, and an opt-in paid three-source regression with
coherent per-fact assertions, exact payment values, and
scenario-specific checks for unsupported signing and company-attribution
claims. Register only those three intentional paid-test skips in the CI
policy.
- Share bounded assertion helpers between offline and paid tests without
importing model or database clients into the offline test module.
- Record the mini/original-Luna/migrated-Luna comparison and remaining
quality gaps. No configuration or schema changes.

### Agents and large language models used

- Codex through T3 Code — gpt-6-astra (implementation and review).
- gpt-4.1-mini and gpt-6-luna — live extraction/ranking evaluation;
gpt-6-luna was the fixed answerer.

### Checklist 📋

#### For code changes:

- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Graphiti non-integration suite after review fixes: 398 passed, one
pre-existing xfail. Thirty assertion false passes and two
coordinated-subject false rejections reproduced across these follow-ups;
retaining the upstream system preamble is also rejected by a checked
negative control.
- [x] Current assertion suite: 65 passed with 100% statement coverage of
the shared helper, three paid cases skipped; includes clause-bound
payment values and signing instructions, explicit counterparty detail
claims, and valid wording controls including negated instructions with
coordinated subjects. Four saved migrated-Luna captures of the
incidental-detail scenario pass offline replay with the tightened checks
(no new model calls).
- [x] Initial focused client/prompt suite: 65 passed; 100% statement
coverage of the prompt dispatcher and adapter. Three paid tests skip by
default.
- [x] Initial opt-in live regression: message, JSON, and text ingestion
all passed against an isolated FalkorDB; six durable details inspected.
This run predates the stricter per-fact assertions.
- [x] Live 31-scenario, 35-probe comparison with
normal/fact-only/memory-off controls, plus ten additional synthetic
cases.
- [x] 40-candidate reranking, three runs per model: Luna recall@4 1.0,
mini 0.5; all completed within eight seconds.
- [x] Native Black/isort/Ruff and scoped Pyright; secret scans and diff
checks passed.
- [x] Revalidated all twelve downloaded CI reports against the corrected
exact skip allowlist for Python 3.11/3.12/3.13.
- [ ] Rerun the tightened paid assertions after the provider daily quota
is available. Production prompt strings and request behavior are
unchanged by the review follow-ups.
- [x] Hosted CI on b729e9b: all 52 checks passed, one skipped, including
the required Check PR Status; backend patch coverage is 93.82%. Native
scoped checks also passed. The full local pre-commit run remains
unavailable because Bash hooks hit Windows/WSL worktree/tool-resolution
errors, the isort hook's Poetry option is unsupported locally, and the
unscoped typecheck scanned Classic with missing dependencies.
- [ ] Full application chat/tool/Dream E2E and repeated
production-representative no-regression evaluation remain unrun.

<details>
  <summary>Test plan scope and reproduction</summary>

Run `poetry run pytest
backend/copilot/graphiti/extraction_prompts_test.py
backend/copilot/graphiti/llm_client_test.py
backend/copilot/graphiti/flex_client_test.py` from the backend
directory. For the paid regression, configure a test FalkorDB and
model/embedding credentials, set `RUN_GRAPHITI_LLM_EVAL=1` and
`GRAPHITI_LLM_MODEL=gpt-6-luna`, and run `poetry run pytest
backend/copilot/graphiti/extraction_quality_integration_test.py -q`. The
fixture uses a fresh graph. The full component evaluation's methodology,
raw/adjudicated counts, and limits are in the report.

</details>

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

<details>
  <summary>Configuration compatibility</summary>

No deployment configuration changes. `RUN_GRAPHITI_LLM_EVAL` opts into
paid tests only; it is not a runtime setting.

</details>

Co-authored-by: gpt-6-astra (Codex) <[email protected]>

---------

Co-authored-by: gpt-6-astra (Codex) <[email protected]>
 2026-10-19 17:32:29 +00:00
Toran Bruce RichardsGPT-6ClaudeNicholas TindleNicholas Tindle
 		23f10ce008 feat(platform): redesign usage limits and review upgrade charges (#15215) 
Resolves SECRT-2865

### Why / What / How

When users exhaust their allowance, they need to understand when they
can continue and what choosing a paid plan will charge. This change
connects plan-specific limit notices, an editable saved draft, billing
settings and explicit payment reviews.

Active Pro trials now offer **Start Pro now**, show the agreed price and
scheduled start date, and make waiting an equally clear choice. The
payment review explains that starting today ends the trial early, shows
the actual invoice and next payment date, and only confirms fresh usage
after authoritative activation and usage refresh. Paid Pro-to-Max
upgrades show a Stripe proration quote before consent and reuse its
proration date when applying the change.

When a trial's first charge fails, checkout refuses a new plan while the
trial subscription is still open, so Start Pro there could only end in
an error. That state now pays the open trial invoice instead: the trial
status returns the amount and Stripe's hosted invoice link, and billing,
the limit popup, the in-chat notice and the paywall show one **Pay $X
now** action. Paying converts the trial through the existing
invoice-paid path.

**Stack:** #15184 → #15208 → this PR. The direct base is
`codex/reset-usage-on-pro-activation`, rebased onto its latest parent
(`c70264fa5`). The parent owns the allowance/reset behavior; this PR
owns its UI and the billing-review disclosures.

### Changes 🏗️

- Consistent trial start date, price and action across the popup,
neutral in-chat notice and billing offer. Legacy rolling trials retain
their actual policy.
- Canceled trials keep the parent’s remaining access and explicit plan
choices. Limit notices no longer promise automatic Pro activation after
cancellation; checkout reconciliation suppresses duplicate purchase
offers.
- Precise daily/weekly reset messages and local dates. Max, Team and
Enterprise have appropriate capacity/support actions. Inactive plans,
failed payments and unavailable usage have explicit recovery actions.
- Trial status adds `payment_due` (`amount_due`, `currency` and
`payment_url`, Stripe's hosted invoice page). It is only filled for an
unconverted trial whose status is `past_due` or `unpaid`, from the trial
subscription's latest invoice, when that invoice is open with an amount
due and the subscription, its `trial_enrollment_id` metadata and the
invoice customer all match the trial. A Stripe error or an unexpected
payload leaves it empty and never fails the status request.
- A failed first trial charge gets one message with the amount and one
**Pay $X now** button on billing, in the limit popup, in the in-chat
notice and in the paywall. The button opens the hosted invoice page in a
new tab, which takes any enabled payment method and handles 3D Secure.
Start Pro and the plan choices are hidden in this state because checkout
refuses a new plan while the trial subscription is open
(`ensure_no_unconverted_trial`). While the message shows, trial status
is rechecked every 15 seconds in a visible tab, and plan and usage
refresh once the trial converts. Without a readable invoice, billing
points to the payment method row and the popup keeps **Review billing
details**.
- Usage status follows the meter: Plenty left, Running low and Allowance
reached. Exhausted meters use orange. Plan choices describe product
benefits and preserve the derived annual-saving label.
- Below 80% of the trial allowance, where the meter switches to Running
low, the billing offer is a plain card that keeps the Pro price and
start date (**Pro starts on Oct 12**, “Your trial keeps going until
then. You can also start today.”) with a secondary **Start Pro now**.
The card that pushes starting today only appears once the trial is
running low or used up. The usage card and the offer read the same
thresholds.
- When usage fails to load, billing shows one error in the Usage slot
with **Try again** and **Contact support**, instead of the generic error
card plus a second error in the side card. The plan card beside it stays
as usual.
- The limit popup names the reset time once; the separate “You can
continue after” box is gone. Max captions say “What you’ve already used
counts toward Max’s higher limit.”, since “today” was wrong for weekly
limits. A cancel-pending trial's limit notice and offer say **Subscribe
now**, the label of the plan cards they open. When #15403 further up the
stack is rebased onto this PR, keep its ResetWaitBox for blocked chats;
only the fallback box was removed here.
- Pro review includes the early trial end, invoice period, tax, amount
due, next payment date, quote-expiry handling and a working
payment-management link. No invented card identity is shown.
- Max review includes the prorated debit, unused Pro credit, tax, amount
due and renewal date. The regular renewal price is explicitly qualified
before discounts/account credits. Quotes are account-bound and expire;
changed terms require a new review. Subscription quantity and the
reviewed proration timestamp are preserved.
- Confirmed Stripe payment failures retire the reviewed Max quote and
require a fresh review/consent after correcting the payment method.
Uncertain transport/5xx responses, rate limits and in-flight conflicts
retain the quote and idempotency key for retry.
- Account-specific charge amounts, payment buttons, accepted trial
prices and renewal details use the design system’s replay masking;
static labels remain readable. The amount owed after a failed trial
charge is masked the same way.
- One platform-level trial-status observer owns the active-trial polling
timer. Other consumers share cached results; stable accounts refresh on
focus/reconnect. The short, bounded checkout-return check is retained
for the parent’s cancel-pending trial flow and pauses in hidden tabs.
The separate pending-activation recovery loop pauses while
hidden/offline and checks on return.
- Every After image below is a browser capture of these application
components, using the real page/composer arrangement, styles and fonts.
The reproducible renderer records source/PNG hashes and checks visible
controls, overflow and draft behavior.

Round two review fixes are in e900125 (pay the open trial invoice),
950f2fc (calmer trial offer, one usage error and clearer limit labels)
and 964b048 (regenerated captures). bc73a2d is an empty commit that
restarted CI.

If a Max preview fails, the UI offers **Retry review** and **Keep Pro**.
It does not charge an undisclosed amount as a fallback. The existing
API's optional quote fields remain compatible with older clients.

### Validation

Round two, on 964b048:

- Backend: **878 tests across 47 files** passed: trial status and
routes, the new open-invoice reader, checkout guards, activation, Stripe
and trial webhooks, trial notices and analytics events. That includes
**18 new tests** for reading the open invoice (trial status, conversion,
invoice status, ownership, enrollment, amount, hosted link and Stripe
errors) and two route tests for `payment_due`. The four cases of
`test_notice_uses_shared_notification_queue` need RabbitMQ and were left
to CI. Ruff, Black, isort and Pyright are clean on the changed files,
and the API schema was re-exported with `TrialPaymentDue`.
- Frontend: `pnpm format`, `pnpm lint` and `pnpm types` pass. The full
`pnpm test:unit` run with coverage (TZ=UTC, as in CI) passed: **10,158
tests across 929 files**, plus 4 marked as expected failures. That run
was on the round two code before one last replay-masking fix, so the 27
test files that import the two components the fix touches were run again
on 964b048: **284 passed**, plus 2 expected failures.
- Captures: all **136** captures (34 states at 1280, 1000, 390 and 320
px) were regenerated from this branch and passed the overflow, control
and draft checks with no page errors. `verify.mjs` passes for **175**
source files and every PNG, and all 34 pairs were recomposed. The source
manifest now leaves out third-party files under `node_modules`, so it
lists the same sources on any checkout. These captures use Playwright's
Chromium 141 instead of 149, so screens whose components did not change
differ from the previous set only in text antialiasing.
- detect-secrets, gitleaks and the large-file check pass on the new
commits.
- CI is green on bc73a2d, an empty commit on 964b048: [Backend
CI](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/38660005365)
(lint, type checks and every test shard on Python 3.11, 3.12 and 3.13,
with RabbitMQ, so the notification test above ran here), [Frontend
CI](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/38660009153)
(lint, knip and all four test shards), [Full-stack
CI](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/38660012724)
(API types, smoke tests and journeys) and
[CodeQL](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/38660015431).
They were started by hand on the branch because GitHub has not run the
pull request checks since the round two push: its test merge for this PR
builds on #15208's, and #15208 now has merge conflicts with its base.

Earlier rounds, on abcb6af:

- Frontend `pnpm format`, `pnpm lint`, `pnpm types` and focused `pnpm
test:unit` passed: **179 tests across 12 files**, with coverage. The
broader regression pass exercised 721 tests across 73 files; all eight
outdated test expectations it identified were corrected and covered by
the passing focused rerun. This covers reviewed payments, replay
privacy, trial polling, activation recovery, billing, both chat stream
paths, saved drafts and checkout returns.
- All **75** backend reviewed-upgrade, invoice and trial-status
regressions passed. They cover fresh consent after confirmed failures,
same-key retries after uncertain responses, ownership isolation, cache
invalidation and accepted trial policies.
- All **132** responsive captures across 33 states passed
overflow/control/draft checks. Hash verification passed for **170**
source files and every PNG. All 33 compact before/after pairs, full-size
comparison sheets and downloadable component images were regenerated.
Browser/context/server cleanup was verified.
- The strict browser-journey guard recognizes only the optional
current-activation GET’s exact no-activation 404 body. Seven boundary
tests prove that other errors still fail. No UI code changed; the
existing 132 source-verified captures remain exact.
- All pre-commit checks passed, including secret scans, schema export,
formatting and backend/frontend types. The local Prisma client and stubs
were refreshed for the updated parent schema.

CI is green on `abcb6afd83d1e34bc58decd76f3b575659388c7a`: Backend CI,
Frontend CI (all four shards), Full-stack CI (30 smoke tests and 11
journeys, all passed without skips or retries), API compatibility,
CodeQL and external checks. The migration-owned policy is excluded from
seed-cache dumps; all 11 seed-cache contract tests and relevant
pre-commit hooks pass. `/review` was posted after CI passed, but the
review bot could not queue it: GitHub’s diff API returns HTTP 406
because this PR exceeds its 300-file limit. No new review verdict is
available.

Rollout depends on the parent activation migration, worker rollout and
Stripe test-mode validation. These local tests and sample renders do not
constitute a live payment test; no production account was changed. The
new **Pay $X now** button still needs one Stripe test-mode check: pay a
past-due trial's hosted invoice and confirm billing switches to Pro.

### Before and after — actual components

All **34** comparisons below have been regenerated. Left: production
components from deployed release
`4473fbbcb1951471df0a05374d47c93c4e39bea6`, verified October 12, 2026.
Right: the implementation in this PR. Both use illustrative account data
and a fixed October 7 sample clock; these are not recordings of
authenticated customer accounts. Missing production screens are
explicitly labelled.

The comparisons below are attached to [this
comment](https://github.com/Significant-Gravitas/AutoGPT/pull/15215#issuecomment-6200139278),
not committed. There are no separate prototype designs in this review
set. The full-size captures, the render harness and the production audit
are no longer in this PR's diff; they stay at
**`964b04864a7e37ee0c3677dc5b943fd122f3b410`**: [Download individual
full-size component
images](https://github.com/Significant-Gravitas/AutoGPT/tree/964b048/docs/engineering/usage-experience/implementation/captures)
· [Capture and verification
instructions](https://github.com/Significant-Gravitas/AutoGPT/tree/964b048/docs/engineering/usage-experience/render)
· [Production state
audit](https://github.com/Significant-Gravitas/AutoGPT/blob/964b048/docs/engineering/usage-experience/production-state-audit.txt).

**01 · Trial · limit popup**

Explain the exhausted trial, automatic Pro date, price, and choice to
wait.

![Trial · limit popup — production versus
implementation](https://github.com/user-attachments/assets/8b75c7bd-08bd-4820-8fbb-82e7701b95e4)

**02 · Trial · in-chat notice**

One neutral notice shows the start date and price; the draft stays
editable.

![Trial · in-chat notice — production versus
implementation](https://github.com/user-attachments/assets/13e64911-e306-4b26-a523-2927aa33e13e)

**03 · Trial · exhausted allowance**

One trial meter, the agreed renewal terms, and a clear option to start
Pro now.

![Trial · exhausted allowance — production versus
implementation](https://github.com/user-attachments/assets/ea0190bb-e346-4e7b-af36-9428525974f4)

**04 · Trial · allowance remaining**

One trial meter with its end date. The Pro card keeps the price and
start date, and only suggests starting early once usage runs low.

![Trial · allowance remaining — production versus
implementation](https://github.com/user-attachments/assets/897ce201-a9cb-47a5-9d45-8368a462793a)

**05 · Pro · daily limit**

Show the actual next plan, benefits, price, and the option to wait.

![Pro · daily limit — production versus
implementation](https://github.com/user-attachments/assets/12193309-6fc8-4df5-939a-6d5efbafac0a)

**06 · Pro · in-chat notice**

Replace the generic billing detour with a contextual Max offer.

![Pro · in-chat notice — production versus
implementation](https://github.com/user-attachments/assets/bcab7017-1084-4e19-8899-575001821439)

**07 · Pro · both limits reached**

The later weekly reset takes priority; both usage windows remain
visible.

![Pro · both limits reached — production versus
implementation](https://github.com/user-attachments/assets/c0862f3a-5ed6-4eaf-af31-05f6c6653927)

**08 · Pro · weekly limit**

Correct the daily-limit wording and identify the weekly reset.

![Pro · weekly limit — production versus
implementation](https://github.com/user-attachments/assets/0ab09c63-76a6-4d26-a0a2-b2deeff01918)

**09 · Max · daily limit**

Put the next usable time first, with a restrained capacity contact.

![Max · daily limit — production versus
implementation](https://github.com/user-attachments/assets/9c8531ff-4751-4c07-8c8a-ee66c2818e24)

**10 · Max · in-chat notice**

A calm reset notice and direct contact replace the generic billing
button.

![Max · in-chat notice — production versus
implementation](https://github.com/user-attachments/assets/b390ee0c-685a-49fa-b916-c9a25276abdc)

**11 · Max · billing and capacity**

Group plan, usage, and capacity help; keep payment details compact.

![Max · billing and capacity — production versus
implementation](https://github.com/user-attachments/assets/345ffd04-617a-45b1-b7c8-18fc04bda245)

**12 · Team · weekly limit**

Identify the plan and blocking week; offer the appropriate support
action.

![Team · weekly limit — production versus
implementation](https://github.com/user-attachments/assets/93379926-a0e1-483b-a942-a1f31e687c88)

**13 · Team · billing and usage**

Usage and organization billing become a coherent page with capacity
help.

![Team · billing and usage — production versus
implementation](https://github.com/user-attachments/assets/5643daf1-afa0-4373-9bb6-f61f54234d25)

**14 · Enterprise · weekly limit**

Clear weekly timing and a direct route to the account team.

![Enterprise · weekly limit — production versus
implementation](https://github.com/user-attachments/assets/c112ce2d-7416-45f5-9f24-d10d114c7a30)

**15 · Enterprise · billing and usage**

Bring usage timing and account-team support together with organization
billing.

![Enterprise · billing and usage — production versus
implementation](https://github.com/user-attachments/assets/1c3e68ef-a4e9-487b-8979-adb9a0e7d7e9)

**16 · Basic · legacy-plan notice**

A contextual Pro action replaces a generic billing destination.

![Basic · legacy-plan notice — production versus
implementation](https://github.com/user-attachments/assets/13e49021-149b-4f09-a27e-5c50d7f710d0)

**17 · Basic · legacy-plan settings**

Keep the existing entitlement visible alongside usage and upgrade
options.

![Basic · legacy-plan settings — production versus
implementation](https://github.com/user-attachments/assets/2b869b8c-1607-4125-83a5-6f068af8a93a)

**18 · No active plan · plan selection**

Explain product benefits, prices, and billing choices before choosing a
plan.

![No active plan · plan selection — production versus
implementation](https://github.com/user-attachments/assets/5e21eac4-1012-4903-87b5-8bb98101d51a)

**19 · No active plan · settings**

Show account context and the path back to a subscription.

![No active plan · settings — production versus
implementation](https://github.com/user-attachments/assets/0a56ac0d-ab9f-4832-8671-cb80e20b0fdc)

**20 · Trial ended · recovery**

Explain ended access and offer a fresh start in the conversation.

![Trial ended · recovery — production versus
implementation](https://github.com/user-attachments/assets/e7c16df7-8b56-4438-a798-9a0d2576e184)

**21 · Trial canceled · recovery**

Respect the cancellation and offer a low-pressure way to return.

![Trial canceled · recovery — production versus
implementation](https://github.com/user-attachments/assets/bdb0dac8-07d3-4dc5-a4ca-be468bbba1c8)

**22 · Payment failure · recovery**

State the amount owed and pay the open trial invoice on Stripe in one
step, instead of a plan checkout that is refused while the invoice is
open.

![Payment failure · recovery — production versus
implementation](https://github.com/user-attachments/assets/374ddbcf-aaa7-41dc-85a9-1dafe7c82c9f)

**23 · Payment failure · settings**

One message with the amount and one button that pays the open invoice.
No refused Start Pro checkout and no duplicate payment cards.

![Payment failure · settings — production versus
implementation](https://github.com/user-attachments/assets/8800b8fd-8064-44a5-ade1-3482db64df01)

**24 · Usage unavailable · in-chat feedback**

Explain the missing usage data and offer retry without displaying a
false zero.

![Usage unavailable · in-chat feedback — production versus
implementation](https://github.com/user-attachments/assets/5f986744-7322-49a6-a4c3-1b63d0c57d62)

**25 · Usage unavailable · settings**

One usage error with a retry in the Usage slot, while the plan card
beside it stays as usual.

![Usage unavailable · settings — production versus
implementation](https://github.com/user-attachments/assets/589bae3f-cfcf-4d7c-b25c-d5c8619110a3)

**26 · Connected provider · continuation**

Keep a connected-provider continuation option next to the upgrade.

![Connected provider · continuation — production versus
implementation](https://github.com/user-attachments/assets/671c18b5-9d34-4adf-9bde-c8ccb54a99af)

**27 · Start Pro · payment review**

Review the trial ending early, exact charge, and next payment before
paying.

![Start Pro · payment review — production versus
implementation](https://github.com/user-attachments/assets/995a8b8e-1041-4bd7-a296-8762bb97a597)

**28 · Pro to Max · upgrade review**

See the prorated charge, credit, tax, and renewal before upgrading to
Max.

![Pro to Max · upgrade review — production versus
implementation](https://github.com/user-attachments/assets/2018ef3c-619a-4b5c-abf5-5652a926a7b9)

**29 · Start Pro · payment processing**

Wait for confirmed payment, entitlement, and usage reset before
continuing.

![Start Pro · payment processing — production versus
implementation](https://github.com/user-attachments/assets/09c4fe2a-d891-4e40-b2ba-9011d9447015)

**30 · Pro activated · return to work**

Confirm successful paid activation and fresh allowances before returning
to work.

![Pro activated · return to work — production versus
implementation](https://github.com/user-attachments/assets/251864ed-2553-4fc8-b846-6f0a78ddde77)

**31 · Pro activated · fresh allowance**

Show fresh allowances after confirmed trial-to-paid activation.

![Pro activated · fresh allowance — production versus
implementation](https://github.com/user-attachments/assets/0fece8f7-c6fb-4c46-9e87-a115dd269517)

**32 · Trial cancellation pending · billing**

Keep remaining trial access and the parent’s explicit plan choices,
without a duplicate automatic-start offer.

![Trial cancellation pending · billing — production versus
implementation](https://github.com/user-attachments/assets/611d2b04-712e-41ff-b917-7387be77afe4)

**33 · Trial cancellation pending · chat notice**

Choose a plan explicitly after cancellation with Subscribe now, the
label of the plan cards it opens; never promise that Pro will start
automatically.

![Trial cancellation pending · chat notice — production versus
implementation](https://github.com/user-attachments/assets/0b63aa82-b02f-464c-a246-f36dee51e57e)

**34 · Payment failure · paywall**

Past-due trials see the same message and pay button in the paywall
instead of plan choices whose checkout is refused.

![Payment failure · paywall — production versus
implementation](https://github.com/user-attachments/assets/b9c43837-b568-4476-b2bc-3e04bbfcbbfe)

### Agents and large language models used

Codex with GPT-6, including delegated coding, validation and render
work.

Claude Code with Claude Opus 5.5 for the round two review fixes
(e900125, 950f2fc and 964b048), including their tests and the
regenerated captures.

### Checklist 📋

- [x] Changes and dependency order are documented.
- [x] A test plan and its results are listed above.
- [x] All 34 comparisons and 136 responsive component captures match the
final source manifest.
- [x] Existing configuration files remain compatible; no new environment
variables or services are required.
- [x] New billing data operations verify the authenticated user and
owned Stripe customer/subscription; quote ownership and expiry are
checked before confirmation. The open trial invoice is only read for the
signed-in user's own trial and must match its Stripe customer and
enrollment.
- [x] All browser contexts, browsers and capture servers created for
this work were closed.


---
_Generated by [Claude Code](https://claude.ai/code)_

---------

Co-authored-by: GPT-6 (Codex) <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: Nicholas Tindle <[email protected]>
Co-authored-by: Nicholas Tindle <[email protected]>
 2026-10-15 19:36:22 +00:00
Krzysztof CzerwinskiandClaude Opus 5.5 15eeb66abe feat(backend): serve the swap proxy from a service that exposes only its two calls (#14865) 
Follow-up to #14645.

### Why / What / How

**Why.** The swap proxy (#14645) reached the backend through
`DatabaseManager`, which serves every method it exposes on one port,
with no caller authentication and no per-caller allowlist. The proxy
needs two of them. Anything that could reach that port could call the
rest too: credit, integration and graph writes, and every user's stored
credentials. The proxy sits next to every box, so it is the component
most likely to be compromised. SECRT-2742: this has to be closed before
the proxy is switched on.

**What.** A small dedicated service, `SwapCredentialService`, that
answers `get_swap_bindings` and `resolve_swap_credential` and nothing
else. The two methods are removed from `DatabaseManager`, and the
proxy's default backend URL points at the new service.

**How.** It is an ordinary `AppService` whose HTTP app has one route per
exposed method: the two swap calls, plus the health checks and metrics
every service has. A call to any other backend method gets a 404,
because this port does not serve it. The service needs what
`DatabaseManager`'s process already has (the database and the encryption
key) and gets little traffic (the proxy caches both answers), so it runs
beside `DatabaseManager` in the same `db` entrypoint, on its own port
(`SWAP_CREDENTIAL_SERVICE_PORT`, default 8012). No new deployment unit.

**Still to do before switch-on, outside this repository:** the network
policy around the proxy must allow that port and not
`DatabaseManager`'s, the `db` deployment must publish the port, and the
port needs its own health probe (the service is a second process in the
entrypoint, so `DatabaseManager`'s probe does not cover it). This PR is
half of what SECRT-2742 accepted. The other half is tracked in
SECRT-2935, which stays open until the policy lands: until then,
`DatabaseManager` still answers every method to a proxy that can reach
it.

To test the served routes without starting a process, `AppService.run()`
now builds its app through a new `build_fastapi_app()`. The routes it
registers are unchanged.

### Changes 🏗️

- `backend/copilot/swap_service.py` (new): `SwapCredentialService`,
exposing only the two swap methods; connects to the database in its
lifespan like `DatabaseManager`, and its health check runs a query and
fails with `UnhealthyServiceError`, as `DatabaseManager`'s does. It
starts no LaunchDarkly client (`reads_feature_flags = False`): nothing
on the swap path reads a flag.
- `backend/db.py`, `backend/app.py`: run it beside `DatabaseManager`.
- `backend/data/db_manager.py`: `get_swap_bindings` /
`resolve_swap_credential` removed (the proxy was their only caller).
- `backend/util/service.py`: app construction moved from `run()` into
`build_fastapi_app()`; same routes.
- `backend/util/settings.py`: `swap_credential_service_port` (8012).
- `.github/scripts/platform-single-container-smoke.sh`: the appliance
smoke test's hostile `backend.json` and pin check include the new port.
- `single-container/common.sh`, `entrypoint.sh`: the appliance pins
`SWAP_CREDENTIAL_SERVICE_PORT` to 8012 with its other service ports, so
the persistent backend JSON cannot move it.
- `swap_proxy/swap_proxy/settings.py`, `swap_proxy/README.md`: default
`SWAP_PROXY_BACKEND_URL` is `http://localhost:8012`. The README no
longer lists `DatabaseManager` exposure as an accepted gap, and says
which port the policy should allow.

Configuration: new backend setting `SWAP_CREDENTIAL_SERVICE_PORT`
(default 8012). `SWAP_PROXY_BACKEND_URL` must now point at that port.
Nothing changes at runtime until `E2B_EGRESS_PROXY_ADDRESS` is set.

### Agents and large language models used

Claude Code with Claude Opus 5.5 (1M context)

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] `swap_service_test.py` (7): the app's POST routes are exactly the
two swap methods plus the health checks; every method `DatabaseManager`
exposes is refused (404/405) when posted to this service; both methods
answer (value and `None`); neither is on `DatabaseManager` any more; the
health check runs a query rather than trusting the connected flag.
  - [x] `swap_credentials_test.py` (31) unchanged and passing.
- [x] `swap_proxy`: 304 tests pass, with the Swap Proxy CI workflow's
steps run locally (lock check with the pinned Poetry, isort, black,
ruff, pyright, pytest).
- [ ] Not run: the `db` entrypoint against a real database with the
proxy calling it. That is part of switching the proxy on.

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
(the default port needs no entry, like the other service ports)
- [x] `docker-compose.yml` is updated or already compatible with my
changes (nothing outside the `database_manager` container calls the new
port locally)
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
 2026-10-15 15:47:22 +00:00
Nicholas TindleandClaude Opus 5.5 c52ce5d6cf fix(platform): refresh Debian packages in image builds daily so security fixes ship (#15464) 
### Why / What / How

**Why:** The single-container image check on `dev` fails its Trivy scan
with `trivy-critical.json contains findings: Vulnerabilities=4`: Debian
`chromium` and `chromium-common` 154.0.8037.92-1~deb13u1
(CVE-2026-106195, CVE-2026-106237), fixed in 155.0.8059.39-1~deb13u1,
which trixie-security already serves for amd64 and arm64. The fix never
reaches the image: `backend/Dockerfile` installs Chromium in an apt
layer, and the build cache keeps reusing that layer (`#46 CACHED` on the
Chromium step in the failing runs), so the image ships the old Chromium
until something above the layer changes. Because the check fails, "Check
PR Status" fails for every PR touching the Dockerfiles, the lockfiles or
the single-container files, and they can't enter the merge queue. The
prod, dev and preview images are built from the same Dockerfile with
layer caches too, so they carry the same stale Chromium: the browser the
Copilot browser tool drives against arbitrary websites.

**What:** A build argument, `APT_REFRESH`, declared right before the
package installs of every stage that installs Debian packages. Every CI
build of these images passes the UTC date.

**How:** BuildKit adds a declared build argument's value to the
environment of every later `RUN` in the stage, which makes it part of
those steps' cache keys. A new date re-runs the stage's package installs
and whatever follows them in that stage; the steps before them, and
stages without package installs (the frontend build among them), keep
hitting the cache. The date changes once a day, so OS packages are at
most a day behind Debian. A build that doesn't pass the argument, such
as a local `docker compose build`, gets an ARG with no value, which adds
nothing to the environment, so its cache keys stay as they are.

### Changes 🏗️

- `autogpt_platform/backend/Dockerfile`: `ARG APT_REFRESH` before the
first package install of `builder` (its Node.js package is copied into
the migrate and server images), `migrate`, and `server-base` (which
installs Chromium).
- `autogpt_platform/single-container/Dockerfile`: `ARG APT_REFRESH`
before the appliance stage's package install (PGDG PostgreSQL, nginx,
Valkey, ...). The comment on the per-build arguments at the end now
names just those.
- `platform-single-container-docker.yml`: a step sets the date (`date -u
+%F`), and both bake builds (CI and release) pass it as
`*.args.APT_REFRESH`, which reaches the linked backend targets as well.
`single_container_cache.py` needs no change: dev pushes still write the
shared `v4-<arch>` registry cache, so the first build of a day re-runs
the package layers and its export hands them to that day's PR builds.
- `platform-fullstack-ci.yml`: the backend bake passes the date, so E2E
runs against the packages that ship.
- `copilot-setup-steps.yml`: the migrate build passes the date.
- Tests: `test_dockerfile_layers.py` checks that every stage with a
package install declares `APT_REFRESH` before its first install, that
the Chromium install follows it, and that every bake build in the
single-container workflow passes the date from an earlier step. The
existing "build arguments follow every layer" test exempts
`APT_REFRESH`, which belongs before the package layers.
`test_e2e_seed_cache_contract.py` checks that the full-stack backend
bake passes it.
- Separate commit: black formatting that was already due in
`check_actions_status.py`, its test, and `test_frontend_environment.py`.
#15449 carries the same formatting and merges with this cleanly.
- Companion infra PR passes the same argument in the prod, dev and
preview backend image builds:
Significant-Gravitas/AutoGPT_cloud_infrastructure#444

### Agents and large language models used

Claude Code with Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] `autogpt_platform/single-container/tests`: new and existing tests
pass. The two `test_ci_smoke_stop` subtests fail locally under macOS's
`/bin/bash` 3.2, the same as on `dev`; CI runs them on Linux.
- [x] Removing `APT_REFRESH` from any of the four stages, from the
release bake, or removing the date step fails the matching test
- [x] `.github/scripts/test_e2e_seed_cache_contract.py` and the
`.github/workflows/scripts` tests pass
- [x] `docker buildx build --check` on both Dockerfiles reports no
warnings; actionlint reports nothing new
- [ ] This PR's "Build, smoke, and scan" jobs re-run the package layers
and the Trivy critical scan passes on amd64 and arm64

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
 2026-10-12 17:47:59 +00:00
Nicholas TindleandClaude Opus 5.5 8d248cd9b0 fix(platform): name the missing field when a graph can't be saved (#15438) 
### Why / What / How

**Why:** Saving a graph that has an Agent Output with an empty **Name**
shows the user a raw Python error. The builder toasts `'Input' object
has no attribute 'name'`, and the save's promise is left unhandled, so
the page also logs an uncaught `ApiError` (patrol finding F9, found on
the self-host appliance; it isn't self-host specific). Nothing tells the
user which block or field is wrong.

Root cause, in two layers:
1. `make_graph_model` dumps the submitted graph, which evaluates the
computed `output_schema`. `_generate_schema` builds each output with
`model_construct` (no validation), so an output without `name` raises
`AttributeError` on `p.name`. #12714 turned that into a `ValueError`,
which the generic handler sends back as a 400 with the raw text. This
happens before validation ever runs.
2. Even past that, `validate_graph` raised a bare `ValueError("This
field is required")`, which names neither node nor field. Its errors
would have been keyed by the node ids `reassign_ids` had just replaced,
so the builder couldn't have marked the field anyway.

**What:** Saving (`POST /graphs`, `PUT /graphs/{id}`) now answers a
refused graph with the same structured 400 the run path already uses:
`{"type": "validation_error", "message": "Agent Output › Name: This
field is required", "node_errors": {"<node id the builder sent>":
{"name": "This field is required"}}}`. The builder shows "Couldn't save
your agent" with that message, marks the field on its node, and keeps
the save from becoming an uncaught error.

**How:**
- `_generate_schema` leaves an unnamed input/output out of the graph's
interface instead of failing the whole dump; validation reports the
missing name.
- A blank name (`null`, `""`, whitespace) counts as missing too, so it
can't add a nameless field to the interface (review follow-up). A name
wired in from a link still counts.
- `_validate_graph` raises the existing `GraphValidationError` with
every node error and a readable first problem (block title or the user's
custom node name, then the field's schema title).
- `reassign_ids` returns `{new id: submitted id}`; the save routes
re-key `node_errors` with it, so the ids match the builder's nodes. The
run path's detail shape is shared via `_validation_error_detail`.
- Frontend: `useSaveGraph` resolves to `null` on failure (after toasting
and marking fields) instead of rejecting, and the run/schedule actions
stop when the save fails, rather than running the pre-edit version. The
API mutator uses a structured `detail.message` as `ApiError.message`
instead of a JSON dump.

### Changes 🏗️

- `backend/data/graph.py`: unnamed I/O skipped in `_generate_schema`;
`GraphValidationError` from `_validate_graph`; `reassign_ids` returns
the id map; message helpers.
- `backend/api/features/graphs/routes.py`: structured 400 on save, keyed
by submitted node ids.
- Frontend: `useSaveGraph` + new `helpers/saveGraphError.ts`;
`useRunGraph`/`useScheduleGraph` stop on a failed save;
`custom-mutator.ts` reads `detail.message`.
- Tests:
- DB-backed `graph_test.py`: create refused with the exact detail and
nothing written, then saved once named; an edit that unnames an output
is refused against the submitted ids, with no new version.
  - `routes_test.py`: create and update through the real validation.
- The old `test_generate_schema_raises_value_error_when_name_missing` is
replaced, since raising there is the bug.
- Hook tests for the new save contract, plus the run and schedule
actions stopping on a refused save.
- Route tests: blank names on Agent Input and Output (null, empty,
whitespace), plus a guard that a linked-in name still counts.
- CI: the journeys leg's `min_tests` floor goes to 11. That's dev's real
hosted inventory of 10 (its pin said 9) plus this journey; the contract
test is updated to match.
  - New journey `builder-save-journey.spec.ts`.

**Risk:** a stored graph with a blank-named Agent Input or Output, which
only API or import could have created, is now refused on fork, copy or
run, with the field named in the message. It was already broken: a
blank-named input never receives a run value (`executor/utils.py`, `if
input_name and …`). The builder can't create one, because
`pruneEmptyValues` drops `""`/`null`. All 183 I/O nodes in the bundled
agents and test fixtures are named.

**User-ID scoping (`data/*.py`):** no query changes. `reassign_ids` and
validation work on the in-memory graph the caller submitted. The save
routes are unchanged in scoping: create stamps the JWT user, and update
loads the versions with `user_id` before anything else. `node_errors`
only ever names the caller's own submitted nodes.

### Agents and large language models used

- Claude Code — Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Journey `builder-save-journey` **fails on origin/dev** (tower
self-host appliance, dev `fb2e05ca7785`): no readable message, and
`pageerror: ApiError: 'Input' object has no attribute 'name'`
- [x] Journey **passes with this branch** on the same appliance rebuilt
from this PR's commit: 1 passed in 11.5 s, and `selfhost-journey` still
passes (46 s). The image is from the pre-squash commit `353491f287`; the
squashed commit adds only the stale-highlight clearing (covered by the
hook test) and the journeys `min_tests` floor, now 11 after merging dev
- [x] New route tests fail on origin/dev's `graph.py`/`routes.py` with
the F9 error and pass here; `routes_test.py` 64 passed locally (after
merging dev). The 6 blank-name cases fail on the previous head:
validation accepts the blank name and the save goes ahead
- [x] Pure `graph_test.py` cases pass locally; the DB-backed ones run in
CI
- [x] Vitest: `useSaveGraph`, `useRunGraph`, `custom-mutator` (40
passed); `useScheduleGraph` 2 passed, and its refused-save case fails
with the guard removed
- [x] ruff/isort/black over the backend, pyright on changed files; `pnpm
format`, `pnpm lint`, `pnpm types`

<details>
  <summary>Example test plan</summary>
  
  - [ ] Create from scratch and execute an agent with at least 3 blocks
- [ ] Import an agent from file upload, and confirm it executes
correctly
  - [ ] Upload agent to marketplace
- [ ] Import an agent from marketplace and confirm it executes correctly
  - [ ] Edit an agent from monitor, and confirm it executes correctly
</details>

#### For configuration changes:

- [ ] `.env.default` is updated or already compatible with my changes
- [ ] `docker-compose.yml` is updated or already compatible with my
changes
- [ ] I have included a list of my configuration changes in the PR
description (under **Changes**)

<details>
  <summary>Examples of configuration changes</summary>

  - Changing ports
  - Adding new services that need to communicate with each other
  - Secrets or environment variable changes
  - New or infrastructure changes such as databases
</details>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
 2026-10-12 08:06:37 +00:00
Nicholas TindleandClaude Opus 5.5 db0b213114 ci(platform): run AutoPilot journeys against a scripted model server (#15431) 
### Why / What / How

**Why:** CI's e2e runs AutoPilot in `CHAT_TEST_MODE`, whose canned reply
skips the whole copilot pipeline. So nothing tests end to end how a turn
builds its context, routes and runs tools, persists, streams and
resumes, or what it costs. That pipeline is where copilot bugs keep
turning up, and each one was found by hand: stream drift (BUILDER-95J),
the double resume, retry double-billing, content-gate holds.

**What:** `autogpt_platform/scripted_model` is a deterministic fake
model provider that plays each model turn from a script. Only the
journeys leg and local test stacks use it. Three copilot journeys drive
the real pipeline against it.

**How:**
- **Two engines, one URL.** The backend runs in OpenRouter mode pointed
at the fake: `CHAT_BASE_URL=http://scripted_model:4010/api/v1`.
- The SDK engine (the prod default) gives the Claude Agent SDK's CLI
that URL without `/v1` as `ANTHROPIC_BASE_URL`
(`backend/copilot/sdk/env.py`), so it speaks Anthropic Messages.
- The baseline engine and side calls (session titles) speak OpenAI chat
completions.
  - One server answers both, streamed, with text and tool calls.
- **Picking a script.** A journey puts `[[script:<name>#<nonce>]]` in
its chat message.
- Turn *n* answers the request whose conversation already holds *n*
assistant messages. Requests carry the whole conversation, so the server
keeps no state: concurrent runs, retries, resumes and queued follow-ups
all land on the right turn.
- A script holds each turn's text, tool calls with exact arguments, and
token usage. Optional checks on tool results and on the user's latest
message are recorded in a journal the journeys assert on.
- **Failing loudly.** A request the script can't answer gets a 400 that
says why: no marker, past the last turn, or a tool the platform didn't
offer. The journey fails instead of hanging.
- **Cost.** The fake prices every call at Anthropic's list prices (the
same as `anthropic_rates.json`) and reports `usage.cost` the way
OpenRouter does.
- Journeys check that the copilot's cost ledger (`PlatformCostLog`)
charged the user exactly what the model billed for its turns: every turn
once, none missed, none doubled.
- Message ids are `msg_…`, not `gen-…`, so SDK turns record the CLI's
estimate synchronously instead of looking the cost up on openrouter.ai.
- **Hermetic.** Graphiti memory is pointed at the fake too, so nothing
leaves the stack. Its calls fail fast and turns go on without memory, as
on any memory failure.

### Changes 🏗️

- **`autogpt_platform/scripted_model/`**: the server.
- Standard library only. Python 3.13-slim Dockerfile, builds in seconds.
  - README.
- Tests that drive it with the real `anthropic` and `openai` clients:
both dialects, tool rounds, usage and cache reporting, refusals, side
calls, follow-ups, concurrent runs, and that every journey script
parses.
- **`autogpt_platform/docker-compose.journeys.yml`** (new): adds the
service and points the backend at it.
- Only the journeys leg and local test stacks add it. `docker compose
up` never reads it, so self-hosted installs are unchanged.
- It also blanks `SENTRY_DSN` for the backend and sets `DISABLE_SENTRY`
on the frontend server. Journeys provoke failures on purpose, so they
must never reach production Sentry, even when a local `.env` carries the
real DSN.
  - `docker-compose.yml` and `.env.default` are untouched.
- **CI (`platform-fullstack-ci.yml`):**
- New matrix key `scripted_model`. The journeys leg resolves compose
with the override and no longer writes `CHAT_TEST_MODE`; smoke is
unchanged.
  - The validator job also runs the fake's tests.
- Journeys `min_tests` goes from 5 to 9 (dev's five plus these four),
pinned in `test_e2e_seed_cache_contract.py`. The leg stays advisory.
- **Journeys** (`src/playwright/journeys/`):
- `copilot-build-agent-journey.spec.ts`: a fresh user asks AutoPilot to
build an agent. The model reads the building guide (`run_capability` →
`skill:agent_building_guide`) and saves the agent (`run_capability` →
`tool:create_agent`); AutoPilot runs both for real. Asserts:
- (a) the streamed text, recorded as the browser received it, equals
what the model said and what was saved, and shows once after a reload;
- (b) the Library shows the agent under the id the chat saved, exactly
once;
- (c) every model turn was asked for once and charged exactly what the
model billed;
- (d) the guard saw no unexpected 4xx/5xx, page error or console error.
- `copilot-follow-up-journey.spec.ts`: a follow-up queued while
AutoPilot answers runs once, after the answer, as its own turn. The
model is shown it, and it's saved once in order. The `CHAT_TEST_MODE`
dummy never drains queued follow-ups, so no journey could check this
before.
- `copilot-reconnect-journey.spec.ts`: the chat stream is cut mid-answer
the way a dropped connection is. The whole answer still shows once,
matches what's saved, and the model ran once.
- The variant where the browser is also offline is marked `test.fail()`
for the bug below.
- `context.setOffline` can't do the cut: Chromium lets an open stream
finish.
  - Helpers:
- `scripted-model.ts`: markers, the journal, the turn and cost checks.
- `copilot-chat.ts`: start a chat, record and cut chat streams, read the
saved chat.
    - `journey-database.ts`: now shared with the tier grant.
- Scripts live in `model-scripts/`. The README explains how to write a
scripted journey.

**Bugs these journeys found (not fixed here):**
- **Duplicate provider requests.** `backend/copilot/service.py` keeps
`_aux_client` and `_main_client` as process-wide `AsyncOpenAI`
singletons, but each copilot executor worker runs its own event loop
(`executor/processor.py:315`). A request on another loop's pooled
connection is sent and answered, then can't be read, so the OpenAI
client sends it again. The provider bills both; the platform records
one.
- Here that hit 11 of 47 chats' title calls. The SDK path uses these
clients only for side calls.
  - The baseline engine's turns use `_main_client` too.
  - So the strict ledger check leaves side calls out until that's fixed.
- **Follow-ups never answered on the baseline engine.** With
`FORCE_FLAG_COPILOT_SDK=false`, a follow-up queued during a reply stays
"Queued" forever, because the baseline engine has no auto-continue
(`baseline/service.py` ~L2495). That covers self-hosters on
`CHAT_USE_LOCAL`, which is baseline only. The follow-up journey fails
2/2 there and passes on the SDK engine.
- **Uncaught error after a network blip.**
`useCopilotPendingChips.ts:190` (`void
getV2GetPendingMessages(...).then(...)`) has no catch, so a
pending-messages peek during a blip surfaces as an uncaught "Failed to
fetch".

**Gaps:**
- CI runs the SDK engine only, the prod default. On the baseline engine,
build-agent and reconnect pass locally (2/2 each); follow-up fails for
the bug above.
- OpenRouter's `/generation` cost reconcile isn't exercised: its URL is
hardcoded to openrouter.ai.
- Memory and embeddings fail as before: there's no embedder.
- The Codex and Microsoft 365 routes and the auto-mode approval gate
aren't covered.

### Agents and large language models used

Claude Code — Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] `scripted_model` tests with the backend environment (real
`anthropic` / `openai` clients), plus the e2e contract tests
- [x] the real bundled CLI (2.1.284) against the fake: reply streamed,
`total_cost_usd` 0.00212 for 1000 in / 12 out, exactly the fake's price
- [x] each copilot journey 10× (`--repeat-each=10`, 2 workers) against a
full stack in the journeys configuration, built from this branch merged
with dev (`a40540cd81`):

    | Journey | Test | Runs | As expected | Median | Max |
    |---|---|---|---|---|---|
| `copilot-build-agent-journey` | builds an agent: streamed = saved, in
the Library, charged what the model billed | 10 | 10/10 | 12.3 s | 29.1
s (first run: cold block registry) |
| `copilot-follow-up-journey` | a queued follow-up runs once, after the
answer | 10 | 10/10 | 17.0 s | 19.8 s |
| `copilot-reconnect-journey` | an answer cut off by a dropped
connection arrives whole, once, as saved | 10 | 10/10 | 14.0 s | 15.0 s
|
| `copilot-reconnect-journey` | the same while the browser is offline
(`test.fail()`, bug above) | 10 | 10/10 failed as expected | 13.2 s |
14.7 s |

    It was also 40/40 on this branch before dev was merged in.
- [x] build-agent and reconnect on the baseline engine
(`FORCE_FLAG_COPILOT_SDK=false`), 2/2 each
  - [x] the journeys leg in CI on this PR: all 8 journeys as expected.
    - The leg took 9 min 55 s, with the Playwright step at 2 min 0 s.
    - It finishes before the smoke leg, so it adds no wall clock.
- [x] a CI dispatch with `journeys_repeat_each=10`
([run](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/38203567789)):
the copilot journeys were 40/40 as expected.
- Medians were 21–25 s; build-agent peaked at 68 s on its cold first
run.
- The leg's one unexpected result of 80 was an existing journey:
`run-correctness` hit a `socket hang up` on `GET /api/proxy/api/credits`
under load.

<details>
  <summary>Example test plan</summary>
  
  - [ ] Create from scratch and execute an agent with at least 3 blocks
- [ ] Import an agent from file upload, and confirm it executes
correctly
  - [ ] Upload agent to marketplace
- [ ] Import an agent from marketplace and confirm it executes correctly
  - [ ] Edit an agent from monitor, and confirm it executes correctly
</details>

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

<details>
  <summary>Examples of configuration changes</summary>

  - Changing ports
  - Adding new services that need to communicate with each other
  - Secrets or environment variable changes
  - New or infrastructure changes such as databases
</details>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
 2026-10-12 05:14:49 +00:00
Nicholas TindleandClaude Opus 5.5 bc671d3718 fix(platform): create a new user's profile on its first read instead of 404ing (#15426) 
### Why / What / How

**Why.** A brand-new user's first page asks for their marketplace
profile (`GET /api/store/profile`) to fill the account menu. Only
account setup (`POST /api/auth/user` → `get_or_create_user`) created
that profile, but Better Auth issues the session the moment the identity
exists, when its hook has inserted only a bare `User` row. So whenever
the first page's request beat setup, it got a **404**. If setup's
best-effort profile insert failed, the account already had its personal
org, so nothing ever re-ran setup, and the profile 404'd on every page
for good.

When does a page load before setup? The app's own email/password sign-up
and sign-in await `POST /auth/user` before redirecting, so they don't
race. A Google or email-link sign-in has a session from Better Auth's
callback until `/auth/callback` runs setup, and keeps it without setup
if that redirect is lost (the case #15135's per-request self-heal exists
for). Even then, the org-scoped requests on the same page set the
account up as a side effect, so the profile read loses only some of the
time. On the CI stack that was 3 of 8 runs.

What the user saw while the profile 404'd (found by reading every
consumer, and in the red CI runs below):
- **Account menu** (sidebar, or navbar on the old layout): placeholder
bars instead of name and email, blank avatar. No error card, no toast.
- **Wallet** (credits) hidden: it only renders once there's a profile.
- **Settings → Profile**: an empty form (the page treats 404 as "no
profile yet"). The legacy `/profile` page shows an error card once React
Query's retries run out.
- **Publish dialog, Creator dashboard**: no creator handle.
- **Console / Sentry**: Chrome's `Failed to load resource: 404`, an info
line from the request logger, and a Sentry **breadcrumb only** (404 is
in `EXPECTED_STATUSES` since #15391), no event.
- No broken "edit profile" link: the account menu's links are static
routes.

React Query's retries (about 1, 2 and 4 s) usually filled the menu in
once setup had run. In the permanent case, nothing did.

**What.** `GET /store/profile` creates the user's default profile when
they have none, using the same idempotent insert account setup uses, and
returns it. Reading your profile no longer depends on setup having run
first.

**How.**
- `data/profile.py` (new; the default-profile code moves here from the
1,500-line `data/user.py`, verbatim):
- `ensure_user_profile(user_id)` reads the user's own `User` row and
calls `ensure_profile`. The default name comes from that row's email,
never from a token's claims, so it's also right under admin
impersonation. It's a no-op for an id with no `User` row.
- `ensure_profile(user_id, email)` is setup's existing race-safe insert,
formerly `_ensure_user_profile`. Concurrent creators collide on
`Profile.userId`'s unique index and read the winner back; a handle
collision retries with a fresh handle. Account setup still calls it
directly with the row it has, so there's no extra lookup.
- `store/db.py`: `get_or_create_user_profile`, one read in the common
case; on a miss, ensure then re-read.
- `store/routes.py`: `GET /profile` uses it. It still 404s only for an
id with no platform user, or if no free handle turns up after retries.

Why this option and not the other two:
- **Refetch on the client after `POST /auth/user`:** the browser never
calls it. It runs in the sign-up and sign-in server actions and in
`/auth/callback`. And refetching after a 404 still sends the 404 and
leaves the permanent case broken.
- **Hold the app shell until setup completes:** the app's own flows
already do. The gap is sessions created outside them, which the shell
can't hold back without blocking every first page on a write. That would
also be an auth-flow change.
- **Lazy create in the read (this PR):** removes the dependency at its
source, and heals accounts setup left without a profile.

### Changes 🏗️

- **Backend:** `GET /store/profile` lazily creates the default profile
(`data/profile.py`, `store/db.py`, `store/routes.py`). Account setup
imports the moved `ensure_profile` (`data/user.py`). `openapi.json`: the
route's description only.
- **Backend tests** (`store/profile_first_load_test.py`, real database
through the `server` fixture):
- A read while account setup is held mid-insert gets the one profile;
setup then finishes cleanly.
  - A read after setup's profile insert failed heals the account.
- A seeded randomized-interleaving test, 200 seeds. Each seed races
reads, setup (whose profile insert fails 30% of the time) and the user
saving their profile. Every profile query gets random yields, and some
handles collide with another user's. The invariants: every read gets a
profile, exactly one row results, and a read never replaces a saved
profile.
- The module reconnects the client before it runs. See **Found along the
way**.
  - Existing tests patch the moved names.
- **Journey** `first-login-journey.spec.ts` (hosted project):
- A fresh user signs up and finishes onboarding, and their first page
loads before account setup has run.
- The journey holds the page's other API calls until the profile read is
answered, so the read lands before setup on every run, not only when it
wins the race.
- On Home, the account menu must name the user with no failed request
and no console error, and with no reload. Marketplace must then work
too.
- `journey-user.ts`: `signUpBeforeAccountSetup`, plus a corrected
comment on the shared fixture's setup step.
  - Journeys `min_tests` 4 → 5, and its contract test.
- **Data access (`data/*.py`):** `ensure_user_profile` acts only on the
id it's given. The route takes that from `get_user_id`: the token's own
subject, or an impersonation target that `get_user_id` has already
verified an admin is asking for. It writes only that user's own
`Profile`, from that user's own row.
- **Other first-load reads checked:**
- Onboarding reads return defaults when there's no row, so they never
404.
- Credits and subscription are org-scoped, so `get_request_context`
self-heals setup for them.
  - Notification preferences read the `User` row.
- In the red runs, the profile was the only failed request on the first
page.

**Found along the way (not fixed here).** In the backend `remainder`
shard, the test process's pooled DB connection is already inside a
transaction an earlier test opened and never closed. Three separate
queries return the same frozen `now()`. It starts around
`skills/routes_test`'s `TestClient` tests, on plain dev too ([probe
run](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/38198540970)).
- Every later DB write in that session is uncommitted.
- The first statement that fails on that connection aborts it (`25P02`)
for every test after it. That's what first broke this PR's real-DB tests
and some later `skill_catalog_test` cases
([run](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/38195334480)).
- The new tests reconnect first. I reproduced it locally: a leaked
`BEGIN` makes the same sequence fail with `25P02`, and a reconnect fixes
it. The leak itself needs its own fix.

### Agents and large language models used

- Claude Code — Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] **Journey on dev code (no fix), red every time.** [Run
38197531268](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/38197531268):
`w41/red-check2` is `origin/dev` plus only this PR's journey,
`--repeat-each=5`. It went **5/5 red**, each with exactly one problem:
`404 GET /api/proxy/api/store/profile` on the first page.
- Before the journey pinned the race, dev went red in 3 of 8 runs
([38193159877](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/38193159877),
[38195251731](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/38195251731)).
- [x] **Journey with the fix:** this PR's `end-to-end tests (journeys)`
is green on `a17e616a2d` and `0854cdd8c0`.
- [x] **Backend real-DB tests** (`profile_first_load_test.py`): green in
all three `remainder` shards on `0854cdd8c0`.
- [x] **`!deploy` preview** (hosted config, Vercel), two fresh
`@previews.agpt.co` users with a real Chrome user agent, preview
undeployed afterwards:
- Real sign-up page: the first page is onboarding, with no failed API
call. After onboarding, Home gets `200 GET /store/profile` and the
account menu names the user.
- Session before account setup: the first page's `GET /store/profile`
returns **200** (the lazy create), and the account menu names the user.
- [x] Locally: `store/db_test.py` (49 passed). `user_test.py`'s
account-setup classes need the Redis cluster, so they run in CI.
- [x] `ruff`, `isort`, `black` over the backend; `pnpm format`, `pnpm
lint`, `pnpm types` in the frontend

<details>
  <summary>Example test plan</summary>
  
  - [ ] Create from scratch and execute an agent with at least 3 blocks
- [ ] Import an agent from file upload, and confirm it executes
correctly
  - [ ] Upload agent to marketplace
- [ ] Import an agent from marketplace and confirm it executes correctly
  - [ ] Edit an agent from monitor, and confirm it executes correctly
</details>

#### For configuration changes:

- [ ] `.env.default` is updated or already compatible with my changes
- [ ] `docker-compose.yml` is updated or already compatible with my
changes
- [ ] I have included a list of my configuration changes in the PR
description (under **Changes**)

<details>
  <summary>Examples of configuration changes</summary>

  - Changing ports
  - Adding new services that need to communicate with each other
  - Secrets or environment variable changes
  - New or infrastructure changes such as databases
</details>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
 2026-10-12 04:00:46 +00:00
Nicholas TindleandClaude Opus 5.5 e09793064a ci(platform): add a journeys leg to e2e for hosted-config user journeys (#15397) 
### Why / What / How

**Why:** CI's e2e job tests a different product from the one customers
use. It runs every flag at its code default (off), with credits
disabled, and the 30 smoke specs check that pages render, not that flows
work. The stability-patrol bug archaeology found ~123 of 300 recent
user-facing bugs would have been caught by a holistic browser journey,
about half of them only with hosted-like configuration. Fix PRs almost
never add one, because there's nowhere cheap to put it.

**What:** A second leg of the e2e job that runs `*-journey.spec.ts`
against the hosted configuration on its own runner, plus the conventions
and the first journey. Other patrol agents are adding schedules,
realtime-after-reconnect and session journeys on top of this.

**How:**
- `e2e_test` becomes a `smoke` / `journeys` matrix with `fail-fast:
false`. Each leg has its own runner, so wall clock is `max(leg)`, not
the sum.
- The smoke leg is unchanged. Only smoke writes the shared caches (GHCR
build cache, Next.js cache, Playwright browsers).
- The journeys leg skips the OpenAI key, so its seed runs ~30 s instead
of ~3 min, and it gets its own seed-cache key (`emb`/`noemb`).
- **Journeys are advisory for now:** `end-to-end tests (journeys)` is in
`ADVISORY_CHECK_RUNS`. We flip it to blocking once it has proven stable.
- **Hosted configuration.** One file,
`src/playwright/journeys/hosted-flags.json`, holds the flags production
serves on for everyone.
- Backend: `.github/scripts/hosted_journeys_env.py` turns it into
per-flag `FORCE_FLAG_*` lines, plus `ENABLE_CREDIT=true` (not
`FORCE_ALL_FLAGS`).
- Frontend: Playwright builds (`NEXT_PUBLIC_PW_TEST=true`) now read flag
values from local storage, which the journey fixtures set before the app
loads. I deliberately didn't use build-time `NEXT_PUBLIC_FORCE_FLAG_*`:
a differently-flagged bundle can't reuse the smoke leg's cached image.
That means a cold 450–600 s Next build on every run, which would make
the journeys leg finish after smoke on frontend-cached runs. Production
builds never read the override.
- `enable-platform-payment` is forced **off**, because journey users
have no subscription and would hit the 402 paywall.
- `EXECUTION_COST_COUNT_THRESHOLD=1` bills every node, so a run's cost
is visible and exactly assertable. This is a deliberate deviation from
hosted.
- **Journey fixtures** (`src/playwright/journeys/fixtures.ts`):
- `journeyUser` signs up a fresh user through the auth API and completes
onboarding, which grants the onboarding credits. It works against any
`PLAYWRIGHT_BASE_URL`, including a `!deploy` preview, and never uses
CI's seeded accounts.
- An automatic **guard** fails a journey on any unexpected `/api/*`
4xx/5xx, `pageerror` or `console.error`, printing the URL, status and a
body snippet. Journeys declare deliberate errors with `test.use({
expectedResponses })`. Known noise goes in a small reviewed allowlist
(`KNOWN_BENIGN`, with reasons).
- **First journey** (`run-correctness-journey.spec.ts`, archaeology J4).
Import a small agent through the Library UI, run it, and assert:
  - the run reaches COMPLETED and its output shows in the run view;
  - the Library run list shows the run;
- the credit ledger charged the run exactly its cost, and the balance
moved by exactly the new ledger entries;
- a run with a failing node ends FAILED (not stuck RUNNING/QUEUED) and
its error is shown.

### Changes 🏗️

- `.github/workflows/platform-fullstack-ci.yml`: the matrix, per-leg
seed cache, cache writes and artifacts (`playwright-report-<suite>`,
`playwright-test-results-<suite>`), `--min-tests ${{ matrix.min_tests
}}` (smoke 28, journeys 3), and Codecov `name: e2e-<suite>`.
- `.github/scripts/validate_playwright_json.py`: new opt-in
`--allow-known-failures`, used only by the journeys leg. A `test.fail()`
known failure is accepted only if it actually failed; smoke stays
strict.
- `.github/workflows/scripts/check_actions_status.py`: the journeys
check is advisory. Adds a test.
- `.github/scripts/hosted_journeys_env.py` and its test; contract tests
updated for the matrix.
- `autogpt_platform/frontend/playwright.journeys.config.ts`
- `src/playwright/journeys/`: `fixtures.ts`, `guard.ts`,
`journey-user.ts`, `credits.ts`, `hosted-flags.json`, `README.md`
(conventions; how to run locally, against a preview and in CI; open
hosted vs self-hosted gaps), and the first journey.
- `src/services/feature-flags/pw-flag-overrides.ts` and
`use-get-flag.ts`: the runtime flag override for Playwright builds, with
unit tests.
- `TaskListItem` / `SidebarItemCard`: a `data-testid` on Library run
items.

**Configuration changes:** none for self-hosters or hosted. The journeys
leg appends `ENABLE_CREDIT=true`, `EXECUTION_COST_COUNT_THRESHOLD=1` and
`FORCE_FLAG_*` to its own CI `backend/.env`.

**Not done here (follow-ups):**
- Scaling `executor`/`copilot_executor` to 2 needs their fixed host
ports (8002/8008) removed in a compose override. Then we'd need to check
that the executor RPC tolerates replicas.
- No scripted model server (W9); chat stays on `CHAT_TEST_MODE`.

**Coordination:** no overlap with #15225 (single-container cache),
#15231 (`login.page.ts`, which journeys don't use) or #15352 (copilot
unit tests).

### Agents and large language models used

Claude Code — Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [ ] I have tested my changes according to the test plan:
- [x] `.github/scripts` validator and contract tests, and
`check_actions_status` tests, pass locally
- [x] Feature-flag unit tests pass (97), including the new override
tests
- [x] Journeys leg passes in this PR's CI (run 38171220939): 3 tests,
validator reports "3 tests passed once without skips or retries"
  - [x] Smoke leg unchanged: 28+ tests, green in every run of this PR
- [x] Journeys pass locally on the patrol stack too (3 passed, one of
them the expected known failure)
- [x] Wall clock: the journeys leg finished before smoke in every run
(frontend rebuilt in all of them, since this PR touches the frontend):

    | Run | journeys leg | smoke leg | journeys Playwright step |
    |---|---|---|---|
    | 38166219280 | 11m31s (failed: spec bug) | 16m49s | 113 s |
    | 38168727001 | 12m00s (failed: findings below) | 19m37s | ~80 s |
    | 38171220939 | **10m02s ✅** | **14m40s ✅** | 71 s |

Without embeddings the journeys seed took 62–93 s; with embeddings,
smoke's takes ~3 min.

### Findings from the first journey (on `dev`, not caused by this PR)

1. **A run whose node fails reports COMPLETED, and the user never sees
the error.** A MatchTextPattern node raises `Invalid regex pattern` with
nothing wired to its error output. The run view then shows "Completed"
and "No output from this run.", and nothing else. Untyped node errors
are non-fatal by design (`backend/executor/manager.py`, the final-status
block). This is the open product question of #14275 / SECRT-2840, so the
journey pins it with `test.fail()`: it goes red the day the behavior
changes, and then the marker comes off. A separate journey checks that
such a run at least *ends* (passes).
2. **A brand-new user's first page 404s on `GET /api/store/profile`**
("User does not have a profile yet"). The profile is only created by
`POST /api/auth/user`, which the app fires on load in parallel with the
navbar's profile query. The guard caught it in 1 of 3 tests. Journey
users now finish account setup first. A first-login journey should
assert this and fail until it's fixed.
3. `importAgentFromFile` in `pages/library.page.ts` returns an empty
`seeRunsUrl`/`id`, because "See tasks" on the Library card is now a
button, not a link. The journey looks the agent up through the API
instead. The page object should get the same fix.

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
 2026-10-11 23:53:27 +00:00
Nicholas TindleandClaude Opus 5.5 fb2e05ca77 fix(platform): self-hosted signup grant, memory page and no-model chat (#15409) 
### Why / What / How

**Why:** Three self-hosted correctness bugs from the stability patrol's
first-hour smoke test (W11, `selfhost-2026-10-11`):
- **F3, a regression from #15135.** The appliance's least-privilege
`autogpt_frontend` role can't `INSERT` into `platform."User"`. The
sign-up hook that creates the User row fails on every signup with
`permission denied for table User` / `Failed to provision platform User
(pg 42501)`, and the failure also goes to Sentry. Accounts only work
because the backend provisions lazily on the first API call, which
breaks the hook's invariant that a session never exists before its User
row.
- **F6.** The appliance forces Graphiti memory on in the backend
(`FORCE_FLAG_GRAPHITI_MEMORY=true`), but the frontend build never sets
the matching flag, so `/settings/memory` is a 404. Users can't see or
erase what was remembered about them, which is a privacy gap.
- **F4.** Copilot with no model configured shows two stacked toasts. The
visible one reads "Couldn't send message:
chat_transport_not_configured", and both vanish in about 5 s. The only
CTA points at Settings → Integrations, which offers ChatGPT and M365 and
nothing about the server-side key a self-hoster actually needs.

**What / How:**
- **F3:** `bootstrap.sh` grants `INSERT (id, email, name, "updatedAt")`
on `platform."User"`, the exact columns the hook writes. Every other
column keeps its default.
- Proved on the live appliance in a rolled-back transaction: without the
grant, `permission denied`; with it, `INSERT 0 1`, then `INSERT 0 0` on
conflict.
- The docker-compose path connects as `postgres`, so it never had the
gap. There's no Supabase path any more.
- The single-container smoke test now asserts the column INSERT grants
and that `metadata` and table-wide INSERT stay denied. It also checks
that sign-up itself creates the User row before any backend call. That's
the check that would have caught #15135.
- **F6:** the appliance's frontend stage sets
`NEXT_PUBLIC_FORCE_FLAG_GRAPHITI_MEMORY=true`, with a comment tying it
to the backend flag.
- **F4:**
- A persistent `NoAIConnectionNotice` above the composer whenever
`/api/chat/transports` reports nothing usable. It says "No AI model is
connected…", links to the single-container docs' "Models and memory"
section, and keeps the Integrations link.
  - The raw code is gone.
- Errors that `createSession` has already toasted are thrown as
`ReportedSendError`, so the composer restores the draft without stacking
a second toast. This also applies to the "still loading" and "choose a
connection" cases.
  - This is UI copy, not LLM-facing text, so no eval cases.

**Journey:** `src/playwright/journeys/selfhost-journey.spec.ts` walks a
self-hoster's first hour on a self-hosted build (BEHAVE_AS=LOCAL, no
keys):
1. A non-allowlisted email gets the neutral message.
2. Sign up, then onboarding with no paywall.
3. Home shows no Earn credits, no Billing and no Agentation.
4. Copilot shows the actionable notice, which outlasts the toast.
5. Build Store Value → Agent Output in the builder.
6. Run it from the Library and read its output, with no Cost shown.
7. `/settings/memory` loads, `/settings/billing` is a 404, and no
Billing link appears.
8. **Zero browser requests to any host but the instance.**

It passes only with this PR **and** #15408 (hosted-only UI).
`playwright.journeys.config.ts` is copied verbatim from W1's pending
journeys infra, so the two add the same file.

### Changes 🏗️
- `single-container/bootstrap.sh`: column INSERT grant on
`platform."User"`.
- `single-container/Dockerfile`:
`NEXT_PUBLIC_FORCE_FLAG_GRAPHITI_MEMORY=true` in the frontend stage.
- `.github/scripts/platform-single-container-smoke.sh`: grant
assertions, plus the User row must exist straight after sign-up.
- `single-container/tests/test_frontend_environment.py`: policy text.
- Copilot:
  - `NoAIConnectionNotice` (new) is mounted in `ChatInput`.
  - `helpers/reportedSendError.ts` (new).
- `useChatSession.createSession` throws `ReportedSendError` with human
copy.
  - `useChatInput` skips the duplicate toast.
- Tests: notice and no-double-toast cases in `ChatInput.test.tsx`;
transports mocks added where the chat module is mocked; the
codex-session-route title updated.
- `src/playwright/journeys/selfhost-journey.spec.ts` and
`playwright.journeys.config.ts` (new).

**Config:** the appliance image's frontend build env (above). No
`.env.default` or compose changes.

### Agents and large language models used
Claude Code — Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] single-container unit tests (173) pass; the copilot ChatInput,
CopilotPage and codex-session-route vitest suites pass.
- [x] Grant proved against the live appliance's Postgres (rolled back).
- [x] Self-host journey: FAILS on the current appliance (dev
`f358a5fc`), PASSES on the appliance rebuilt from this PR plus #15408
(`6d5046ce`): **1 passed (48.8s)**.
- The before-run's soft failures were: the closed-beta modal; Earn
credits; Agentation ×2; Billing in the menu and sidebar; "Cost: $";
"Pick a plan" / "Add credits"; `/settings/memory` 404; no no-model
notice; requests to tally.so and twemoji.maxcdn.com.
- [x] F3 on the rebuilt appliance: `docker logs` shows **0** `42501` /
`Failed to provision platform User` lines after the journey's signup.
Before, every signup logged one.

### CI
The journey can't reach tower from GitHub. CI's existing compose stack
runs as `BEHAVE_AS=LOCAL` with payments off, which is the self-hosted
configuration. So once W1's journeys leg lands,
`selfhost-journey.spec.ts` can run as a `journeys-selfhost` matrix leg
on that stack's cached images, with `PLAYWRIGHT_SELFHOST_EMAIL_DOMAIN`
set to a domain on the CI allowlist and **no** `hosted_journeys_env.py`
flags. It runs in parallel on its own runner, so it adds no wall clock,
and it should stay advisory at first. Two caveats:
- the compose stack ships no Graphiti, so the memory step needs
`FORCE_FLAG_GRAPHITI_MEMORY` on both sides or a skip;
- the stack has no single-container bootstrap, so F3 stays covered by
the single-container smoke test.

That's a follow-up rather than part of this PR.

#### For configuration changes:
- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
 2026-10-11 21:56:59 +00:00
Nicholas TindleandClaude Opus 5.5 b12c70dd8a fix(backend/copilot): charge a retried turn only for the attempt that succeeded (#15399) 
### Why / What / How

**Why.** We don't charge users for our failures. When a turn hits a
transient provider error (a 529, a 5xx, a dropped connection) and the
platform relaunches the CLI to retry it, the user should pay for the
attempt that succeeded, not for the one that failed.

Today the retry charges both. #15305 (in production through the #15342
hotfix) charges each SDK turn its own spend, measured against the
session total the CLI restores on `--resume`. To keep a relaunched
process's spend, it also kept the turn's starting baseline across
transient retries. The retry resumes the same session, the CLI restores
a total that already includes the failed attempt's spend, and the
retry's result charges the difference: the failed attempt plus the
retry.

It is inconsistent too. If the retries run out and the turn ends in an
error, the failed attempts' spend is never recorded, so the user pays
nothing for it. The same failure was charged only when the retry
succeeded.

The billing e2e from #15379 measured it with the real CLI. When the
connection dropped after a tool call, the retried turn was charged
42,127 µ$, where the attempt that succeeded cost 21,618 µ$. The failed
attempt's tool call came on top.

**What.** A transient retry now charges only its own spend. Spend that a
completed query reported before the failure stays charged. A
building-mode relaunch is deliberate, not a failure, so it still carries
its unreported spend over, as #15305 intended. Context retries (prompt
too long) are unchanged.

**How.** Both transient-retry branches go through
`_do_transient_backoff`, which now clears
`state.usage.cli_session_total_usd`. `_run_stream_attempt` then re-reads
the baseline from the session the relaunch resumes, so the failed
attempt's spend lands in the baseline instead of the charge. If the CLI
never persisted that spend, neither the baseline nor the restored total
has it, and the charge is the retry's either way.

The provider's charge for the failed attempt is now ours, and nothing
records it: it reaches neither the user's budgets nor `PlatformCostLog`.
Logging it as absorbed cost would need a row type of its own.

The billing e2e's retry rows now expect a relaunched turn to owe only
the calls billed after the failure. Each recorded call carries the
number of the request it answered, and a scenario whose fault relaunches
the turn owes only the calls after the fault. The baseline row's OpenAI
client retries the refused request within the same attempt, so that row
still owes every call. The token gap those rows pinned disappears: the
cost row and session usage already held only the retry's tokens, which
is now exactly what the turn owes. Every scenario passes, so the suite's
two expected-failure entries go.

### Changes 🏗️

- `backend/copilot/sdk/service.py`: `_do_transient_backoff` clears the
CLI cost baseline. Its docstring and the launch comment in
`_run_stream_attempt` say which relaunches carry spend over.
- `backend/copilot/sdk/p0_guardrails_test.py`: the backoff keeps
reported usage and clears the baseline.
- `backend/copilot/sdk/turn_cost_retry_test.py`: a raised and a handled
transient failure each charge only the retry. The two context-retry
cases are unchanged.
- `backend/copilot/billing_e2e/`: calls record their request number
(`invoice.py`, `provider.py`), scenarios mark faults that relaunch the
turn and work out what the turn owes (`scenarios.py`), and the charge
and token checks use that (`audit.py`). No known gaps remain.
- `.github/scripts/backend-allowed-skips.json`: drops the suite's two
expected-failure entries, which CI would otherwise reject as stale.

### Agents and large language models used

- Claude Code with Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Billing e2e, real bundled CLI against the fake provider, in the
backend's Linux image: 16 passed, no expected failures. With the
one-line fix reverted, both after-tool retry rows fail with `turn 1:
charged [42127] microdollars; the turn owes 21618 (invoice) for 1
call(s)`.
- [x] `turn_cost_retry_test.py`, `turn_cost_test.py`,
`turn_cost_restart_test.py` (building-mode relaunch, unchanged) and
`cli_cost_state_test.py`, against a one-node Redis cluster: 16 passed.
- [x] `p0_guardrails_test.py`, `turn_cost_e2e_test.py`,
`openrouter_cost_test.py` and `service_test.py`: 240 passed.
- [x] `retry_scenarios_test.py` makes no cost assertions. It needs the
database-manager service, so CI runs it.
- [x] `ruff`, `black` and `isort` clean; `pyright` shows no new errors
on the changed files.

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

No configuration changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
 2026-10-11 19:37:39 +00:00
Nicholas TindleandClaude Opus 5.5 54cea3114d fix(backend/copilot): retry Anthropic 5xx and overload errors the Claude CLI reports (#15381) 
### Why / What / How

**Why.** When Anthropic is overloaded (529) or returns a 5xx, the
platform is meant to treat the failure as transient. If nothing has
streamed yet, it relaunches the turn with backoff (up to 3 times). If it
can't, it ends the turn with a friendly error and a retry button.

Today it does neither. After the CLI's own retries run out, the user
gets the CLI's raw error as the assistant's reply, and the turn is
recorded as a success:

> API Error: Repeated 529 Overloaded errors. The API is at capacity —
this is usually temporary. Try again in a moment.

The CLI reports a failed API call as an `AssistantMessage` with two
parts: `error` holds a category and the text holds a message. The turn
retried only if one of them matched `_TRANSIENT_ERROR_PATTERNS`.
- `"rate_limit"` happens to match a pattern, so 429s still retry.
- `"server_error"` matches none.
- The 5xx patterns look for `status code 5xx`, which appears nowhere in
the bundled CLI (2.1.284). It words a 529 as `API Error: 529 Overloaded.
This is a server-side issue…`, or as the "at capacity" message above
once its retries are exhausted.

Every occurrence logs `AssistantMessage has error=server_error` at
ERROR, so Sentry has the count.

**What.** `server_error` and `rate_limit` now count as transient by
category, whatever the text says. Overloads and 5xx get the platform's
retry, and if retries run out, the friendly error with a retry button.
Other categories (`authentication_failed`, `billing_error`,
`invalid_request`, `unknown`) still need a transient text to retry, as
before.

**How.** `is_transient_assistant_error(category, content)` sits beside
`is_transient_api_error` in `constants.py`. The `AssistantMessage`
branch in `sdk/service.py` uses it in place of the two text checks.

The text patterns are unchanged. They deliberately avoid phrases such as
"overloaded" because those can appear in ordinary SDK messages. The CLI
sets the category only on a failed API call, so keying on it carries no
such risk, and it doesn't move when the CLI rewords its messages.

Found by the billing e2e from #15379, which drives the real bundled CLI
against a fake provider and pinned this bug as a known gap. This PR
updates that suite to match the fix:
- `sdk-anthropic-overloaded-on-resume` loses its known gap and passes.
The 529 on the resumed turn's first call is retried, and the turn is
charged only the call that succeeded.
- `sdk-anthropic-overloaded-after-tool` now retries like its
dropped-connection sibling. The turn is charged the call before the 529
plus the retry, and it carries that sibling's known gap: a retried
turn's cost row and session usage keep only the retry's tokens.

### Changes 🏗️

- `backend/copilot/constants.py`: `is_transient_assistant_error` and the
two retryable categories.
- `backend/copilot/sdk/service.py`: the `AssistantMessage` error branch
uses it, passing the category and text by name.
- `backend/copilot/sdk/p0_guardrails_test.py`: retryable categories with
the CLI's two 5xx wordings verbatim, and the other categories still
needing a transient text.
- `backend/copilot/sdk/retry_scenarios_test.py`: the transient-retry
test runs for `server_error` with the CLI's 529 text as well as
`rate_limit`.
- `backend/copilot/billing_e2e/scenarios.py`: the two 529 scenarios
above.
- `.github/scripts/backend-allowed-skips.json`: the allowed expected
failure moves from `[sdk-anthropic-overloaded-on-resume]` to
`[sdk-anthropic-overloaded-after-tool]`. Pytest reports an expected
failure as a skip, and CI rejects unlisted skips and stale entries
alike.

### Agents and large language models used

- Claude Code with Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Read both 5xx templates and the `PQ = "Repeated 529 Overloaded
errors"` constant out of the bundled CLI 2.1.284 binary. The test
strings are its exact wording.
- [x] `p0_guardrails_test.py`, transient tests: 53 passed. The new
category cases fail on `dev` (the function doesn't exist there).
- [x]
`retry_scenarios_test.py::…test_handled_stream_error_transient_retries_then_succeeds[server_error]`:
passed in CI (`test (3.13, copilot)` JUnit report), alongside the
`rate_limit` case and the 10 new category cases. It reaches the
database-manager service, so it doesn't run in my local environment.
- [x] The billing e2e on this branch merged with `dev`, real CLI against
the fake provider, in the backend's Linux image: 14 passed, 2 xfailed.
`sdk-anthropic-overloaded-on-resume` passes; both expected failures are
the retried-turn token gap, on `sdk-anthropic-overloaded-after-tool` and
`sdk-anthropic-dropped-after-tool`.
- [x] `ruff`, `black`, `isort` and `pyright` clean on the changed files.

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

No configuration changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
 2026-10-11 17:09:51 +00:00
Nicholas TindleandClaude Opus 5.5 c9cd58e1c6 dx(backend/copilot): check each chat route charges a thread what its provider billed (#15379) 
### Why / What / How

**Why.** #15342 (the hotfix of #15305) fixed resumed SDK turns being
charged the CLI session's running total instead of their own cost. The
bug lived in the gap between our code and the CLI's semantics. Our unit
tests scripted `total_cost_usd` as per-turn, while the real CLI reports
a cumulative total that it restores on `--resume`. Mocks written from
the same belief as the code can't catch that kind of bug. There was also
no check that spans a whole thread, or every way a turn is priced. There
are six on the SDK engine, the SDK over Codex, three on the baseline
engine, and Microsoft 365.

**What.** A new suite, `backend/copilot/billing_e2e/`, runs the same
three-turn chat through every route and checks every turn's charges
against the provider's own invoice:

1. A one-call turn.
2. A tool turn: two calls.
3. A turn on a "new pod": the CLI session is restored from storage and
resumed. This is the path #15342 fixed.

The suite fails if any turn is charged anything other than its own
calls, in any of the three places a charge lands: the user's
daily/weekly spend counters, the turn tree's budget, and the
`PlatformCostLog` row (cost, provider label, user, chat and tokens).

**How.** Mocks stop at the network, not at our functions. One localhost
server stands in for every upstream: Anthropic Messages, OpenRouter
(Messages, chat completions and `/generation`), Anthropic's and Ollama's
OpenAI-compatible endpoints, ChatGPT's Codex Responses API, and Graph's
Copilot chat. It answers in each wire format and keeps its own invoice,
priced from tables written out in the test rather than read from
`anthropic_rates.json`, the catalog or `moonshot.py`. That way code that
misprices a call can't also misprice the invoice it's checked against.
OpenRouter's prices are a tenth above Anthropic's list on purpose, so a
turn charged the CLI estimate where it should have been charged the
invoice can't match by coincidence.

The real clients run against it. On SDK rows that's the bundled Claude
Code CLI, with the real Codex gateway and `CodexHttpSession` on the
Codex row. Baseline rows use a real `AsyncOpenAI`, and the Microsoft 365
row the real Graph client. Only three writes are replaced:
`record_cost_usage`, `charge_turn` and `log_platform_cost`. Everything
that decides what to charge runs as in production: the cost baseline on
resume, the OpenRouter reconcile and its fallback, Moonshot repricing,
cache-TTL pricing, and Codex and Microsoft 365 recording tokens without
charging.

| Route | Charge basis |
| -- | -- |
| SDK, direct Anthropic | invoice |
| SDK, OpenRouter (Sonnet, Kimi), reconcile resolved | invoice |
| SDK, OpenRouter, `/generation` refused or reconcile off | CLI estimate
(per turn) |
| SDK, OpenRouter Kimi, reconcile off | Moonshot rate card |
| SDK over the Codex gateway | none (user's ChatGPT plan); tokens logged
|
| Baseline, OpenRouter / direct Anthropic | invoice (`usage.cost` / rate
card at the requested cache TTL) |
| Baseline, Ollama | none; tokens logged |
| Microsoft 365 Copilot | none; estimated tokens logged |

Five rows inject provider failures: a 529, or a connection dropped
before any work. On the four SDK rows the CLI's own retries are off, so
the failure reaches the platform's handling. A dropped connection
mid-tool-round makes the platform relaunch the CLI. That turn must still
be charged the call billed before the drop, plus the retry, and nothing
else. On the baseline row, the OpenAI client retries the 529 itself, and
the retry must be charged once.

Each run also checks that it took its planned path: SDK launches after
the first resumed the chat's session, every planned fault fired, and
nothing unexpected hit the provider. A broken harness therefore can't
pass vacuously.

The suite needs no database or Redis. SDK rows need the bundled CLI,
which ships in the Linux wheel, so they skip on Windows. The rest run
anywhere. All 16 take about 17s in the backend's Linux image.

**What it found.** Both findings are pinned as known gaps. Each runs as
a strict expected failure confined to its own check, so cost regressions
in the same scenario still fail the test. A gap that stops reproducing
also fails the test, asking for the gap to be removed.

1. **A retried turn's cost row and session usage carry only the retry's
tokens.** The charge is correct (the hotfix keeps the pre-turn baseline
across the retry), but the failed attempt's tokens are dropped. #15305's
repair notes mention this. It also makes cost-vs-token queries, such as
that repair SQL, read a retried turn as overcharged.
(`sdk-anthropic-dropped-after-tool`, check `tokens`.)
2. **The platform no longer recognises the bundled CLI's 5xx errors as
transient.** CLI 2.1.284 words a 529 as `API Error: 529 Overloaded. This
is a server-side issue…`, with `AssistantMessage.error=server_error`.
`_TRANSIENT_ERROR_PATTERNS` looks for `status code 529`, which no longer
appears in the CLI binary. Once the CLI's own retries are exhausted, the
platform doesn't retry either: the user gets the CLI's raw error text as
the reply instead of a retry or the friendly message. Billing stays
correct (an unbilled turn is charged $0). The fix is #15381.
(`sdk-anthropic-overloaded-on-resume`, check `path`.) Whichever of the
two PRs lands second updates this suite. With #15381 this scenario
passes, and `sdk-anthropic-overloaded-after-tool` retries like its
dropped-connection sibling, picking up gap 1. Verified with the real
CLI: 14 passed, 2 xfailed.

### Changes 🏗️

- `billing_e2e/provider.py`, `wire.py`, `replies.py`, `invoice.py`: the
fake upstream, its wire formats, what its "model" answers, and its price
tables and invoice.
- `billing_e2e/ledger.py`: records the platform's three writes per turn
and waits for the deferred ones (reconcile task, cost-log tasks).
- `billing_e2e/sdk_turns.py`, `baseline_turns.py`, `m365_turns.py`:
drive one turn on each engine with only persistence, locks, transcript
storage, the copilot's own tools and per-user lookups stubbed.
- `billing_e2e/scenarios.py`, `audit.py`, `thread_billing_test.py`: the
route matrix, the per-turn checks, and the test.
- `.github/scripts/backend-allowed-skips.json`: lists the two known-gap
rows. Pytest reports an xfail as a skip, and CI rejects skips that
aren't listed. Each row fails the test once its gap is fixed, at which
point its entry goes too.
- No production code changes.

### Agents and large language models used

- Claude Code with Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] All 16 scenarios in CI's Linux image built from
`backend/Dockerfile` (builder stage), with the bundled CLI: 14 passed, 2
strict xfails (the gaps above). Also passes with no Redis or Postgres
reachable.
- [x] On Windows (no bundled CLI): the 5 non-SDK scenarios pass, the 11
SDK scenarios skip.
- [x] Alongside `sdk/turn_cost_*_test.py`, `cli_cost_state_test.py` and
`token_tracking_test.py` in one pytest session, with CI's shared session
loop (this suite runs on its own function-scoped loop): 61 passed, 2
xfailed.
- [x] Mutation check: each of these, applied alone to the code under
test, turns the listed scenarios red:

| Mutation | Scenarios that fail |
| -- | -- |
| Resumed SDK turn charged the CLI's restored total (the #15342 bug) |
all 5 direct-Anthropic SDK rows, both OpenRouter fallback rows |
| OpenRouter reconcile records the CLI estimate despite resolved lookups
| `sdk-openrouter`, `sdk-openrouter-kimi` |
| Baseline counts `usage.cost` twice | the 3 charged baseline rows |
| Baseline keeps only the last tool round's cost | the 3 charged
baseline rows |
| Baseline prices 1h cache writes at the 5m rate | `baseline-anthropic`
|
| Codex turn charges the platform | `sdk-codex` |
| Turn tree not charged | all 13 charged rows |
| Chat spend skips the daily budget | all 13 charged rows |
| Cost-log rows lose their chat | all 16 |
| SDK cost row drops cache-read tokens | the 6 SDK sync-write rows |

  - [x] `ruff`, `black`, `isort` and `pyright` clean on the new package.

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

No configuration changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
 2026-10-11 16:15:46 +00:00
Reinier van der LeerandClaude Opus 5.5 0560947a06 feat(backend/copilot): id every stream entry, checkpoint every persist, and resume a turn from a cursor (#15084) 
The copilot stream now lets a client resume a turn exactly where it left
off, and it carries every row the turn persists, in order. Every event
frame carries its Redis entry id, both engines publish a
`data-checkpoint` event after each persist of the turn's rows that
succeeds, and `GET /sessions/{id}/stream?turn=<turn>&after=<entry>`
serves exactly the entries after that cursor. Queued follow-ups and
reasoning now reach the stream as the rows they become. This is the
backend layer (L0) of the chat's own stream converter; today's chat
client ignores the ids and checkpoints and sees the same follow-ups and
reasoning it sees now.

### Why / What / How

**Why:** today a client can resume a turn only by replaying it from its
first event, and even that replay can come back incomplete:
- a running turn's stream is capped at 10,000 entries, so a long turn's
replay can open in the middle of a text block (W5);
- a finished turn's replay stops after 200 entries and then invents a
`finish` (W9);
- the server says nothing about which of the turn's rows it has already
persisted;
- the stream and the rows disagree in five places: a follow-up drained
mid-turn is published straight to Redis, ahead of events the engine sent
before it; an auto-continue turn's user row never reaches the stream;
reasoning can be persisted without being streamed; the SDK engine stores
a post-tool message's reasoning after its text; and the SDK adds a tool
call that follows a tool result to the earlier assistant row, where the
baseline opens a new one.

The converter we plan to build needs three things the backend does not
provide:
- a cursor to resume from;
- a server-side record of what is persisted, to compare the client's
view against;
- a read that returns exactly the entries the client has not seen yet.

**What:**
- **Entry ids.** Every data frame on the POST stream and on the resume
GET starts with `id: <turn_id>:<entry_id>`. Comment frames (`:
heartbeat`, `: usage`) carry no id, because an SSE parser drops the id
of a frame that has no data.
- **Checkpoints.** `data-checkpoint {rows, sequence, digest}` is sent
with `transient: true`. It means "the turn's first `rows` rows are
persisted, starting at DB `sequence`". It is published after:
  - the SDK engine's intermediate flush and its turn-end persist;
- the baseline engine's mid-turn persist (when it drains a queued
follow-up) and its turn-end persist;
  - the test-mode dummy engine's persist.
  
An engine publishes one only if the persist landed and no text or
reasoning block and no tool call is open. The last checkpoint's entry
id, row count and sequence are recorded for the turn and returned on
`GET /sessions/{id}` as `active_stream.checkpoint`.
- **Follow-ups on the stream, as their rows.** The engines yield
`data-pending-drained` at the point they append the user rows, with each
row's own content. That covers:
- the turn-start drain and approved held calls, as the first entries
after `start`;
- the SDK's follow-up after a tool result, and the baseline's drain
between rounds;
- an auto-continue call. It now persists one user row per queued
message, as the turn-start drain does, instead of one combined row.

  The drain no longer publishes the hint itself.
- **Reasoning always streamed.** `render_reasoning_in_ui` is retired.
Nothing deployed sets `CHAT_RENDER_REASONING_IN_UI`, so it was True
everywhere; turning it off would only have made the stream and the rows
disagree. The SDK engine now stores a post-tool message's reasoning row
before its text row, in the order the stream carries them.
- **One row rule on both engines.** A new assistant row opens after tool
results, for a tool call as well as for text (design §1.4). The baseline
already did this; the SDK used to add the call to the earlier row.
- **What a reload shows.**
- The new row for a tool call after a tool result is invisible to
today's renderer (`convertChatSessionToUiMessages.ts`). The renderer
draws no bubble for a tool row and merges consecutive assistant rows
into one bubble, so either shape reloads as the same bubble with the
same parts in the same order; only the bubble's internal id moves to the
last row's sequence. `drift/tool-row-render.test.ts` pins this over the
recorded SDK turn, and it fails if the renderer stops merging across a
tool row.
- Two of the other changes do show on reload, and in both cases the
reload now shows what the live stream already showed. An auto-continued
turn's queued messages reload as one bubble each, not as one combined
bubble. A post-tool SDK message's reasoning now sits before its text in
the bubble. These two are read from the renderer, not screenshotted.
- **A turn's rows** start right after the message that triggered it.
Follow-ups drained at turn start therefore belong to the turn, and
checkpoints name that first row.
- **Cursor read.** `GET /sessions/{id}/stream?turn=<turn>&after=<entry>`
(`after` defaults to `0-0`) returns one of:
- `200`: the stored entries after the cursor, then the live ones, with
no frame the server made up;
- `409 {reason: "trimmed", checkpoint}`: entries after the cursor were
trimmed;
- `410 {reason: "expired"}`: the stream is gone, or the turn belongs to
another session.
  
  The GET without `turn` behaves as before.
- **Trimming.** A running turn's stream is never trimmed. When the turn
completes, its stream is trimmed back to the last checkpoint. A finished
turn's replay returns its whole tail, ending with the stored `finish`
(W9).

**How:**
- **Queue entries.** The subscriber queue carries `(id, chunk)`. The id
is `<turn>:<entry>`; it is `None` for a heartbeat or an end-of-stream
marker that the registry made up. The turn is part of the id because a
duplicate POST subscribes without knowing its turn. `sse_frame` in the
registry writes the frame, and both routes and the fixture recorder use
it. The Discord and Slack bots and the session waiter unpack the pair
and are otherwise unchanged.
- **Per-turn hash.** A new hash, `chat:stream:<turn>:meta`, holds the
turn's `session_id` and its last `checkpoint`. It exists because the
session meta is overwritten by the next turn. With it, the route can
serve a finished turn after the next one has started, and can refuse a
turn that belongs to another session.
- **Detecting a trim.** A stream counts as trimmed when `XINFO STREAM`
reports `entries-added > length`. A cursor older than the first
remaining entry then gets a 409, and that entry is the checkpoint the
trim kept. `max-deleted-entry-id` looks like the obvious signal, but on
Redis 7 `XTRIM` does not update it: it read `0-0` after a `MINID` trim
of two entries, measured locally.
- **Where the checkpoint is sent from.** Engines yield the checkpoint
rather than publishing it directly. `wrap_stream_with_heartbeat` drains
the engine ahead of the publisher, so a direct publish could overtake
events yielded before the persist. The SDK engine yields its turn-end
checkpoint after its `finally`, because an async generator cannot yield
while it is being closed.
- **Is it safe to send?** `turn_checkpoint` returns nothing unless every
row from the turn's start has a sequence, no row still waits on a
tool-call back-fill, and the sequences have no gap. That rules out a
failed persist, a failed back-fill (which only logs and leaves
`tool_calls_pending_save` set), and a retry that left rows in the DB
which the in-memory session no longer holds.
- **Digest.** SHA-256 over the compact JSON of `[role, content
(tool_call_id for a tool row), [[call id, call name], ...]]` for each
row. The browser can reproduce it with `JSON.stringify`.
- **Fold check.** `stream_drift/fold.py` folds a stored stream by the
design's row rules. Every recorded turn, fresh or committed, must fold
to the row count and digest each checkpoint names, and at the end to
every persisted turn row.
- **Recorded fixtures.** Eight turns are recorded: a dummy text turn, a
baseline tool round, the SDK's late tool result, a baseline turn with a
turn-start and a mid-turn drain plus reasoning, an SDK turn with
reasoning and a follow-up after a tool call, an SDK auto-continue, and
one turn per engine with a tool call right after a tool result. They are
what the converter's replay-equivalence test will compare against, so
they come from the real persist path. Each recorder test creates a user
and a session in Postgres, runs the engine with its own persistence, and
reads `rows.json` back the way the session GET does. The committed files
were recorded on a throwaway `pgvector/pgvector:pg16` migrated with the
branch's migrations (`RECORD_COPILOT_STREAM_FIXTURES=1`, with
`DATABASE_URL`/`DIRECT_URL` pointed at it). In CI, the same tests check
the files against the CI database.
- **Config.** `stream_max_length` is removed; nothing reads it now.
`stream_replay_count` is now the replay batch size, not a cap.

**Drift markers (#15060):** L0 fixes one of #15060's pinned failures.
- Now passes: W5 in the backend. Its xfail and its allowlist entry are
removed. The test publishes 10,500 entries, because redis-py's `maxlen`
is approximate and a 10,000 cap only starts trimming whole 100-entry
nodes past that point.
- Still pinned: W2, W3, W6, W8 and the approval-wake case. These are
other fixes' work.
- Unchanged: every frontend `today:` pin. Those wait for the converter
(L1) and the runtime (L2).

**Where this departs from the design:**
- The checkpoint is recorded in the per-turn hash, not in the session
meta hash.
- The queue id includes the turn.
- Comment frames carry no id.
- The design says the bots and the session waiter read Redis directly.
In fact they read `subscribe_to_session`'s queue, so they now unpack its
pairs.
- The baseline persists mid-turn only when it drains a follow-up. Its
turns therefore carry fewer checkpoints than the design's per-tool-round
persist would give.
- An auto-continue persists one user row per queued message, not the
combined message. The model still receives them combined.

**Review triage.** CodeRabbit's review of `490f9619bd` had one
outside-diff 🟡 Minor and no inline threads. It is fixed in `e98158b848`:
a checkpoint waits while a tool-call back-fill is pending, and
`stream_checkpoint_test.py` fails without the guard. Sentry's review of
`e98158b848` had one inline LOW finding, which I declined and resolved:
an empty recorded stream in `recording.py` raises `IndexError`, and in
that case the test has to fail anyway. CodeRabbit's review of
`c8af9dbe1c` has no outside-diff findings and one nitpick, which I
declined. It asked to wrap the recorder fixture's cleanup in
`try/finally` and to disconnect Prisma. pytest already resumes the
fixture after a failing test, so the `try/finally` changes nothing.
Disconnecting would break every later test that shares the session's
client. Its static-analysis notes were also declined as not applicable
to this code: Flask's `jsonify`, the in-process `http://test` client,
and the OpenAPI file's global-security setting.

**Open:**
- Nobody has measured `XLEN` on the largest Dev streams; that needs a
shell on the Dev Redis. Until someone measures it, a running turn's
stream grows with the turn, with no cap.

### Changes 🏗️

- `backend/copilot/stream_registry.py`:
  - `(id, chunk)` queue entries and `sse_frame`;
- whole-tail replay and `subscribe_to_turn`, with `TurnStreamGone` and
`TurnStreamTrimmed`;
  - the per-turn hash;
- no trimming while a turn runs, and the trim to the last checkpoint at
completion;
  - the listener stops once the session meta names a later turn.
- `backend/copilot/stream_checkpoint.py` (new): `turn_checkpoint` and
`rows_digest`.
- `backend/copilot/response_model.py`: `StreamCheckpoint`
(`data-checkpoint`).
- `backend/copilot/sdk/service.py`, `baseline/service.py`,
`sdk/dummy.py`: checkpoints after persists, drain hints where rows are
appended, the turn's first row after its trigger, one row per
auto-continued message, and reasoning always streamed. The SDK also
orders reasoning before a post-tool placeholder, and opens a new
assistant row for a tool call after tool results.
- `pending_messages.py`: the drain no longer publishes the hint.
`pending_message_helpers.drained_rows_entry` builds it.
- `config.py`, `baseline/reasoning.py`, `sdk/response_adapter.py`:
`render_reasoning_in_ui` removed, along with the tests of its
suppression.
- `backend/api/features/chat/routes.py`:
  - ids on the POST and resume frames;
  - the `turn` and `after` query parameters and the 409/410 answers;
  - `active_stream.checkpoint`.
- `bot/bot_backend.py`, `sdk/session_waiter.py`: unpack the queue pair.
- `config.py`: `stream_max_length` removed.
- `stream_checkpoint.py`: `canonical_rows` and `canonical_digest`, which
the fold check shares; no checkpoint while a tool-call back-fill is
pending (`stream_checkpoint_test.py`).
- `stream_drift/`:
- the recorder stores the whole untrimmed stream, writes frames with
ids, and reads the rows from Postgres;
  - the SDK scripted turn runs the real consume loop, flush included;
- `saving_into`, a stand-in save that numbers rows like the DB does, for
the engine checkpoint tests;
- `fold.py`, the reference fold, and `sdk_service_turn`, which runs the
whole SDK engine over a scripted CLI;
- the frontend's `drift/tool-row-render.test.ts`: the recorded SDK turn
reloads the same as its old joined shape;
  - three new recorded turns.
- Tests: `stream_registry_resume_test.py`, `chat/stream_resume_test.py`,
`stream_drift/checkpoints_test.py`,
`retry_scenarios_test.py::TestTurnEndCheckpoint`, the frontend's
`drift/stream-wire-compat.test.ts`, and the re-recorded fixtures.
- `frontend/src/app/api/openapi.json`: regenerated.
- `.github/scripts/backend-allowed-skips.json`: W5's entry removed.
`.pre-commit-config.yaml`: detect-secrets skips the recorded stream
fixtures, whose checkpoint digests read as high-entropy hex, as it
already skips the style eval's hash baseline.

### Agents and large language models used

- Claude Code with Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] The cursor read over each of the three recorded turns, at every
cursor, returns exactly the committed frames after it. The trimmed and
gone answers carry the right body.
- [x] Both engines publish a checkpoint only after a persist that
landed. Each checkpoint equals the one the most recent save describes,
and none sits inside a block or tool call. This is checked through the
SDK consume loop's real flush, the SDK `finally`, and the baseline's
mid-turn and turn-end persists.
- [x] Every recorded turn folds to its persisted rows at every
checkpoint and at the end.
- [x] The AI SDK's own parser builds the same message with and without
the ids and checkpoints, on all eight recorded turns.
- [x] Fourteen mutations each turn a named test red, including the seven
that undo one of the stream-and-rows fixes (see the evidence comments).

**Verified:** I ran these locally against the local Redis cluster:
- the new tests: registry cursor 9, route cursor 5, engine checkpoints
and the fold check over all eight recorded turns 20, SDK turn-end 2, and
wire compatibility on all eight turns 8 (vitest);
- #15060's frontend drift suite (11 tests), which passes unchanged;
- on a throwaway `pgvector/pgvector:pg16` migrated from the branch: the
recorder and all of `stream_drift/`, and the copilot suites the change
touches, 1,844 passed and 6 xfailed. Those cover the registry, bot,
waiter, drift, executor (including the engine re-entry sweep),
pending-message, SDK service, persistence, adapter and guardrail,
baseline, held-call, e2e and architecture tests;
- `backend/api/features/chat` (246), `tools/capability_tools_test.py`
(40) and `test_block.py` (3,600);
- ruff, pyright, `tsc` and eslint;
- after the row-rule alignment: the SDK suites (service, persistence,
adapter, guardrails, interrupted-partial, tool adapter, turn-end
checkpoint), the drift checks and the engine re-entry sweep, 457 passed
and 8 xfailed; the recorder on a throwaway Postgres, 8 passed.
- after the rebase onto dev: the channel bot already reads the
subscriber queue's `(entry_id, chunk)` entries, but its held-call card
test from #15069 still queued bare chunks and now queues entries; with
it fixed, `backend/copilot` plus `architecture_test.py` gave 8,250
passed and 11 xfailed (the one red, `benchmark_test.py`'s hit-rate
check, reads this machine's data and passes in CI), and `test_block.py`
3,666 passed.

Fourteen mutations each turned a named test red. Eight of them undo one
of the stream-and-rows fixes, and each of those fails the recorder's
fold check.

Not exercised: a real browser; the Dev `XLEN` measurement; and the whole
of `retry_scenarios_test.py`, of which only the new class ran here.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
 2026-10-06 22:22:27 +00:00
Reinier van der LeerandClaude Opus 5.5 e170df9d2a dx(platform/copilot): reproduce each known cause of the chat drifting from the backend (#15060) 
Adds a test suite that reproduces, on today's code, each known way the
copilot chat's screen drifts from what the backend holds, so every fix
to one of them is proven by a test that flips, and no fix can quietly
break another.

### Why / What / How

**Why:** the chat can show something other than what the backend has:
text painted twice after a reconnect, "Connection lost" on a healthy
turn, a reloaded turn that never renders, a reply that vanishes when the
turn ends, a chat stuck on "running". The fixes in flight or planned for
these (#15044, the approval-wake fix, a stream converter of our own) had
no test that shows each defect today and shows it gone after.

**What:** one test per cause, each against the same invariants, in the
state it is in today:

| Cause | Where | Today |
|---|---|---|
| W1 a tab shown again after 30 s resumes over a live stream | frontend
| first text block painted twice |
| W4 the 30-minute load-balancer cut (simulated as a clean close) |
frontend | "Connection lost" toast, first block painted three times |
| W5 a resume onto a trimmed stream | frontend + backend | parser dies,
an intact later block never renders while the turn runs, toast; backend:
a running turn's stream is trimmed, replay opens on 84 orphan deltas |
| W7 a tool call silent for 70 s | frontend | stall watchdog toasts
"Connection lost" |
| W8 the final persist fails | frontend + backend | the streamed reply
vanishes at the end-of-turn hydrate; backend: the baseline engine
finishes the turn as a success |
| chained turn (#15044) | frontend | "Connection lost" toast |
| approval wake: cards answered during a turn | frontend + backend | two
"Connection lost" toasts; backend: the executor drops the woken turn as
a duplicate and it reads running forever |
| W2 a second dispatch while a turn runs | backend | `create_session`
hands the meta to the new turn |
| W3 an executor dies mid-turn | backend | the turn reads running until
its meta expires |
| W6 a silent turn outlives its meta | backend | heartbeat gaps
10/20/30/7,160 s against a 3,600 s TTL |
| #14762 finish-step with an open block | frontend guard | holds |
| a sent turn, and a reload into a running turn | frontend guards | hold
|

The completion writers' turn guard (the other half of W2) is already
pinned by `TestCompletionOnRealRedis` from #14974.

**How:**
- **Recorded fixtures.** `backend/test/fixtures/copilot_stream/<turn>/`
holds `entries.jsonl` (the turn's Redis stream as XRANGE returns it),
`frames.jsonl` (what the resume route writes per entry) and `rows.json`
(what the session GET returns). Three turns are recorded through the
real registry and Redis: the dummy SDK engine, the baseline engine on a
scripted provider with a tool round, and the SDK engine's own adapter
and row builder replaying Dev session c68992f9's late tool result.
`recorded_turns_test.py` fails when the pipeline stops producing the
committed files; `RECORD_COPILOT_STREAM_FIXTURES=1` re-records them.
- **Frontend harness** (`__tests__/drift/`). The real `CopilotChatHost`
and its hooks run against a simulated backend that publishes a recorded
turn frame by frame into a stream every connection replays, with cuts,
trims, 10 s heartbeats and chained turns. The oracle is a fresh page
load from the persisted rows. Each case is held to six invariants: no
connection toast, no text block painted twice at any moment, the settled
chat equals the reload, nothing the user watched stream in disappears,
intact text renders while the turn still runs, and a cleanly finished
turn is not shown as failed.
- **Expected failures pin their cause.** A frontend case lists the
invariants a live bug breaks today (`today: [...]`) instead of using
`test.fails`, which stays green on a harness error too. Every other
invariant must hold, and a pinned one that holds again fails the case
with "Now holds: …, remove it from `today`". Backend causes are
`xfail(strict=True, raises=AssertionError)` for the same reason, and are
allowlisted in `backend-allowed-skips.json` because JUnit reports an
xfail as a skip. Whoever fixes a cause removes its pin, or its xfail
marker and allowlist entry.

### Changes 🏗️

- `autogpt_platform/backend/backend/copilot/stream_drift/`: the
recorder, scripted engines, fixture check and the backend cause tests.
- `autogpt_platform/backend/test/fixtures/copilot_stream/`: three
recorded turns.
-
`autogpt_platform/frontend/src/app/(platform)/copilot/__tests__/drift/`:
the simulator, the drift report and three test files.
- `.github/scripts/backend-allowed-skips.json`: the six backend expected
failures.

No production code changes.

### Agents and large language models used

- Claude Code with Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Each frontend expected failure, run with its pins removed, reports
exactly the pinned invariants; each backend one, run with `--runxfail`,
fails on its assertion.
- [x] Each guard turns red under one mutation (see the evidence
comment).
- [x] Every frontend case passed in at least three runs; backend CI's
skip validator accepts the report with the allowlist entries and rejects
it without them.

**Verified:** I ran every case locally on `origin/dev` `fade9292ff`: the
11 frontend cases, each passing in at least three runs, the 9 backend
tests against the local Redis cluster, each expected failure unpinned or
with `--runxfail` to read the assertion it fails on, three guard
mutations (removing the end-of-turn hydrate, a 1 s stall watchdog,
reverting #14762 in the SDK adapter and re-recording),
`architecture_test.py`, `test_block.py`, the copilot source sweeps
`executor/entry_points_test.py` and `tools/capability_tools_test.py`
(the scripted baseline turn carries a root envelope as the executor's
does), the whole copilot vitest directory, eslint, tsc, ruff, pyright.
Not exercised: real browsers (tab hiding is simulated with
`visibilitychange`; the load-balancer cut with a clean close), and the
SDK engine's own persist-failure path, which W8 covers on the baseline
engine only.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
 2026-10-06 21:01:37 +00:00
BentlyNicholas TindleClaude Opus 5.5
 		c23b67a643 ci(platform): let single-container PR builds read the GHCR layer cache (#15013) 
### Why / What / How

**Why:** Single-container PR builds run cold every time: 386-462 s for
the image build on recent PR runs, against 151-177 s for a warm `dev`
push. #14969 moved the dependency layers (apt, Node, RabbitMQ/Erlang,
FalkorDB, frontend) under the code layers, so a PR that only touches
code could reuse almost all of them. PR runs just never read the cache.

**What:** PR runs now read the shared `dev` GHCR layer cache
(`v4-{arch}`), the same trick as the e2e job in #14892. They still never
write it.

**How:** Stacked on #14969, which has now merged into `dev`.
- `single_container_cache.py` no longer returns early for
`pull_request`. PR runs get the three `cache-from` lines
(`backend-server-base`, `backend-server`, `single-container`) for
`v4-{arch}` and no `cache-to`. Only a push to `dev` and
`workflow_dispatch` export, as before.
- The workflow's "Authenticate to GHCR build cache" step now runs on PRs
too, with `continue-on-error: ${{ github.event_name == 'pull_request'
}}`. On a PR, a failed login (for example a fork's read-only token
without package access) gives a cold build, not a red job. On a `dev`
push or a dispatch, a failed login still fails loudly.

### Changes 🏗️

- `.github/scripts/single_container_cache.py`: drop the `pull_request`
early return.
- `.github/workflows/platform-single-container-docker.yml`: the GHCR
login step drops `if: github.event_name != 'pull_request'` and gains
PR-only `continue-on-error`.
- `autogpt_platform/single-container/tests/test_ci_registry_cache.py`:
- `test_pull_requests_never_read_or_write_registry_cache` becomes
`test_pull_requests_read_but_never_write_registry_cache`: exactly the
three `cache-from ...:v4-amd64` lines, and nothing with `cache-to`
(arm64 too).
- New `test_only_pull_requests_tolerate_a_failed_cache_login`: the login
step has no `if:`, and its `continue-on-error` is PR-only.
- `docs/platform/contributing/tests.md`: the Caching section said PRs
don't read or write registry caches. It now says they read but never
write, and that on a PR a failed login gives a cold build.

No `.env`, compose or permission changes. The job keeps `packages:
write`: PR runs never get a `cache-to` line, and fork PRs get a
read-only token anyway, the same shape as the e2e job.

### Evidence

Collected on a tester VM against `dev` 6b97e95 (#14969 merged) plus this
branch:

| | `dev` | this branch |
|---|---|---|
| `single_container_cache.py`, `event=pull_request` | empty set block |
3 `cache-from ...:v4-{arch}` lines, no `cache-to` |
| `single_container_cache.py`, `event=push` to `dev` | 3 `cache-from` +
1 `cache-to` | unchanged |
| new tests on `dev`'s script/workflow | 2 of 7 FAIL | - |
| login test with plain `continue-on-error: true` | FAIL | - |
| `autogpt_platform/single-container/tests` (whole suite) | - | 173 run,
OK (1 allowed skip) |

- **Unreadable cache means a cold build, not an error.** On the VM, with
no GHCR credentials, a `docker buildx bake` using the PR-run
`cache-from` lines (docker-container driver, buildkit v0.32.0 as in CI)
logged `ERROR: failed to configure registry cache importer ... 401
Unauthorized`. It then kept building cold: 58 steps finished before I
stopped it at 300 s.
- **This repo's token can read the package.** `dev` push run 36479435743
(linux/amd64) logs in with `GITHUB_TOKEN`, runs `importing cache
manifest from
ghcr.io/significant-gravitas/autogpt-single-container-buildcache:v4-amd64`
(55 `CACHED` steps), then runs `exporting cache to registry`. The
package is private (anonymous pull gets 401).

### Caveats

- **Fork and Dependabot PRs are unproven.** They read the cache only if
the `autogpt-single-container-buildcache` package gives this repo
Actions read access. The `dev` push run above shows this repo's token
reading and writing the package, but I could not read the package
settings: my token lacks `read:packages`. This PR is a same-repo branch,
so its own run does not exercise the read-only token. If access is
missing, those PRs log the 401 and build cold, as they do today.
- The tester VM had no GHCR credentials, so it did not show a warm cache
hit. This PR's own `pull_request` run is the live proof of the warm
path.

### Agents and large language models used

- Claude Code with Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] `single_container_cache.py` output for `pull_request` and `push`,
before and after
- [x] The changed and new tests fail on `dev` and pass with the change.
Whole single-container suite: 173 OK
- [x] An unauthenticated bake with the PR `cache-from` lines builds cold
instead of failing
- [ ] This PR's own single-container run shows cache hits and a shorter
build than the 386-462 s PR baseline

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

---------

Co-authored-by: Nicholas Tindle <[email protected]>
Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
 2026-09-28 23:30:16 +00:00
Nicholas TindleandClaude Opus 5.5 6b97e95ffd refactor(platform): put the single-container image's code layers last (#14969) 
### Why / What / How

**Why:** The single-container image was built on top of the finished
backend `server` image. Everything the appliance adds (Postgres, Valkey,
nginx, supervisor, Node, RabbitMQ/Erlang, FalkorDB and the frontend,
about 0.8 GB) sat above AutoGPT's code. So every commit re-ran those
install steps and stored all of it again. On a self-hosted box that
builds branch images, a one-line backend change cost 878 MB of Docker
disk.

**What:** Reorder the image so dependencies sit below the code. The
cloud `server` target still runs the same steps in the same order.

**How:**
- `backend/Dockerfile` splits the `server` stage where the code starts:
- `server-base`: packages, Python, Node, Prisma engines, Chromium and
the dependency venv.
  - `server`: `FROM server-base` plus the unchanged code steps.
- The appliance builds `FROM server-base`, adds its own dependencies,
then the frontend, then AutoGPT's code.
- The code is copied from the `server` image with its venv dropped, so
it is exactly the files the backend image ships. The appliance then
re-runs `poetry install --only-root` to create the entry points.
- `IMAGE_VERSION`/`VCS_REF` and the OCI labels move below the last
layer, so the per-build arguments no longer invalidate cached `RUN`
steps.

**Side effect on the image scans:** Trivy treats every layer below the
last `CMD` in the image history as a base image and skips secret
scanning there. The `server` stage's `CMD ["rest"]` used to sit in the
appliance's history, so Trivy never scanned the backend's dependencies
or code for secrets. It now does, and two fixes follow:
- One pre-existing false positive: yt-dlp's Shahid extractor embeds that
site's public client key.
`autogpt_platform/single-container/.trivyignore.yaml` ignores that one
rule in that one file, in all four scan steps.
- The report verifier required a `Type` on every Trivy result, but Trivy
never writes one for secret results. It now accepts clean secret results
without `Type`, and any secret finding still fails.

### Changes 🏗️

- `autogpt_platform/backend/Dockerfile`: split `server` into
`server-base` and `server`. No steps changed.
- `autogpt_platform/single-container/Dockerfile`:
  - Builds on `server-base`, with a new `backend-code` stage.
- The frontend, backend code and entry points go last; the build
arguments and labels go at the end.
- Two cleanup commands (the `config.json` and `.next/cache` symlinks)
moved next to the files they touch. No other command changed.
- `autogpt_platform/single-container/docker-bake.hcl`: new
`backend-server-base` target, passed in as the `autogpt-backend-base`
context.
- `.github/scripts/single_container_cache.py`: the new target reads the
registry cache like the other two.
- `autogpt_platform/single-container/.trivyignore.yaml` (new) and
`.github/workflows/platform-single-container-docker.yml`: one ignore
entry (rule `aws-access-key-id` in `yt_dlp/extractor/shahid.py`), used
by the CI scans and the release digest scans.
- `.github/scripts/verify_single_container_reports.py`: secret results
need only `Target` and `Class`.
- Tests:
- `test_dockerfile_layers.py` (new): `server-base` holds only
dependencies; the appliance copies everything the `server` target adds;
the build arguments come after every layer; the bake wiring is in place.
- `test_ci_scan_ignores.py` (new): every Trivy step uses the ignore
file.
- `test_ci_registry_cache.py`: every bake target reads the validation
cache.
- `test_ci_report_verifier.py`: a clean secret result passes without
`Type`; a secret finding fails; a package result without `Type` still
fails.

**CI caching:**
- The e2e GHCR cache from #14892 and #14877 is keyed by compose target
(`backend-server`, `backend-migrate`). Those targets are unchanged and
every existing step keeps its cache key.
- The single-container registry cache rebuilds the moved appliance
layers once after merge.
- PR builds don't use the registry cache.

**Measured** on the same builder, building `dev` and this branch, then
each again with a one-line backend change:

| | `dev` | this PR |
|---|---|---|
| New Docker disk per commit | 877.7 MB | 45.5 MB |
| Rebuild after a one-line change | 72 s | 38 s |
| Image size | 3.52 GB | 3.52 GB |

### Agents and large language models used

- Claude Code with Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Built `dev` and this branch with the single-container bake file on
the same builder, then each again with a one-line backend change, and
compared new data per commit and rebuild time (table above)
- [x] Compared every file in the old and new single-container images
(path, type, mode, owner, size, symlink target, sha256) and their
config. They are identical apart from the edited `backend/Dockerfile`
and timestamps in the apt/dpkg logs and the ldconfig cache, which also
differ between two builds of unchanged `dev`
- [x] Compared the backend `server` target before and after: the first
14 of 17 layers are reused, and every file is identical except the
edited `Dockerfile` it ships
- [x] Reproduced the CI secret finding with Trivy 0.70.0 and fresh
caches: `dev`'s image has 0 findings, this image has 1 (`shahid.py`, in
the venv layer)
- [x] Checked the ignore entry: with the real entry both CI scans exit
0; a decoy entry for a different rule still exits 1
- [x] Ran the verifier on those real reports: it passes with the ignore
entry and fails with `Secrets=1` without it
- [x] Ran the single-container unit suite on Linux: 172 passed (163
existing plus 9 new). The layer-order and scan-ignore tests fail on
deliberate regressions
- [x] Ran `.github/scripts/platform-single-container-smoke.sh` against
the new image: all phases passed (boot, contracts, 7 s clean stop,
replacement, FalkorDB recovery, watchdog restart)
  - [x] Ran pre-commit on the changed files

<details>
  <summary>Example test plan</summary>
  
  - [ ] Create from scratch and execute an agent with at least 3 blocks
- [ ] Import an agent from file upload, and confirm it executes
correctly
  - [ ] Upload agent to marketplace
- [ ] Import an agent from marketplace and confirm it executes correctly
  - [ ] Edit an agent from monitor, and confirm it executes correctly
</details>

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

<details>
  <summary>Examples of configuration changes</summary>

  - Changing ports
  - Adding new services that need to communicate with each other
  - Secrets or environment variable changes
  - New or infrastructure changes such as databases
</details>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
 2026-09-28 21:12:32 +00:00
Nicholas TindleClaude Opus 5Reinier van der Leergpt-daybreak-blue-latest
 		43a7bc38d2 ci(platform): speed up and harden single-container validation (#14282) 
### Why / What / How

The appliance build was exporting a large GHA cache that was evicted
before reuse, while smoke tests spent substantial time polling boot
state. This workflow layer uses architecture-scoped GHCR validation
caches, overlaps immutable-image scans with runtime smoke tests, and
preserves fail-closed reports for both architectures.

Stack base: #14281; next: #14284. The maintainer-closed overlap layer
and already-landed CodeQL change are excluded from the remaining stack.
Historical timings and approvals do not validate this rewritten head.

Pwuts's earlier comparison measured appliance wall time at 20m46s ->
14m15s (-31%) and runner-minutes at 37.57 -> 25.40 (-32%). These were
single samples with cache-sensitive build time, not a new benchmark on
this head. Every-dev-push validation also adds runs for pushes that
previously missed the broad source filter; that cost is intentional.

### Changes 🏗️

- Replace GHA buildx caching with GHCR `mode=max` per architecture. Dev
pushes refresh a shared validation tag; dispatches refresh a
branch-scoped tag. PRs neither read nor write these registry caches.
- Make cache export best-effort (`ignore-error=true`) without making
image builds, smoke tests, report validation, or security scans
optional.
- Build release digests cold: no writable CI-cache imports, no GHCR
login, no package-read permission in the publication job. Smoke-test and
scan the exact pushed digest before the multi-platform manifest is
published.
- Run appliance validation on every dev push, release, and dispatch;
restrict PR triggering to the explicitly listed packaging inputs.
Source/schema/migration-only PRs no longer trigger this workflow;
full-stack CI still validates the composed platform, and appliance
validation runs on the resulting dev push. This coverage tradeoff was
explicitly approved by the evaluator.
- Manual dispatch validates but does not publish Docker Hub test images.
Publication is release-only; this is an intentional externally visible
behavior change, not just a performance change.
- Add `SCHEDULER_STARTUP_EMBEDDING_BACKFILL`, defaulting to **true**.
Existing production startup behavior remains enabled; isolated
full-stack tests may opt out. Regression tests pin the actual Config
default as well as enabled/disabled scheduling behavior.
- Poll initial health every second, retain the 30-second steady-state
watchdog cadence and three-real-failure restart threshold, and permit
smoke tests to advance cadence via USR1.
- Fix the smoke race by snapshotting all failure ordinals before
stopping nginx and accepting new scheduled or forced evidence for each
ordinal. A scheduled check can no longer cause the smoke test to wait
forever for a skipped forced ordinal. Missing, stale, or wrong-ordinal
evidence still fails.
- Record boot/phase timings and preserve original failing reports and
`validated: false` status on validation failures.

### Agents and large language models used

Codex: unknown runtime model; gpt-daybreak-blue-latest contributed to
earlier review/fixes. Earlier contributions and human reviewer
attribution remain in the commit trailers.

### Checklist 📋

- [x] Changes, publication policy, path-filter tradeoff, and test plan
listed.
- [x] Linux appliance helper suite through the actual CI JUnit reporter:
162 tests, 161 passed and the one pre-existing allowed Bash 3 skip.
- [x] Scheduler service-free unit suite on the rebased head: 115 passed.
In-memory default=False mutation correctly fails the new default-pinning
regression.
- [x] Changed backend files pass formatting/import-order and
Linux-targeted type checking.
- [x] Publication self-tests and smoke-script Bash syntax pass.
- [x] Same-head GHCR cold/warm dispatch pair passed both architectures
on pre-restack `1be28e2d`: builds 10m49s -> 2m39s amd64 and 9m20s ->
2m25s arm64, 35/98 warm cached vertices each. Full evidence and phase
timings are in the September 22 review follow-up; the upper layer is
unchanged, but new-head checks below remain required.
- [x] Predecessor `77d09c54ce32f6424adf54426abd3c20b8e4130c`: [amd64 and
arm64 builds, full smoke tests, scans, and
reports](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35781920373)
passed; publication skipped. [Backend
CI](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35781920605)
passed all 18 jobs, including the [original duplicate suite
rerun](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35781919712/attempts/2).
- [x] Current `628a444a79080fe77ef07fb852a70b19219a6beb`: [appliance
CI](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35793258786)
passed both architecture builds, complete smoke tests, scans and report
gates; publication skipped. [Backend
CI](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35793258641)
passed all 18 jobs. Range-diff confirms this PR's own layer is
unchanged.
- [x] Inherited full-stack [35793258577 attempt
2](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35793258577/attempts/2)
passed all 28 tests with zero failures/skips/flaky results on unchanged
`628a444a79`; aggregate status is green. First warm attempt failed a
Copilot session-creation HTTP 504 (27/28 passed), despite successful SQL
restore and all eight fresh-key credential checks. The rerun missed the
evicted cache, so it is a cold success, not proof that the first warm
failure was fixed. Root #14281 independently passed same-head warm E2E.
Failed report/trace retained; no failure waiver or timeout/retry
changes, and no concrete link to this layer was found in Daybreak's
review.
- [ ] Fresh review, resolved threads, and two quiet post-CI polls.
- [x] `.env.default` and Compose are compatible: production default
remains enabled without an environment override.

Local limitation: the broad Windows backend format/type command reaches
six errors in unchanged files (including POSIX-only APIs and missing
optional Playwright fixtures). No unrelated fixes or type suppressions
are included. Hosted Linux checks now pass as linked above. Earlier
comments about this layer's backend tree being unchanged predate the
scheduler setting and are no longer the validation argument for this
head.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Changes release-only Docker Hub publishing, CI gating, and scheduler
startup embedding behavior; mistakes could block releases or alter
production cold-start search indexing.
> 
> **Overview**
> **Hardens and speeds up single-container CI** by switching validation
builds to architecture-scoped GHCR registry caches, overlapping image
build, smoke tests, and Trivy scans, and requiring machine-readable
JUnit/Trivy reports with a verifier before the job succeeds.
> 
> **Publication policy changes:** Docker Hub publishing is
**release-only**; manual `workflow_dispatch` no longer publishes
(validation-only). Release digest builds stay cold (no writable CI
cache). The publish script drops dispatch/`PUBLISH_REQUESTED` paths and
tightens release-tag and digest-enumeration error handling.
> 
> **Smoke and watchdog:** The smoke script polls health every second,
logs boot/phase timings (sanitized milestones), syncs on concurrent
scans, hardens cleanup and negative probes, and exercises watchdog
restart via USR1 with race-safe failure ordinals. `watchdog.sh` adds
forced checks on USR1 and a 1s initial health poll while keeping the 30s
steady-state cadence and three-failure limit.
> 
> **Scheduler:** Adds `scheduler_startup_embedding_backfill` (default
**true**); when false, the first embedding coverage job waits six hours
instead of running at startup.
> 
> **Triggers:** Every `dev` push runs full appliance validation; PRs
only run when listed packaging paths change.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
628a444a79. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Co-authored-by: Claude Opus 5 <[email protected]>
Co-authored-by: Reinier van der Leer <[email protected]>
Co-authored-by: gpt-daybreak-blue-latest (Codex) <[email protected]>
 2026-09-23 08:11:48 +00:00
Nicholas TindleClaude Opus 5Reinier van der Leergpt-daybreak-blue-latest
 		d46e93bacb ci(platform): overlap full-stack builds and harden E2E validation (#14281) 
### Why / What / How

Full-stack CI was serializing independent setup and image-build work and
could accept incomplete test evidence. This layer overlaps independent
work, reuses deterministic seed data, and validates every Playwright
result and coverage upload without retries or weakened assertions.

This PR contains only the full-stack workflow and its supporting
scripts/UI/test fixes. The signup hydration and sidebar changes are
necessary to make zero-retry browser tests reliable: signup waits for
auth initialization, task creation no longer depends on unrelated list
queries succeeding, and failed lists expose a usable retry path instead
of leaving the detail panel loading forever. They are not generic UI
refactoring.

Stack order after the September 22 restack: this PR -> #14282
(appliance) -> #14284 (docs). Classic, backend, and frontend layers have
already merged. #14283 remains closed and is excluded. #14508 was
redundant with merged #14277 and has been closed. Older
approvals/timings are historical, not validation of this head.

Pwuts's earlier comparison measured full-stack wall time at 14m30s ->
9m19s (-36%) and runner-minutes at 18.53 -> 12.58 (-32%). This is
historical single-wave evidence from the review, not a new benchmark on
the September 22 head.

### Changes 🏗️

- Overlap BuildKit/image builds and independent setup; preserve current
dev's `make init-env` fresh-secret bootstrap.
- Limit Docker cache export to dev pushes or explicit
`publish_build_cache` dispatches; retain the ability to cache test
commits.
- Key seed-data reuse on its transitive inputs, and keep database
restore errors blocking.
- Evaluate the seed cache immediately after checkout, before fresh
secrets or resolved Compose files exist. A regression pins this
ordering; old runs of the same source used different keys when
runtime-generated files were hashed.
- Refresh only the eight marked dummy integration credentials after cold
seeding or cache restore, preserving their IDs and verifying decryption
under the fresh run key. Missing/wrong fixtures and failed updates block
CI; no runtime keys are cached or reused.
- Require 28 Playwright tests (current inventory: 28 tests in 9 files),
one successful attempt each, no skips/flakes, and validated coverage
before upload.
- Cover per-test report errors even when aggregate statistics look
clean.
- Fix signup hydration, task-list errors/recovery, and async
navigation/form assertions without test retries.
- Select successfully created runs/schedules/webhook triggers directly,
avoiding stale-empty-list count races; three new regressions fail
against the old handler.

### Agents and large language models used

Codex: unknown runtime model; gpt-daybreak-blue-latest contributed to
earlier review/fixes. Earlier work also credited Claude Opus 5 and human
reviewer Reinier van der Leer; existing commit attribution is retained.

### Checklist 📋

- [x] Changes and test plan listed.
- [x] Full-stack helper/validator regressions: 31 passed locally. The
cache-ordering test fails against the prior workflow and passes with the
fix; changed Python format/import/type checks pass and YAML parses.
- [x] Credential/embedding regressions: 19 passed locally, including
fresh-key readability, incomplete/wrong fixture sets, database failures,
failed encryption verification, CLI routing, and disconnect-on-failure.
An in-memory no-op refresh mutation fails the new regression. Daybreak
reviewed the change and found no blockers.
- [x] Frontend format, lint, and type checks passed locally.
- [x] Complete frontend suite with the first-item fix: 7,850 tests
passed, zero failures/skips/todos. After a test-only props-interface
cleanup, format/lint/types and all 56 affected sidebar/signup tests
passed again.
- [x] Earlier 6-failure hosted E2E run diagnosed and reproduced; no
browser assertion, retry or test-inventory weakening. CI verified the
application fix on predecessor
`3541822cd9835bb69b2b4f9fb08b2c868aeef6ee`; the new cache-ordering
commit changes no application code.
- [x] Predecessor [full-stack E2E and schema
checks](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35781921471)
passed: downloaded report confirms 28 passes, zero failures/skips/flaky
results; report and coverage validators passed.
- [x] Cold full-stack run on `47e3d7625d`:
[35792006029](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35792006029)
passed in 8m31s, including verification of all eight seeded credentials.
Seed creation took 1m49s; key was saved for reuse.
- [x] Hosted frontend shard failure on `47e3d7625d` isolated to a
synchronous input assertion racing the detail hook's effect. Two
remaining assertions now await their exact populated values, without
retries or timeout changes. All 26 trigger tests, formatting, lint and
type checks pass; Daybreak confirmed the fix preserves coverage.
- [x] Current `83f34c613f7ff2d2ca0c2a42170cfec308ef6319`: [same-head
cold/warm
E2E](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35793038476)
passed all 28 tests with zero skips/failures/flaky results. Wall time
9m19s -> 7m01s; identical key restored, seed creation skipped, all eight
credentials verified under the fresh key. Detailed phase timings are in
the review follow-up.
- [x] Current-head
[frontend](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35793038495):
7,856 tests passed, zero failures/errors/skips, verified from all four
JUnit artifacts.
[Backend](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35793038502):
all 18 jobs passed; coverage settled green.
- [x] Inherited [appliance
checks](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35793038488)
passed both architectures, smoke tests and scans; publication skipped.
Aggregate PR status and all current-head checks are green.
- [ ] Current-head approval. All 17 inline threads are resolved; Pwuts
is requested. `/reapprove` cannot restore bot-only approvals, and the
bot's fresh-review quota is exhausted. No approval or limit bypass is
used.
- [ ] Two quiet post-CI/review polls for the complete stack.
- [x] Existing generated environment setup and Docker Compose
configuration retained.

This layer changes backend test seeding and its focused tests, not
production backend source. Earlier comments about complete backend-tree
equivalence were correct for their recorded heads, but their run IDs are
not claimed as fresh evidence here. #14282 changes scheduler
configuration and needs its own backend validation.

The fresh `/review` request was answered with the repository's
three-reviews-per-five-hours limit. That limit is unchanged. Existing
bot approval and the independent Daybreak reviews are recorded
separately; no claim is made that the rate-limited bot reviewed the two
newest fixes.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Changes CI orchestration, credential re-encryption in test seeds, and
auth/library UX paths relied on by E2E; production signup behavior
shifts slightly but scope is mostly CI and test reliability.
> 
> **Overview**
> **Full-stack CI** is reworked for faster, stricter E2E: the shared
frontend `setup` job is removed, BuildKit pull / image bake / DB startup
run in parallel with Playwright install, and Docker **cache export**
only runs on `dev` pushes or manual `publish_build_cache` dispatches
(compose cache script now allows an empty `--cache-to`).
> 
> **E2E seeding** moves the SQL cache step immediately after checkout
with a broader `e2e-test-data-v3` key, restores the DB with
`ON_ERROR_STOP`, and always runs `e2e_test_data.py
--refresh-credentials-only` so cached credentials decrypt under each
run’s fresh encryption key. Contract tests pin those workflow
guarantees.
> 
> **Test evidence** is tightened: Playwright runs with `--retries=0`, a
new `validate_playwright_json.py` enforces a 28-test floor and rejects
flakes/skips/multi-attempt results (even when aggregate stats look
clean), Cobertura is validated before Codecov upload, and validator
suites run in the API-types job.
> 
> **Product fixes for zero-retry E2E:** signup shows a loading shell
until auth has initialized once (preserving draft input on refresh); the
library agent view surfaces sidebar fetch errors with retry, keeps **New
agent task** usable when lists fail, and selects newly created
runs/schedules/triggers directly instead of racing empty counts.
Playwright helpers wait for hydrated signup and harden upload/navigation
assertions.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
83f34c613f. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Claude Opus 5 <[email protected]>
Co-authored-by: Reinier van der Leer <[email protected]>
Co-authored-by: gpt-daybreak-blue-latest (Codex) <[email protected]>
 2026-09-23 08:11:46 +00:00
Reinier van der LeerandClaude Opus 5 cb1d3bb0c7 ci(backend): let secret-gated skips pass on fork PRs (#14606) 
### Why / What / How

Backend CI can pass on a pull request from a fork again: a test that
skips because a repository secret is absent no longer fails the skip
gate. Forks get no secrets, so
`backend/copilot/tools/run_agent_test.py::test_run_agent_with_llm_credentials`
skips on every one of them (`OPENAI_API_KEY is not set`,
`_test_data.py:205-207`), its ID is not in the allowlist, and the three
copilot shards fail at `Validate test reports` — not at the tests.
`Validate backend skip policy` then fails downstream because it needs
all twelve artifacts, `Check PR Status` goes red, and since that is the
only required check, nothing from a fork can enter the merge queue. The
merge queue's own run has the secret, so a PR ejected for any other
reason reads as fixed and still cannot re-enter.

The skip policy gains a `secret_gated` section naming the environment
variable each ID depends on:

```json
"secret_gated": {
  "OPENAI_API_KEY": ["backend.copilot.tools.run_agent_test.test_run_agent_with_llm_credentials"]
}
```

Enforcement splits along where the secrets actually are. The per-shard
`Validate test reports` step runs inside the `test` job, which has
`OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}` in its env, so it is the
strict check: the skip is allowed only while the variable is absent or
empty, and a run that holds the secret and skips anyway still fails. The
aggregate `Validate backend skip policy` job holds no secrets and cannot
tell the two cases apart, so it accepts those IDs and leaves them out of
its staleness sweep. Nothing else moves: a skip nobody declared still
fails both checks, exactly as before.

**Allowlisting the ID in `common` is the obvious fix and is the wrong
one.** The test executes when the secret is present, so the aggregate
job's staleness sweep would report the allowance as obsolete and turn
every push to `dev` and every merge-queue run red — trading a red on
fork PRs for a red on everything else. Verified against the two runs
below:

```
$ python3 .github/scripts/validate_backend_skips.py --allow-skips-from <ID added to "common"> --reports-dir <dev run 35033474274>
Invalid backend skip reports: Python 3.11 has stale skip IDs in ...:
backend.copilot.tools.run_agent_test.test_run_agent_with_llm_credentials. Remove obsolete allowances or ...
```

Also considered and rejected: accepting the skip on the strength of its
message (`"<VAR> is not set"`) alone. It would need no env at all, but
it turns a reworded skip reason into a CI failure and it accepts the
skip even on runs that do have the secret, which is the case worth
catching.

### Changes 🏗️

- `backend-allowed-skips.json`: new optional `secret_gated` section,
mapping an environment variable to the skip IDs it gates. One entry
today.
- `validate_junit.py`: parses the section, rejects a variable name that
is not an uppercase identifier and an ID that is also unconditionally
allowed, and adds the gated IDs to the allowed set only when their
variable is absent or empty in the job environment.
- `validate_backend_skips.py`: accepts gated IDs regardless, and keeps
them out of the stale-ID sweep.
- Tests in both suites, plus a workflow test asserting every
`secret_gated` variable is really supplied to the `test` job from
`secrets.*` — a variable that is never set is not secret-gated, it
belongs in `common`.

No workflow change: the `test` job already has the secret in scope, and
the aggregate job deliberately does not.

#### Inventory: every environment-gated skip in the backend

`OPENAI_API_KEY` is the only repository secret any backend test gates
on. The whole workflow references `secrets.` exactly twice —
`CODECOV_TOKEN`, which no test reads, and `OPENAI_API_KEY` — so every
other gate resolves identically on a fork and on `dev` by construction.

| Variable | Tests | Set in CI? | Fork vs non-fork | Treatment |
| --- | --- | --- | --- | --- |
| `OPENAI_API_KEY` | `copilot/tools/_test_data.py` → 1 |
`secrets.OPENAI_API_KEY` | **differs** | new `secret_gated` entry |
| `OPEN_ROUTER_API_KEY` | `copilot/baseline/service_test.py`,
`copilot/service_test.py` → 2 | never | same | already in `common` |
| `AIRTABLE_API_KEY` | `blocks/airtable/_api_test.py` → 5 | never | same
| already in `common` |
| `RUN_E2E` | `blocks/mcp/test_e2e.py` → 5 | never | same | already in
`common` |
| `CI` | `test/test_data_creator.py` → 1 | always `true` | same |
already in `common` |
| `TRIAL_TEST_DATABASE` | `data/subscription_trial_*` | always `"1"` |
same | tests run; nothing to allow |
| `E2E_REDIS_CLUSTER_RESTART` | `data/e2e_redis_restart_test.py` | set
for the `data` shard | same | tests run; nothing to allow |

The remaining allowlisted skips are gated on a missing browser, an
absent FalkorDB port or an uninstalled package rather than on an
environment variable, and are unaffected by who opened the pull request.

### Verified

I ran both validators against the real shard reports of two CI runs —
[run
34851839105](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/34851839105)
on fork PR #14545, whose only failures are the three copilot shards and
the aggregate job, and [run
35033474274](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/35033474274)
on `dev` — and confirmed eight behaviours: the fork reports pass
per-shard and in aggregate; the same reports still fail when the secret
is present; the `dev` reports pass either way and raise no stale ID; and
an unlisted skip injected into the fork reports still fails both checks.
I then removed each half of the change in turn and watched the matching
test go red, including the workflow wiring test. 59 tests in
`.github/scripts` pass; `ruff check` is clean. Full evidence in a
comment below.

I did not reason about anything I could execute here, with one
exception: that this makes a fork PR's `Check PR Status` green
end-to-end is inferred from run 34851839105's job list, where the only
non-success jobs are the four this fixes. A same-repo PR cannot
reproduce a secretless run.

### Agents and large language models used

Claude Code with Claude Opus 5.

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Both validators run against the 12 real shard reports of a blocked
fork run and pass
- [x] Both validators run against the 12 real shard reports of a `dev`
run and pass, with no stale ID
- [x] The per-shard validator still rejects the gated skip on a run that
holds the secret
- [x] An unlisted skip injected into the fork reports is still rejected
by both validators
- [x] Each half of the change removed in turn, and the matching test
observed failing
  - [x] `.github/scripts` suite: 59 passed

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 <[email protected]>
 2026-09-16 16:23:58 +00:00
Nicholas TindleClaude Opus 5Reinier van der Leergpt-daybreak-blue-latest
 		14f8d6539a ci(frontend): shard integration tests with verified coverage (#14280) 
### Why / What / How

Frontend integration tests previously ran as one serial job behind a
shared setup job. This PR runs the same Vitest suite as four independent
shards, keeps each shard's coverage and JUnit evidence, and rejects
missing, malformed, or inconsistent reports before coverage upload.

All frontend jobs use standard GitHub-hosted `ubuntu-latest` runners.
The separate Full-stack CI workflow and its existing `big-boi` E2E job
are unchanged; this PR does not multiply paid E2E jobs or move shards
onto that runner.

### Changes 🏗️

- Run four native Vitest shards with coverage collection, per-shard
artifacts, and mandatory JUnit and Cobertura validation. Each active job
installs its own dependencies; removing the shared setup job removes the
serial dependency.
- Enforce the standing frontend no-skip policy. Skipped or omitted tests
fail validation; legitimate policy exceptions require review rather than
silent quarantine.
- Validate Cobertura structure, counters, nested packages, line
evidence, and rounding before Codecov upload. Failed validation still
preserves available artifacts.
- Add shared command/unittest JUnit reporters and their regressions.
Exact skip IDs, discovery crashes, fixture errors alongside passing
tests, repeated subtest failures, and later cleanup errors are covered.
Every failure is retained in the report, with duration counted once.
- Make report-write failures return a clear nonzero result and
diagnostic. Command failure/spawn exit statuses remain intact even when
the report cannot be written.
- Keep Chromatic disabled. Re-enabling it must require `(github.ref ==
'refs/heads/dev' || github.base_ref == 'dev')` and a paths-filter match
on `autogpt_platform/frontend/src/components/**`; the removed serial
setup job should not be restored just for that gate.

### Validation

- Current cleanup: 55 local tests pass (37 CI helpers and 18 reporter
tests). The report-write test covers ten command/unittest and
invalid-destination combinations.
- Three deliberately broken reporter variants are caught: any nonempty
allowlist permitting all skips, discovery crashes reporting success, and
fixture-error records being omitted. Two new subtest/cleanup regressions
failed before their source fix and pass afterward. All ten report-write
cases failed before their fix and pass afterward.
- Whole CI-helper/appliance-test Black formatting, diff checks, and
normal commit/push hooks pass. The cleanup's workflow change is comments
only; parsed frontend workflow configuration is identical to the
approved head, and Full-stack CI is byte-for-byte unchanged.
- Actionlint reports only the intentional existing constant-false
Chromatic gate; its earlier removal request was explicitly withdrawn
after baseline behavior was explained. No lint rules or CI gates were
weakened.
- Historical measurement from the earlier shard implementation: wall
time 14m07s → 4m32s and standard runner usage 16.58 → 20.01 minutes
(+21%). These are historical samples, not a new billing guarantee.
Actual current-head hosted CI and fresh review remain required.
- Full local frontend/application and live Docker E2E suites were not
rerun for this CI-helper cleanup; hosted workflows validate the
application matrix.

### Scope and provenance

Targets `dev`; former prerequisite #14279 is merged. Downstream CI PRs
and #14114 are outside this PR. The current diff includes the previously
consolidated frontend validators/shared reporters; earlier historical
review and CI claims are not substituted for fresh-head gates.

### Agents and large language models used

- Codex with GPT-6: review-comment validation, regression tests,
diagnostics fixes, workflow/run inspection, and PR maintenance.
- Earlier history credits Codex (model unavailable), Codex with
`gpt-daybreak-blue-latest` for independent review, and Claude Opus 5
(originating platform not recorded). Existing attribution is preserved.

### Checklist 📋

#### For code changes:

- [x] Changes and ownership are listed.
- [x] Test plan and validation limitations are described.
- [x] Relevant local CI-helper tests, formatting, and normal hooks pass.
- [ ] Current-head hosted workflow checks pass.
- [ ] Fresh independent review covers the current head.

#### For configuration changes:

- [x] Existing environment and Compose configuration remain compatible.
- [x] Runner, sharding, report-validation, and disabled Chromatic
behavior are listed above.

---------

Co-authored-by: Claude Opus 5 <[email protected]>
Co-authored-by: Reinier van der Leer <[email protected]>
Co-authored-by: gpt-daybreak-blue-latest (Codex) <[email protected]>
 2026-09-14 04:31:52 +00:00
Nicholas TindleandGPT-6 80c4e7af7b ci(backend): reject stale skip allowances across shard reports (#14536) 
### Why / What / How

An obsolete backend skip allowance can survive after its test is fixed
and silently permit a later regression to skipping. Each individual
shard cannot identify unused allowances because most allowed tests
belong to another shard.

Validate the complete four-shard report set separately for every
configured Python version, then reject allowed IDs absent from that
version's combined skipped-test set. The CI job also requires every
producer shard to have succeeded so partial reports cannot pass as
complete evidence.

Fixes #14533. This implements the follow-up from #14279.

### Changes 🏗️

- Add one lightweight `ubuntu-latest` job after the backend test matrix.
It downloads the existing report artifacts and performs validation
without starting services or installing application dependencies;
timeout is five minutes.
- Preserve the existing 136 allowed skip IDs as common policy entries
and add explicit Python-version-specific exception lists. Per-shard and
aggregate validation share the same loader, including rejection of
unexpected skips, malformed policies and unknown versions.
- Require exactly the configured versions × four shard artifact
directories and valid, nonempty, successful reports. Report stale IDs
separately for each Python version.
- Add regression coverage for stale/new skips, version isolation,
missing/malformed reports, artifact naming, failed/cancelled producers,
workflow triggers and policy/matrix agreement.
- Include two behavior-preserving lint fixes found by whole-helper
checks in `detect_overlaps.py` (standard-library import order and a
redundant f-string).

### Agents and large language models used

- Codex with GPT-6: implementation, independent code review and
validation.

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [ ] I have tested my changes according to the test plan:
- [x] 42 local CI-helper tests and 10 workflow-status tests pass (52
total).
- [x] All 12 real reports from successful backend queue run
[34779048952](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/34779048952)
pass. Each Python version independently matches all 136 allowances.
- [x] Mutating copies of those inputs rejects a stale ID, a missing
shard report, and an allowance belonging to another Python version;
explicit matching version allowances pass.
  - [x] Whole-helper formatting/lint and normal pre-commit checks pass.
- [ ] Hosted CI validates the full matrix and the new aggregate job on
this PR.

<details>
  <summary>Example test plan</summary>

- [x] Confirm an allowance used in another shard passes aggregate
validation.
- [x] Confirm stale allowances fail with an actionable ID and
Python-version diagnostic.
- [x] Confirm incomplete shard input and unsuccessful producer jobs fail
closed.
- [x] Confirm version-specific allowances do not hide a regression in
another version.
- [ ] Confirm the new job succeeds with artifacts produced by
current-head hosted CI.
</details>

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

<details>
  <summary>Examples of configuration changes</summary>

  - CI skip policy now has common and explicit Python-version sections.
- Aggregate validation uses a standard Ubuntu runner and existing report
artifacts.
- Application configuration, services and database schemas are
unchanged.
</details>

Local validation limitation: actionlint 1.7.12 still reports its two
existing unsupported-syntax diagnostics for GitHub's native
`background`/`wait` steps. The new job adds no such diagnostics; no
tracked validation rule was weakened. Full application tests run in
hosted CI, not locally for this CI-helper change.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Changes CI gating for skipped backend tests across the full Python
matrix; misconfiguration or incomplete artifacts can fail merges, but
application runtime code is untouched.
> 
> **Overview**
> **Tightens backend CI skip allowlisting** so obsolete entries cannot
linger after tests stop skipping, and version-specific skips cannot mask
regressions on other Python versions.
> 
> `backend-allowed-skips.json` moves from a flat list to **`common`**
plus explicit **`python_versions`** (3.11–3.13). `validate_junit.py`
gains shared **`SkipPolicy` / `load_skip_policy`** and requires
**`--python-version`** when using the new policy shape during per-shard
report validation.
> 
> A new **`validate_backend_skips.py`** aggregates all four shard JUnit
artifacts per Python version, enforces complete artifact sets, rejects
stale allowlist IDs not observed as skips, and still blocks unapproved
skips. Backend CI adds a lightweight **`validate-skip-policy`** job
(after the test matrix, fail-closed if shard jobs did not all succeed)
and path filters for the new scripts. Regression tests cover policy
shape, workflow wiring, and edge cases. **`detect_overlaps.py`** only
gets minor lint fixes.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
73b55c9aa5. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Co-authored-by: GPT-6 (Codex) <[email protected]>
 2026-09-14 03:47:39 +00:00
Nicholas TindleClaude Opus 5Reinier van der Leerunknown
 		bb4b2e488a ci(backend): shard tests with fail-closed reports (#14279) 
### Why / What / How

Backend tests take about 20 minutes per Python version when run as one
job. Split each Python 3.11/3.12/3.13 leg into four disjoint path groups
(`data`, `copilot`, `util-executor`, `remainder`) and validate each
JUnit report so missing tests, failed tests, and unexpected skips cannot
produce a successful shard. This PR targets `dev`.

All jobs use standard GitHub-hosted `ubuntu-latest` runners. The
measured tradeoff is **20.3 → 9.7 minutes elapsed** and **58.7 → 102.9
combined backend-test runner-minutes**, comparing [the historical dev
run](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/34516284145)
with [the historical sharded
run](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/34528703062).
These are measurements of those runs, not a guarantee for every run or a
current-head CI result.

### Changes 🏗️

- Run 12 backend test jobs with isolated Postgres databases, RabbitMQ
vhosts, and real Redis clusters. Preserve all Python versions,
`TRIAL_TEST_DATABASE=1`, and service dependencies.
- Set `E2E_REDIS_CLUSTER_RESTART=1` in the data shard. This activates
`test_subscriber_survives_shard_restart`, which previously did not run
in normal CI.
- Discover remainder tests exhaustively, including new backend and
repository-root test locations. Regression tests bind the workflow
matrix to the shard definitions and protect pytest exit propagation and
restart-test activation.
- Validate JUnit counts, missing/empty/malformed reports, failures, and
exact allowed skip IDs. The shared-library job forbids skips. Protect
the zero-test guard and identify the affected nested suite in
invalid-count diagnostics.
- Include the supporting fixes with their tests: queue factories honor
`Config().rabbitmq_vhost` so each shard uses its isolated broker
namespace; Docker image pulls use bounded retries for service startup;
console formatters copy `LogRecord` before modification so formatting
does not leak changes into other handlers. These are explicit behavior
changes carried with the CI work, with paired regression coverage.
- Preserve on-demand coverage attachment through the `pr_number` input
and document both dispatch forms. Quote the runner PATH output file.
- The `data/*.py` changes are test fixtures; user-ID authorization paths
are unchanged. Environment defaults and Compose configuration remain
compatible.

Pwuts' cross-shard stale-skip-allowlist check is tracked separately in
[#14533](https://github.com/Significant-Gravitas/AutoGPT/issues/14533).
The current per-shard validator rejects new skips; detecting entries
unused across all four reports is not implemented here.

### Validation

- 23 CI-helper tests and 10 workflow-status helper tests passed locally
on Python 3.12.
- Deliberately introducing a shard absent from the workflow, removing
the restart export, swallowing pytest failure, or removing the
empty-report guard is caught by the new regression tests.
- Eight malformed/negative nested-count diagnostic cases failed before
the label correction and pass afterward.
- Whole CI-helper directory Black formatting and normal commit/push
hooks pass. Independent code audit found no blockers in the cleanup.
- Local actionlint 1.7.12 reports only its two unsupported-syntax
diagnostics for existing `background`/`wait` steps. These are
[documented GitHub Actions
syntax](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#jobsjob_idstepsbackground);
[actionlint support is tracked
upstream](https://github.com/rhysd/actionlint/issues/693). No other
workflow or ShellCheck findings remain. The workflow retains those
parallel steps.
- Fresh hosted CI must validate the full backend/service/Python matrix
and coverage on the current head; it remains the integration gate. The
full application matrix was not rerun locally.

### Agents and large language models used

- Codex with GPT-6: current review fixes, validation, and description.
- Earlier work: Codex (model details unavailable), Codex with
`gpt-daybreak-blue-latest` for an independent earlier review, and Claude
Opus 5 as attributed in original commits. Existing human/coauthor
credits are preserved.

### Checklist 📋

#### For code changes:

- [x] Changes and bundled runtime behavior are listed.
- [x] Test plan and measured CI tradeoff are documented.
- [x] Local helper tests, regression mutation checks, formatting, and
normal hooks pass.
- [x] Current-head hosted workflow checks pass.
- [x] Fresh independent GitHub review covers the current head.

#### For configuration changes:

- [x] Existing `.env.default` and Compose configuration remain
compatible.
- [x] Runner, shard, service-isolation, and dispatch configuration
changes are listed.

---------

Co-authored-by: Claude Opus 5 <[email protected]>
Co-authored-by: Reinier van der Leer <[email protected]>
Co-authored-by: unknown (Codex) <[email protected]>
 2026-09-13 19:52:53 +00:00
Nicholas TindleandClaude Opus 5 4b3df42c50 refactor(backend): run Codex over HTTP and drop the 391 MB bundled CLI (#14195) 
## Why

Codex ran through the Codex CLI. That pulled `openai-codex-cli-bin` into
the image (**391 MB**), needed a synthetic `$HOME` per login, and —
because one subprocess owned a credential — **served one chat at a time
per ChatGPT connection**.

None of it was necessary. Everything the CLI brokered is reachable over
ordinary HTTPS. This PR moves sign-in, refresh, the model catalog, rate
limits and inference to HTTP, and deletes the binary.

Every endpoint and response shape was verified against the live backend
with a Pro account, not inferred from another client's source.

## What the probing found

**Full Responses works on every model, including the
`use_responses_lite` ones.** This was the main open question and it
removes the model-pinning problem entirely:

| model | `use_responses_lite` | classic `tools:[...]`, 10 tools |
|---|---|---|
| `gpt-5.5` (control) | false | 200, tool called |
| `gpt-5.6-sol` | **true** | 200, **2 parallel calls** |
| `gpt-5.6-terra` / `-luna` | **true** | 200, tool called |
| `gpt-daybreak-blue-latest` / `gpt-reserve` | **true** | 200, tool
called |

`use_responses_lite` is a flag Codex reads from its *own* catalog to
reshape its *own* requests — a client-side token optimisation, not a
server contract. No lite implementation, no model pin, whole 5.6 family
available. (opencode's four-model allowlist is avoidance, not a solved
lite path.)

**The stock OpenAI SDK works against it.** The endpoint *is* the
Responses API, so `AsyncOpenAI` pointed at the ChatGPT base URL streams,
calls tools and reports usage with no bespoke SSE handling.

**Required headers are few**: `Authorization`, `ChatGPT-Account-Id`,
`content-type`, `accept`, `originator`, `user-agent`, `session_id`. No
`client_metadata`, no `OpenAI-Beta`, no `x-codex-*`.

**Rate limits ride on every inference response** as `x-codex-*` headers
— nothing extra to poll, and the numbers arrive with the work that spent
them.

**`/models` is reachable** (requires a `client_version` param it does
not validate — `0.0.0` returns the same catalog as a current build).

**No embeddings.** `/embeddings` returns an HTML 403 on every path tried
— the edge signature for a route that does not exist, not an entitlement
error — and `model_specialty` across the catalog is only `[null,
"cyber"]`. The subscription backend exposes `/responses` and `/models`,
nothing else.

## Sign-in is not RFC 8628

It resembles it enough to be dangerous. Verified differences: the poll
is keyed by **`(device_auth_id, user_code)`**; `interval` arrives as the
**string** `"5"`; the deadline is an **ISO timestamp** in `expires_at`;
there is **no `verification_uri`** in the payload; and an approved poll
returns an **`authorization_code`** still needing the ordinary PKCE
exchange.

**Pending is HTTP 403; an unknown or expired login is HTTP 404.**

```
403 → code "deviceauth_authorization_pending"   ← still waiting
404 → code "deviceauth_not_found"               ← terminal
```

Classification is on the error `code`, never the status. Treating 404 as
pending polls a dead login to its deadline; treating 403 as failure
tells someone sign-in failed while they are looking at the approval
screen that was about to work. Published descriptions of this flow say
"403/404 as pending" — that is wrong in the second case.

## Concurrency

`CodexCredentialBusyError` existed because a subprocess owned the
credential — "a ChatGPT connection serves one chat at a time" was a
property of the process, not the account. Over HTTP one credential
serves as many concurrent turns as the account allows. The type is kept
for callers that still distinguish it, but acquisition no longer raises
it.

## Two details in the tool loop

Both fail silently rather than raising, so both have tests:

- **Every output item is replayed** into the next request, reasoning
items included, with `include=["reasoning.encrypted_content"]` so they
come back replayable. Feeding back only the tool calls degrades
reasoning models quietly.
- **A tool timeout is reported to the model** as tool output rather than
killing the turn, so it can recover.

The loop also stops after 32 rounds rather than spending quota on
something that will not converge.

## Refresh ownership

Codex moves from `refresh_strategy="provider_runtime"` to
`"oauth_handler"`, putting it behind the credentials manager's Redis
mutex. OpenAI rotates the refresh token on every exchange and treats
replay as compromise, so the handler sets `ROTATES_REFRESH_TOKEN` and
refreshes serialize. A refresh response that omits a new refresh token
keeps the old one. Credentials written before this change stay readable,
so existing grants survive.

## On the client id

It is OpenAI's public Codex client — an identifier, not a secret; `grep
-rni client_secret` across the CLI's `login/`, `codex-api/` and
`model-provider/` returns nothing transmitted (the one hit is a
log-redaction list). PKCE protects the exchange.

The binding constraint is its **redirect allow-list** (`localhost:1455`,
`localhost:1457`, `auth.openai.com/deviceauth/callback`), which we
cannot add to. That is why device flow is the only viable door for a
server — not merely the convenient one.

`originator` is sent as **`autogpt`**. The backend serves any originator
(codex sends `codex_cli_rs`, opencode `opencode`, Zed `zed`), so
claiming to be the Codex CLI would be impersonation for no benefit.

## What was deleted

`runtime.py`, `launcher.py`, `temporary_home.py`, the runtime pool, the
capacity semaphore, and the `openai-codex` dependency.
`codex_compat_gateway.py` imported the SDK's notification type;
`CodexStreamEvent` replaces it, so nothing outside the codex package
reaches for it now.

**Net −3,779 lines.**

## Testing

- **83 new tests**; 123 passing across `integrations/codex` +
`integrations/oauth`, plus the codex gateway suites
- `ruff` and `pyright` clean on every changed file
- Live-verified end to end through `CodexTransport`: account, 10-model
catalog with the right default, chained multi-hop tool calls
(`get_weather` → `convert_temp`, correctly threading the first result
into the second), streaming deltas, accumulated usage, and rate limits

`test_transport_failure_logs_useful_bounded_context` fails **identically
at HEAD** — a local ANSI-in-log-message artifact, verified by checking
the file out at HEAD and re-running. Not introduced here.

`poetry.lock` was regenerated with **Poetry 2.2.1** to match the file's
own header, since CI reads its Poetry version from the lockfile — a
1.8.3 regeneration would have silently downgraded CI. The diff is 1
insertion / 35 deletions: the two packages and the content hash.

## Still unverified

Whether Cloudflare 403s these calls from a **datacenter IP**. All
probing ran from a residential connection. `!deploy` answers it.

## Checklist

- [x] Endpoints verified live rather than copied from another client
- [x] Tests for each silently-failing path
- [x] Pre-existing credentials remain readable
- [x] Pre-existing test failure verified as pre-existing
- [x] Lockfile regenerated with the version CI expects

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **High Risk**
> This is a full swap of authentication, token refresh, and inference
for a user-billed integration; regressions would break ChatGPT/Codex
login or copilot turns, though legacy credential shapes are explicitly
preserved.
> 
> **Overview**
> **Replaces the Codex CLI/subprocess stack with plain HTTPS** for
ChatGPT device sign-in, token refresh, model catalog, and agent turns
via the Responses API (`AsyncOpenAI` against the ChatGPT Codex base
URL). The large `openai-codex` / `codex_cli_bin` path and supporting
modules (`runtime`, `launcher`, `temporary_home`, runtime pool) are
removed; container smoke now asserts the HTTP runtime and that
`openai_codex` / `codex_cli_bin` are not importable.
> 
> **Sign-in and credentials:** New `chatgpt_auth` + `device_login`
implement OpenAI’s non-RFC-8628 device flow (poll by `device_auth_id` +
`user_code`, classify on error codes). New credentials use
`refresh_strategy="oauth_handler"`; **`provider_runtime` stays
accepted** for existing grants. API lease checks and codec round-trips
were updated accordingly.
> 
> **Inference and copilot:** `CodexHttpSession` runs streaming turns
with a tool loop (reasoning item replay, rate limits from response
headers). **`CodexStreamEvent`** replaces CLI `Notification` types in
the Anthropic compat gateway and tests. **`PooledCodexRuntimeLease` is
renamed to `CodexCredentialLease`** through model routing and SDK chat
paths.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
01a7b58e35. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
 2026-09-02 09:38:28 +00:00
Nicholas TindleandClaude Opus 5 8c65f748b1 fix(platform): shut the single container down inside Docker's stock timeout (#14077) 
### Why / What / How

**Why.** `docker stop` on the single-container appliance never completed
under Docker's stock 10-second timeout. Docker SIGKILLed the container
at 10s — **exit 137** — with PostgreSQL, RabbitMQ, Valkey and FalkorDB
still running. On Unraid that is a product blocker: the stock timeout is
what a default host sends, and operators should not have to raise a
host-wide Docker setting to run one appliance.

**What.** Fixes the bug that wedged shutdown, restructures the
supervised stop into bounded tiers with a budget that accounts for
supervisor's own cost, and makes CI able to catch a regression.

**How.** Three compounding causes:

1. **The scheduler never exited.** The entrypoint sets `APP_ENV=dev`, so
the scheduler calls `initialize_launchdarkly()`. With no SDK key — the
case for every self-hosted deployment — that logs a warning and returns
*without* calling `ldclient.set_config()`. On shutdown `cleanup()` calls
`shutdown_launchdarkly()`, which does `ldclient.get()` and raises
`Exception("set_config was not called")` out of service teardown,
leaving the process alive. `rest_api.launch_darkly_context` pairs
init/shutdown on `app_env` the same way, so the guard lives in
`feature_flag.py` and covers both.

2. **Supervisor stops process groups one at a time**, in descending
priority order, waiting for each group to stop completely before
signalling the next. The default layout puts every program in its own
group, so one stuck program strands every program behind it — including
the data stores, which stop last by design. The wedged scheduler was 2nd
in the app tier with `stopwaitsecs=60`; the six data stores were never
signalled before the SIGKILL. Explicit `runtime` and `state` groups
collapse this to two phases plus the event listener.

3. **The budget was unbounded, and CI could not see it.** `stopwaitsecs`
ran 30–120s against a 10s window, and the smoke test stopped with
`--timeout 360` — generous enough that an overrun still reported exit 0
in CI and only failed on a real host.

**Sizing the budget.** The phases add up, and so does supervisor's own
cost: each phase needs at least one more poll iteration of
`runforever()` (which polls with `timeout=1`) to reap what it stopped.
Benchmarked against supervisor 4.2.5 with every program ignoring
SIGTERM, so each phase runs its `stopwaitsecs` out and escalates to
SIGKILL:

| Layout | `sum(stopwaitsecs)` | Measured wall time |
| --- | --- | --- |
| 2 phases | 4s | 5.30s |
| 3 phases | 7s | 8.38s |
| 3 phases | 8s | 9.47s |
| 4 phases | 8s | 9.27s / 10.34s |

Wall time is `sum(stopwaitsecs) + ~1.4s` regardless of the values. The
budget is therefore **1s runtime / 5s state / 1s listener = 7s**,
measuring 8.38s worst case.

The split is weighted hard toward the data stores, because measurement
on a *seeded* appliance showed PostgreSQL's shutdown checkpoint is both
the largest and by far the most variable term (see below). The stateless
tier gives up nothing that holds state: nginx and `next` exit in ~15ms,
notification and websocket in ~1.5s, and the six heavy services are
pinned at ~5.6s by a PostHog join (see follow-ups) that no affordable
cap can cover, so they are SIGKILLed either way.

That table also rules out restoring nginx-first draining with a fourth
`edge` tier: it measured 10.34s, which is exit 137. `priority` no longer
orders anything within a tier — `ProcessGroup.stop_all()` signals every
member in one pass — so that trade-off is documented in the config
rather than reversed.

### Changes

- `backend/util/feature_flag.py` — guard `shutdown_launchdarkly()` on
`_is_initialized`; record a separate `_is_shutdown` sentinel in a
`finally` rather than clearing `_is_initialized`, which `get_client()`
reads as "never started" and would use to build a fresh client, with new
streaming threads, inside the shutdown window.
- `single-container/supervisor/supervisord.conf` — add `[group:runtime]`
and `[group:state]`; bound every `stopwaitsecs` to the budget above;
PostgreSQL gets `stopsignal=INT` (SIGTERM is its *smart* shutdown, which
waits for every client to disconnect and never completes on a deadline)
and `stopasgroup=false` (`run-service.sh` execs the postmaster, so
`killpg` would send SIGINT to backends, where it means cancel-query,
racing the postmaster's own fast shutdown).
- `single-container/healthcheck.sh` — grouping renames status lines to
`group:program`, so the required-program list is group-qualified. An
un-updated list would silently match nothing and pass every program.
- `single-container/entrypoint.sh` — opt out of `mem0` and
`graphiti-core` vendor telemetry.
- `.github/scripts/platform-single-container-smoke.sh` — stop assertions
use the stock 10s timeout via `assert_clean_stop`, which bounds elapsed
time as well as exit code. Exit code alone only changes at the cliff, so
drift from 6s toward 9.9s would otherwise pass identically.
supervisorctl calls use group-qualified names.
- `single-container/tests/test_supervisor_config.py` (new) — 6 tests;
the budget test models per-phase supervisor overhead and rejects the
previous 8s budget.
- `backend/util/feature_flag_test.py`, `tests/test_entrypoint.py`,
`tests/test_runtime_config.py` — shutdown coverage, telemetry assertion,
and three `groupname:` fixtures that grouping made unreachable.

**Configuration changes:** two new environment variables exported by the
appliance entrypoint, `MEM0_TELEMETRY=false` and
`GRAPHITI_TELEMETRY_ENABLED=false`. Both `mem0` and `graphiti-core`
embed their own PostHog write keys and report anonymous usage to their
vendors by default; a self-hosted appliance should not send anything to
a third party the operator never chose. No ports, services, or secrets
change.

### Test plan and results

| Scenario | Before | After |
| --- | --- | --- |
| `docker stop -t 10`, fresh volume, first boot | `10569ms`, **exit
137** | `5864ms`, **exit 0** |
| `docker stop -t 10`, restart on existing volume | — | `6507ms`, **exit
0** |
| Worst case, all programs ignoring SIGTERM | — | `8383ms`, **exit 0** |
| **Seeded appliance, stopped mid-write** | — | **`4640ms`, exit 0** |

Loaded runs seeded 516MB into PostgreSQL (`pgbench -s 20`), ~200k keys
per Valkey node, 300k in FalkorDB and 50k persistent messages on a
durable RabbitMQ queue, then stopped the container with a `pgbench`
write workload still running. Every data store still reported `exit
status 0`.

The reason the state tier carries 5 of the 7 seconds is PostgreSQL's
shutdown checkpoint, measured across those runs:

| Dirty state at stop | Checkpoint | `sync` component | Wall time |
| --- | --- | --- | --- |
| 82MB distance | 0.858s | 0.171s | 4597ms |
| 256MB distance | 1.735s | 0.699s | 4640ms |
| 516MB distance | **3.190s** | **2.378s** | 7257ms |

It is the largest term and it varies by 3.7x with accumulated dirty
buffers, with `fsync` — the disk-bound part — dominating the expensive
case.

> [!IMPORTANT]
> **These were measured on NVMe. They do not establish that 5s is enough
on a parity array.** The expensive checkpoint above spent 2.378s of its
3.190s in `fsync`, which is exactly the term that grows on slow or
parity-backed storage, and six `state` programs flush concurrently to
the same volume.
>
> The design fails safe there rather than regressing: if the checkpoint
overruns its cap, supervisor SIGKILLs PostgreSQL **inside our own
budget** and it recovers from WAL on the next boot — a slower start, not
lost data. That is strictly better than the behaviour this PR replaces,
where Docker SIGKILLed all 21 programs mid-checkpoint at 10s. An
operator on very slow storage may still see recovery on restart.

- [x] Reproduced the original failure on the published image: exit 137,
`OOMKilled=false`.
- [x] Confirmed the root cause in the container logs — `Exception:
set_config was not called` from `scheduler.py` `cleanup()`, and
supervisor's `waiting for ... to die` listing every program behind the
scheduler.
- [x] Post-fix, every data store reports clean shutdown: PostgreSQL logs
`checkpoint complete` then `database system is shut down`,
RabbitMQ/Valkey/FalkorDB all `exit status 0`.
- [x] Verified the scheduler completes cleanup (`Cleanup done`, 131ms
after SIGTERM) with no LaunchDarkly exception.
- [x] Benchmarked supervisor 4.2.5 across 2-, 3- and 4-phase layouts to
derive the overhead term (table above), and confirmed the unit test
rejects the previous 8s budget.
- [x] Verified each new test fails against the pre-fix implementation,
including mocking an SDK key present so the re-initialization regression
is actually exercised.
- [x] Confirmed both telemetry variables reach the running services via
`/proc/<pid>/environ`.
- [x] Confirmed supervisor's `processname` event field is unaffected by
grouping, so `fatal_listener.py`'s bootstrap check still fires.
- [x] `pytest` — 36 backend and 14 single-container tests pass;
remaining single-container failures are pre-existing and
platform-specific (`os.fchmod`), identical in count to `origin/dev`.
- [x] `poetry run format` (isort/black/pyright) and `ruff check` clean.
- [x] CI: `Build, smoke, and scan` passes on `linux/amd64` and
`linux/arm64` with the stock-timeout assertion.

### Known cost

`database-manager` previously exited cleanly at ~3.5s and is now
SIGKILLed at the 1s cap, as are notification and websocket (~1.5s). It
is a Pyro service layer, so PostgreSQL rolls back its connections, but
it is a change from the 3s configuration.

The stateless cap is also not free for the executors: they consume run
messages with `auto_ack=False` and release cluster locks at the end of a
`cleanup()` that polls on a longer interval than the cap allows, so a
stop with work in flight leaves messages to be redelivered — the run is
**re-executed on next boot rather than resumed**, repeating its side
effects. Bounding the stop is still the right trade against being
SIGKILLed wholesale by Docker, but the cost lands on in-flight runs.

### Follow-ups (not in this PR)

1. **PostHog blocks every service shutdown by ~4.3s.**
`mem0/memory/telemetry.py` constructs a PostHog client *at module
import* with mem0's own write key, registering
`atexit.register(self.join)`. Confirmed with py-spy: the main thread
sits in `join (posthog/client.py)` while the consumer polls its queue.
`MEM0_TELEMETRY=false` does **not** remove it — mem0 sets `.disabled =
True` only *after* construction; measured 4.31s either way. This affects
every AutoGPT service, not just the appliance, and fixing it would
change the budget arithmetic here.
2. **`bootstrap` can be SIGKILLed mid-`prisma migrate deploy`.** Its cap
dropped from 30s, and no cap that fits a 10s window survives a
multi-minute migration. Prisma writes the migration row before applying
and updates it after, so an interrupted first boot can leave
`finished_at IS NULL` and fail every subsequent boot until an operator
runs `prisma migrate resolve`. Auto-resolving is not safe — it can mark
partially-applied DDL as rolled back — so this needs a deliberate
repair-or-detect design.
3. **A short explicit cancel path for the executors**
(nack-without-requeue plus lock release well inside the cap), which
overlaps the in-flight resume-dropped-runs work.

### Checklist

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan

#### For configuration changes:
- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Changes shutdown ordering, stop timeouts, and LaunchDarkly lifecycle
across scheduler/REST and all supervised services; in-flight executor
runs may not resume cleanly after the shorter runtime stop cap.
> 
> **Overview**
> Fixes **`docker stop`** on the single-container appliance so it
finishes inside Docker’s default **10s** window (avoiding **exit 137**
with data stores still running).
> 
> **LaunchDarkly teardown** in `feature_flag.py` no longer raises when
no SDK key was configured, closes clients that never connected, and uses
**`_init_attempted`** so unconfigured self-hosted deploys don’t re-init
on every flag read or rebuild a client during shutdown.
> 
> **Supervisor** is reorganized into **`runtime`** and **`state`** stop
groups with a bounded budget (**~1s + 5s + 1s** `stopwaitsecs` plus
overhead): stateless services stop together first, then
PostgreSQL/RabbitMQ/Valkey/FalkorDB; PostgreSQL uses **fast shutdown
(SIGINT)** and nginx **QUIT**.
> 
> CI smoke tests use **`assert_clean_stop`** (exit **0** and elapsed
time under the same margin as unit tests) and **group-qualified**
`supervisorctl` names; **healthcheck** lists match grouped names.
> 
> **Bootstrap** detects interrupted Prisma migrations and fails with
operator guidance; **README** documents clean stop and first-boot
migration risk. **Entrypoint** opts out of **mem0** / **graphiti**
vendor telemetry. New **`test_supervisor_config.py`** and expanded
**`feature_flag`** shutdown tests guard the budget and LD behavior.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
d61abdceb7. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
 2026-08-20 06:41:03 +00:00
Nicholas TindleBentlybroClaude Fable 5
 		dac59d9184 feat(platform): add ChatGPT/Codex subscription transport preview (#13761) 
> [!IMPORTANT]
> Preview release for user-owned ChatGPT/Codex usage. `LOCAL`
deployments allow it for every authenticated user. `CLOUD` deployments
allow it for AutoGPT `MAX`, `BUSINESS`, and `ENTERPRISE` users. There is
no Codex feature flag or staff-only allowlist, so deploying this branch
to hosted production exposes discovery, sign-in, and execution to every
user in those tiers. The connected ChatGPT account's own plan, workspace
policy, allowance, and credits still determine what Codex can execute.

### Why / What / How

This PR adds a user-funded ChatGPT connection as a distinct Codex
subscription transport, without requiring an OpenAI API key.

| Supported surface | Execution path |
|---|---|
| AutoPilot chat | Existing Claude Agent SDK and bundled Claude Code
CLI, with a loopback Anthropic Messages gateway translating text and
tool rounds to Codex App Server |
| AutoPilot graph block | The same AutoPilot path when a ChatGPT
credential is explicitly selected |
| Code Generation block | Direct Codex App Server invocation when `Codex
App Server` is selected |

Existing platform-funded, self-hosted, and OpenAI API-key routes remain
available and keep their existing behavior.

The frontend presents **Sign in with ChatGPT** under the existing OpenAI
integration card and uses OpenAI branding. The stored credential and
backend auth provider remain `codex`, while the API-key method remains
`openai`. Keeping those canonical identities separate prevents a ChatGPT
subscription credential from being mistaken for an OpenAI API key.

### Changes 🏗️

#### Authentication and credentials

- Add ChatGPT device-code sign-in, login status and cancellation,
authenticated account and rate-limit read endpoints, and disconnect with
provider logout where available. This PR does not add a frontend
plan/rate-limit dashboard.
- Store one active user-owned Codex connection per user in the existing
encrypted `IntegrationCredential` payload as
`OAuth2Credentials(provider="codex",
refresh_strategy="provider_runtime")`.
- Strictly validate ChatGPT auth bundles and reject direct API creation
of Codex or provider-runtime OAuth credentials.
- Let the Codex runtime own token refresh, then checkpoint rotated auth
back into the encrypted credential while holding the existing
distributed credential lease.
- Materialize `auth.json` only inside an isolated temporary
`CODEX_HOME`. RabbitMQ messages, Redis login state, chat sessions, and
Claude CLI environments contain credential IDs or short-lived
capabilities, not ChatGPT access, refresh, or ID tokens.
- Starting another sign-in supersedes the previous AutoGPT attempt
through Redis-backed state. Popup close, timeout, unmount, explicit
abort, and blocked-popup paths request server cancellation exactly once;
successful completion does not. This clears AutoGPT-owned stale state
but cannot bypass OpenAI-side device-code throttling or cooldowns.
- Recheck entitlement before persisting a completed login and recheck
authoritative credential ownership, provider, type, and validity at
execution boundaries.

#### Runtime and execution

- Pin `openai-codex==0.144.4` together with its bundled Codex runtime.
- Add bounded App Server capacity, startup, control, invocation, turn,
and tool timeouts; lease heartbeats; refresh checkpointing; redacted
errors; and temporary-home cleanup on success, failure, timeout,
cancellation, or shutdown.
- Keep every Codex-backed AutoPilot mode flowing through the existing
Claude Agent SDK and bundled Claude Code CLI. A request-scoped gateway
bound to `127.0.0.1` translates Claude text and MCP tool rounds to Codex
App Server `dynamicTools`. Claude Code retains the MCP tool loop,
permission/security hooks, transcripts and resume, compaction,
attachments, and sub-sessions. The CLI receives only a random per-turn
capability.
- Keep the Code Generation block separate from the Claude harness:
subscription-backed Code Generation invokes Codex App Server directly
and exposes no workspace or tool access.
- Reuse one process-local Codex runtime actor per user and credential.
Overlapping AutoPilot turns reaching the same Copilot executor process
use separate Codex threads while sharing the actor and its exclusive
credential lease; canceling one borrower does not stop its siblings.

#### Routing and model behavior

- Add an authenticated `/api/chat/transports` inventory and validate
every requested route against what is currently usable for that user and
deployment.
- Keep **AutoGPT Platform** as the default on hosted deployments.
Connecting ChatGPT adds it as another route.
- Keep **Self-hosted chat** as the default when a local chat provider is
configured. Connecting ChatGPT adds it as another route.
- On a keyless self-host with one valid ChatGPT connection,
automatically use that sole route and hide the selector. With no usable
route, show setup guidance instead of inventing a platform route.
- Show the route selector only when more than one usable route exists
and refetch the inventory after pairing or disconnecting.
- Keep the selected provider and credential ID immutable for the life of
a session, including retries, queued turns, and resume. Entitlement,
ownership, and credential validity are still checked at execution time.
Downgrade, deletion, revocation, or malformed state makes the session
fail visibly; it never reroutes to platform funding, another credential,
or another user's account.
- Prefer these AutoPilot routes when the connected account advertises
the corresponding model and effort:

| Mode | Preferred model | Effort |
|---|---|---|
| Fast / Balanced | GPT-5.6 Luna | low |
| Fast / Advanced | GPT-5.6 Terra | medium |
| Thinking / Balanced | GPT-5.6 Terra | high |
| Thinking / Advanced | GPT-5.6 Sol | xhigh |

If a preferred model or effort is unavailable, routing uses an allowed
account-advertised default or available OpenAI model and a
supported/default effort.

For Code Generation, the model dropdown applies only to the OpenAI API
transport. The subscription transport lets App Server choose its live
account-compatible model while still applying the selected supported
reasoning effort.

#### Entitlement and cost behavior

- Add the shared `require_entitlement(user_id, entitlement)` policy
shell.
- In `LOCAL` mode, allow `codex_subscription_transport` before any
subscription database lookup.
- In `CLOUD` mode, read the authenticated user's authoritative AutoGPT
`User.subscriptionTier` through `DatabaseManager` on every check in this
PR and allow `MAX`, `BUSINESS`, and `ENTERPRISE`.
- Hide Codex from optional discovery when tier state cannot be resolved.
Actual connect and execution requests fail with a retryable
service-availability error rather than treating an outage as a valid
route.
- Keep pending-login cancellation and credential deletion available even
after downgrade so users can clean up.
- Do not separately resolve organization-seat entitlements in this PR;
cloud access is based on the user's authoritative `subscriptionTier`.
- Keep entitlement reads deliberately uncached here. Shared caching and
tier-mutation invalidation are isolated in follow-up
[#14004](https://github.com/Significant-Gravitas/AutoGPT/pull/14004).
- Record subscription-backed token usage with
`billing_mode=user_subscription` and the execution path. Do not report
public OpenAI API USD pricing as AutoGPT provider spend.
- Skip platform model-paywall and platform-provider USD usage windows
for Codex AutoPilot. Subscription-backed Code Generation does not apply
the API-token wallet charge. Ordinary graph execution, turn concurrency,
queueing, bridge compute, and infrastructure policies still apply, and
usage consumes the connected ChatGPT account's allowance or credits.
- Preserve existing OpenAI API-key billing behavior.

### Configuration / rollout

No new secret, Codex rollout flag, or Codex-specific frontend build
argument is required.

- Cloud previews must still set `FRONTEND_BASE_URL`, build-time
`NEXT_PUBLIC_FRONTEND_BASE_URL`, and runtime `BETTER_AUTH_URL` to the
exact public origin. Otherwise the device-login popup can use the wrong
host or lack the preview authentication cookie.
- Outside the single-container appliance, `CODEX_TEMP_ROOT` is optional
and otherwise falls back to the operating-system temporary directory.
- Local Compose sets `CODEX_TEMP_ROOT=/run/autogpt-codex` and mounts a
128 MiB memory-backed temporary filesystem in the REST, graph-executor,
and Copilot-executor containers.
- The single-container appliance fixes
`CODEX_TEMP_ROOT=/dev/shm/autogpt-codex`, creates it as `0700` for the
unprivileged backend user, and verifies that REST, graph executor, and
Copilot executor inherit it. Docker's `/dev/shm` keeps auth homes out of
the writable layer; the documented quick-start sizes it to 2 GiB.
- Narrow Git attributes keep the appliance shell scripts LF-normalized
in Windows checkouts so a Windows source build uses the same executable
bytes as Linux CI.
- Capacity, process startup/control, checkpoint, invocation,
Copilot-turn, tool, and login intervals have bounded defaults and
optional `CODEX_*` overrides.
- Deployments need outbound HTTPS access to OpenAI. Device-code login
does not require an inbound OpenAI callback.

### Explicit exclusions and known limitations

This PR does not route the following through a user's ChatGPT
connection:

- Orchestrator or shared LLM blocks;
- image generation or embeddings;
- `ClaudeCodeBlock` / E2B;
- builder-panel-bound sessions;
- background or system-owned LLM work.

It also does not add credential pooling, silent fallback to a platform
credential, multiple active Codex credentials per user, per-user
Kubernetes resources, persistent Codex homes, or persistent volumes.

Operational limits that remain:

- Device-code authorization must be enabled for the ChatGPT account or
workspace. A pending AutoGPT attempt is bounded by
`CODEX_LOGIN_TIMEOUT_SECONDS`, 15 minutes by default.
- Login actors are process-local. Redis shares status, supersession, and
cancellation across REST replicas, but a restart of the owner process
interrupts the active sign-in.
- Runtime actors are process-local. A different Copilot executor process
or replica cannot join the current owner and may receive the bounded,
retryable `codex_credential_busy` error. Multi-replica owner routing or
a dedicated bridge is not included.
- Codex App Server and device-login actors currently run inside the
ordinary REST, graph-executor, and Copilot-executor containers. A
dedicated non-root, least-privilege bridge with memory-only homes,
restricted egress, process limits, and credential-owner routing remains
follow-up work.
- Commercial/policy approval for hosted subscription passthrough and the
Claude-harness compatibility route is a deployment decision outside this
implementation; this PR does not claim that approval.

### Validation

At review-fix parent `9cbe549d36dd3197b22871dbb7ae79d85c26c03f` against
`dev@5c980c2f743ae2431e30f5511ce304413782c163`, the formatted tree
passed the following focused validation:

- Backend auth/runtime suite: 95 passed, with 3 Windows-specific skips.
- Backend model routing, compatibility gateway, and SDK environment
suite: 74 passed.
- Backend platform-cost, metrics, and Code Generation suite: 94 passed.
- Backend billing/cost-leak suite: 19 passed.
- Authenticated integrations API suite: 25 passed.
- Isolated Linux executor/gateway suites: 65 passed, including processor
cleanup, lease ownership, Codex billing-gate, and compatibility-gateway
behavior; some cases overlap the focused Windows suites above.
- Frontend: 83 tests across five affected files, TypeScript `tsc
--noEmit`, and targeted Prettier checks pass.
- Backend isort, Black, Ruff, AST/import compilation checks, and Git
diff hygiene pass for every changed Python file.

Final head `aa4bcb445c450d2da1c0de2a4c1ea03b9c3b70e5` restores the
framework-compatible optional AutoPilot credential annotation,
regenerates the public provider-discovery OpenAPI contract, safely
canonicalizes macOS system temp aliases, and includes the exact
generated Code Generation documentation sync emitted by CI. On that
final diff, the AutoPilot schema/import assertion passes; the temp-home
suite passes 12 tests with 6 expected Windows symlink skips; Ruff,
Black, Prettier, Git diff hygiene, and full TypeScript `tsc --noEmit`
pass. The final test-only commit also updates the shutdown mock to patch
the manager module's bound transport symbol; that exact regression
passes locally. Current-head CI remains the authoritative broad
verification below.

The prior head `9eb8283db22d6ddf624f6c891c5b1579cdd07516` also completed
a real-account, keyless local single-container smoke: four Claude Agent
SDK turns used the Codex subscription route successfully, persisted 40
chat messages, recorded subscription token usage, and recorded zero
platform model cost. The review-fix head changes authentication cleanup,
model-routing structure, compatibility edge cases, cost reporting, and
selector cleanup, so exact-head external-account smoke remains
explicitly unchecked below.

- [x] Focused backend tests cover auth-bundle validation, temporary-home
isolation, refresh/checkpointing, login supersession and cancellation,
entitlement enforcement, authoritative credential identity, transport
inventory, immutable session routing, process-local runtime sharing,
queue/executor behavior, costs, Code Generation, and AutoPilot.
- [x] The bundled Claude Agent SDK/CLI conformance suite covers
streaming text, tool call/result, cancellation, and resume against the
compatibility gateway.
- [x] Frontend tests cover OpenAI/ChatGPT presentation, credential
selection, popup cancellation, transport inventory, selector defaults
and visibility, setup guidance, and session creation.
- [x] Current-head backend, frontend, full-stack, dual-architecture
single-container, CodeQL, Classic, overlap, documentation, and PR-status
automation passed on `aa4bcb445c450d2da1c0de2a4c1ea03b9c3b70e5`.
- [ ] Fresh deployed-preview device-code sign-in at the current head
with a real ChatGPT account.
- [ ] Fresh deployed-preview Code Generation and AutoPilot smoke at the
current head with a real ChatGPT account.

The unchecked real-account checks are intentionally not represented as
automated coverage; they consume an external account's allowance and
should be completed on the exact deployment selected for release.
### Checklist 📋

#### For code changes:

- [x] Changes and explicit exclusions are listed above.
- [x] Test and rollout evidence is documented above.
- [x] Local focused backend, frontend, conformance, formatting, and diff
checks finish successfully.
- [x] Credential access is user-scoped and execution revalidates
authoritative credential identity.

#### For configuration changes:

- [x] Existing environment defaults remain compatible; no new required
secret or rollout flag was added.
- [x] Compose supplies memory-backed temporary storage for local
execution.
- [x] Required cloud-preview origin settings and optional Codex runtime
settings are documented above.






<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **High Risk**
> Touches authentication, credential storage/refresh, chat session
routing, and billing/paywall behavior for a new user-funded LLM path. A
routing or entitlement bug could leak paid platform usage or expose
ChatGPT credentials incorrectly.
> 
> **Overview**
> Adds a **user-funded ChatGPT/Codex subscription transport** so
AutoPilot chat, the AutoPilot graph block, and Code Generation can run
on a connected ChatGPT plan instead of platform-funded or OpenAI API-key
routes.
> 
> **Auth & credentials.** Introduces ChatGPT device-code sign-in under
the existing OpenAI integration card, stores one `codex` OAuth
credential with runtime-owned refresh/checkpointing, and blocks direct
API creation of Codex credentials. Discovery, connect, and execution are
entitlement-gated (`LOCAL` for all users; `CLOUD` for
Max/Business/Enterprise).
> 
> **Routing & billing.** Adds `/api/chat/transports` and persists an
immutable `llm_auth_provider` / `llm_credential_id` on each session.
Codex AutoPilot skips the platform paywall and USD usage windows;
subscription usage is recorded as `user_subscription` rather than
platform spend. Builder-bound sessions stay platform-only.
> 
> **Runtime.** Pins the Codex App Server runtime, isolates auth homes
under `CODEX_TEMP_ROOT`, and reuses a process-local runtime actor under
an exclusive credential lease. AutoPilot still goes through the Claude
Agent SDK via a loopback compatibility gateway; Code Generation can
invoke App Server directly when that transport is selected.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
1441864f7f. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Bentlybro <[email protected]>
Co-authored-by: Claude Fable 5 <[email protected]>
 2026-08-12 05:40:12 +00:00
Nicholas Tindle 5c980c2f74 ci(platform): publish single-container images (#13994) 
### Why / What / How

The single-container distribution is built and validated on `dev`, but
it needs a production Docker Hub release path. This publishes only the
exact multi-architecture image content that passed the complete
appliance smoke and security scans, and gives self-hosters a usable
first-run account flow.

Manual publication from `dev` uses the reviewer-gated
`dockerhub-publish` environment and creates only an immutable
`sha-${GITHUB_SHA}` tag. A stable GitHub release named
`autogpt-platform-beta-vX.Y.Z` automatically uses the tag-restricted
`dockerhub-release` environment and creates that SHA tag, immutable
`vX.Y.Z`, and `latest` without a separate deployment approval.
Prereleases, malformed platform tags, other releases, and non-`dev`
dispatches cannot reach the Docker credentials or push jobs.

Native AMD64 and ARM64 jobs push by digest, pull those exact registry
digests back, rerun the full smoke and Trivy scans, and pass only the
verified digests to manifest assembly. Existing tags are reusable only
when they contain the exact tested digests and source revision. Registry
reads are retried with bounded validation, release tags are immutable,
and `latest` cannot move backward or replace the same version with
different content.

The successful stable-release manifest job also syncs the checked-in
single-container README to the Docker Hub Overview. The full workflow
must be present on the default branch before the next GitHub release is
published so GitHub can deliver the release event.

Fresh single-container installs start with account registration open,
matching the frontend default and making literal `docker run` usable.
Startup and documentation warn that anyone who can reach the app can
register until the operator promotes the intended administrator and
recreates the container with `AUTH_ALLOW_NEW_ACCOUNTS=false`. Canonical
Docker commands bind only to localhost.

### Changes 🏗️

- Publish native `linux/amd64` and `linux/arm64` images through separate
manual and stable-release environments.
- Publish immutable `sha-*` tags from approved manual `dev` dispatches
without changing `latest`.
- Map stable `autogpt-platform-beta-vX.Y.Z` GitHub releases
automatically to immutable `vX.Y.Z` plus monotonic `latest`.
- Fail closed before credential use for unsupported events, refs,
prereleases, and malformed tags.
- Verify exact smoke-tested child digests, OCI source revisions,
manifest platforms, repository visibility, tag immutability, and release
ordering.
- Retry registry digest resolution after writes and reject missing or
malformed digest responses.
- Keep publication policy and fixture coverage in a standalone
ShellCheck-tested helper.
- Sync a checked-in quick start, tag guide, resource guidance, and
FalkorDB memory description to Docker Hub.
- Open first-run signup by default, with explicit warnings and a tested
`false` override.

### Checklist 📋

#### For code changes:

- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] `actionlint
.github/workflows/platform-single-container-docker.yml`
- [x] ShellCheck and `bash -n` for both changed publication/smoke
scripts
- [x] Publication fixtures cover authorization, malformed tags, Buildx
metadata forms, attestations, repository visibility, bounded registry
retries, empty optional paths, release annotations, semantic-version
ordering, and rollback rejection
- [x] 59/59 single-container appliance unit tests, including
default-open and explicit-closed signup behavior
- [x] Bake target resolves the current dedicated single-container
Dockerfile and linked backend target
- [x] Protected canary run
[31454312554](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/31454312554)
built, pushed, pulled, fully smoked, scanned, and assembled both
architectures without changing `latest`
- [x] Final canary run
[31519892189](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/31519892189)
passed both native pre-push and exact-registry-digest smoke/scan lanes,
published the correct attested manifest, and exposed the post-write
verification race now covered by regression tests and a successful
live-manifest replay
- [x] Public canary manifest was pulled and remained healthy locally on
ARM64 and on Unraid AMD64 using persistent named volumes
- [x] Disposable-account parity flow: signup/sign-in, real two-block
Builder execution, actionable missing-provider error from the normal
stream API, signup closure, existing-account sign-in, and rejected
second signup
- [x] Current-head PR CI builds, smokes, and scans natively on AMD64 and
ARM64
- [ ] Full workflow replaces the inert default-branch registration frame
before the next stable GitHub release

#### For configuration changes:

- [x] `.env.example` documents the open-first-run account policy and how
to close it
- [x] No Compose files are changed
- [x] No shared backend service source is changed
- [x] Configuration changes are listed under **Changes**
 2026-08-11 19:38:05 +00:00
Nicholas TindleandClaude 18608f5b22 feat(platform): add single-container distribution (#13758) 
### Why / What / How

**Why**

AutoGPT Platform should be runnable as a complete local appliance with
one `docker run`, including FalkorDB-backed memory, without changing the
images or deployment behavior used by hosted infrastructure.

**What**

Adds a dedicated single-container appliance containing the frontend,
backend services, PostgreSQL, a three-node Valkey cluster, RabbitMQ,
FalkorDB, nginx, Supervisor, and Tini.

The appliance persists application and service state under `/data`,
generates installation secrets on first boot, and exposes only nginx on
container port `3000`.

**How**

Buildx Bake builds the existing backend `server` target, then uses it as
the base for `autogpt_platform/single-container/Dockerfile`. Internal
services bind to loopback and Supervisor manages their lifecycle. Health
checks and a fail-stop watchdog allow Docker restart policies to recover
the appliance when a required service cannot recover internally.

The frontend server runs as its own Unix user with a clean environment
and a passwordless local PostgreSQL role limited to the Better Auth
tables and required `User` columns. It does not receive PostgreSQL
superuser credentials. Internal AppService traffic stays loopback-only
inside the appliance trust boundary.

The effective PR diff adds only `autogpt_platform/single-container/`
plus its CI workflow and smoke script. Existing backend source, frontend
source, Dockerfiles, the root `.dockerignore`, and Compose files do not
differ from `dev`.

### Changes 🏗️

- Adds the dedicated appliance Dockerfile, runtime scripts, nginx
configuration, Supervisor configuration, and Bake targets.
- Enables FalkorDB-backed Graphiti memory by default.
- Persists generated application secrets, databases, queues, caches, and
memory data under `/data`.
- Runs startup migrations without downloading Prisma tooling at runtime.
- Adds health checks, required-service recovery, clean shutdown, and
whole-container fail-stop handling.
- Defaults new-account signup to disabled until explicitly enabled by
the operator.
- Isolates the frontend and nginx with dedicated operating-system users
and a least-privilege frontend database role.
- Pins internal topology to loopback so persisted backend configuration
cannot expose internal listeners.
- Adds native `linux/amd64` and `linux/arm64` build, runtime-smoke,
vulnerability-scan, and secret-scan CI.
- Keeps the validation workflow non-publishing.

### Test plan

- [x] Run all 59 appliance Python tests.
- [x] Run ShellCheck for the appliance and smoke scripts.
- [x] Run `bash -n` and `git diff --check`.
- [x] Run the repository pre-commit suite under Node 24, including
backend/frontend type checks and API generation checks.
- [x] Build and smoke-test the exact current-head image on `linux/amd64`
and `linux/arm64`.
- [x] Verify literal `docker run`, `/data` reuse, generated-secret
persistence, FalkorDB graph persistence, frontend database isolation,
and service recovery on the exact current head.
- [x] Pass current-head Trivy vulnerability and embedded-secret scans.

### Checklist 📋

#### For code changes:

- [x] I have clearly listed my changes in the PR description.
- [x] I have made a test plan.
- [x] I have tested my changes according to the complete test plan
above.

#### For configuration changes:

- [x] `.env.default` is already compatible with these changes.
- [x] `docker-compose.yml` is already compatible with these changes.
- [x] I have included a list of configuration changes under **Changes**.

### Stack navigation

This appliance is based directly on `dev`. Publication and end-user
documentation are handled separately.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Large new packaging surface with auth, secrets, and DB privilege
boundaries; mitigated by extensive smoke tests and isolation checks, but
it is still a new distribution path operators may run in production.
> 
> **Overview**
> Introduces an **experimental single-container AutoGPT Platform** you
can run with one `docker run`, bundling the frontend, backend services,
PostgreSQL, Valkey, RabbitMQ, FalkorDB (Graphiti memory on by default),
nginx, and Supervisor. State and generated secrets live on a **`/data`
volume**; only **port 3000** is exposed.
> 
> The image is built via **Docker Bake** from the existing backend
`server` target plus a new `autogpt_platform/single-container/` stack
(entrypoint/bootstrap, runtime config, least-privilege **frontend DB
role**, loopback-pinned internal ports, health checks, watchdog).
Operators get **`.env.example`** for public URL, signup policy, and LLM
keys.
> 
> **CI** adds `platform-single-container-docker.yml`: unit tests for
appliance helpers, per-arch image build, a long **end-to-end smoke
script** (health, auth, hostile `backend.json`, isolation,
FalkorDB/nginx recovery), and **Trivy** critical vuln + secret scans.
Hosted multi-container deployments are unchanged.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
2d368953ae. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Claude <[email protected]>
 2026-08-10 20:41:46 +00:00
Otto 965b7d3e04 dx: Add PR overlap detection & alert (#12104) 
## Summary

Adds an automated workflow that detects potential merge conflicts
between open PRs, helping contributors coordinate proactively.

**Example output:** [See comment on PR
#12057](https://github.com/Significant-Gravitas/AutoGPT/pull/12057#issuecomment-3897330632)

## How it works

1. **Triggered on PR events** — runs when a PR is opened, pushed to, or
reopened
2. **Compares against all open PRs** targeting the same base branch
3. **Detects overlaps** at multiple levels:
   - File overlap (same files modified)
   - Line overlap (same line ranges modified)
   - Actual merge conflicts (attempts real merges)
4. **Posts a comment** on the PR with findings

## Features

- Full file paths with common prefix extraction for readability
- Conflict size (number of conflict regions + lines affected)
- Conflict types (content, added, deleted, modified/deleted, etc.)
- Last-updated timestamps for each PR
- Risk categorization (conflict, medium, low)
- Ignores noise files (openapi.json, lock files)
- Updates existing comment on subsequent pushes (no spam)
- Filters out PRs older than 14 days
- Clone-once optimization for fast merge testing (~48s for 19 PRs)

## Files

- `.github/scripts/detect_overlaps.py` — main detection script
- `.github/workflows/pr-overlap-check.yml` — workflow definition
 2026-02-13 15:45:10 +00:00