mirror of
https://github.com/Significant-Gravitas/AutoGPT.git
synced 2026-11-13 06:21:32 -10:00
dev
556
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Krzysztof CzerwinskiandClaude Opus 5.5
|
aace9458c6 |
docs(platform): list what an account erasure leaves in Stripe, cost records and handed-over agents (#16794)
### Why / What / How **Why:** we ran two account erasures end to end on dev with test accounts (one through the hourly job, one through the CLI), then searched every table, the buckets, FalkorDB, PostHog and Stripe test mode for what was left. Everything the runbook says goes was gone. A few things that stay were not in the runbook, or were described differently from what dev showed. **What:** `docs/engineering/account-deletion.md` only: - **Stripe:** the saved payment methods, subscriptions, subscription schedules and Checkout Sessions stay too. The customer keeps its email, name, any billing address and saved cards; invoices and completed Checkout Sessions keep the email (invoices also the name and address they copied); the customer, subscriptions, schedules and the trial's Checkout Session keep the user ID in their metadata. - **`PlatformCostLog`:** besides the nulled user ID, rows keep the chat, run, agent and expert IDs, and an AutoPilot turn's metadata keeps the trial's ID. The chat, run, expert and trial rows are gone; the agent ID can still name an agent that stays, and the trial ID matches the `trial_enrollment_id` kept in Stripe. - **Handed-over agents:** an agent that goes over keeps its organization and team IDs even when the erasure deletes that organization, as do its listing versions that carry one; the listing's own organization is set to null. It still opens and runs for the accounts using it. - **Library creator:** nothing sets `LibraryAgent.creatorId` today, when an entry is added, so an entry added from the Marketplace reports `creator_name` "Unknown" (to Otto and the API) before and after the erasure; the "Deleted user" relabel only applies to entries that name one. **How:** what the dev run showed, written into "Agents other accounts use" and "What it keeps, and why". Part of SECRT-3462. ### Changes 🏗️ - `docs/engineering/account-deletion.md`: the four points above. No code changes. ### Agents and large language models used - Claude Code with Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Each statement matches what the dev erasures left (database, Stripe test mode, a run of the handed-over agent by another account) #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**): none 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Krzysztof CzerwinskiandClaude Opus 5.5
|
7d47ba528a |
fix(platform): catch keys and Checkouts made in the seconds after an account closes (#16765)
### Why / What / How **Why:** closing an account promises to end its API keys, app grants and billing, and **Keep my account** promises that what closing stopped stays stopped. Each REST worker caches "is this account closed" for 30 seconds, and only the worker that handled the close clears its own cache. For up to 30 seconds after a close, a token issued before it still works on the other workers. An API key, OAuth grant, auto top-up setting or Stripe Checkout made in that window survives the close. A key or grant works again after Keep. A Checkout paid later subscribes a closed account, and the erasure left that subscription running, because it skipped the account's own Stripe customer. **What:** each place a leftover from that window can show up now catches it: right after the close, where keys, grants, auto top-up and Checkouts are made, at Keep and at erasure. The auth hot path is not changed. **How:** - **Close** (`service.delete_account`): once `close_account` wins, the API keys and OAuth grants are revoked again, the plan cancel (which first expires the open Checkouts) runs again and auto top-up is switched off again. All are best effort (bounded, logged, never raised), because the account is already closed, and each is skipped if the owner has kept the account by then. - **Making things**: `POST /api/api-keys`, the OAuth authorize route, `POST /credits/auto-top-up` and the plan, trial and top-up Checkout paths read the closed state from the database first (`refuse_closed_account`, uncached), before any Stripe customer, trial seat or charge. A closed account gets the same 401 "This account is closed." the auth dependencies already return. There is no new copy. - **Checkout race**: after Stripe returns a new Checkout, the closed state is read again (`expire_if_account_closed`). If the account closed meanwhile, the Checkout is expired and the call gets 401. Otherwise the close came after the Checkout existed, and the sweep after the close expires it. - **Keep** (`reopen_account`): API keys and OAuth grants made since the close are revoked in the same transaction that deletes the request. Only the call that reopens the account revokes anything: a second Keep, or one too late, changes nothing. A failure rolls back and leaves the account closed, so the owner can try again. - **Erasure**: the Stripe step now covers the account's own customer as well as its sole orgs' customers, in both the dry run and the apply, as a backstop. The step is renamed `org_subscriptions` → `subscriptions` to match. `STRIPE_API_KEY` already reaches the database manager through the shared config. - **Docs**: the runbook, `restore.py` and `reopen_account` now describe what happens in the window. Sign-ins are not ended by Keep: a token issued before the close (up to an hour) and the browser's cached session cookie (up to 5 minutes) are refused only while the account is closed, and work again once it is kept. Overlap: #16567 checks that the session is still live on the routes that create credentials. That covers the API-key part for session tokens. This PR touches none of its files. Out of scope: reading the closed state through a shared cache on every request (a separate ticket), revoking stateless tokens, and refunds. Resolves SECRT-3459. ### Changes 🏗️ - `account_deletion/service.py`: revoke, cancel and switch auto top-up off again after the close, best effort, while the account is still closed. - `data/account_deletion.py`: `refuse_closed_account` (uncached, raises 401); `reopen_account` revokes keys and grants in its transaction. - `data/subscription_checkout.py`: `expire_if_account_closed`, called after each Checkout is made. - `data/credit.py` (top-up and plan Checkout) and `data/subscription_trial_checkout.py` (trial Checkout, new or resumed): refuse a closed account first, and expire a Checkout made as it closed. - `api_keys/routes.py`, `oauth.py` (authorize), `billing/credits/routes.py` (auto top-up): refuse a closed account. - `data/account_erasure/{preflight,models,subscriptions,routine}.py`: the account's own customer is included; the step is renamed to `subscriptions`. - `docs/engineering/account-deletion.md`: what closing does about the window, and erasure step 3. The proxy timeout comment's worst case is updated (180 s still covers it). - Tests: - mock-only tests for each step, the reopen's revoke, and each route's refusal; - Checkout tests: refused first (no customer, no trial seat), and expired when the account closes while Stripe makes it; - a DB route test: a key made after the close is revoked by Keep; - the existing Checkout tests patch the new check. No configuration changes. ### Agents and large language models used - Claude Code with Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [ ] I have tested my changes according to the test plan: - [x] Mock-only unit tests pass locally: account deletion service, reopen and Keep, the erasure steps, preflight and routine, the trial, plan and top-up Checkout paths, the API-key, OAuth authorize and auto top-up routes - [x] ruff, black, isort and pyright pass on the changed files - [ ] The DB-backed route tests pass in CI (`account_deletion/routes_test.py`) - [ ] On dev, with the QA account: close the account while an old token keeps creating API keys, then Keep. Every key is revoked. - [ ] On dev, with the Stripe test key: the `erase-account` dry run of a closed test account with a test customer counts its own subscriptions 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Krzysztof CzerwinskiandClaude Opus 5.5
|
a68d90ca1d |
fix(backend): ban records match every spelling of an address, and Keep refuses a banned account (#16764)
### Why / What / How **Why:** a ban record (kept when a banned account is erased) should cover the mailbox, not one spelling of it. Today it hashes the address only trimmed and lower-cased, so `name+x@domain` or a dotted Gmail address is a different record. And "Keep my account" let a banned account reopen itself with an access token issued before the ban, while the runbook and `restore.py` say a banned person can't keep the account. Ticket: [SECRT-3461](https://linear.app/autogpt/issue/SECRT-3461). **What:** ban records hash a canonical form of the address, the lookup also matches records written before this change, and Keep answers 403 `account_banned` while a ban is in force. **How:** - `canonical_address`: trim, lower-case, drop everything from the first `+` in the local part (every domain), and on `gmail.com`/`googlemail.com` drop the dots and read both domains as `gmail.com`. A part that would come out empty is left as it is. - `keep_ban_record` writes the canonical hash. `is_banned_address` asks for `{"in": [canonical, old]}`, where "old" is the trimmed, lower-cased hash existing records hold. Existing rows are not rewritten. - `restore_account` checks `is_user_banned` first and raises `BannedAccountError` before touching anything; the route maps it to 403 with `ACCOUNT_BANNED_DETAIL`, which the frontend's request layer already handles by signing the person out. A lapsed temporary ban is not a ban, so that account can still be kept. ### Changes 🏗️ - `backend/data/account_erasure/ban_record.py`: `canonical_address`, `email_hashes`; `email_hash` now hashes the canonical form; lookup matches both hashes. - `backend/api/features/account_deletion/restore.py`, `routes.py`: Keep refuses a banned account (403 `account_banned`); docstrings updated. `openapi.json` regenerated (the 403 description). - `backend/api/conftest.py`: the new `is_user_banned` call site joins the list the no-bans fixture patches. - Tests: `ban_record_test.py` (+tag, Gmail dots, googlemail, an old record still matching, a non-Gmail dot still distinct), `restore_test.py` (banned refused with nothing changed, not banned kept), `routes_test.py` (DB-backed: banned account refused with a token from before the ban; a lapsed ban can keep). - `docs/engineering/account-deletion.md`: the canonical form under "Ban records", `email_hashes` under "Changing a ban record", and Keep refusing a banned account. ### Agents and large language models used - Claude Code with Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] `ban_record_test.py` and `restore_test.py` pass locally (mock-only) - [x] `poetry run format` (ruff, isort, black, pyright) clean - [ ] `routes_test.py` (database-backed) in CI 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Krzysztof CzerwinskiandClaude Opus 5.5
|
c6533779f0 |
fix(backend): free a crashed account erasure's lease within minutes, log CLI runs and page on stalled erasures (#16766)
### Why / What / How **Why:** a run of `erase-account` that crashes or is killed (for example when its pod is OOM-killed) never releases its per-account Redis lease, and the lease lasted an hour. Until it ran out, every rerun said "another run is erasing it" and exited 1. A CLI erasure also left no log line, since the CLI switched INFO logging off, and nothing noticed an erasure that started and never finished: the hourly job logs its own failures, but it is off by default, and a failed CLI run only printed to its terminal. **What:** the lease now lasts 5 minutes and a heartbeat keeps a live run's lease, the CLI keeps the erasure's own log lines, a scheduler check pages on stalled erasures, and the runbook covers all three plus how to run the CLI in production carefully. **How:** - `lease.py`: `LEASE_TTL_SECONDS` drops from 1 h to 5 min. `renew_in_background` renews the lease every 60 s (5 beats per lifetime, and above the lock's own rate limit of a tenth of the TTL) while the erasure runs, because one step (the files sweep of a large account) can run for minutes. The check before each step still stops an erasure whose lease is gone. A failed beat is logged and the next one tries again. - CLI: `logging.disable(logging.INFO)` becomes the backend's logging with the root logger at WARNING and `backend.data.account_erasure` at INFO, so the routine's `Account erasure of <user ID>: …` line (IDs only) is printed and logged while Prisma and client INFO chatter stays out of the report. It reaches Cloud Logging when the CLI is the pod's own process. Under `kubectl exec` it only reaches the terminal, which is why the runbook now says to save the output. - `monitoring/account_erasure_monitor.py`: `report_stalled_account_erasures` asks the database manager (`stalled_account_erasures` → `request.stalled_user_ids`) for requests with `startedAt` older than `ACCOUNT_ERASURE_STALL_ALERT_SECS` (default 3 h) and no `completedAt`, and pages Discord and Sentry with their user IDs, at most once a day per account, like the auth identity monitor. It is registered every `ACCOUNT_ERASURE_STALL_CHECK_INTERVAL_SECS` (default 1 h), whether or not the erasure job is on, because the CLI claims requests too. - `docs/engineering/account-deletion.md`: the 5-minute lease and what "another run is erasing it" means after a crash, the stalled-erasure check and how to clear it, and production care: check `kubectl top pod` first, one command at a time, save the terminal output of every prod run. A small piece of [SECRT-3460](https://linear.app/autogpt/issue/SECRT-3460). A second PR adds closing without the email. Running the CLI outside a serving pod is not in this PR. ### Changes 🏗️ - `backend/data/account_erasure/lease.py`: 5-minute TTL, `HEARTBEAT_SECONDS = 60`, `renew_in_background` context manager. - `backend/data/account_erasure/routine.py`: an apply holds the lease under the heartbeat. - `backend/cli/erase_account.py`: `_log_the_erasure_only()` replaces `logging.disable(logging.INFO)`. - `backend/data/account_erasure/request.py`: `stalled_user_ids`. - `backend/data/db_manager.py`: `stalled_account_erasures` RPC (manager and sync client). - `backend/monitoring/account_erasure_monitor.py` (new), exported from `backend/monitoring`; scheduler job `report_stalled_account_erasures` and manual trigger `execute_report_stalled_account_erasures`. - `backend/util/settings.py`: `account_erasure_stall_check_interval_secs` (default 3600, min 60) and `account_erasure_stall_alert_secs` (default 10800, min 1800). - Tests: lease lifetime and heartbeat (renews during a long step, retries a failed beat, stops with the erasure or on an error), routine holds the heartbeat for the whole apply and not for a dry run, the CLI logs the erasure's line and no INFO chatter, the stalled query, the monitor (quiet, pages with IDs only, re-pages once a day), and the scheduler registering the check with the job on or off. - Runbook as above. Configuration: two new optional settings with defaults (above). No `.env.default` or `docker-compose.yml` change is needed. ### Agents and large language models used - Claude Code with Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] `pytest backend/data/account_erasure/lease_test.py routine_test.py request_test.py backend/cli/erase_account_test.py backend/monitoring/account_erasure_monitor_test.py` and the scheduler registration test pass locally (mock-only) - [x] `poetry run format` and `poetry run lint` (ruff, isort, black, pyright) pass - [ ] CI runs the database-backed erasure tests - [ ] On dev after merge: kill a CLI `--apply` mid-erasure, and a rerun gets through within 5 minutes - [ ] On dev after merge: a request claimed over 3 h ago with no `completedAt` pages once #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Krzysztof CzerwinskiandClaude Opus 5.5
|
e7f074f161 |
feat(backend): one-off to delete bucket folders of deleted accounts (#16767)
### Why / What / How
**Why:** accounts deleted by hand before the erasure routine existed
left their files in the buckets, and the privacy policy says we don't
keep data of people who are gone. A read-only count found a few such
folders on prod, and the call is to delete them. Ticket: SECRT-3463
(Linear).
**What:** a one-off script, `scripts/delete_orphan_bucket_folders.py`,
and a runbook section in `docs/engineering/account-deletion.md` for
running it.
**How:** in every configured bucket, it lists the folders under an
account's prefixes (`workspaces/{workspace id}/`,
`workspaces/cli-sessions/{user id}/`, `uploads/users/{user id}/`,
`users/{user id}/`, `oauth-apps/{app id}/`, the ones the erasure sweeps)
and checks each folder name against its table with SELECTs. A folder
whose row is gone is deleted, except:
- a `users/` folder whose media a listing, a library entry, or an org,
org profile, expert or profile avatar still shows, checked again just
before deleting;
- a folder with a file from the last 7 days, which may belong to a row
created after the check;
- a folder whose row exists again on a second check just before
deleting;
- a folder not named by a UUID. `workspaces/chat-transcripts/` itself is
one: an earlier count read it as an orphan workspace, and deleting it
would have taken every account's old transcripts.
`workspaces/chat-transcripts/`, where transcripts lived until #12804, is
left entirely alone by default, and the output says it was skipped:
whether to delete the transcript folders of gone accounts is still being
decided. `--include-chat-transcripts` adds
`workspaces/chat-transcripts/{user id}/` to the sweep.
It is a dry run unless `--apply`, and prints counts per bucket and
prefix, never an ID or a file name. It deletes folder by folder; a
folder with a failed delete stays, and failures are counted by error
type and status. A prefix that hits an error prints only the error's
type and status, and the run exits non-zero. It is piped into the
database manager pod using the least memory, by name (`poetry run python
-u - < script`), so it runs on whatever image is deployed. Just before
deleting a folder it lists it again and keeps it if a file arrived since
the first listing.
### Changes 🏗️
- `autogpt_platform/backend/scripts/delete_orphan_bucket_folders.py`:
the one-off.
-
`autogpt_platform/backend/scripts/test_delete_orphan_bucket_folders.py`:
mock-only tests against a fake bucket.
- `docs/engineering/account-deletion.md`: "One-off: files of accounts
deleted before the erasure", with the commands, what it keeps, and why
dev shows preview accounts' folders as orphans (previews write to dev's
bucket with a database of their own).
### Agents and large language models used
- Claude Code with Claude Opus 5.5
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Unit tests: dry run counts and deletes nothing; apply deletes only
orphans and keeps shown, recent, reappearing and non-UUID folders; a 404
on delete is fine, another error is counted; the report line names no
folder; a prefix that fails prints no folder and fails the run; the
media-owner pattern matches every stored URL form; a file written
between the listings keeps its folder; `workspaces/chat-transcripts/` is
skipped (and says so) unless `--include-chat-transcripts`
- [x] Dry run in dev's database manager: counts per prefix, nothing
deleted
- [ ] `--apply` is run by the operator, after reading the dry run's
counts
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
|
||
|
Krzysztof CzerwinskiandClaude Opus 5.5
|
ab03e6a06a |
fix(backend): delete the PostHog person after the account's rows, and let a rerun repeat it (#16763)
### Why / What / How **Why:** the account erasure deleted the PostHog person before the account's rows. An event captured in that gap (Postmark's delivery webhook for the account-closed email is the usual one with `--close --now`, a Stripe webhook can do it too) makes PostHog create the person again, and the request is still marked done. A later `erase-account --apply` then printed "nothing ran: erased already", so there was no way to clean it up with the CLI. Ticket: [SECRT-3458](https://linear.app/autogpt/issue/SECRT-3458). **What:** the PostHog step now runs after the database step (it needs only the user ID; once the rows are gone, webhooks can no longer find the account to capture events for it). And because an event already on its way can still land after the delete, an `--apply` on a finished erasure now repeats the PostHog delete. **How:** - `routine._erasure_steps`: `posthog` moved from before `mailerlite` to after `database`, before `request`. A PostHog step that fails or isn't configured still stops the apply before the request is marked done; the next run resumes (the earlier steps find nothing, and PostHog runs again). - `routine._run`: when the claim fails because the request is completed, it runs `_repair_steps` (the PostHog delete only) under the same lease, instead of nothing. `not_claimed` stays `erased already` and `completedAt` is not touched. The other outside steps find their data through the rows (or the address they held), and nothing re-creates it once the rows are gone, so they are not repeated. The final log line also says when a run only repeated the delete. - CLI: the report prints `erased already: repeats only the PostHog delete` with the step line, and the command fails when that delete failed (it still exits 0 when it went through). - Runbook `docs/engineering/account-deletion.md`: step order, the Confirm step waits two minutes and reruns `--apply` if `posthog persons` is above 0, a note for erase-at-once requests on releases before this fix (close first, wait, then `--close --now`, then check), and the user ID instead of the email wherever the rows may be gone. ### Changes 🏗️ - `backend/data/account_erasure/routine.py`: step order, repair run on a finished erasure, docstrings. - `backend/cli/erase_account.py`: report line and exit code for the repair run, docstring. - Tests: `routine_test.py` (new order, PostHog after the rows, repair run repeats only PostHog, failed repair), `cli/erase_account_test.py` (repair output and exit codes), `account_erasure_db_test.py` (the rerun of a finished erasure now repeats only the PostHog step). - `docs/engineering/account-deletion.md`. No configuration changes. ### Agents and large language models used - Claude Code with Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] `routine_test.py`, `cli/erase_account_test.py`, `steps_test.py`, `sweep_test.py`, `third_parties_test.py` pass locally (mock-only) - [x] ruff, black, isort; pyright shows no new errors on the changed files - [ ] `account_erasure_db_test.py` (database-backed) in CI - [ ] After the merge, on dev: `erase-account <user ID> --apply` on a finished erasure with a leftover PostHog person deletes it, and the dry run two minutes later shows `persons=0` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Krzysztof CzerwinskiandClaude Opus 5.5
|
b81a04fc1d |
fix(backend): account erasure removes AutoPilot transcripts from before April 2026 (#16769)
### Why / What / How
**Why:** AutoPilot stored chat transcripts under
`workspaces/chat-transcripts/{user id}/` until #12804 moved new ones to
`workspaces/cli-sessions/{user id}/`. The older files were kept, but the
account erasure's files step only sweeps `cli-sessions`, so an erased
account that chatted before the move keeps those transcripts.
**What:** the erasure now sweeps both transcript folders, in every
bucket and, on a self-hosted stack, on disk. The existing
`chat-transcripts/` folder itself stays: this changes only what an
erasure removes.
**How:** one `_TRANSCRIPT_FOLDERS` tuple feeds both `_prefixes`
(buckets) and `_local_dirs` (the local-disk fallback), behind the same
`_TRANSCRIPT_ID` check as before, since the old code sanitised the user
ID in its paths the same way (hex and hyphens only).
Ticket: [SECRT-3469](https://linear.app/autogpt/issue/SECRT-3469)
### Changes 🏗️
- `backend/data/account_erasure/storage.py`: add
`workspaces/chat-transcripts/{user id}/` to the bucket prefixes and the
local folders of the files step.
- `backend/data/account_erasure/storage_test.py`: the fake bucket and
the self-hosted folder hold old transcripts of the account and of
another account; the tests check the account's are swept (dry run counts
and real delete) and the other account's stay.
- `docs/engineering/account-deletion.md`: step 6 lists the old prefix.
### Agents and large language models used
- Claude Code with Claude Opus 5.5
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] `pytest backend/data/account_erasure/storage_test.py` (mock-only):
21 passed
- [x] ruff, black, isort and pyright clean on the changed files
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
|
||
|
Krzysztof CzerwinskiandClaude Opus 5.5
|
6006036d8b |
feat(backend): close and erase an account without the confirmation email (erase-account --no-email) (#16768)
### Why / What / How **Why:** closing an account always sends the account-closed email, and an erasure sends it late when closing could not. Sometimes nobody should hear from us about it, but an operator had no way to close and erase an account without that email except a one-off script with the sender patched out. **What:** `poetry run cli erase-account <account> --close --now --no-email` closes and erases the account without the confirmation email. The flag works only with `--close --now`, for an owner erased at once. **How:** - New column `AccountDeletionRequest.confirmationSkipped` (boolean, default false). The migration is a catalog-only `ADD COLUMN ... DEFAULT false`, with no table rewrite. - `delete_account(..., send_confirmation=False)` skips `_send_confirmation` and `close_account(..., confirmation_skipped=True)` writes the column. Settings and the paywall keep the default (email sent). - The erasure's `confirmation` step reads the column (`Subject.confirmation_skipped`) and sends nothing: `emails=0`, note "closed without the email (--no-email)". - For an account closed earlier whose email has not gone out (`confirmationSentAt` null), `request.skip_confirmation` marks the request before the erasure runs, so the erasure doesn't send it late. - CLI: one usage check, `--no-email goes with --close --now.`, behind the `_NO_EMAIL_ONLY_WITH` message. With the flag, the CLI prints a line saying no email goes to the owner. - Runbook `docs/engineering/account-deletion.md`: a short "Without the email" part under step 3. Part of [SECRT-3460](https://linear.app/autogpt/issue/SECRT-3460). The lease, log line and stalled-erasure alert are in #16766. User IDs: `skip_confirmation` updates only the row of the given `userId`, and only operators reach it through the CLI, so no request-level ownership check applies. ### Changes 🏗️ - `schema.prisma` and migration `20261112120000_account_deletion_confirmation_skipped`: `confirmationSkipped Boolean @default(false)`. - `backend/data/account_deletion.py`: `close_account(..., confirmation_skipped=False)`. - `backend/api/features/account_deletion/service.py`: `delete_account(..., send_confirmation=True)`. - `backend/data/account_erasure/request.py`: `close_for_email_request(..., send_email=True)`, `skip_confirmation`, and the confirmation step honouring the column. - `backend/data/account_erasure/models.py` and `preflight.py`: `Subject.confirmation_skipped`. - `backend/cli/erase_account.py`: the `--no-email` option, its check, and a docstring line. - Tests: - service: no email, and the column recorded. - request: close without the email, `skip_confirmation`, and no late email on dry run and apply. - CLI: an open account, a closed account marked, a dry run marks nothing, and the flag refused without `--close --now`. - DB-backed (CI): three closed accounts. Only the one neither closed with the skip nor marked gets the late email. Configuration: one database migration (an additive column with a default). No env or compose change. ### Agents and large language models used - Claude Code with Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Mock-only tests pass locally: `erase_account_test.py`, `request_test.py`, `preflight_test.py`, `service_test.py` and `routine_test.py` - [x] `poetry run format`, ruff and pyright pass on the changed files - [ ] CI runs the DB-backed erasure test and the migration - [ ] On dev after merge: `--close --now --no-email --apply` on a QA account sends no email, and its request shows `confirmationSkipped` #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Nicholas TindleandClaude Opus 5.5
|
938143fae0 |
fix(blocks/exa): verify webhook secrets on preset triggers and check signature timestamps (#16760)
### Why / What / How **Why.** Two gaps in the Exa webhook verifier (`backend/blocks/exa/_webhook.py`): 1. A Search Monitor trigger set up as a preset stores its `webhook_secret` nested under `_node_input_mask_<node id>`. Every setup path writes it that way (library Run modal, `setup_agent_webhook_trigger`, `/presets/setup-trigger`). `_search_monitor_secrets` read it from the top level of `preset.inputs`, didn't find it, and verified the delivery as if no secret had been set. 2. `verify_exa_signature` signs over the `t=` timestamp but never compared it with the clock. **What.** - `_search_monitor_secrets` reads preset trigger configs through `preset_trigger_configs()`, as `GenericWebhooksManager._configured_secret` and the All Quiet manager already do. Nested configs are read, and a legacy flat preset is read whole. - `verify_exa_signature` refuses (403) a delivery whose `t` is more than 300 s from now in either direction, or doesn't parse as an integer. It checks the signature first, so a timestamp refusal only happens for a delivery Exa really signed, and it logs the webhook id and the clock offset. This is the window Exa's [verification guide](https://exa.ai/docs/websets/api/webhooks/verifying-signatures) suggests, and the one the Stripe, RMFG and All Quiet verifiers use. **How.** The documented "URL only until the secret is pasted" behaviour stays: the secret only exists once the monitor is created from the trigger's URL. A secret set on any node or preset of the webhook is enforced. No event-ID deduplication (it needs shared state; no verifier in the tree does it). ### Changes 🏗️ - `backend/blocks/exa/_webhook.py`: preset reader, `SIGNATURE_TOLERANCE_SECONDS = 300`, timestamp check and log line. - `backend/blocks/exa/exa_webhook_manager_test.py`: Search Monitor signature tests run for node-attached, nested-preset and flat-preset layouts; a node and a preset sharing a webhook; `TestSignatureTimestamp` for both webhook types (±300 s edges, stale, future, unparseable); rejection tests assert their reason. - `backend/api/features/integrations/webhook_ingress_test.py`: fresh timestamps in `TestExaSignature`; new `TestExaSearchMonitorPresetTrigger` end to end through the ingress route, with the webhook built from the rows trigger setup writes. - `docs/integrations/block-integrations/exa/search_monitor_trigger.md`, `webhook_blocks.md`: one sentence on the window each. ### Agents and large language models used - Claude Code with Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] `exa_webhook_manager_test.py` + `webhook_ingress_test.py` in the Linux backend image: 154 passed - [x] `webhook_blocks_test.py`, `exa_platform_credentials_test.py`, `webhook_preset_trigger_test.py`, `test_block.py -k Exa`: pass - [x] `scripts/generate_block_docs.py --check`: in sync - [x] Mutation checks: old preset reader, no timestamp check, no `abs()`, `>=` for `>`, a 0 s window, and checking the time before the signature each fail the tests meant to catch them - [ ] Before release: one real Exa Search Monitor and one Websets delivery, plus one forced retry, to confirm `t` is Unix seconds and is re-signed on retry #### For configuration changes: - [x] Not applicable: no configuration changes --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Nicholas TindleandClaude Opus 5.5
|
c58c7f0b22 |
docs(platform): Microsoft sign-in is set up on both Entra apps and in Vercel (#16787)
### Why / What / How **Why:** `docs/engineering/microsoft-local-e2e.md` still said dev's Entra app lacked step 5, and that prod's app hadn't been set up for the Microsoft connection (#16268) or Continue with Microsoft (#16269). Both shipped to prod in v0.11.2 on 2026-11-11, and the setup was finished on 2026-11-12. The guide also told people to rotate a client secret by adding the new one first, and that no longer works. **What:** the guide's Entra section now records what each environment has, and how a secret is rotated given Microsoft's limit. **How:** docs only. The facts come from the Entra apps' manifests and pages, Vercel's environment settings, and read-only checks against Microsoft's authorize endpoint and dev-builder. ### Changes 🏗️ - **Step 4 (Certificates & secrets):** - An app that accepts personal Microsoft accounts can hold only two client secrets. Entra refuses a third with "credential limit has been reached". - Each app now has two: the bot's, used by the backend as `MICROSOFT_CLIENT_SECRET`, and Vercel's, used for `AUTH_MICROSOFT_CLIENT_SECRET`, which expires in 2028-11. - So rotating one means deleting it first. - **Status:** - Dev's app: on 2026-10-26 it got steps 1–4, for localhost and the local test stack. On 2026-11-12 it got `dev-builder.agpt.co`'s two redirects and step 5. - Prod's app: all five steps on 2026-11-12. Until then, personal accounts couldn't connect on prod. - Four Copilot permissions need admin consent in each organization. agpt.co hasn't granted it. - Browser sign-in is set in the infra shared config (infra #498). - Continue with Microsoft and browser sign-in are live on dev. Prod has every setting and switches both on with its next deploy. ### Agents and large language models used - Claude Code with Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Each fact checked against the live systems on 2026-11-12: - Microsoft's `/consumers` authorize endpoint serves the sign-in page for both apps' registered callbacks, and still rejects an unregistered one. - Both apps' API permissions pages list the 17 delegated permissions. - Prod's Token configuration lists `xms_edov`. - Dev-builder's `/api/auth/login/with-provider` returns Microsoft's authorize URL: the dev app's client ID, dev-builder's callback, scope `openid profile email`. - A headless browser sees "Continue with Microsoft" on dev-builder's `/login` and `/signup`, but not on platform.agpt.co yet. - Dev's API and WebSocket servers' shared ConfigMaps hold `MICROSOFT_BROWSER_SIGN_IN=true`. - The two-secret limit is Entra's own error on the dev app. #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**): none in this repository. The Entra, Vercel and infra settings it describes are already applied. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]> |
||
|
Reinier van der LeerandClaude Opus 5.5
|
0f48ec163e |
fix(blocks/discord): read at most 1 MiB of a text attachment in Read Discord Messages (#16719)
Read Discord Messages now adds at most the first 1 MiB of a `.txt` or
`.py` attachment to `message_content`, and ends the text with a note
when it cut a longer file.
Resolves [SECRT-3432](https://linear.app/autogpt/issue/SECRT-3432)
### Why / What / How
**Why:** a workflow that reads Discord messages usually hands
`message_content` to a model block, and a long log or source file
attached to a message went into it in full, more text than that model
can take.
**What:** an attachment of up to 1 MiB is added exactly as before. A
longer one is cut at 1 MiB and followed by `[File truncated to its first
|
||
|
|
90a7c6aebe |
feat(platform): let Otto answer with visual cards (#16677)
### Why / What / How **Why:** Otto's answers read like documents: headings, bullets and tables in markdown. Tool calls already get rich cards, but Otto's own answers don't. **What:** A new `render_ui` tool lets Otto show visual cards in a message: stats, charts, workflow steps, images, timelines, tables and one-click options. Everything ships behind `copilot-visual-answers`, off by default. This is the first half of the plan; a declarative "canvas" for one-off interactive visuals follows in a separate PR. **How:** We already use the AI SDK's generative UI pattern (typed tool parts mapped to React), so this needs no new SDK. The tool validates a typed list of blocks with Pydantic and returns them, and the chat renders the `tool-render_ui` part as cards outside the tool chain. Blocks hold only plain text, numbers and `workspace://` image refs. There is no URL, HTML or style field, so nothing a model writes can leave the page. An option sends its visible title as the reply, so what someone clicks is what gets sent, and any action it leads to still goes through the usual approval. ### Changes 🏗️ **Backend** - `render_ui` tool and block models, with caps of 6 blocks and about 16 KB per call. - The tool is hidden on chat platforms, for callers that keep only the final text (subtasks and the AutoPilot block, through a new `text_only_result` session field), and when the flag is off. Scheduled follow-ups keep it, because people read those chats. - The schema adds about 4 KB to each turn, only where the flag is on. Budgets in `tool_schema_test.py` are raised by that measured amount. **Frontend** - `tools/RenderUI/` with one component per block. Recharts loads on demand. - Options work only on the newest message, when no turn is running, and outside shared chats. - New `visual_option_clicked` PostHog event, added to the tracking plan. - The cards use soft gradients and shadow rings. The timeline has a connecting line, status dots and date pills; the table has a header band and number columns aligned right; chart fills fade to the axis; workflow steps stack on phones. - Every block type appears on `/copilot/styleguide` under "Tool: Visual answers", so the cards can be checked without a live chat. **Login and signup (out of scope, small)** - From the `lg` breakpoint the auth pages are exactly one screen tall and no longer scroll. The login page's "What's new" list scrolls inside its card; on a short window the signup panel scrolls on its own (`AuthSplitLayout`, `AuthMarketingPanel`, `MarketingHeader`, `LoginWhatsNew`). **Configuration** - Create the `copilot-visual-answers` flag in PostHog in each environment. It's backend-only and defaults to off. The chat always draws stored cards, so turning it off doesn't break old conversations. ### Agents and large language models used Claude Code with Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [ ] I have tested my changes according to the test plan: - [x] `render_ui_test.py`: validation, caps, null optionals, schema shape and who is offered the tool - [x] `tool_schema_test.py`, `capability_tools_test.py`, `permissions_test.py`, `gate/policy_test.py`, tool names fixture - [x] `RenderUIParts.test.tsx`: every block, sending, disabled states, unknown blocks, stopped turns, plain-text safety - [x] A shared fixture (`renderUIOutput.json`) ties the real tool output to what the frontend renders, in place of a browser end-to-end test - [x] Ran locally with the flag forced on: Otto answered a launch-plan prompt with a timeline and a table card - [x] Every block type checked on `/copilot/styleguide` at 1440 and 390 px, with no card overflowing - [x] Login and signup measured at 1440×900, 1440×700 and 390 px: the page height equals the window, and nothing overlaps - [ ] Check a real Otto answer on a preview with the flag on #### For frontend changes (design system, see `docs/engineering/design-system.md`): - [ ] `pnpm lint` passes, including the `design-system/*` rules. eslint and oxlint pass on the changed files; the full command was not run, so CI is the check - [x] No `dark:`/`light:` classes, raw palette classes (`bg-zinc-200`) or hex/`rgb()` colours (`bg-[#fff]`); colours use role tokens (`bg-surface`, `text-foreground-secondary`). Recharts SVG attributes are the exception, listed in `ds-colour-exempt.json` like the other charts - [x] No imports from `src/components/__legacy__`, `legacy/` paths or `@/components/ui`, and no `LegacyButton` / `legacy-*` Button variants - [x] Copy uses the `Text` atom with a `variant`/`tone`; primitives get a `variant`/`size`. `className` holds layout, plus on `Text` only truncation, line clamping, wrapping, breaking, `text-balance`/`text-pretty` and alignment - [ ] Before/after screenshots at 1440 and 390 px are attached for changes to `src/components/**`, `globals.css` or `tailwind*.ts`, or I wrote "no pixel change" and how I checked. `src/components/auth/AuthSplitLayout/**` changed; screenshots to be attached - [x] User-facing copy says workflow and Otto, never agent or graph (root `AGENTS.md`, "User-facing copy") #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]> Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: Claude Sonnet 4.5 (Cursor Code) <[email protected]> |
||
|
|
df71544e2d |
feat(platform): support native and browser Sign in with Apple (#16683)
### Why / What / How Native and browser clients need Sign in with Apple on the session foundation in #15384. This follow-up adds native identity-token exchange and browser authorization-code login to the existing server-side Better Auth service. Both use the same accounts, signup policy, bans, session issuance, access-JWT renewal and logout. Stacked on #15384 (`codex/secrt-2881-native-mobile-backend`). Native login uses the app's registered bundle identifier. Browser login uses a Service ID grouped under the same primary App ID, server-side signing credentials and an HTTPS Apple callback. The existing browser login API accepts `provider: "apple"`; this PR supplies backend support without adding client login controls. ### Changes 🏗️ - Register native, browser-only or combined Apple configuration. Require the nonce for direct ID-token exchange and validate browser callback identity against the Service ID. - Complete Apple's form POST through Better Auth's state-bound callback into HttpOnly browser cookies. Limit Apple's trusted origin to that callback POST; retain the foundation's native bearer export rules. - Generate a fresh five-minute ES256 client-secret JWT for each outbound Apple OAuth operation, including code exchange after delayed consent. Keep signing out of shared auth initialization; malformed or wrong-algorithm keys return a controlled HTTP 503 for browser Apple login while email and native Apple login remain available. - Recover omitted profile data only from the account linked to the verified Apple subject. Preserve private relay addresses, signup restrictions, bans and verified-email linking rules. - Add blank `AUTH_APPLE_APP_BUNDLE_IDENTIFIER`, `AUTH_APPLE_CLIENT_ID`, `AUTH_APPLE_TEAM_ID`, `AUTH_APPLE_KEY_ID` and `AUTH_APPLE_PRIVATE_KEY` settings. The browser return URL is `https://<auth-host>/api/auth/callback/apple`. [Configuration and client contract](https://github.com/Significant-Gravitas/AutoGPT/blob/codex/secrt-2881-apple-sign-in/docs/engineering/native-mobile/apple-sign-in.md). - Add real-handler integration tests with ephemeral Apple signing keys and the existing browser-provider route regression. Fully disabled deployments return the standard provider-not-found response; audience lists contain distinct configured identifiers. - No dependency change, schema migration, production secret or Python API change. Scanner baseline updates only track existing test-fixture line numbers. ### Agents and large language models used - Codex via T3 Code: gpt-6-astra (implementation, review and validation). - Claude Code: Claude Opus 5.5 (base-branch merge and live browser QA video). ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [ ] I have tested my changes according to the test plan: - [x] 942 auth/session/API-client/login/signup tests, including 55 Apple cases, with zero skips (after merging the base branch's Continue with Microsoft). - [x] Native Apple identity → signed session bearer → access-JWT renewal → logout; repeat login and private relay addresses preserve the same account. - [x] Browser authorization → cross-site form callback → state-bound completion → cookie session → logout. Grouped native and browser identities resolve to the same account. - [x] Invalid token claims/signatures/nonces, unavailable keys, state mismatch/replay, denied consent, rejected codes, untrusted redirects, signup policy and partial configuration. - [x] Client-secret verification after consent takes over five minutes; multiline and escaped-newline private-key configuration. Three malformed-key regressions first reproduced initialization failures, then verified email/native sign-in and the controlled browser error without credential logging. - [x] Whole frontend `pnpm format`, `pnpm lint`, `pnpm types`, then relevant `pnpm test:unit`; eight documentation-layout tests. - [x] Live browser acceptance against real Apple ([video](https://github.com/Significant-Gravitas/AutoGPT/pull/16683#issuecomment-6785819490)): QA App ID, Service ID and signing key with an HTTPS callback, run against this branch's auth service and an isolated database. Covers Hide My Email, platform user provisioning, an access JWT verified via JWKS, reload persistence, logout, and repeat login returning the same account. - [ ] Live native acceptance: a signed iOS build with the Sign in with Apple entitlement against a deployed backend, including shared native/browser account identity. Tests use the production auth configuration and real Better Auth cryptographic/session handling, with origin and CSRF checks enabled. Apple network responses, database storage, DNS and platform provisioning are controlled test boundaries. The automated tests use no real Apple credentials; the live browser run linked above used real Apple authorization with dedicated QA identifiers. <details> <summary>Example test plan</summary> - [ ] Create from scratch and execute an agent with at least 3 blocks - [ ] Import an agent from file upload, and confirm it executes correctly - [ ] Upload agent to marketplace - [ ] Import agent from marketplace and confirm it executes correctly - [ ] Edit an agent from monitor, and confirm it executes correctly </details> #### For frontend changes (design system, see `docs/engineering/design-system.md`): - [x] `pnpm lint` passes, including the `design-system/*` rules; I did not raise any count in `scripts/eslint-rules/design-system-baseline.json` except for a moved, renamed or split file (explained under **Changes**) - [x] No `dark:`/`light:` classes, raw palette classes (`bg-zinc-200`) or hex/`rgb()` colours (`bg-[#fff]`); colours use role tokens (`bg-surface`, `text-foreground-secondary`) - [x] No imports from `src/components/__legacy__`, `legacy/` paths or `@/components/ui`, and no `LegacyButton` / `legacy-*` Button variants - [x] Copy uses the `Text` atom with a `variant`/`tone`; primitives get a `variant`/`size`. `className` holds layout, plus on `Text` only truncation, line clamping, wrapping, breaking, `text-balance`/`text-pretty` and alignment - [x] Before/after screenshots at 1440 and 390 px are attached for changes to `src/components/**`, `globals.css` or `tailwind*.ts`, or I wrote "no pixel change" and how I checked - [x] User-facing copy says workflow and Otto, never agent or graph (root `AGENTS.md`, "User-facing copy") No pixel change: only server-side auth TypeScript, tests, environment defaults and engineering docs change. No components, markup, styles or client sign-in controls were changed; the markup-related checklist items are not applicable. #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) <details> <summary>Examples of configuration changes</summary> - Changing ports - Adding new services that need to communicate with each other - Secrets or environment variable changes - New or infrastructure changes such as databases </details> --------- Co-authored-by: GPT-6 Astra (Codex via T3 Code) <[email protected]> Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]> |
||
|
|
b16d667ded |
feat(platform): add native mobile server foundations (#15384)
### Why / What / How **Breaking change when the legacy-token cutoff is enabled:** old JWTs without `sid` must be renewed. Deploy the signer first, then set one shared cutoff at least one access-token lifetime (one hour) after the signer rollout completes. Unset preserves the old signature/expiry checks for legacy tokens; `0` enforces immediately. Never renew the cutoff on worker restart. Automatic native outbox replay remains disabled until durable cancellation and Redis-loss recovery are delivered. Installed mobile and desktop clients need to authenticate without browser cookies and retry queued sends without creating duplicate conversation rows. This foundation adds signed Better Auth session bearers, session-authority checks in REST/WebSocket authentication, and stable queued-message identity through Redis and PostgreSQL. The retry work ships alongside authentication because reconnecting clients repeat accepted sends. Preserving those identities requires recovery to keep accepted batches through cancellation, failed writes and buffer refill. The engine changes preserve existing approval results and transient-error handling around those recovery boundaries. This PR can land independently of the client applications; it does not certify their interfaces, OAuth callbacks, push, purchases or offline delivery. [Client contract and rollout](https://github.com/Significant-Gravitas/AutoGPT/blob/codex/secrt-2881-native-mobile-backend/docs/engineering/native-mobile/backend-handoff.md) · [Server validation](https://github.com/Significant-Gravitas/AutoGPT/blob/codex/secrt-2881-native-mobile-backend/docs/engineering/native-mobile/server-validation.md) · [SECRT-2881](https://linear.app/autogpt/issue/SECRT-2881) Part of SECRT-2881. ### Changes 🏗️ - Accept only signed session bearers and issue access JWTs containing `sid`. Native credential export requires `X-AutoGPT-Native: 1` without Cookie, Origin or browser Fetch Metadata headers. Browser sign-in and refresh retain HttpOnly cookies without exporting a native credential. - Validate session ownership, expiry, revocation and account bans on authenticated REST requests and new WebSocket connections, including the nested v1 and v2 API apps. Share validation only within one request. Keep service-token verification separate and use one shared ban predicate while retaining the authoritative database read. - Preserve deployment compatibility for older signed JWTs when `AUTH_LEGACY_TOKEN_DEADLINE` is unset. JWTs containing `sid` always validate the session. An explicit fixed cutoff, or `0`, requires renewal of tokens without `sid`: HTTP 401 with `X-Error-Code: session_token_renewal_required`, or retryable WebSocket 1013. The web client renews without losing pending subscriptions. - Require an authenticated heartbeat before treating a WebSocket as connected. Share bounded reconnect backoff across closing sockets and temporary token/JWKS failures; invalid, revoked and banned sessions remain terminal. Give each send a 30-second connection deadline while background reconnection continues; expired sends are never delivered later. Provider unmount disconnects its client, cancelling retry and heartbeat timers and ignoring late token responses. - Preserve scoped client message IDs through queue admission and database persistence. Full queues return an explicit 429 with documented `X-Error-Code` and `Retry-After` headers. Pending-message history lookup failures return 503 with `Retry-After: 30` before admission, including expert kickoff. The composer retains the draft with concise recovery messages for full queues and temporary service outages. - Couple receipts and accepted claims to queue expiry. Restore already accepted batches ahead of newer arrivals without admission eviction, retain ownership through preflight and mid-turn writes, and drain a finite initial backlog. Preserve completed approvals across cancellation. After cancellation just after commit, both engines avoid repeating already-loaded history in the next model prompt while retaining recovered images. - Exercise production Better Auth and real PostgreSQL/Redis journeys in a dedicated Ubuntu lane. Its coverage uses the separate `native-server` flag; the backend coverage flag retains exactly its twelve shard uploads. Harness startup diagnostics retain the cause while redacting connection URLs and configured secrets. - Account closure (#15459) composes with session authority: a new WebSocket rejects a closed account first, with 4003 `This account is closed.` (1011 when the lookup fails), through the same accept-then-close path so the client receives the code. Signed-in REST routes validate the session, then return 401 for a closed account. The client contract lists both. - A baseline turn refused under the running-sandbox limit puts the queued messages it drained back with the same restore failed turns use: ahead of newer arrivals and deduplicated by id. - Keep the client contract and executable validation guide; remove stale planning inventories and the audit of another client PR. Replace line-number secret-fixture fingerprints with two narrow path-and-rule exceptions; annotate synthetic test credentials and an upstream curve-class false positive at their exact lines. Register the native harness as a Knip entry point. Include formatter fixes, a type correction in an upstream Gmail test, and portable workspace paths in existing SDK cleanup tests found during whole-project validation. Data ownership: auth session lookup checks both session ID and JWT subject. Pending-message reconciliation is scoped to an already authorized conversation. Only internal recovery can restore accepted messages beyond the admission cap. There is no application-data migration or new production secret. Operational behavior: session validation adds an uncached indexed PostgreSQL read per authenticated request/new socket; an auth database outage returns 503/1013. Optional-auth store/marketplace requests with an unverifiable bearer also return 503; requests without a token remain anonymous. Failed WebSocket authentication completes the protocol upgrade before closing, so proxy upgrade counts increase while authenticated-connection metrics remain unchanged. The account-deletion routes (#15459) validate the session too. Better Auth's ban revokes every session, so a banned account can no longer use an unexpired pre-ban JWT to reach them, as it could on dev for the rest of that JWT's hour. ### Agents and large language models used - Codex via T3 Code: GPT-6 Astra (implementation, review and validation); gpt-daybreak-blue-latest (earlier focused retry/auth reviews). - T3 Code claudeAgent: Claude Fable 5.1 and Claude Opus 5.5 (earlier planning and inventories). - Claude Code: Claude Opus 5.5 (dev merges, conflict resolution, review follow-ups and validation). ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Local runs with Python 3.12 and Node 24 against PostgreSQL, a Redis cluster, RabbitMQ and ClamAV: this PR's auth, WebSocket and pending-message suites, the account-deletion, ban and closed-account suites, the chat routes, the architecture and block guards and the native lane pass; the frontend suite, format, lint and types pass. Each behavioural change was undone in a scratch copy and its test failed. Per-run figures are in the PR conversation. - [x] Browser/native authentication: real signed email sessions, renewal, logout, expiry, bans, ownership mismatch, unavailable lookups, unsigned-bearer refusal, and JWKS recovery; browser sign-in and daily refresh do not export native credentials. - [x] Rolling deployment: legacy JWTs before/at/after a fixed cutoff, service-token separation, and WebSocket renewal preserving pending subscriptions. - [x] Queued messages: concurrent HTTP retries, full-buffer refusal, receipt/list/claim expiry, rollback after refill, finite drains, repeated cancellation and persistence exactly once. Real PostgreSQL/Redis cancellation immediately after commit followed through the next model input, for text and images. - [x] The separate WebSocket process authenticates through database-manager RPC without its own usable database URL. - [x] Whole backend/library formatting and type checks; whole frontend format, lint and types; API schema and generated client regeneration. Local validation uses Python 3.12 and Node 24.18 with isolated test services. Hosted checks and review are evaluated on the published head. The repository's Knip report remains informational and reports existing unused-code findings. Native-device acceptance and live identity/payment/push providers are outside this server foundation. #### For frontend changes: - [x] `pnpm lint` passes, including design-system rules; no baseline counts were raised. - [x] No new legacy components, raw colours, dark/light classes, or styled primitive overrides. - [x] No pixel change: the product diff changes auth, WebSocket behavior and error copy; component markup and styling are unchanged. - [x] User-facing copy follows workflow/Otto terminology; identifiers retain their API meaning. #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] Configuration changes are listed above: optional fixed `AUTH_LEGACY_TOKEN_DEADLINE`, independent native coverage flag, pinned CI actions and test harness configuration. --------- Co-authored-by: GPT-6 Astra (Codex via T3 Code) <[email protected]> Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]> |
||
|
Nicholas TindleandClaude Opus 5.5
|
1cbcb80e66 |
fix(platform): self-host auth DB connection budget, visible token failures, build memory (#15440)
### Why / What / How **Why.** These are the backend and appliance findings from the self-host smoke on dev `92b44302` (patrol W51). None is a regression. - **N2 (S2, intermittent): concurrent sign-ups ran the frontend out of Postgres connections.** During a burst of sign-ups the appliance logged `FATAL: too many connections for role "autogpt_frontend"`, then `Failed to get auth token`. The affected requests then went to the backend **without a token**, and the backend's 401 reads as a logout. - Root cause: a production Next build carries **two copies of `lib/auth/auth.ts`**: route handlers in one chunk, and the pages' server actions in another. In the appliance image they are `chunks/1657.js` and `chunks/2443.js`, each running its own `new Pool(...)`. - The `globalThis` cache that should make the pool a singleton only ran outside production. That gives **two pools of 10 against a role limit of 10** (`bootstrap.sh`), so a sign-up (server action) and a page's API calls (route handlers) push the role past its limit. Current dev still has that production-only gap. - On top of that, `getServerAuthToken` turned every failure into `null`, which means "not signed in", so nothing on the way failed visibly. - **N4: building the appliance from source peaked at ~8.2 GiB**, and one attempt was OOM-killed in an 8 GiB builder during "checking validity of types". - The Dockerfile's `NODE_OPTIONS=--max-old-space-size=8192` never applied: `pnpm build` runs `cross-env NODE_OPTIONS=--max-old-space-size=16384 next build`, which replaces it. - **F10 (a warning on every flag read) is no longer in this PR.** It fixed LaunchDarkly with no SDK key. Dev has since removed LaunchDarkly (#15853, merged 11-02), and its PostHog path is already quiet without a key: one warning when the flag client is first built, then no SDK client and no user lookup on any read. A proof run on dev `5d13d3bc` shows it (see the test plan) and goes red when either guard is taken out. Dev's own `TestPostHogWithoutAKey` and `test_an_unconfigured_deployment_does_not_retry_forever` already pin both halves, so nothing is added. Two edges stay as they are on dev: an experiment-arm read (`evaluate_experiment_variant`) still looks the user up, cached for a day, before finding no client; and the `@feature_flag` decorator warns on every call without a key, but nothing outside tests uses it. **What / How.** - **One auth pool per process, with headroom.** - `lib/auth/auth-db-pool.ts` caches the pool on `globalThis` in every environment, sets an explicit `max: 10`, and adds a 10 s acquire timeout, so a starved request fails instead of hanging. It keeps dev's changes to the pool: no fallback connection string (#15970), and Better Auth 1.7's Kysely `PostgresDialect` with `transaction: false` (#16419) runs on this pool. - `bootstrap.sh` sets the role's limit to **20**, twice the pool, because Postgres counts a connection until its backend exits. The CI smoke check and a single-container budget test (`role limit >= 2 x pool max`) follow it. - **Docker-compose self-hosting** (checked): the frontend connects as the `postgres` superuser, so no role limit applies there. It did carry the same two-pools bug (up to 20 connections), and that is now 10. - Unchanged and pre-existing: compose's backend Prisma pools (`DB_CONNECTION_LIMIT=12` x 7 services, 8 with the `bot` profile), the scheduler's 3, and the frontend's 10 can in theory reach 97-109 against Postgres's default `max_connections` of 100. That needs its own measured change; it's flagged for a follow-up. - **A token that can't be minted is an error, not "signed out".** Both `getServerAuthToken` variants return `null` only when there is no session: no cookie, or Better Auth's 401. Anything else throws `AuthTokenUnavailableError`, an `ApiError` with status 503. The in-process variant decides "Better Auth said no" with #15386's `isSessionRejection`, the same check the session read uses. - The proxy, chat stream, upload and transcribe routes answer it with **503 + `Retry-After`** and never forward the request. The custom mutator stops sending server-side calls without the token. - React Query retries the 503. A 401 would have signed the user out. - Limitation: Better Auth (still in 1.7.16, where `getSessionFromCtx` turns any error into "no session") reports a failed *session lookup* inside the token mint as a 401, so only failures after it can be told apart. That covers the signing-key read, where the 5-minute session cookie cache sends most mints, and it is where the appliance failed. - Not covered, unchanged from dev: the `getWebSocketToken` server action catches the error and returns `{ token: null, error }`, and its callers (the copilot's auth headers, direct upload, the legacy WebSocket) treat any `error` or missing token as signed out, as they already do for a `null` mint on dev. Those callers can now tell the two apart from `error`; that is a follow-up. - **Build memory (N4): not lowered, documented.** Measured on tower's 8 GiB builder: a build with an *effective* 6 GiB cap passed but still peaked at **7.13 GiB** anon in the builder, against **7.68 GiB** for W51's uncapped build (concurrency differed, so not a controlled comparison). Its main `next build` process sat at the cap (6.2 GiB RSS). A lower cap therefore doesn't get the build under 8 GiB and only removes headroom. - The Dockerfile drops the dead `NODE_OPTIONS=8192`, which never applied: `pnpm build` runs `cross-env NODE_OPTIONS=--max-old-space-size=16384` (still so on dev), and V8 reports a 16576 MiB limit. No behaviour change. - `docs/platform/single-container.md` and `docs/platform/getting-started.md` (docker compose builds the same frontend) now say building from source needs **more than 8 GiB** available to Docker. "At least 8" would read as "8 is enough", and one of the three builds measured in an 8 GiB builder was OOM-killed. - A controlled frontend-only A/B on the same builder (dev's line against an effective 6 GiB cap) couldn't run, because the shared build host's builder went down. The decision doesn't depend on it: a cap that saved a few hundred MiB would still leave the build over what an 8 GB host can give Docker. ### Changes 🏗️ - `frontend/src/lib/auth/auth-db-pool.ts` (new), `auth.ts`: the singleton pool, `max: 10`, `connectionTimeoutMillis: 10_000`. - `single-container/bootstrap.sh`: `CONNECTION LIMIT 20`; the `.github/scripts/platform-single-container-smoke.sh` check and `single-container/tests/test_frontend_environment.py` (the budget invariant) follow it. - `frontend/src/lib/autogpt-server-api/helpers.ts`: `AuthTokenUnavailableError`; the SSR token variant throws on a non-401 failure. - `frontend/src/lib/auth/server/getServerAuthToken.ts`, `authTokenUnavailableResponse.ts`, and the `proxy`, `chat/.../stream`, `workspace/files/upload` and `transcribe` routes: answer 503. - `frontend/src/app/api/mutators/custom-mutator.ts`: no unauthenticated fallback for a signed-in request. - `single-container/Dockerfile`: the dead `NODE_OPTIONS` goes. `docs/platform/single-container.md`, `docs/platform/getting-started.md`: the memory a build from source needs. - Tests: - `lib/auth/__tests__/auth-db-pool.test.ts`, the seeded randomized test the patrol asks for on concurrency fixes. Over 300 seeds it imports the pool module as 1–3 "bundle copies" and runs 10–70 concurrent queries with random arrival, hold times and statement failures. These go through the real `pg-pool` against a fake Postgres that enforces `bootstrap.sh`'s limit and keeps a closed connection's slot for a tick. Invariant: no refusal, and the peak never exceeds the role limit. **Red with the old pool** (seed 2: 5x `too many connections for role "autogpt_frontend"`); green now; stable over repeats. - `playwright/journeys/signup-burst-selfhost-journey.spec.ts` (selfhost project, not in CI): users sign up through the UI in two overlapping waves while those already in hammer the API. Every sign-up must land in onboarding and every signed-in call must answer 200. - Against an appliance built from this branch with #15439 (`patrol-w53` on tower): **5/5 green** (6 users ×3, 10 users ×2). Postgres logged 0 `too many connections` and the frontend logged 0 failed token mints. - Against dev's own image (`autogpt-patrol-selfhost:bc671d371823`, this branch's base then) on fresh data: **fails at 6 users**. Postgres logged **36** `too many connections for role "autogpt_frontend"` and the frontend **18** `Failed to get auth token`, and 15 signed-in API calls came back **401**: the unauthenticated-backend symptom from W51, reproduced on demand. - The auth-token tests (both variants, proxy route, custom mutator) pin the new contract: 401 gives null; a DB error, a 500 or a network failure gives 503; outside a request, null; the proxy returns 503 without calling the backend. - Review follow-ups: - The in-process mint reads `cookies()` inside a try, like the SSR variant. - The randomized pool test lets closed connections free their slots before the next seed. Before, `open` went negative and `peak` was under-counted. It asserts `peak > 0` and that every connection closes, and has a 30 s budget. - Merge of dev `5d13d3bc` (11-09), 850 commits: - `auth.ts`: dev's Better Auth 1.7 imports and Kysely dialect stay; the pool comes from `auth-db-pool.ts`, which takes #15970's no-fallback connection string. - Proxy route and its test: both sides' imports. Custom-mutator test: dev's banned-account tests and this branch's server-side test both stay; the server-side test runs last and resets the two mocks it sets. - Backend: all of this branch's backend changes were F10 and its LaunchDarkly test helpers, so the backend now matches dev exactly (see F10 above). ### Risk - **Hosted.** One pool per serverless process instead of up to two, so fewer connections. A pool that is out of connections now fails after 10 s instead of waiting forever. A failed token mint gives a retryable 503 instead of a 401 that looks like a logout. - **Self-hosted.** On the next boot, `bootstrap.sh` re-applies the role and raises its limit to 20. The budget is max_connections 100: backend Prisma pools at 5 per service, plus the scheduler pool, plus 20. - No backend code changes any more, and no flag behaviour changes. ### CI note The old head's red "Build, smoke, and scan (linux/amd64)" (and "Check PR Status" with it) was not this change: the build failed importing a layer from the GHCR build cache (`failed to authorize: Unavailable: error reading from server: EOF`) at `COPY autogpt_libs`, before any of this PR's files mattered, while arm64 of the same commit passed. That is the buildkit cache-session race described in #15225, which is still open, so the job can still flake that way; a re-run clears it. ### Agents and large language models used Claude Code — Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [ ] I have tested my changes according to the test plan: - After the dev merge (head `4636deae98`): - [x] Vitest over `src/lib/auth`, `src/lib/autogpt-server-api` and `src/app/api` (58 files, plus `src/lib/auth-redirect.test.ts` through the path filter): 850 passed, the randomized pool test included - [x] `pnpm format` (no changes), `pnpm lint` and `pnpm types` over the whole frontend - [x] `single-container/tests` through CI's runner: 259 passed, 1 allowed skip; includes the role-limit budget invariant - [x] Backend lint as CI runs it: `poetry run lint --skip-pyright`, and `pyright` with 0 errors. The full `poetry run lint` also points pyright at `backend/test/` and fails there on 4 errors dev has as well; this branch has no backend diff - [x] `backend/util/feature_flag/` and `backend/data/subscription_trial_config_test.py`, the files the merge resolved (now dev's): 263 passed - [x] Dev's guards on a private Postgres/Redis/RabbitMQ: the whole of `backend/api` (both paywall route guards included), `backend/util/architecture*_test.py` and `copilot/tools/tool_schema_test.py`: 6528 passed, 12 xfailed, 18 failed. The same 18 fail on a pure dev `5d13d3bc` snapshot. 17 need ClamAV and pass on both dev and this branch once it runs; the last needs CI's 3-shard Redis cluster - [x] F10 proof on pure dev `5d13d3bc`: with no PostHog key, startup plus 35 calls across seven read APIs log one warning, never build an SDK client and never look the user up. The same test fails when the client stops remembering that it tried (a warning per read) or when reads skip the no-client short-circuit (20 lookups) - Before the dev merge (head `033e3eafc7`, 10-12), not re-run since: - [x] Appliance built from this branch on tower: role limit 20, one pool per process in both auth chunks - [x] Signup-burst journey: fails against dev's appliance (36 FATALs, 15× 401), passes 5/5 against this branch's (0 FATALs) - [x] The randomized pool test is red with dev's pool and green with this one #### For frontend changes (design system, see `docs/engineering/design-system.md`): - [x] `pnpm lint` passes, including the `design-system/*` rules; I did not raise any count in `scripts/eslint-rules/design-system-baseline.json` except for a moved, renamed or split file (explained under **Changes**) - [x] No `dark:`/`light:` classes, raw palette classes (`bg-zinc-200`) or hex/`rgb()` colours (`bg-[#fff]`); colours use role tokens (`bg-surface`, `text-foreground-secondary`) - [x] No imports from `src/components/__legacy__`, `legacy/` paths or `@/components/ui`, and no `LegacyButton` / `legacy-*` Button variants - [x] Copy uses the `Text` atom with a `variant`/`tone`; primitives get a `variant`/`size`. `className` holds layout, plus on `Text` only truncation, line clamping, wrapping, breaking, `text-balance`/`text-pretty` and alignment - [x] Before/after screenshots at 1440 and 390 px are attached for changes to `src/components/**`, `globals.css` or `tailwind*.ts`, or I wrote "no pixel change" and how I checked: no pixel change; the diff touches no component, stylesheet or Tailwind config, only route handlers, `lib/` modules and tests - [x] User-facing copy says workflow and Otto, never agent or graph (root `AGENTS.md`, "User-facing copy") #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**): the appliance's frontend role connection limit goes from 10 to 20; no new settings. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]> |
||
|
|
9789d6d655 |
fix(backend): let a run leave an input blank when Advanced is on (#16706)
### Why / What / How **Why:** Optional workflow inputs don't work (#10614). An input only counted as optional when it had a default value. The Advanced toggle only took effect when a default was set, and the builder won't save a blank default, so there was no way to make an input that is both empty and optional. The library run form kept Run disabled until it was filled. Experts have likely built workflows with Advanced on expecting the input to be optional, and those are broken today. There was a second problem behind the first: a blank input sent nothing on, so every block wired to it waited forever and the run finished with no outputs. **What:** Turning on Advanced now makes an input optional. A run can leave it blank, and the blocks it feeds still run, using their own pin defaults. Workflows already built with Advanced on start working without being edited. **How:** - `BaseGraph._generate_schema` asks each input for `is_optional()` instead of checking `value is None`. An input is optional when it has a default or Advanced is on, so it leaves `required`. The library run form, the external API (`assert_inputs_match`) and Otto's `run_agent` (`missing_inputs`) all read that list, so no frontend change is needed. - `AgentOutputBlock.Input.is_optional()` keeps the old rule, so the output schemas are unchanged. The dry-run simulator reads their `required` list. - A blank optional input sends `None`. The link then counts as filled, so the next block isn't left waiting. `Block` builds its input model with `None` values dropped (`_base.py`, around line 1081), so an optional pin falls back to its own default. A required pin given `None` is skipped, as before. ### Changes 🏗️ - `backend/blocks/io.py`: `is_optional()` on the input and output block schemas. The base input block and the file input send `None` when blank and optional. Removed the Google Drive input's `run`, which copied the base one. New help text for the toggle: "Whether the input is optional, so a run can leave it blank." - `backend/data/graph.py`: `advanced` and `required` in the graph input schema now come from `is_optional()`. - Tests: - New `backend/data/graph_optional_inputs_test.py`: which inputs end up required, outputs unchanged, and what a blank input sends. - New run test `test_a_run_can_leave_an_optional_input_blank` in `backend/executor/manager_test.py`: one required input and one blank Advanced input feed a Store Value block; the run gets only the required value, and the block must complete with it. - `test_get_input_schema` and the `grph_in_schm` snapshot asserted the old rule (an Advanced input with no default was required), so both are updated. - `docs/integrations/block-integrations/basic.md`: the new toggle text, edited by hand to match what `generate_block_docs.py` would write. The script couldn't import locally because of an out-of-date Prisma client. Follow-up, not in this PR: the toggle still reads "Advanced", and Otto's setup card hides Advanced inputs behind "Show advanced fields". Renaming it to "Optional" means a schema change. Additional review fixes: - Blank optional tables send `None`; an explicit empty table stays `[]`. - A literal `None` output also populates links that select a dictionary key or list item. This applies to every block that produces `None`, matching direct-link behavior; missing children on non-null outputs still leave their links waiting. - Tests cover blank Drive inputs, table defaults, and six full-run cases with direct and nested links. A second downstream block checks its non-null schema default through its output. - Existing inputs older than a day still count when filling a delayed execution; a new database regression test covers this. - Public docs explain the change for existing workflows and how to keep an input required. - `data/graph.py` only changes schema generation. The `data/execution.py` query retains its node and run filters; callers keep the same access checks, so no new user ID check is needed. - Local validation: 25 routing and optional-input tests pass; the delayed-input database regression passes. Changed files pass pyright. The final architecture/routing/input run passes all 144 cases. The broad architecture/block run had 4,172 passes; all 12 scanner-related failures pass on retry once the local scanner is healthy. All eight docs-layout checks pass. Full-run tests need CI because local service ports are in use. ### Agents and large language models used - Claude Code with Claude Opus 5.5 (`claude-opus-5-5`) - Codex with GPT-6 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [ ] I have tested my changes according to the test plan: - [x] `backend/data/graph_optional_inputs_test.py` passes locally (15 passed) - [x] black, isort, ruff and pyright are clean on the changed files - [ ] `test_get_input_schema` and `test_a_run_can_leave_an_optional_input_blank` need the database and test server; CI runs them - [ ] `backend/blocks/test/test_block.py` (the Google Drive input now uses the base `run`); CI runs it - [ ] Manual: in the builder, add an input block with Advanced on and no default, wire it into an optional pin, and check that the library run form allows Run with it blank and the workflow completes <details> <summary>Example test plan</summary> - [ ] Create from scratch and execute an agent with at least 3 blocks - [ ] Import an agent from file upload, and confirm it executes correctly - [ ] Upload agent to marketplace - [ ] Import an agent from marketplace and confirm it executes correctly - [ ] Edit an agent from monitor, and confirm it executes correctly </details> #### For frontend changes (design system, see `docs/engineering/design-system.md`): - [ ] `pnpm lint` passes, including the `design-system/*` rules; I did not raise any count in `scripts/eslint-rules/design-system-baseline.json` except for a moved, renamed or split file (explained under **Changes**) - [ ] No `dark:`/`light:` classes, raw palette classes (`bg-zinc-200`) or hex/`rgb()` colours (`bg-[#fff]`); colours use role tokens (`bg-surface`, `text-foreground-secondary`) - [ ] No imports from `src/components/__legacy__`, `legacy/` paths or `@/components/ui`, and no `LegacyButton` / `legacy-*` Button variants - [ ] Copy uses the `Text` atom with a `variant`/`tone`; primitives get a `variant`/`size`. `className` holds layout, plus on `Text` only truncation, line clamping, wrapping, breaking, `text-balance`/`text-pretty` and alignment - [ ] Before/after screenshots at 1440 and 390 px are attached for changes to `src/components/**`, `globals.css` or `tailwind*.ts`, or I wrote "no pixel change" and how I checked - [ ] User-facing copy says workflow and Otto, never agent or graph (root `AGENTS.md`, "User-facing copy") #### For configuration changes: - [ ] `.env.default` is updated or already compatible with my changes - [ ] `docker-compose.yml` is updated or already compatible with my changes - [ ] I have included a list of my configuration changes in the PR description (under **Changes**) <details> <summary>Examples of configuration changes</summary> - Changing ports - Adding new services that need to communicate with each other - Secrets or environment variable changes - New or infrastructure changes such as databases </details> Fixes #10614 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: GPT-6 (Codex) <[email protected]> Co-authored-by: Cursor Agent <[email protected]> |
||
|
Krzysztof CzerwinskiandClaude Opus 5.5
|
06340e194f |
fix(backend): account erasure deletes the account's chats (SECRT-2854) (#16745)
### Why / What / How **Why:** account erasure (#15487) left an account's chats to the `User` row's `ON DELETE CASCADE`. Testing it on dev showed that dev's database lacks that foreign key (`ChatSession_userId_fkey`; the schema and the migration that created it both still have it, so the database has drifted). So after an erasure the account's chats and their messages stayed, and the search backfill re-indexed their titles under the erased account's ID. Chats are the most personal data an account has, and the privacy policy promises they go. **What:** the erasure's database step now deletes the account's chats itself instead of relying on the database to delete them with the account. **How:** one more entry in the explicit deletions (`chat_sessions`, `ChatSession` where `userId` is the account), in the same transaction as the private search embeddings, so the backfill never sees a session whose embedding is gone. Everything else that hangs off a session (messages, shared files, links, background tasks, callback tokens) cascades from `ChatSession`, and those foreign keys are present on dev. ### Changes 🏗️ - `account_erasure/database.py`: delete `ChatSession` rows of the account explicitly, before the orgs, beside the private embeddings. - `database_test.py`: `chat_sessions` is among the rows deleted explicitly. - `account_erasure_db_test.py`: the end-to-end erasure now seeds a titled chat with a message and checks that the chat, its message and every embedding of the account are gone. - Runbook (`docs/engineering/account-deletion.md`): chats are deleted by the erasure itself, and why. Follow-up outside this PR: dev's database is missing that foreign key. Whether prod is too is being checked separately; this PR makes the erasure correct either way. ### Agents and large language models used - Claude Code with Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [ ] I have tested my changes according to the test plan: - [x] Black, isort and ruff on the changed files - [ ] `database_test.py` and the end-to-end `account_erasure_db_test.py` (database-backed: CI) - [ ] On dev after merge: erase a test account that has a titled chat, then check no `ChatSession` or `UnifiedContentEmbedding` row of its ID is left, including after the next search backfill 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Reinier van der LeerandClaude Opus 5.5
|
b6e79fc245 |
docs(platform): document every rate limit, and how the Platform finds a caller's IP address (#15872)
Adds a docs page that lists every rate limit the Platform enforces, and a second page that explains how the Platform determines a caller's IP address, which several of those limits count by. It also tells contributors to update the pages with any limit they change: a line in the backend and frontend `AGENTS.md`, and a comment above each limit's constants pointing at its page. Resolves #15869. Part of [REQ-65](https://linear.app/autogpt/issue/REQ-65). ### Why / What / How **Why:** someone whose chat is paused, whose request gets a `429`, or whose store search starts returning plainer results can't find out today which limit they hit, what it counts, or when it resets. The limits are spread over a dozen modules, and their numbers and settings appear nowhere outside the code. An API integrator planning load, or a self-hoster deciding what to set, has the same gap. Several limits count per IP address, and the backend and frontend read that address in different ways; three settings decide whether it is the visitor's real one, and no page says so. And a page like this goes stale the first time someone changes a limit without knowing the page exists. **What:** `docs/platform/rate-limits.md` gives each limit what it covers, what it counts per (account, IP address, or everyone together), its default number and window, what a caller sees on hitting it (status, message, `Retry-After`, or a queued or simpler result), and the setting that changes it. Sections: - Chat with Otto and your Experts: - the daily and weekly allowance - the trial allowance - resets with credits - tasks running at once - follow-up messages - workflow runs per chat - Experts: - the weekly budget - spend approval - results posted to the Expert's chat - avatar generation - Workflows: - runs at once, and the share of an executor's workers one account's runs may hold, sub-workflows included - schedules per account - generated workflow images - emails the Send Email To Me block sends to the account - Sandboxes: how many run at once per account, and the optional ceiling per install. - Search: global search and store search. - Plans and billing: the billing page, and free trial places. - Onboarding personalization. - Sign-in with a code, for integrations. - Signing in and account emails: the sign-in and password-reset attempt limits, the per-IP and per-address email caps, and Better Auth's own limits by path. - Closing your account: answers to the exit question. - The External API: v2's shared per-account and per-IP limits, v1's two per-account tiers, and running a block on its own. `docs/platform/client-ip.md` covers: - the backend: the connecting address, or `X-Forwarded-For` only from proxies in `FORWARDED_ALLOW_IPS`; - the External API's own per-IP limits, which read the forwarded chain by `TRUSTED_PROXY_COUNT` instead; - the frontend: `X-Forwarded-For` only on Vercel, with `TRUST_X_FORWARDED_FOR=true`, or with `TRUSTED_PROXIES` naming the proxies in front of it; - the signed `X-Client-IP-Token` the frontend passes to the backend for store search; - the five things on the sign-in path that read the address, each with the two conditions it needs; - a table of what to set for each kind of install. The rate-limit page links to it wherever a limit counts per IP address. To keep the pages current, `backend/AGENTS.md` and `frontend/AGENTS.md` each gain a Code Style line: a PR that adds, removes or changes a rate limit updates `rate-limits.md`, and one that changes how a caller's IP address is found updates `client-ip.md`, in the same PR. Each limit's constants or settings, and each reader of the client IP, carry a one-line comment naming the page that documents them: 51 comments in 27 files. **How:** every number, message and setting was read from the code at `dev` `dc36bbef61` and re-checked at `c27bf0b975`, `531bf37603` and `2dbf60b8ec`; comments on this PR give a `file:line` for each one. Better Auth's built-in limits were read from the installed package, better-auth 1.7.16, which is what `dev`'s lockfile pins. Choices made along the way: - **Defaults, not Cloud's figures.** The page gives the code's defaults. For a Cloud user's own allowance it points to the pricing page and the chat's **Usage limits** popover, because Cloud sets plan allowances outside the code and can change them without a release. - **The sign-in path is a table of two conditions, not prose per limit.** Everything there reads the address through Better Auth, which needs `X-Forwarded-For` to carry exactly one entry unless `TRUSTED_PROXIES` names the proxies; three of the five also need the frontend to trust the header. The table is one row per reader, with what happens when there is no address. - **The External API section gives v2's shared limit and v1's two tiers.** v2's per-endpoint limits are already listed in the API's own conventions page, so this page links there rather than repeating the numbers. - **Size limits are left out**, because they have no time window. That covers upload sizes, storage per plan and query length. So are limits of other services, such as an AI provider's 429s, which the Platform passes on unchanged. - **Server capacity limits are left out too**: file previews, virus scans and the executors' worker pools. They never refuse a caller, only make one wait. The share of an executor's workers one account may hold is per account, so it is on the page with runs at once. - **A pointer comment per limit, not a test.** A test that fails when a limit is missing from the page needs one place every limit goes through, and there isn't one: seven limiters call `record_rate_limit_hit` and four use the shared `RateLimiter`, while the chat allowance, the turn caps, the worker share, the Expert budgets, the schedule cap, the device-code throttle and the frontend's sign-in limits share nothing. The page also names limits by what a user does, not by an identifier a test could match. ### Changes 🏗️ - New `docs/platform/rate-limits.md`, listed under "Using the Platform" in `docs/platform/SUMMARY.md`. - New `docs/platform/client-ip.md`, listed under "Self-Hosting". - `autogpt_platform/backend/AGENTS.md` and `autogpt_platform/frontend/AGENTS.md`: the line above. - A one-line comment naming its page above each documented limit's constants or settings and each client-IP reader, in 22 backend modules and 5 frontend files. No code changes. ### Verified I read every limit on the pages in source at `dev` `dc36bbef61`, re-checked each against `dev` `c27bf0b975`, `531bf37603` and `2dbf60b8ec` after merging them, and ran `.github/workflows/scripts/test_docs_layout.py`: 8 passed. With `client-ip.md` removed from `SUMMARY.md` it fails and names the page. On the comment-only backend changes I ran `backend/util/architecture*_test.py` and `backend/blocks/test/test_block.py` (4175 passed, 84 skipped), and black, isort and ruff on the touched modules (ruff reports only findings that were already there). Pages render only on GitBook, so nothing here was rendered locally. What the IP page claims, and where each came from: - **Better Auth needs exactly one `X-Forwarded-For` entry unless `TRUSTED_PROXIES` is set.** `getIPFromHeader` in the installed `@better-auth/[email protected]` returns `null` unless the header holds one address, when no trusted proxies are configured (`dist/utils/ip.mjs:190`); with some, it returns the first entry from the right they don't cover (`:180-188`). Its rate limiter keys every caller without an address on one bucket and logs a warning (`better-auth/dist/api/rate-limiter/index.mjs:244-248`). - **nginx's `$proxy_add_x_forwarded_for` gives a browser exactly one entry**, because it appends the connecting address to whatever arrived and a browser sends nothing; it reaches two only through a second proxy, or a request that already carried the header. With no `proxy_set_header X-Forwarded-For` line nginx passes the header through as it arrived. - **The five sign-in readers and their conditions:** Better Auth's own limiter and the per-IP email cap (`ip-email-cap.ts:26`) call `getIP` alone; the attempt limits (`attempt-limits.ts:69-71`), the sign-in history and IP blocks (`auth.ts:125-129`, `:253`) wrap it in `getTrustedClientIP`, which returns `null` unless the frontend runs on Vercel or has `TRUST_X_FORWARDED_FOR=true` or a non-empty `TRUSTED_PROXIES` (`client-ip.ts:15-21` at `c27bf0b975`). - **`FORWARDED_ALLOW_IPS` accepts a CIDR range.** uvicorn 0.53.0, the version `dev`'s lockfile pins, parses each entry as an address or a network: `"172.18.250.9" in _TrustedHosts("172.18.0.0/16,127.0.0.1")` is `True` and `"172.19.0.1"` is `False`. Its default is `127.0.0.1,::1` (`uvicorn/config.py:363`). - **`FORWARDED_ALLOW_IPS` under Docker Compose is not the proxy's own address.** Measured with a throwaway container published on a Docker bridge network: the backend saw the connection from the network's gateway, not `127.0.0.1` or the proxy's address. - **`TRUSTED_PROXY_COUNT` defaults to 3** (`backend/util/settings.py:539`), and the External API's per-IP limit takes the entry that many from the right, falling back to the connection address when the header has fewer (`external/v2/global_rate_limit.py:190-212`). `single-container/entrypoint.sh:156` sets it to 1. - **The External API's shared numbers** are 200 a minute per account and 5 per IP, with 60 per IP for its docs paths (`external/v2/global_rate_limit.py:47-65`), which match what the API publishes in its own description (`external/v2/app.py:35-43`). - **The sign-in and password-reset attempt limits** are `ATTEMPT_LIMITS` in `attempt-limits.ts:24-42`; the refund on a successful sign-in is `refundSuccessfulAttempt` (`:88`), and the message and `X-Retry-After` header are at `:126-132`. The "100 per 10 seconds" default for other `/api/auth/` paths is `better-auth/dist/context/create-context.mjs:173-174` in 1.7.16. ### Agents and large language models used - Claude Code with Claude Opus 5.5 - Claude Code with Claude Opus 5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Every number, message and setting traced to source (evidence comment) - [x] `test_docs_layout.py` passes with both pages listed - [x] Every cross-page anchor the two pages use resolves to a heading on the target page 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Krzysztof CzerwinskiandClaude
|
2dbf60b8ec |
feat(platform): erase closed accounts after #15459's 14 days, keeping a ban record (erase-account CLI and job) (#15487)
> **Stacked on #15459** (SECRT-2877, self-serve "Delete account"). The base of this PR is #15459's branch, so the diff shows only the erasure. Once #15459 merges, this PR gets retargeted to `dev`, and it must merge after #15459. ### How it fits #15459's closing flow #15459 closes the account without banning sign-in; the owner has **14 days** (`eraseAfter`) to sign in and keep it, and `startedAt` on the request marks the erasure's claim. The erasure follows that contract point by point: 1. **Selection:** pending requests are `completedAt IS NULL AND eraseAfter <= now()`, oldest first, not filtered on `startedAt` (a stopped erasure is resumed). The 15-minute `PENDING_GRACE` is gone. 2. **Claim before any step:** `request.claim_request` runs its own short statement, `UPDATE ... SET "startedAt" = COALESCE("startedAt", now() AT TIME ZONE 'UTC') WHERE "userId" = $1 AND "completedAt" IS NULL AND "eraseAfter" <= now() AT TIME ZONE 'UTC' RETURNING "id"` (the columns hold UTC without a zone). No row → nothing runs (`report.not_claimed`: kept by its owner, not due, or erased already). No lock is held across the erasure. The claim reads the database clock while Keep compares with the app server's clock; a skew only moves the edge of the 14 days, since each statement re-checks its condition on the locked row and Keep deletes only an unclaimed row. 3. **`startedAt` is never cleared.** Recovery is a rerun of the idempotent routine. 4. **CLI:** `erase-account <account>` refuses inside the 14 days (new `not_due` blocker, with the date). `--close` now **only closes** (the owner then has 14 days, as from Settings) and prints the date. 5. **Late confirmation:** when the erasure sends the email closing could not, the 14 days are over, so it says "everything in your account is being erased now" instead of "Sign in by <date>". The `EMAIL` reason line stays. 6. **`complete_request`** marks the claimed row only (its ID, `startedAt` set). 7. **No banned refusal:** `AccountBannedError`, `--allow-banned`, `banned_for_another_reason` and the `banned` blocker are gone; #15459's `close_account` (no ban fields) is used as it is. A banned account may delete itself. 8. **Ban record:** when the erased account is still banned, the erasure keeps an `AccountBanRecord` (new table): a keyed hash of the address, the date and a category, until the ban would have ended and **3 years** at most; each sweep purges expired ones. Sign-up checks it (below). 9. **Frontend:** no change to #15459's sign-in or sign-out code; the only frontend change is the sign-up check. 10. **Memory crons:** closing now leaves them registered (so they resume if the account is kept), so the erasure's work step removes the nightly-batch, community-rebuild and skill-learning crons of the account and each of its experts. 11. **Status:** `GET /api/account/deletion` already answers `too_late` once `startedAt` is set; no new status. 12. **Runbook** rewritten for all of the above. And for running unattended: - **Bounded retries:** PostHog and MailerLite calls retry a 5xx at most three times, then fail the step. - **Rows that would outlive the account:** a preset's saved inputs and a notification batch's events (`ON DELETE SET NULL`) are deleted explicitly. - **Errors outside the steps** (reading the account, the lease, the claim) skip only that account, logged by type; the ban-record purge runs on its own (and in `--pending --apply`). - **Team agents** are handed over only while another active team member can open them; one only the account could open is deleted. - **One erasure per account at a time:** a Redis lease per account (`lease.py`, 1 hour TTL, renewed before each step; the erasure stops if it was lost) around each apply keeps two database manager replicas, or the CLI beside the job, off the same account. With Redis down nothing is erased. - The memory step builds its FalkorDB client off the event loop with the transport deadlines; the work step fails while schedules outlive `stop_user_work`; a failed step logs only its exception type for the confirmation step. ### Why / What / How **Why:** [SECRT-2854](https://linear.app/autogpt/issue/SECRT-2854). #15459 lets people delete their account from Settings or the paywall. It **closes** the account (cancels the plan, disconnects apps, stops work and email, ends sessions, takes listings down, writes an `AccountDeletionRequest` with `eraseAfter` 14 days on) and the owner can keep it until then. This PR is the erasure that follows: it deletes the account's data, stored files included, the PostHog person and the MailerLite subscriber, hands over workflows other people use, and deletes the rows Marketplace `ON DELETE RESTRICT` keys block. **What:** - The erasure processes #15459's request rows once their 14 days are over. For each one it claims the row, then: - reruns `stop_user_work` and removes the memory crons; - sends `send_account_deleted_email` if `confirmationSentAt` is null, then records it; - erases the data (files, memory, PostHog, MailerLite, marketplace, the shared-workflow handover); - keeps a ban record if the account is banned; - deletes the rows, the auth identity with the `User` row; - sets `completedAt` on the claimed row. - **A periodic job** does this inside the database manager, behind `ACCOUNT_ERASURE_SWEEP_ENABLED` (**off by default**). - **The `erase-account` CLI** erases one account or every pending one (dry run unless `--apply`), and `--close` closes an account for a request made by email. `--close --now` is for an owner who asked to be erased at once: it closes the account with no 14 days to keep it (or ends a closed account's 14 days now) and erases it in the same run. - **Sign-up refusal:** Better Auth's sign-up checks (email sign-up and a first social sign-in, in `lib/auth/auth.ts`) ask the backend whether the address has a live ban record and refuse a match with the existing generic "Sign-up isn't available" message. **How** *The hand-off.* The `AccountDeletionRequest` row is the whole contract (the docstring of `account_deletion/service.py`, whose hand-off paragraph now says selection does not filter on `startedAt`). The row outlives the `User` (no foreign key), so backend auth keeps refusing a token issued before the close. *Keep vs. erase.* Keep deletes the row only while `startedAt` is null and `eraseAfter` is ahead; the erasure claims only once `eraseAfter` has passed. Whichever statement commits first wins; the other matches no row. A dry run never claims. *The ban record.* `AccountBanRecord { emailHash @unique, category, recordedAt, deleteAfter }`. The hash is HMAC-SHA256 of the trimmed, lower-cased address, keyed with a new dedicated secret, `BAN_RECORD_HASH_KEY` (backend only; the frontend never holds it). It must never be rotated, or the records stop matching; rotating `ENCRYPTION_KEY` no longer touches them. `make init-env` and the single-container image generate it (runtime config v3). The category is `ADMIN` (admins are the only source of bans today); the admin's free-text reason is never copied. `deleteAfter` is the ban's own expiry, or 3 years, whichever is first. The frontend asks `POST /api/auth/email/ban-record` with a frontend service token (scope `auth-sign-up:check`), the same pattern as the suppression lookup; the lookup fails open (3 s timeout), like the access blocks. An admin creating an account from /admin/users is not checked. *The job.* `scheduler.erase_closed_accounts` is registered only when `account_erasure_sweep_enabled` is on and checks the switch again when it fires. It calls `start_account_erasure_sweep` on the database manager over RPC, which starts the sweep in the background (at most one per process). A blocked account is logged at error level, a failed step stops that account until the next run, a skipped step (setting not configured) is logged as a warning, and an unclaimed one (kept meanwhile) at info. **The database manager pod then needs** (also in the runbook): the bucket settings and GCS credentials, FalkorDB (`GRAPHITI_FALKORDB_*`), the PostHog personal key (`POSTHOG_PERSONAL_API_KEY`, `POSTHOG_PROJECT_ID`, `POSTHOG_APP_HOST`), MailerLite (`MAILERLITE_API_TOKEN`), Stripe, and `BAN_RECORD_HASH_KEY` (same value as the API pods). A step is skipped only when its service is not configured at all. A skipped step stops the erasure before the rows go, and the request stays open, unless the stack lists that service in `ACCOUNT_ERASURE_UNUSED_SERVICES`; any error fails the step the same way. Without `BAN_RECORD_HASH_KEY` nothing is skipped: the ban record step fails for an account with a ban in force, and that erasure stops before the rows go; an account with no ban in force needs no record and is erased as usual. **Edits to #15459's files:** - `AccountDeletionSource.EMAIL` (`schema.prisma`, migration `20261020120000_account_deletion_source_email`), its `_SURFACES` entry and its reason line in `notifications/auth.py`. The API's `source` is still `settings`/`paywall`. - `send_account_deleted_email` picks the "being erased" line once `erase_after` has passed (optional `now` for tests); its tests now pin the clock. - The hand-off paragraph of `account_deletion/service.py`'s docstring (selection not filtered on `startedAt`). - `lib/auth/auth.ts` calls the ban record check; `lib/auth/access-blocks.ts` exports its generic message. **What only we do** (unchanged from before): - **Files:** every bucket prefix of the account (private, public, legacy), or the local data directory on a self-hosted stack. Media a handed-over listing or another account's library entry still shows is kept. - **FalkorDB memory:** Otto's graph and one per expert. Skipped only when `GRAPHITI_FALKORDB_HOST` is empty. - **PostHog person** (`persons/bulk_delete`) and **MailerLite forget**. - **Credit ledger** kept six years (no foreign key), metadata cut to IDs and amounts. - **Sole-member orgs** go with the account, their own Stripe customers' subscriptions cancelled. - **Shared-workflow handover** to "Deleted user": workflows other accounts use (library, presets, runs, experts, org or team members) go to the placeholder with all versions and sub-workflows; their listings stay taken down. - **Published things** (Toran, 22 Oct): off the Marketplace, keep working for people who have them. Skill listings are deleted (installs are copies). OAuth apps others use block the erasure (`oauth_apps_in_use`) until moved or disabled. ### Changes 🏗️ - `backend/data/account_erasure/request.py`: `pending_user_ids` by `eraseAfter`, `claim_request`, `complete_request` by the claimed ID, `send_confirmation` passes `erase_after`, `close_for_email_request` returns the erase date. `PENDING_GRACE` removed. - `routine.py`: lease and claim before any step; `closing` removed; `ban_record` step before `database`. - New `lease.py` (per-account Redis lease). `storage.py`: sweeps the local data directory whenever the private bucket is unset. `memory.py`: off-loop client. `database.py`: preset inputs and notification events. `shared_agents.py`: the team clause. `third_parties.py` and `notifications/mailerlite.py`: `retry_max_attempts=3`. - `preflight.py`: `not_due` blocker; loads `eraseAfter` and the identity's ban. `models.py`: `request_id`, `erase_after`, `banned`, `ban_expires`, `ErasureReport.not_claimed`. - `work.py`: removes the memory crons (`remove_scope_memory_jobs`) for the account and each expert. - New `ban_record.py`: `keep_ban_record`, `is_banned_address`, `purge_expired_ban_records`, `email_hash`. - `sweep.py`: no grace; purges expired ban records each run; logs an unclaimed account at info. - `cli/erase_account.py`: `--close` only closes; erasing respects the 14 days; prints `not_claimed`. - `schema.prisma` + migration `20261102120000_account_ban_record`: `AccountBanRecord`, `AccountBanCategory`. - `api/features/auth_email/routes.py`: `POST /api/auth/email/ban-record` (`checkAuthEmailBanRecord`); `openapi.json` regenerated. - `notifications/auth.py`: the late wording (`ACCOUNT_BEING_ERASED_PREHEADER`). - Frontend: new `lib/auth/ban-record.ts`, called from `assertEmailSignUpAllowed` in `lib/auth/auth.ts`. The earlier changes to #15459's sign-in/sign-out code are reverted. - `docs/engineering/account-deletion.md`: the runbook. - Unchanged from before: the package rename to `account_erasure`, the `erase-account` CLI (no `delete-account` alias), the scheduler job and settings, `credit.py`/`credit_admin_lookup.py`. - `mailerlite.py`: the erasure's lookup and `forget_subscriber` use their own client with at most three attempts; the shared client is unchanged, so the backfills and the audience consumer still wait out rate limits. **User-visible copy changed:** - Account-closed email sent by the erasure after the 14 days: "The 14 days to keep it are over, so everything in your account is being erased now." in place of the "Sign in by <date>" line. - Sign-up with the address of an erased, banned account: the existing "Sign-up isn't available from this network or email address. If you think this is a mistake, please contact support." **Configuration changes:** - `ACCOUNT_ERASURE_SWEEP_ENABLED` (off by default) and `ACCOUNT_ERASURE_SWEEP_INTERVAL_SECS`, on the scheduler. - The database manager needs the bucket, FalkorDB, PostHog personal key, MailerLite, Stripe settings and `BAN_RECORD_HASH_KEY` before the switch is turned on, and the REST API pods need `BAN_RECORD_HASH_KEY` too. - New table `AccountBanRecord` (migration). No change to `.env.default` values or `docker-compose.yml`. **Rollout:** the switch stays off until #15459 ships; turn it on (with the database manager settings above) before the first closed accounts reach their 14 days, so each is erased on time (we promise 30 days). ### Still open - [ ] Refunds (business decision; the runbook says to ask the owner before `--close`). - [ ] The org ledger (`OrgCreditTransaction`) of a sole-member team org is still deleted with the org. - [ ] Ban categories: only `ADMIN` exists today. Finer categories (spam, fraud, ...) would need admins to pick one when banning. - [ ] For Kris: email changes to a banned address; the org-ledger metadata scrub; access to submission media. (Decided: a step skips only when unconfigured, and a skip stops the erasure unless the service is listed as unused; a dedicated ban-record key; fail-open sign-up check; trim + lower-case normalisation.) ### Not covered (listed in the runbook) - E2B volumes, Mem0, the Linear customer, the Ayrshare profile, LaunchDarkly, Sentry. - Redis keys and Langfuse traces (expire within 90 days). - Provider grants without a revoke endpoint. ### Risks - **Turning the job on without the database manager settings** stops every erasure at the first unconfigured service, with an error in the sweep's log, and the requests stay open until the setting is in. Nothing is marked done with data left behind. - **Rotating `BAN_RECORD_HASH_KEY`** makes existing ban records stop matching (the runbook and `.env.default` say never to). - **A backend outage at sign-up** lets a banned address back in (the check fails open by design). - **Handed-over workflows keep their content** as the creator saved it; their credentials go with the account. - **Very large accounts** could hit the 5-minute transaction timeout; only executions are batched. - **A blocked account** (sole owner of a shared org, or the developer of an OAuth app others use) is logged at every sweep until resolved. ### Agents and large language models used - Claude Code (Claude Opus 5.5) ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [ ] I have tested my changes according to the test plan: - [x] Backend unit tests (mock-only, all passing locally): pending selection by `eraseAfter`, the claim statement (conditional, `COALESCE`, database clock, never clears), an unclaimed request runs no step, the claim before any step and never on a dry run, completion of the claimed row only (`request_test.py`, `routine_test.py`); `not_due` and its date (`preflight_test.py`); memory crons removed for the account and each expert, an unreachable scheduler failing the step (`steps_test.py`); ban record: keyed hash, only a ban in force, 3 years or the ban's end, dry run, no key, lookup, purge (`ban_record_test.py`); a record purged between the conflict and the restart is created again, a second conflict fails the step (`ban_record_test.py`); a Redis client that can't connect is `ErasureLeaseUnavailable` (`lease_test.py`); a failed step logs no `exc_info` (`routine_test.py`); only the erasure's MailerLite client bounds its retries (`mailerlite_forget_test.py`); sweep purge and info log (`sweep_test.py`); CLI `--close` only closes, a kept account fails the command, `--pending --apply` purges even when every account is blocked (`erase_account_test.py`); late email wording (`notifications/auth_test.py`); the ban-record route and its service token (`auth_email/routes_test.py`). - [x] Frontend (Vitest, 484 auth tests passed): the real Better Auth handler refuses a sign-up from a recorded address with `SIGN_UP_BLOCKED` and creates no account, and lets it in when the lookup fails (`access-blocks-flow.test.ts`); `ban-record.test.ts` (scope, request, generic message, fail-open). `pnpm types` and `pnpm lint` clean. - [ ] `account_erasure_db_test.py` (DB-backed, left to CI; backend CI does not run while this PR's base is a feature branch, so it runs once #15459 merges and this PR retargets to `dev`): pending only after the 14 days, claimed and completed, rerun not claimed and unchanged; a preset's saved input and a notification event deleted; claim vs. Keep on the real table (no claim inside the 14 days, Keep wins before the claim, too late after even on a clock inside the 14 days, a second claim keeps the first date); a team agent handed over only with another team member; a banned account erased leaves a findable record, a non-banned one none, and the purge removes it. - [x] isort and black clean; pyright clean on the changed backend files (after `gen-prisma-stub`, as CI runs it). - [ ] On dev, with the switch on and the database manager settings in place: close a test account with files, a shared workflow, a schedule and a sole-org subscription; move its `eraseAfter` into the past; confirm the sweep erases it within the hour, `startedAt` and `completedAt` set, buckets empty except kept media, PostHog and MailerLite cleared, the memory crons gone. - [ ] On dev, a banned test account erased the same way: sign-up with its address refused; after deleting its `AccountBanRecord`, allowed. - [ ] On dev, `erase-account <email> --close --apply` closes and prints the date; `erase-account <email>` refuses with `not_due`. #### For frontend changes (design system, see `docs/engineering/design-system.md`): No UI changes: the only frontend change is in `lib/auth` (sign-up check) and its tests; no pixel change. #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]> |
||
|
Nicholas Tindle
|
0640de04ca |
feat(backend): add Claude Fable 5.2 support (#16699)
Adds `claude-fable-5-2` (Anthropic, released 2026-11-09) to the LLM catalog, the `LLMModel` enum and the block docs. - Price: $8/$40 per 1M (20% below Fable 5.1), cache reads $0.20/1M, 5m cache writes $10/1M, at the standard 1.5x credit margin (1200/6000 credits). Source: Anthropic's Fable 5.2 pricing table. - Specs: 1M native window (capped at 200K like the rest of the Claude 5 family), 128K max output, tools, JSON and reasoning. - Anthropic documents that Fable 5.2 rejects forced `tool_choice`, so it joins the auto-tool-choice list next to Fable 5.1. - Fable 5.2 matches the existing `claude-fable-5` family prefix, so the temperature and tokenizer handling already apply. - No defaults change: Fable 5.1 was never a default alias. Opus 5.5, Sonnet 5.5 and Haiku 5.5 remain the current defaults. - Tests: pinned catalog rates, price card cache prices, forced-tool-choice cases. |
||
|
Krzysztof CzerwinskiandClaude Opus 5.5
|
82bf3e9201 |
feat(backend): set the plan tier on the PostHog person (SECRT-3411) (#16702)
### Why / What / How **Why:** `chat_blocked` (#16698) carries a `tier` only for the cost limits; the per-minute and running-turn caps send it as null, and PostHog persons had no tier at all. So refusals couldn't all be split by plan, which SECRT-3411 asks for. **What:** the PostHog person now carries `subscription_tier`, the stored plan tier (`NO_TIER`, `TRIAL`, `PRO`, `MAX`, `BUSINESS`, `ENTERPRISE`, legacy `BASIC`). Any event, `chat_blocked` included, can then be broken down by plan through the person. The refusal path is unchanged: no tier lookup is added there. **How:** the tier rides on the existing lifecycle sync (`backend/data/posthog_lifecycle_sync.py`). It already reads the user row, runs in the background, never raises, and coordinates syncs per user so the newest read is the one sent. What changes is who schedules it, and one extra column in the daily sweep. The name follows the tracking plan, which calls the plan tier `subscription_tier` on every event and keeps the person's billing properties under `subscription_*`. ### Changes 🏗️ - `posthog_lifecycle.py`: `LifecycleUser` and `LifecycleSnapshot` get `subscription_tier`; `person_update()` `$set`s it. A tier the caller didn't read is neither sent nor unset (MailerLite's backfill builds users without it). The status still never reads the tier. - `posthog_lifecycle_sync.py`: the per-user sync passes the row's tier; the daily sweep's query selects it. New `set_tier_on_sign_in()` for sign-ins (below). - Tier writes now schedule the sync: - `credit.set_subscription_tier`: cancel without a Stripe subscription, a plan changed in place, the lazy reconcile. A new `track_lifecycle=False` keyword is used by the webhook path, which already decides for itself, and by the 6-hourly tier sweep. - `rate_limit.set_user_tier` (the admin's tier override). - The 6-hourly tier sweep schedules nothing, as before. One person sync per corrected user would fan out a Stripe call each when a run corrects many at once. Every user it touches has a Stripe customer, so the daily lifecycle sweep brings their persons up to date. - An account deletion's cancel still syncs nothing. A sync would bring back the person the erasure deletes. - Sign-in (`POST /api/auth/user`, existing accounts): `$set` of the stored tier. Accounts the daily sweep doesn't cover (no Stripe customer, no trial) get it without waiting for a tier change, and a sync PostHog never got is repaired. A tier change landing during a sign-in can be overwritten by the older read until the next sign-in or sync, which is documented. - Impersonation: the sign-in `$set` goes through `posthog_client.capture`, which drops `$set` while an admin acts as the user. The sync itself still runs then, as before: it mirrors the stored tier, trial and Stripe state, not the admin's action. - Fire-and-forget throughout: syncs run in background tasks, and the sign-in fill is wrapped, so a PostHog failure never reaches a tier write or a request. - `docs/engineering/tracking-plan.md`: `subscription_tier` under Person properties (values, when it is set, that it means the plan the limits apply rather than payment, and that a person property shows the current plan, not the plan at the time of the event), plus a pointer from `chat_blocked.tier`. - Tests: the mapping per status, the sync and sweep payloads, the sign-in `$set` (empty tiers skipped, dropped while impersonated, PostHog failing), every scheduling hook (tier write, admin override, account-deletion cancel on both paths, the webhook-internal and sweep writes, the unchanged sweep), and a tier write surviving a failing sync with the real scheduler. The old tests that pinned "a tier write schedules no sync" are rewritten. No configuration changes. Existing accounts get the property at the daily sweep (accounts with billing history), their next sign-in, or their next tier change. `scripts/backfill_posthog_lifecycle.py --send --all-users` sets it on everyone at once. ### Agents and large language models used - Claude Code, Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] `posthog_lifecycle_test`, `posthog_lifecycle_sync_test`, `posthog_lifecycle_hooks_test`, `posthog_lifecycle_trial_cancel_test`, `test_backfill_posthog_lifecycle`, `credit_account_deletion_test`, `v1_stripe_webhook_test` (without its `server` test), `stripe_reconciliation_test`, `stripe_activation_sweep_test` and the five changed `credit_subscription_test` cases pass locally (mock-only) - [ ] CI: the rest of `credit_subscription_test`, `user/routes_test` and the database-backed suites - [ ] After merge, on dev: change the QA account's tier and check its PostHog person shows `subscription_tier`; a `chat_blocked` insight breaks down by that person property Linear: SECRT-3411 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]> |
||
|
|
83f6ce5776 |
feat(platform): add in-app feedback prompts (#16348)
### Why / What / How **Why:** The app collects almost no feedback at the moments that matter. Today there is the Tally form in the account menu, thumbs on Otto replies and a Sentry "Report Error". Nothing asks why someone cancels a trial, leaves checkout, or found a finished task disappointing. **What:** Six feedback prompts, live in the app with no feature flag, reshaped after the PM review (trial checkout and brain dump skip prompts cut, trial cancel back to one dialog). This PR started as Storybook prototypes and now wires them in, saving every answer through our backend. **How:** One `FeedbackPrompt` table records each showing, and an atomic `claim` applies the limits, so they hold across tabs and devices. Page-level prompts claim, then save each tap, and the backend forwards to PostHog. The browser PostHog client sends nothing until someone accepts analytics cookies, so answers never go that way. Two new Otto tools claim on the model's behalf and render chat cards, whose ratings use the existing reply-feedback path. A prompt never blocks what someone came to do: if saving fails, the trial still cancels, onboarding still moves on, and no toast appears. Linear: SECRT-3217 ### Changes 🏗️ | Prompt | Where it shows | Limit | Where the answer goes | |---|---|---|---| | Trial cancel | Inside the one "Cancel your trial?" dialog: optional reason chips, Keep trial and Cancel trial always shown | None | Database + PostHog, and Stripe `cancellation_details` | | Left upgrade checkout | On return to Billing. Only "My card didn't go through" offers "Try another card", which reopens the same checkout | None | Database + PostHog | | Work check | Under the reply that finished the work, once the turn ends, in Otto and expert chats | Shared wait | `$ai_metric`, as a `work_check` score | | Otto mistake | Inside the reply when Otto decides it got something wrong; no card when the user already said what went wrong | Once per chat | `$ai_metric`, as an `otto_mistake` score, plus `otto_mistake_flagged` | | Top-up | Inside the out-of-credits dialog, after "Not now" | Every 30 days, plus the shared wait | Database + PostHog | | Day 14 NPS | Any page, from day 14, for paying accounts that finished a workflow run or an Otto chat that week | Once, plus the shared wait | Database + PostHog | **Shared wait:** work check, top-up and NPS share one 7-day wait, and two "Not now"s in a row among them mean 30 days quiet. One advisory lock covers all three, so two tabs cannot both pass it. **Backend** - New `FeedbackPrompt` table and migration; a nullable `reason` column on `ChatMessageFeedback`. - `POST /api/feedback-prompts/claim`, `PATCH /api/feedback-prompts/{id}`, `GET /api/feedback-prompts/{id}`. - `data/feedback_prompts.py`: every query is scoped by `userId`, and another user's row returns 404. - New Otto tools `request_work_check` and `flag_mistake`, eager so they can render cards, and available in expert chats. `flag_mistake` takes an optional `reason` from the shared list; with one, no card asks again (the tool schema budget rises by 150 characters for it). Work checks are off on chat platforms, automation and delegated sessions. Mistake flags still reach PostHog there, with no card. - `POST /api/credits/trial/cancel` takes an optional `{reason, comment}` and passes it to Stripe as `cancellation_details` (`too_expensive`, `low_quality`, `missing_features`, `unused`, `other`). - New PostHog events: `feedback_prompt_shown`, `feedback_prompt_answered`, `otto_mistake_flagged`. An admin acting as a user creates no rows or events. **Frontend** - Prompts in `organisms/FeedbackPrompts/` take real callbacks. Options now have fixed values, so code no longer compares labels. - Mounted in the trial dialog, Billing, the top-up dialog, the chat and a new `PlatformShell`, which keeps an open NPS card alive across route changes. - The billing cancel URL now carries `plan` and `cycle`, so "Try another card" can reopen the same checkout. - Chat rating requests run in order per prompt and mark the card answered only after the rating saves. Delayed-write and failed-request tests cover both paths. - Review fixes: scores follow each prompt kind; unattended turns do not claim weekly work checks; mistake analytics omit free-form chat content; pending microphone starts cannot replace skip reasons; checkout prompts record dismissal and close on account changes; notes stop at 2,000 characters; cancellation attempts claim separate showings. - Polish pass: no pixel change. Checked the JSX and className diff: styles and layout are unchanged. The fixes change behavior, form limits, link destinations, generic tool labels, tests and generated docs. - Pixels: the PM review changes the look of the trial cancel dialog (one step), the out-of-credits dialog (a "Not now" button and a question step) and the NPS card (two rows of scores under 640px). Screenshots still to attach. ### Agents and large language models used - Claude Code with Claude Opus 5.5 - Codex with GPT-6 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [ ] I have tested my changes according to the test plan: - [x] `black`, `isort`, `ruff` and `pyright` on the changed backend files - [x] `prettier`, `oxlint` and `eslint` on the changed frontend files - [x] 424 targeted frontend tests across 19 files pass locally (one worker) - [x] Backend route/tool registry: 25 tests pass; feedback route/tool/rating checks: 105 tests pass locally - [x] Required backend architecture and block tests: 4,175 pass, 84 known skips - [x] Full backend and E2E CI suites pass on `39d1ac5619`; repo review approved - [x] Review fixes (`6d9fdd7637`): 79 feedback route/chat tests pass; required backend architecture/block sweep passes 4,175 tests with 84 known skips (local ClamAV disabled); frontend format/lint/types and targeted Python format/lint/types pass. Full CI is running. - [ ] Each prompt checked by hand in a running app #### For frontend changes (design system, see `docs/engineering/design-system.md`): - [x] `pnpm format`, `pnpm lint` and `pnpm types` pass locally, including the `design-system/*` rules; I did not raise any count in `scripts/eslint-rules/design-system-baseline.json` except for a moved, renamed or split file (explained under **Changes**) - [x] No `dark:`/`light:` classes, raw palette classes (`bg-zinc-200`) or hex/`rgb()` colours (`bg-[#fff]`); colours use role tokens (`bg-surface`, `text-foreground-secondary`) - [x] No imports from `src/components/__legacy__`, `legacy/` paths or `@/components/ui`, and no `LegacyButton` / `legacy-*` Button variants - [x] Copy uses the `Text` atom with a `variant`/`tone`; primitives get a `variant`/`size`. `className` holds layout, plus on `Text` only truncation, line clamping, wrapping, breaking, `text-balance`/`text-pretty` and alignment - [ ] Before/after screenshots at 1440 and 390 px are attached for changes to `src/components/**`, `globals.css` or `tailwind*.ts`, or I wrote "no pixel change" and how I checked - [x] User-facing copy says workflow and Otto, never agent or graph (root `AGENTS.md`, "User-facing copy") #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: GPT-6 (Codex) <[email protected]> |
||
|
Nicholas TindleandClaude Opus 5.5
|
25e1ea300c |
feat(platform): Continue with Microsoft, with the Microsoft 365 offer as a movable onboarding step (#16269)
> Builds on #16268, which merged into `dev` on 2026-10-30. This PR now
targets `dev`, and its diff is only its own changes.
### Why / What / How
**Why:** people whose work runs on Microsoft 365 had no way to sign in
with Microsoft, and connecting their mail and files came later as a
separate step most never found. The onboarding wizard also had a fixed
order, so trying the connection offer earlier or later meant a code
change.
**What:**
- **Continue with Microsoft** on the login and signup pages. It asks
Microsoft only for name and email (`openid profile email`, no Graph
permission).
- A new account's onboarding asks **"Also connect Microsoft 365?"**
using the panel from #16268. Skipping it is fine.
- The wizard's order is data, so an experiment can move the step without
a code change.
**How:**
- **Who signs in** (`lib/auth/microsoft-login.ts`,
`microsoft-sign-up-policy.ts`): Better Auth keys accounts on Microsoft's
object ID (`oid`), never the email address. An address counts only when
Microsoft vouches for it: a personal account's own Outlook.com, Hotmail,
Live or MSN mailbox, or a work address its organization verified
(`xms_edov`). A personal Microsoft account can carry any address and
keeps it after its owner loses the mailbox, and a work address without
`xms_edov` is whatever an admin typed in. A guest's address is never
trusted either: a personal account invited into someone else's
organization signs in with that organization's tenant, whose `xms_edov`
would vouch for it. A token counts as a member's only when its `idp` is
absent or exactly the issuing tenant's issuer. Any other address neither
joins an existing account nor creates one. Both answer Better Auth's
`account_not_linked`, the same redirect to the byte, so a refusal
doesn't reveal whether an AutoGPT account uses the address. /login gives
one message for both and points to email sign-up. Emailing a
confirmation link would not be safe: the mailbox's owner could confirm
an account that the other person's Microsoft login still opens.
- **Team addresses** (`microsoft-sign-up-policy.ts`): an AutoGPT team
address (`@agpt.co`) gets an account only through Google, however
Microsoft vouches for it. A refused one lands on
`/login?provider=microsoft&error=team_email_requires_google` with its
own message. A team member's existing account still signs in with
Microsoft. Microsoft's checks run first in the user-create hook, so its
sign-ups get these codes. #16445 applies the team-address and
verified-email rules to every provider; whichever of the two PRs merges
second keeps that hook order (Microsoft's check, then the team-address
check, then the verified-email check), and a trial merge of both passes
their sign-up tests together.
- **Only the redirect flow signs in**: Better Auth's ID-token sign-in
(`POST /sign-in/social` with `idToken`) is switched off for Microsoft,
because it creates accounts without the callback's checks; the sign-up
check also follows the provider rather than the callback path.
- **Refusals**: a Microsoft sign-up the signup gate refuses shows the
"Email Not Allowed" dialog on /login. Better Auth sends a refusal from
our checks back with its code as `error` (`SIGN_UP_NOT_ALLOWED` for the
gate, which `isWaitlistSignInError` reads). A blocked network, a
disposable address or one mail can't reach say the sign-up isn't allowed
instead of "try again", and a blocked sign-in says so too. The page
never shows the URL's own text. A refusal keeps the page's `next`, so a
deep link still opens after logging in another way. Since #15459 every
provider's refusal comes back to /login; Microsoft's also says
`provider=microsoft`, and only then does /login use the Microsoft
wording.
- **The access step** (`onboarding/steps/AccessStep/`): it shows when
the account signed in with a provider whose data Otto can use.
`accessProviders.tsx` maps each provider to its connection, so Google or
GitHub can join with one entry. `GET /api/auth/sign-in-providers` names
the account's sign-in providers, looked up once per account; Better
Auth's own `/list-accounts` stays disabled because it returns account
details.
- **Movable order** (`onboarding/store.ts`): `DEFAULT_STEP_ORDER` lists
the steps. The `onboarding-step-order` flag's payload names the steps to
lead with; the rest keep their default order, and `preparing` stays
last. With payments on, the access step always follows the subscription
step, whatever the experiment asks: a new account has no plan until
then, and connecting a service needs one. The order is frozen per
account when the wizard loads, through `useFrozenPerUser`, which the
payment snapshot now uses too (dev's ref read during render raised 14
`react-hooks/refs` warnings). Progress is saved by step name, so a
resumed wizard lands on the right step even if the order changed.
### Changes 🏗️
- Frontend auth: the Microsoft login provider (shown only when
`AUTH_MICROSOFT_CLIENT_ID`, `AUTH_MICROSOFT_CLIENT_SECRET` and
`NEXT_PUBLIC_AUTH_MICROSOFT_ENABLED` are set), the sign-up refusal for
unconfirmed addresses, error messages on /login, and Microsoft sign-ups
counted as such in analytics.
- Onboarding: the access step, `resolveStepOrder` (with the paywall
rule) and `buildStepLayout` with an `order`, `useFrozenPerUser`, and
`access` as a saved step name (backend `WizardStep`).
- Backend: the `onboarding-step-order` payload flag, and a seeded e2e
user who signed up with Microsoft.
- From CodeRabbit's and Sentry's reviews, once this targeted `dev`:
- Login and signup start one provider sign-in at a time.
- The sign-in-provider lookup gives up 5 seconds after it starts, its
retry included, so it holds onboarding for at most 5 seconds. A quick
failure gets one retry in what's left of that time; a lookup that timed
out isn't retried.
- A saved draft keeps the role pick once the person moves past the role
step, in whatever order the wizard used. It reads that order from
`completedSteps`. With an experiment that asks the role first, drafts
saved on the steps after it used to leave the pick out until
`painPoints`.
- A regression from #16268: the Copilot executor again reports a refused
Microsoft refresh as `microsoft_365_copilot_credential_unavailable`, not
`…_not_found`. Since #16268 the refused refresh raised
`MicrosoftAuthError`, a `ValueError` subclass.
- With Better Auth 1.7.12 (#16419) merged from `dev`:
- The Microsoft refusals carry codes (`account_not_linked`,
`team_email_requires_google`). 1.7's OAuth callback only redirects an
error that has one; any other ends on a raw 403 JSON page.
- /login explains the codes dev's own sign-up checks now send.
- Accounts are keyed on `oid`, as 1.7 does, instead of `${tid}:${oid}`.
Dev and prod have no Microsoft login rows yet, so nothing needs
migrating.
- The Playwright spec signs in through the API. The login form can be
replaced just after it loads, which dropped the email the spec typed
(smoke failed once on CI).
- From Pwuts's review:
- An address Microsoft didn't confirm gets `account_not_linked` with no
message, as described above.
- Sign-in no longer fetches the Microsoft profile photo, so it reads
only name and email.
- The Outlook.com domains are the 113 whose primary MX is Outlook.com's
consumer servers. That adds `hotmail.se`, `live.se` and the other
country domains, and drops `outlook.co.uk` (its mail goes to a third
party) and `hotmail.nl`, `hotmail.cz` and `outlook.cz` (a Microsoft 365
organization first).
- Both sign-in buttons' logos are decorative (`alt=""`), so a screen
reader says each provider once.
- `openapi.json` regenerated; `.secrets.baseline` line numbers updated.
- `docs/engineering/microsoft-local-e2e.md`: sign-in, the access step,
moving it, and the extra Entra settings.
**Deploying:** this builds on #16268, whose Entra step 1 (token version
2, then personal accounts in the sign-in audience) has to happen before
this release reaches an environment. The Entra app also needs the
`{frontend}/api/auth/callback/microsoft` redirect and the `xms_edov`
optional ID-token claim. Without the claim, work and school accounts
can't use Continue with Microsoft. Dev is configured; prod is not.
### Agents and large language models used
- Claude Code with Claude Opus 5.5: design, implementation, tests and
live testing.
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] `lib/auth/__tests__/microsoft-sign-in-flow.test.ts` runs Better
Auth's real handler with Microsoft's endpoints stubbed: an Outlook.com
personal account signs up; a verified work address joins the account
that uses it; a personal account's other address, an unverified work
address and a guest's address are kept out of it; an unconfirmed address
creates nothing, with email verification on or off; an ID token for an
Outlook.com address, signed with a test key served as Microsoft's key
set, can't sign in directly; and a waitlist refusal comes back as a code
/login recognizes. Before the fixes, the guest case joined the existing
account and the ID token created one; with `disableIdTokenSignIn`
removed, the ID token still signs in.
- [x] `vitest` over onboarding, login, signup, `lib/auth`, the Microsoft
panel and feature flags: 935 passed, including the access step offering
only "Continue" once connected and following the paywall when payments
are on. `pnpm types` and `pnpm lint` pass.
- [x] Backend: `pytest backend/data/onboarding_wizard_test.py
backend/util/feature_flag test/test_e2e_seed_credentials.py`: 284
passed.
- [x] After #16268 merged, with `dev` merged in (only the shared
engineering doc conflicted; this PR's copy, which extends #16268's, was
kept): backend 674 passed (onboarding wizard, feature flags, Microsoft
integrations, Copilot transports), black, isort and pyright clean, and
`openapi.json` matches the export. Frontend `pnpm types` and `pnpm lint`
pass, and all four `vitest` shards pass under `TZ=UTC` (12,700 tests).
- [x] The team-address rule: `microsoft-sign-in-flow.test.ts` drives the
real callback for an `@agpt.co` work address (refused, nothing created)
and for a team member's existing account (signed in), and the login page
test shows the message.
- [x] Screenshots, before (dev-builder) and after (this branch) at 1440
and 390 px, of /login and /signup with the Microsoft button, /login
after a refused sign-in, and the access step:
https://github.com/Significant-Gravitas/AutoGPT/pull/16269#issuecomment-6560941949
- [x] Playwright `microsoft-sign-in-happy-path.spec.ts`: a seeded
Microsoft sign-up opens onboarding on the offer, and "Not now" moves on.
It passed locally against a full stack and in CI's smoke run on
|
||
|
Krzysztof CzerwinskiandClaude Opus 5.5
|
30b70e7679 |
feat(backend): send a PostHog event whenever a usage limit refuses or pauses work (SECRT-3411) (#16698)
### Why / What / How **Why:** when a usage limit refuses or pauses someone's work, PostHog hears nothing, so we can't see how often limits bite, for whom, or where (SECRT-3411). **What:** one backend event, `chat_blocked`, every time a limit refuses, pauses or queues work, from every surface. The name and its `reason` values (`usage_limit`, `rate_limit`) come from the analytics plan; the event now also fires for scheduled, bot, block, voice, API and background refusals, not just the chat. **How:** the event is sent where each refusal is answered, not where the limit is read. `check_rate_limit()` is also used by probe-only checks (for example `at_reset`), and only the caller knows the surface. `RateLimitExceeded`, `TurnRefusal`, `ConcurrentTurnLimitError` and `InflightCapExceeded` now carry the reading (tier, used, cap), so each call site is one line. Sending goes through `posthog_client.capture` (queued on the client's background thread), and every helper is guarded end to end, reading the refusal included, so a refusal returns exactly what it returned before, just as fast, even when PostHog raises. Server events have no consent check, as for the other backend events. ### Changes 🏗️ - `backend/copilot/limit_tracking.py` (new): `track_chat_blocked` and helpers for cost limits, turn caps, turn pauses and queue/scheduler refusals; the surface of an executor turn is worked out from its origin. - Properties: `reason`, `limit` (`daily`, `weekly`, `trial`, `task`, `rate`, `concurrency`), `outcome` (`refused`, `paused`, `queued`), `surface` (`chat`, `schedule`, `chat_bot`, `block`, `voice`, `api`, `background`), `tier`, `used` and `cap` (dollars for cost limits, counts for rate/concurrency), `resets_at` (null for a trial, a window that allows nothing, and the turn caps). `distinct_id` is the user id. No message content or personal data. - Where it fires: - Chat: the stream route's cost check (`enforce_platform_allowance`, also used by Keep going), the concurrency 429s and the queue fallback (`queued`), the queued-message per-minute cap, and the Keep going concurrency 429. - Chat bots: the usage gate and the in-flight cap. - Voice (`/speech`), external API/MCP tool calls, and the nightly memory pass (weekly). - Scheduler: usage refusals and the turn cap, and "Send/Resume at reset" turns a used-up window holds back (`queued` to the next reset, `refused` when nothing refills) or the turn cap refuses. - Queue promotion: a queued turn a usage window holds back (`queued`), checked once per pass over the queue. - Executor: the pre-turn allowance gate (`refused`), plus every pause the engine streams: a landed `daily`/`weekly`/`trial`/`task` pause (`paused`) and a refused auto-continue (`refused`). A `steps` check-in is not a limit and sends nothing. - Sub-sessions and the AutoPilot block at the turn caps, approval wakes, `message_session` wakes, and per-call platform spend (copilot tools, dry-run blocks, the simulator). - Readings: a pause the engine streams keeps the reading its check made (on the stream part as a private attribute, never serialised to the client or the stored marker), and a turn that lands on its budget carries its plan tier. Used and cap stay null for a mid-turn landing. - Surface: checks deep inside a turn (a tool's platform spend) report the surface the executor resolved, so a chat bot's turn is `chat_bot` and unattended work `background` there too. - An ended trial reads as `trial` (cap 0) and a plan with no allowance as `weekly` (cap 0) at every site, through one helper. - A turn the executor refuses before it starts is marked `impersonated` when an admin started it. - Each backend process sends at most one `chat_blocked` a minute per user, limit, outcome and surface, so a client retrying a refused request sends one event, not one per retry. The window is in memory: it never blocks or fails the refusal. - The queued-turn event carries the queued turn's own impersonation flag, stored when it was queued, not that of the turn whose end ran the queue. - `PostHogEvent.CHAT_BLOCKED` added and pinned; `docs/engineering/tracking-plan.md` lists the event and drops it from "not sent yet". - A `chat_blocked_events` test fixture in `backend/conftest.py`, and tests at each refusal point that check the properties sent. Out of scope: `/usage/reset` 429s and the generic `api/utils/rate_limit.py` limiter for non-chat routes (not usage limits on work). Also left out: outages that leave the limits unreadable (503s), paywall/ban/Codex-access refusals (not usage limits), the engine-switch continuation dispatch and the AutoPilot block's recovery re-enqueue (best-effort system retries). Post-merge checks: - [ ] Build a PostHog insight on `chat_blocked`, broken down by `limit`, `surface` and `outcome`. - [ ] Compare one day of Sentry 429s on `POST /api/chat/sessions/{id}/stream` with `chat_blocked` where `surface = chat` and `limit` is `rate`, `concurrency` or a cost limit. ### Agents and large language models used - Claude Code, Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Unit tests for `limit_tracking` (properties, reason mapping, refill rules, surfaces, never raises, even on a refusal it can't read) - [x] A refusal answers the same when PostHog raises, at real call sites (MCP tool call, executor gate) - [x] Event assertions at each refusal point: chat route, Keep going, queued messages, chat bots, voice, MCP, dream, scheduler and at-reset turns, queue promotion, executor gate and streamed pauses, sub-sessions/AutoPilot block, approval wakes, `message_session`, platform spend - [x] Existing suites for every touched module pass locally (DB-backed tests left to CI) - [x] pyright, ruff, black and isort clean on the changed files 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
|
9d04ac1a8d |
feat(platform): self-serve account deletion with an optional exit question (#15459)
### Why / What / How **Why.** Today the only way to delete an AutoGPT account is to email [email protected], and people without a plan can't reach Settings at all (the paywall covers `/settings/*`). SECRT-2877 asks for self-serve deletion from Settings › Account and from the paywall, with an optional "Why are you leaving?" step. **What.** A two-step Delete account dialog: the optional exit question (each reason offers one alternative, such as Manage connected apps or cancelling a trial), then a confirmation where the person types their email. Confirming closes the account at once. The data is erased 14 days later; until then the owner can sign in and choose **Keep my account**. Erasing the data is #15487 (SECRT-2854), which also holds the ban record that outlives erasure. The `self-serve-account-deletion` flag stays off until that erasure runs in prod. **How.** `POST /api/account/deletion` (`account_deletion/service.py`): 1. Check the typed email (trimmed, any case), else 400. Already closed: 200, nothing else runs. 2. Stripe: expire open Checkouts, cancel every subscription now with no invoice or proration, turn off auto top-up. A Stripe error returns 502 and the account stays open. 3. Best effort: disconnect and revoke every connected app (10 at a time, 10 s each), revoke API keys and OAuth grants, remove schedules and the Briefing, turn off webhook triggers, unlink chat platforms, cancel AutoPilot turns, stop runs, turn off product email, opt out of MailerLite. 4. One transaction: `AccountDeletionRequest` with `eraseAfter` = now + 14 days, all sessions deleted, Marketplace listings soft-deleted, reviews deleted, share links off, Discord/Telegram/Slack links removed. 5. Email "Your AutoGPT account is closed" ("Sign in by {date} and choose Keep my account"), then one anonymous count. While closed, every API route answers 401 "This account is closed." (optional-auth routes treat it as signed out; WebSocket and external API refuse it). `is_user_refused` gates all background work: scheduler, memory crons, graph runs, AutoPilot, webhook ingress, Briefing. The closed state is cached 30 s and dropped on close and keep. The creator page is hidden. Closing does not ban; a banned person can still delete, but can't sign in to keep. Keeping: sign-in lands on `/account-closed` with **Keep my account** and **Sign out**. `POST /api/account/deletion/restore` deletes the request only while `startedAt` and `completedAt` are null and `eraseAfter` is ahead, in one statement (409 otherwise); #15487 claims a row by setting `startedAt` first. Only the account comes back: plan, apps, keys, schedules, triggers, share links, listings and reviews stay off, and product email stays off until switched on. Feedback (`POST /api/account/deletion/feedback`) goes to PostHog through the new `capture_anonymous()`: no user id, email or IP; text capped at 1,000 characters, never logged; 20 calls per user per hour. The dialog is `ph-no-capture`. ### Changes 🏗️ Backend - `api/features/account_deletion/`: routes, service, step modules, `restore.py`. - `data/account_deletion.py`: close, reopen, request record. - `autogpt_libs/auth/closed_accounts.py`, `dependencies.py`: closed check in every auth dependency; `get_user_id_allowing_closed`. - `data/user.py`: `is_user_refused`; no billing or marketing email to a closed account. - Refusal gates in `executor/`, `copilot/`, `integrations/router.py`, `api/ws_api.py`, `api/external/middleware.py`, `platform_linking/`. - `data/credit.py`, `data/subscription_checkout.py`, `notifications/lifecycle.py`: immediate cancel tagged `autogpt:account_deleted`, with no "ended" emails or MailerLite moves. - `integrations/router.py`: `delete_and_revoke_credentials` extracted, route unchanged. - `integrations/oauth/google.py`: revoke sends the refresh token (all Google disconnects). - `notifications/auth.py`: `account_deleted` email. - `api/features/store/`: hidden creator page, cache clear. - `data/auth/`: bulk revoke for API keys and OAuth. - `util/posthog_*`: `capture_anonymous()`, five events. Frontend - `components/organisms/DeleteAccountFlow/`: dialog, steps, panels, stories, tests. - `DeleteAccountCard` (Settings), `DeleteAccountButton` (paywall). - `app/(no-navbar)/account-closed/`: the keep page; `lib/auth/closed-account.ts` routes closed accounts there. - `QuestionOptionList`: `editable` prop; `Dialog`: opt-in drawer `autoFocus`. - `services/billing/subscription-plan.ts`: shared with `useYourPlanCard`. - `openapi.json`; `docs/engineering/tracking-plan.md`. **Configuration changes** - Flag `self-serve-account-deletion`, off by default (`FORCE_FLAG_` / `NEXT_PUBLIC_FORCE_FLAG_` overrides). Off: no entry points, delete and feedback 404. Status and restore stay reachable so nobody is stranded. - Migrations `20261010120000_account_deletion_request`, `20261030120000_account_deletion_grace_period`. - Routes: `POST`/`GET /api/account/deletion`, `POST .../restore`, `POST .../feedback`. Not behind the paywall; 403 for an admin impersonating (except status). - Protected page `/account-closed`. - Proxy: 180 s timeout for `POST api/account/deletion`. - CI: E2E forces the flag on; smoke count 28 → 29. - No new secrets or env vars. **Known gaps** - Cloudflare's ~100 s origin timeout is under the 180 s proxy timeout: many apps can show a gateway error; a retry returns 200. - Open Stripe invoices stay open, with collection stopped. - `opened` shares the 20/hour feedback cap with answers. - A schedule fire or webhook delivery already under way can still add one run or turn. - Deploy the DB manager with or before the chat bot (new `open_only` lookups). ### Agents and large language models used - Claude Code with Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [ ] I have tested my changes according to the test plan: - [x] Backend, frontend and `autogpt_libs` tests pass in CI on the current head. - [x] Playwright: delete from Settings, sign in, `/account-closed`, Keep my account, back to Settings. - [x] Dialog at 320×568 to desktop, 200% and 400% zoom: no sideways scroll, footer reachable, focus never hidden. - [ ] Manual delete from Settings and paywall, then Keep, on the current head. - [ ] `/account-closed` at the same sizes; iOS Safari and Android Chrome. - [ ] Dev deploy with the flag on: plan cancelled with no invoice, Google grant gone, schedule gone, email arrives, PostHog events have no person, a double call sends one email. #### Screenshots [The screenshot page](https://claude.ai/artifact/M4jqDhdBQvgfCAdbEFExD7) has the dialog at all seven sizes, both entry points, and the keyboard-focus checks. #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) Resolves SECRT-2877 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Krzysztof Czerwinski <[email protected]> |
||
|
Nicholas Tindle
|
76e4d5ce58 |
feat(backend): add Meta Muse Ember (#16700)
Adds `meta/muse-ember` (Meta, released 2026-11-09) via OpenRouter, alongside the other Muse models. - Price: $0.10/$0.50 per 1M, cache read $0.004/1M, from the live OpenRouter listing, at the standard 1.5x credit margin (15/75 credits). OpenRouter bills it by cost, so the per-1M credits are display only. - Specs: 1M context, 943K max output, tools, structured outputs and reasoning. - Tests: pinned catalog rates; block docs updated. |
||
|
|
e6a3d35514 |
feat(platform/onboarding): Otto replies on the brain dump's not-enough screen (#16654)
### Why / What / How **Why:** A large share of onboarding brain dump recordings land on the "We didn't catch enough of that" screen, and most people there skip or leave instead of talking again. Rejected takes are tiny (mic checks, greetings, filler), and today's screen reads like an error: it never shows what was heard, and its headline sounds like the microphone failed. Context: [SECRT-3361](https://linear.app/autogpt/issue/SECRT-3361). **What:** The screen becomes a short, labeled chat ("Otto replies"): what Otto heard, his answer, and one big, easy question, "What do you want off your plate?", with an example for the role. Talk stays the main action, and talking or writing afterwards keeps the question in view. The backend now sends back what was heard on an `insufficient_content` reject, and known speech-to-text silence phrases become `no_usable_speech` for voice, so the screen never shows words nobody said. **How:** The rejection fills the existing `transcript_preview` field; the only API addition is the optional `otto_reply_kind` enum ( |
||
|
Bently
|
c27bf0b975 |
docs: add deployment environment variables guide (#12122)
## Summary Documents all environment variables that must be configured when deploying AutoGPT to a new server or environment. ## Changes - Added `docs/platform/deployment-environment-variables.md`: what to change to run the Docker Compose stack on a server that people reach from other machines, and why. - Listed it in `docs/platform/SUMMARY.md`. - Brought up to date with `dev` (merged, not rebased): - Dropped the step that made Web Push (VAPID) keys by hand. `make init-env` generates the pair since #16416, and the frontend fetches the public key from the backend. Only `VAPID_CLAIM_EMAIL` is left to set. The steps after it are renumbered. - Added `TRUSTED_PROXIES` (#16522) to the proxy section and the checklist, for a chain of proxies or one that adds to `X-Forwarded-For`. - `TRUST_X_FORWARDED_FOR` also feeds the per-IP sign-in and password-reset attempt limits, and the guide now says so. - Added `BETTER_AUTH_INTERNAL_URL=http://localhost:3000`, found by deploying the guide for real (see Testing). Without it, the frontend's server fetches a signed-in visitor's token through the public hostname. Sign-up then fails without an error whenever the container can't resolve that name or trust its certificate. - Said which lines have to be added because they aren't in the `.env` files yet (`BACKEND_CORS_ALLOW_ORIGINS`, `BETTER_AUTH_INTERNAL_URL`). - Gave the no-proxy API example its `/api` path (`NEXT_PUBLIC_AGPT_SERVER_URL=http://<server>:8006/api`). ## What's Documented - Which `.env` file each setting lives in, and which values the compose files override - What `make init-env` generates, and what breaks without each secret - The public URLs to set, with `BETTER_AUTH_INTERNAL_URL` for the frontend's calls to itself - A TLS reverse proxy (Caddy example), the ports to close, `TRUSTED_PROXY_COUNT`, `TRUST_X_FORWARDED_FOR` and `TRUSTED_PROXIES` - Limiting sign-ups, and the mailer that email verification and password reset need - OAuth callback URLs for integrations and for social sign-in - Using your own database or Redis - A checklist ## Closes - Closes #10961 - Closes OPEN-2715 ## Testing Documentation only, no code changes. - Followed the guide from the top on a clean checkout on a LAN test machine. Set only what it says, behind Caddy with TLS on two hostnames, and used a browser on another machine. - Sign-up, log-in, a server-rendered page while logged in, the builder, and an agent run (completed, output shown) all work. Live updates connect over `wss://`. - The sign-in history records the visitor's real address, and a spoofed `X-Forwarded-For` is dropped. - Before `BETTER_AUTH_INTERNAL_URL` was added, sign-up failed (`ENOTFOUND` on the public hostname inside the frontend container). - `python -m unittest discover -s .github/workflows/scripts`: 101 tests OK, including `test_docs_layout.py` (8). <!-- greptile_comment --> <details><summary><h3>Greptile Summary</h3></summary> Adds comprehensive deployment environment variables guide documenting all configuration settings required for deploying AutoGPT to new environments. The guide includes a quick reference table of critical URLs, security key generation commands, service configuration details, OAuth provider setup, and a complete deployment checklist. Documentation is well-organized with clear sections for URL configuration, security keys, database setup, service dependencies, default ports, OAuth callbacks, optional services, and Docker vs external services guidance. </details> <details><summary><h3>Confidence Score: 5/5</h3></summary> - This PR is safe to merge with minimal risk - Documentation-only PR with comprehensive and accurate environment variable reference. Cross-verified all variables against actual `.env.default` files, confirmed file structure matches codebase, and validated all internal documentation links. Only one minor default value discrepancy found. - No files require special attention </details> <sub>Last reviewed commit: 8d557d3</sub> <!-- greptile_other_comments_section --> <!-- /greptile_comment --> |
||
|
|
bba2ff341a |
feat(platform/onboarding): plan-first trial: starter tasks and experts on the paywall, hire step removed, team hired on trial start, Home starts with ranked tasks owned by experts (#16308)
### Why / What / How **Why.** Since Oct 16, 954 people signed up, 444 saw the paywall, 209 opened Stripe and 29 started a trial ([funnel](https://eu.posthog.com/project/117338/insights/ugcxrNQU)). 68% of the people who reached the hire step hired nobody, and only 3 people have ever clicked a suggested first task on Home. The hire step asked people to pick experts before they had seen what those experts would do, the paywall then asked for a card without a reason, and Home's suggestions had no owner and no order. **What.** The paywall becomes the plan: what Otto heard, the first three tasks ranked, each owned by the expert who will do it, beside the 7-day trial. The hire step is gone. The plan's team is hired on the preparing screen once the trial is confirmed, so a user who abandons Stripe hires nobody. Home opens on the same tasks as a list: "Start here" on the first open row, each row naming who takes it, x removes a task, and Hide / Show collapses the list. **How.** - **Backend.** After the greeting and the team are stored, one TypeSafe Jev call ranks the greeting's tasks and picks an owner for each (a call to TypeSafe's hosted model, which carries the tasks, the team and up to 6,000 characters of the transcript); a deterministic fallback keeps the greeting's order and matches owners by icon and title. The team is capped to the three people the step shows. `GET /onboarding/brain-dump/plan` serves the plan (or a role plan for a skipped, failed or empty brain dump), `POST /onboarding/brain-dump/plan/settle` makes the plan the step shows the take's plan when the step stops waiting, and `POST /onboarding/brain-dump/plan/hire` hires its team idempotently. A suggested task's `expert_id` is read fresh: an open task's from the owner's active hire, a started task's from the chat it opened. - **Onboarding.** `PlanTrialStep` (`PlanPanel` beside `TrialPanel`, or `PaidPanel` without a trial offer, or the plan alone with Continue when there is no paywall) replaces the hire step and the trial welcome. `usePlanHire` hires on the preparing screen after the trial (or plan) is confirmed. - **Home.** `OnboardingIntroCard` goes back to `SuggestedTaskList`; the #16236 cards are deleted. A task is sent to its owner the way the recipient picker sends to an expert, so its chat is the expert's. ### Progress - [x] **Phase 0**: branch, baseline counts, before screenshots. Frontend `vitest` 12,282 passed + 4 expected failures (1,127 files); backend `onboarding_dump` + `experts` 904 passed, 29 skipped. - [x] **Phase 1** `bd3dc3cc feat(backend/onboarding): plan with ranked, owned tasks and a hire endpoint` (+ `f879fb20` review follow-ups). - [x] **Phase 2** `ba232616 feat(frontend/onboarding): plan + trial step replaces hire and subscription` (+ `7fb3dce4`, the plan read waits for a Skip still in flight). - [x] **Phase 3** `61708f3c feat(frontend/onboarding): hire the plan's team after the trial is confirmed`. - [x] **Phase 4** `130b92ad feat(frontend/home): suggested tasks ranked and owned by experts, list restored, remove and collapse` (+ `3782dfc4`, a started task names the expert its chat is with). - [x] **Phase 5** `e075809d` tracking audit fixes and the tracking plan, `177f4fad` the Playwright helper, `044c4fdb` edge-case coverage (a Jev outage logs one warning, an unset key warns once at startup, an empty team hires nobody, the walkthrough lands on the pill's row), `6eda537b` focus after Remove never drops to the page (found by the final review), `8badd23e` one ref write out of render (a lint warning this PR added), `d113cf2c` the plan's hire checks the paywall and the plan read is listed open (CI's `paywall_routes_test` requires one or the other for every route). - [x] **The plan step's wait** (found testing the preview) `9532c2b7` the step stops waiting on work the plan does not use and a plan it gives up on is the plan, `2abf8821` the settle route listed open while paywalled, `418ae59c` review fixes. See "The plan step's wait" below. - [x] **Merged with dev again** `8d81826a`, for dev's expert-roster test update (#16443). Checked on the merge: the exported API schema matches `openapi.json`; frontend types and lint pass, and onboarding, Home's task list and analytics run 1,053 tests, all passing; backend `experts` + `onboarding_dump` + `paywall_routes_test` has 1,170 passed, and the one failure needs ClamAV, which this environment does not run. - [x] **Kept current with dev** `075f17a9` (checked: schema matches `openapi.json`, frontend types and lint pass, 506 test files / 6,510 tests pass in onboarding, copilot, services and hooks, backend `onboarding_dump` + `paywall_routes_test` 637 passed), `538606ae`, `28e82949`, `6a8e6052`, `21ec4514`, `89750735`, `28a7f663`, `9a7fa159`, and `c365e277` (a port of #16489, dev's fix for the MCP tests it had broken). `9a7fa159` took dev's colour-role refactor (#16511): the hire-step Playwright fixture it recoloured stays deleted, and its `ds-colour-exempt.json` entry for `OttoCover.tsx`, a file this branch deletes, is removed. Test-only ports for CI flakes outside this PR: `52a5f637` (#16449, the PostHog consent test's capture interval) and `36527ca2` (#16566, the shared auth mock's session exports; #16565, the external API concurrency test's Redis timeout). They no-op once dev carries those PRs. The conflicts: dev's Langfuse prompt client in `intro.py` (#16026); a storage key both sides added (#16527); dev's Text, Icon and Button props on the Suggested Tasks components this branch rewrote (#16513: kept this branch's versions, moved two colour classes to `tone`, regenerated the design-system baseline); dev's `ENABLE_PLATFORM_PAYMENT` setting (#16459); and dev's removal of unused exports (#16163), which dropped `SELF_HOST_STEPS` from the onboarding store while this branch had changed it (`075f17a9`: kept this branch's plan step in `NO_PAYWALL_STEPS` and left `SELF_HOST_STEPS` deleted, since nothing imports it). Dev's palette removal (#16612) also added a lint-rule test case that relies on the exemption for `OttoCover.tsx`, which this branch deletes; `000a3f88` points that case at `CodeRenderer.tsx`'s allow-list instead. The paywall test's PostHog mock now returns the event, as dev's own tests do since #16518. - Since #16459, the no-paywall version of the plan step is what an install gets while `ENABLE_PLATFORM_PAYMENT` is off, which is the default. The plan step is step 5 either way; only the paywall beside it changes. - Checked on the last merge: the exported API schema matches `openapi.json`; frontend types and lint pass, and onboarding, copilot, services, hooks and the paywall gate run 485 test files (6,442 tests), all passing; backend `onboarding_dump`, `experts`, `paywall_routes_test` and dev's payment-setting and trial tests: 1,326 passed, and the one failure needs ClamAV. - [x] **The greeting has a fast backup, so the brain dump shapes the plan** (found testing the preview) `2c416169`, with review fixes in `a50305f3`. After a real brain dump, the plan step kept showing the role's starter tasks. The greeting, which writes the plan's tasks, either failed (the generic greeting) or was still running when the step stopped waiting at 30 s. Sonnet writes long answers for a long take, and this PR turned its thinking off, which is proven on Haiku 5.5 but not on Sonnet 5.5. `greeting_race.py` runs Sonnet first, and Haiku 5.5, which the team job already calls the same way, joins it. Haiku starts at once when Sonnet fails or gives an unusable answer, or alongside Sonnet once it has run 12 s. The first full answer wins and the other call is cancelled. Each attempt logs its model and why it failed, and the winner logs its model and time. Sonnet is now asked for by OpenRouter's own id, `anthropic/claude-sonnet-5.5`. A two-lens adversarial review confirmed three defects, now fixed in `a50305f3`: a malformed reply could escape the race and fail the dump, a test passed for the wrong reason, and the env example was stale. Checked: `onboarding_dump` + `paywall_routes_test`, 654 passed. - [x] **Every plan starts with a team of three** (asked for after testing the preview) `d6ea5524`. Without a brain dump, the team came from the wizard role alone: two experts for Sales/BD, Operations and HR/People, none for Other. Each role now names three skills that cover its starter tasks. Sales/BD adds Quinn for the account research; Operations and HR/People add Kai, for the self-chasing tracker and the interview scheduling. A brain-dump team with fewer than three is topped up with the wizard's picks after the model's own. Checked: `onboarding_dump`, `experts` and `paywall_routes_test`, 1,255 passed. - [x] **Tasks go to the experts whose roles fit them** (found testing the preview) `29476fbb`, with review fixes in `676a6b34`. A product manager's plan named Alex, Quinn and Nadia as the team and every task as Otto's. The owner match compared exact role strings ("research", "bookkeeping") with the catalog's roles, which now read "Research, Data & KPI Analysis", "Market & Competitor Intelligence" and "Product", so nobody matched. Now a role's skills come from the words it contains (`role_skills.py`), and the role sets, the title and icon hints and the fallback team all name skills. Against the live roster, every role's starter plan is its team's work: Product gets Alex, Alex, Nadia, Quinn, Alex. The one exception is Engineering's customer-questions task, which no engineering expert covers. Among teammates with a skill, the most focused one takes the task, so Maya (Marketing) writes the newsletter, not Clip (Motion Graphics). Title hints read the task's subject, not a word in passing: a "feature" in a marketing title or a sales "pitch" no longer decides. New hints cover CRM, contracts, security, bookkeeping and fundraising work. A task whose best match is cut from the three-person team goes to the next match who stays, not to Otto. An empty seat goes to a later task's owner before someone with no task. A four-lens adversarial review checked this against the live roster, with each finding verified. The greeting's failure log now names the exception type, because a timeout's message is empty. - Checked: the onboarding suite (613 tests) passes; experts + `paywall_routes_test` pass (638 passed, 30 skipped); pyright is clean on the changed package. The new `role_skills_test.py` pins owners and teams against the catalog's 33 roles. See "The greeting fell back on the preview" below. - [x] **Review round (kcze)** `4015926a`, then merged with dev `6c365f60`. 18 of the 19 findings fixed, each with a test that failed first; `d286bd1f` fixes the 19th, with `e438b230` and `4ea3cc78` correcting that commit: - Settling keeps the tasks the greeting already wrote instead of swapping in the role plan, and the role plan's template read is cached for 60 s. - `plan_experts_hired` moved to the backend and counts only experts that joined in that call. - Preparing retries a hire that got no answer inside its 15 s window, toasts who couldn't join, refreshes Home's task owners, and waits (bounded) for a ready plan when none was shown. - The plan step bounds and retries its settle, keeps polling behind a fallback the server never stored, ignores stale answers, and remembers only stored plans. - Checkout is back to "within a trial usage allowance". - Home holds owned tasks while the roster read fails, doesn't carry a new-chat mode over to another recipient, and only restores a recipient it changed. - `d286bd1f`, `e438b230` and `4ea3cc78` then cover the last finding and two gaps. `hire_plan` refuses a plan that `settle_plan` could not settle (409, nobody hired, no event) and the client retries it inside its window. This replaces the earlier choice to hire anyway: a user with nobody is now the cost only if every retry loses too, and then the toast says so. A refresh mid-hire re-asks once the first call's time is up, with its own retry window: the answer does the hire if the first call was lost, says who did not join, and refreshes Home. If no answer comes, it shows no toast, because the first call may have worked. Home's owners refresh whatever the answer. A hire nobody could join is not tried again, because the answer can't tell a blip from the expert limit. `plan_experts_hired` counts a failure once, through a Redis claim that gives up after 1 s and reports anyway. - Left as is: when no answer ever comes on the first hire, the toast names the whole plan even though a lost answer may have hired them (rare, and Home's roster still refreshes). A refresh whose first call was lost and whose re-ask never gets an answer leaves nobody hired and shows no toast; only Sentry sees it (rarer still). Failed whole calls (a 409 or 5xx) send no `plan_experts_hired`, as the tracking plan says. - Checked on the merge: the exported API schema matches `openapi.json`; frontend types and lint pass; backend `onboarding_dump`, experts routes and db, and `posthog_events` tests pass (1,025 passed, 19 skipped); frontend Vitest runs 1,220 files (13,685 passed, 4 expected failures). - [x] **Review round (Capy, CodeRabbit)** `32bac59b`, after merging dev in `0d7379cf`. Each of the 31 findings was checked against the code: 8 are fixed, and the other 23 are answered on the PR with the reason each stays as is. - Home's suggested-task reads fail when the roster can't be read and a task names its owner, instead of handing every owned task to Otto. The plan read and the plan hire keep the lenient read. - Each plan-hire try gets a 7 s deadline while another try still fits in the 15 s window, so a stalled request is sent again. The last try waits for its answer; the proxy's 30 s timeout ends it with the failure toast. - The plan step's settle retries stop once the step is left, so they can't land on a newer take. Paid checkout stays locked while the trial is starting. - Removing the last task on a touch screen moves focus to a composer button, not the text box that opens the keyboard. The folded list is inert while it closes. A roster read still running doesn't count against the task-switch give-up. - Checked: CI green on `32bac59b` (project coverage 90.61%, +0.04%). - [x] **Merged with dev** `88fefd94` (the conflicts were dev's design-token and Tailwind 4 refactors on files this branch deletes or rewrote) and `ef8ccdb6` (this branch's components on dev's design-system lint; see "Where this differs"). Final counts: | | Before (dev, Phase 0) | After (`6eda537b`) | After the dev merge (`ef8ccdb6`) | | --- | --- | --- | --- | | Frontend `pnpm test:unit` | 12,282 passed + 4 expected failures, 1,127 files | 12,471 passed + 4 expected failures, 1,135 files | 12,834 passed + 4 expected failures, 1,164 files | | Backend `onboarding_dump` + `experts` | 904 passed, 29 skipped | 1,073 passed, 29 skipped | 1,172 passed, 29 skipped (with `paywall_routes_test` and dev's `onboarding_role` and `onboarding_wizard` tests) | The deleted hire-step, card and trial-welcome tests are inside those numbers; the count went up by 189 regardless. `8badd23e` (after the full run) moves one ref write into an effect; the 622 onboarding tests pass on it. `d113cf2c` adds one backend test; `onboarding_dump` + `paywall_routes_test`: 526 passed. ### Before and after All screenshots, side by side, desktop (1440 × 900) and phone (390 × 844): **[Plan-first trial: before and after](https://claude.ai/artifact/N4aZGHKWh3Ug3upSVKNA83)** (a private Claude page: it opens for people it has been shared with). It covers: | Screen | Before (dev) | After (this PR) | | --- | --- | --- | | The step after the brain dump | Hire step, then the trial welcome | One plan + trial step (brain dump and skipped; paid and no-paywall variants; pending skeleton; 1280 × 720 and 1366 × 768 without scrolling; 320 × 640 with the sticky trial button) | | Preparing | Checklist | Checklist with "Hiring Maria, Max and Mina" | | Home under the greeting | Two-column cards | The ranked, owned list with the Your team row | | Home after the greeting | Hide all | Hide / Show, "{n} more tasks", a started task, the "Task removed" toast | | The walkthrough's last step | Lands on the first card | Lands on the row with the Start here pill | The screenshots come from `next dev` with every API call mocked (no backend, no Stripe here). The app renders light only (`forcedTheme`), so there are no dark captures. ### Changes 🏗️ **Backend (`onboarding_dump`)** - `plan.py`: one Jev call ranks the greeting's tasks and picks owners; every failure (no key, timeout, raise, cut state, malformed answer) returns None and the plan falls back. `plan_owners.py`: the deterministic fallback, the team cap of three, and owners matched by skill among the three the plan keeps. `role_skills.py`: a template's skills, read from the words of its role, so a renamed role still matches. `plan_job.py`: the last step of the background pipeline, a conditional write on `updatedAt`, retried, Jev asked once. `plan_service.py`, `plan_models.py`, routes: `GET /plan` (open while paywalled: it is shown on the paywall step), `POST /plan/hire` (checks the paywall, so a user who left Stripe without paying hires nobody, whatever the client sends). - `role_tasks.py` / `role_task_sets.py`: five starter tasks per wizard role, each naming the skills that take it, owned from the role's fallback team. `recommend_experts.py`: the fallback team is the roster's most focused template for each skill, starting with the role's first skill, then the user's pain points, then the role's other skills. Each role's skills cover its starter tasks. - `SuggestedPrompt` gains `expert_template_id`, `why`, `expert_name`; `SuggestedTask` gains read-time `expert_id`. `suggested_tasks.with_expert_ids` reads a started task's expert from its chat (`get_chat_session_expert_ids`), so firing that expert, or hiring the owner after Otto took the task, does not relabel the chat. `expert_hired` gains `via: plan`. **Onboarding (frontend)** - `SubscriptionStep/PlanTrialStep.tsx` with `PlanPanel` (polls every second and shows the server's progress as a checklist, settles the plan at 30 s, never swaps a shown plan, restores it on a Stripe cancel return), `TrialPanel`, `PaidPanel`. The trial date math moved to `TrialPanel/helpers.ts` with its tests. Deleted: `HireStep/`, `SubscriptionStep.tsx` and the trial welcome components (#16036) the new step does not use. - `store.ts` drops the hire step; saved progress that names it resumes on the plan step. - `usePlanHire.ts` + `usePreparingStep.ts`: one hire after confirmation, retried inside its window (sessionStorage guard against a remount), "Hiring {names}" or "Setting up your team" in the checklist, the bar holds up to 15 s and then goes on, roster queries refreshed. **Home (frontend)** - `SuggestedTaskList` + `components/TaskRow`, `TaskOwner`, `YourTeamRow`; `SuggestedTasksBlock` with the Hide / Show toggle (`useCollapsedSuggestions`, per user in `localStorage` under `home.suggestedTasks.collapsed`); `useSuggestedTasks` split into `useSuggestedTaskList` and `useTaskRecipient`. Deleted: `SuggestedTaskCard.tsx`, `SuggestedTaskCards.tsx`. - Sending: the composer's recipient becomes the owner (`?expertId=`) and the task goes out once the page can send; a switch that does not land in 3 s gives up with a failure toast and puts the recipient back. An owner who is not on the team: Otto takes it, with "{Name} isn't on your team, so Otto has this one." only when the roster actually loaded. - Accessibility: rows are named "Start with {Name}: {title}" / "Open the chat with {Name}: {title}" and described by their pill and why; a starting task is announced; focus moves to the neighbouring row on remove and back to the row when a start fails; inset focus rings; text and icons at AA contrast. **Tracking, docs, tests** - `posthog-events.ts`, `plan-analytics.ts`, `experts-analytics.ts`, `expert-thread-source.ts`; `docs/engineering/tracking-plan.md` updated (below). - `src/playwright/utils/onboarding.ts`: `completeOnboardingWizard` waits for `plan-panel` to lose `plan-panel-pending`, presses Continue, then connect or preparing. - Two new optional settings, both commented out in `.env.default`: `BRAIN_DUMP_GREETING_BACKUP_MODEL` (the greeting's fast backup) and `BRAIN_DUMP_PLAN_JEV_MODEL` (the Jev model; unset means the SDK default). No new feature flag, no data migration. `openapi.json` is regenerated (the generated client is gitignored). ### Tracking Both walks ran in the mocked app with every `posthog.capture()` call recorded (twice each, identical). App events only; `$autocapture` and `$pageleave` are left out. **The walk ran before `e075809d`**, which renamed two properties after the audit (see "Where this differs"); the lists show the names the code sends now, and the unit tests pin them. <details><summary>Path A: brain dump typed (20 app events)</summary> | # | Event | Properties | When | | --- | --- | --- | --- | | 1 | `onboarding_step_viewed` | `step: team` | /onboarding | | 2 | `onboarding_step_viewed` | `step: autopilot` | Next | | 3 | `onboarding_step_viewed` | `step: role` | Next | | 4 | `onboarding_step_viewed` | `step: pain_points` | role Next | | 5 | `brain_dump_typed_fallback` | `reason: chose_to_type, prompt_set: Founder/CEO` | Write | | 6 | `brain_dump_completed` | `input_mode: typed, chars: 283, prompt_set: Founder/CEO, prompts_seen: 1` | I'm done | | 7 | `paywall_viewed` | `surface: onboarding` | plan step opens | | 8 | `onboarding_step_viewed` | `step: subscription` | plan step opens | | 9 | `plan_viewed` | `plan_source: llm, skipped_dump: false, tasks: 3, experts: 3, wait_ms: 84, gave_up: false` | plan step opens | | 10 | `trial_offer_viewed` | `trial_offer_version: trial-v1, subscription_tier: PRO, trial_duration_days: 7, surface: onboarding` | plan step opens | | 11 | `onboarding_step_viewed` | `step: preparing` | Start my free trial → Stripe → back | | 12 | `plan_experts_hired` (backend) | `hired: 3, already_on_team: 0, failed: 0` | preparing | | 13 | `intro_path` | `path: A` | Home | | 14 | `home_suggestion_clicked` | `position: 0, surface: intro, remaining: 5, owner: expert, expert_template_id: tpl-maria, rank: 0` | task 1 | | 15 | `intro_followup_sent` | `chars: 115` | task 1 | | 16 | `expert_thread_created` | `expert_id: exp-maria, thread_source: suggested_task` | task 1 | | 17 | `home_viewed` | — | back to Home | | 18 | `home_suggestion_dismissed` | `position: 1, surface: home, remaining: 4, scope: one` | x on task 2 (Undo sends nothing) | | 19 | `home_suggestions_collapsed` | `open: 5` | Hide | | 20 | `home_suggestions_expanded` | `open: 5` | Show | </details> <details><summary>Path B: brain dump skipped (19 app events)</summary> | # | Event | Properties | When | | --- | --- | --- | --- | | 1–4 | `onboarding_step_viewed` | `step: team`, `autopilot`, `role`, `pain_points` | as in A | | 5 | `brain_dump_skipped` | `prompt_set: Founder/CEO, prompts_seen: 0` | Skip for now | | 6 | `paywall_viewed` | `surface: onboarding` | plan step opens | | 7 | `onboarding_step_viewed` | `step: subscription` | plan step opens | | 8 | `trial_offer_viewed` | `trial_offer_version: trial-v1, subscription_tier: PRO, trial_duration_days: 7, surface: onboarding` | plan step opens | | 9 | `plan_viewed` | `plan_source: fallback, skipped_dump: true, tasks: 3, experts: 2, wait_ms: 103, gave_up: false` | plan step opens | | 10 | `onboarding_step_viewed` | `step: preparing` | Start my free trial → Stripe → back | | 11 | `plan_experts_hired` (backend) | `hired: 2, already_on_team: 0, failed: 0` | preparing | | 12 | `intro_path` | `path: B` | Home | | 13 | `home_suggestion_clicked` | `position: 0, surface: intro, remaining: 4, owner: expert, expert_template_id: tpl-max, rank: 0` | task 1 | | 14 | `intro_followup_sent` | `chars: 63` | task 1 | | 15 | `expert_thread_created` | `expert_id: exp-max, thread_source: suggested_task` | task 1 | | 16 | `home_viewed` | — | back to Home | | 17 | `home_suggestion_dismissed` | `position: 1, surface: home, remaining: 3, scope: one` | x on task 2 (Undo sends nothing) | | 18 | `home_suggestions_collapsed` | `open: 4` | Hide | | 19 | `home_suggestions_expanded` | `open: 4` | Show | </details> None of the removed events appeared. Sent by the backend, so not in a browser walk here: `signup_completed`, `checkout_started` (`checkout_kind: trial, surface: onboarding`), `trial_started`, `trial_ending`, `trial_canceled`, `trial_converted`, `payment_succeeded`, `expert_hired` (3 in A, 2 in B, `surface: onboarding, via: plan`). Not reached by the walk's actions, covered by unit tests: `plan_selected` (paid path, `PlanTrialStep.paywall.test.tsx`), `checkout_abandoned` (cancel return), `welcome_dialog_closed` (the walkthrough was marked seen). `capability_card_*` were retired before this branch (SECRT-3149) and stay reserved. **Removed with the hire step:** `expert_recommended`, `expert_recommendation_clicked`, `hire_all_clicked`, `hire_step_continued` (in `RETIRED_EVENT_NAMES`, rows in the tracking plan's Removed table). `hire_started` loses its `onboarding` surface (the plan's hires have no click); it stays for `expert_page`. **After deploy:** four saved PostHog insights read the removed events (u2ZK2Isk, QfcpXRmM, 2OSZpNcv and ugcxrNQU). SECRT-3257 has the steps to freeze each one at the release and the tested queries for the plan-first versions, to do once this is live on prod. ### Langfuse Tracked in SECRT-3225, to do before this reaches prod. The greeting's instructions come from the Langfuse prompt **Brain Dump Greeting** (`BRAIN_DUMP_GREETING_PROMPT_NAME`) whenever it is reachable; the local copy in `intro.py` is only the fallback. The local copy now asks for one optional key per suggested task: > `"why"` (optional): one short clause, at most 60 characters, second person, naming the thing they said this task fixes ("Because listings eat your evenings."). Leave the key out when nothing they said ties to the task. The same line needs adding to the Langfuse copy, or every task in prod arrives without a why line (nothing breaks; the line is just absent). The parser keeps a `why` of 1 to 60 characters and drops a longer one rather than cutting it. Nothing else in the greeting prompt changes. ### Settled decisions - The paywall is replaced entirely by the plan + trial step; `useSubscriptionStep`, `useTrialCard`, `tracking.ts`, the trial date math (moved, tests kept) and the Stripe return handling stay. - Tasks are ranked and assigned by TypeSafe Jev with a deterministic fallback. Jev is a hosted model call: the request carries the role, pain points, tasks, team and up to 6,000 characters of the transcript, with a 5.5 s budget. - Experts are hired on the preparing screen after the trial is confirmed; a user who abandons Stripe hires nobody (with no paywall, when the user continues past the plan). - The list is the design, the #16236 cards are removed; x removes a task, Hide all became a collapse. ### Where this differs from the handoff, and why - **`plan_source` and `thread_source`, not `source`.** `docs/engineering/tracking-plan.md` reserves `source` for the emitter: every browser event carries `source: web` as a super property, and an event property of the same name replaces it. The repo settled this clash before with `team_source` and `workflow_source`, so `plan_viewed` sends `plan_source` and `expert_thread_created` sends `thread_source`. - **A started task keeps naming who has it.** Edge case 11 (a fired expert's tasks go to Otto) holds for open tasks. A task already started links to a chat with that expert, so its row keeps their name; the server reads it from the chat. Without this, firing Maria relabelled her chat as Otto's, and hiring an owner after Otto took a task relabelled Otto's chat. - **Contrast.** The spec's "muted" grey (`#83838C`) is under 4.5:1 for small text, so the why line, the role, the sent rows' title and the toggle use `foreground-secondary`, and "In progress with {Name}" uses `success-foreground` rather than `success`. - **Undo focus.** Focus moves to the neighbouring row on remove. After Undo, the toast (sonner) returns focus to where it was before the toast, which is that neighbour, rather than to the restored row; fighting the toast's own focus return would make focus jump twice. - **The brain dump step** gained one read-only accessor (`whenSkipSettled`), so the plan read waits for a Skip still in flight (`7fb3dce4`). Its behaviour is unchanged. - **Module split.** The backend plan lives in seven small modules instead of one `plan.py`, because backend AGENTS.md caps files at about 300 lines. - **Jev request budget.** Owner candidates are the recommendation's first 6, each task's prompt is cut to 600 characters / 1,200 bytes, and the transcript to 6,000 characters / 12,000 bytes, then 6,000 and 3,000 bytes if the state still does not fit. A character-only cut sent every long Japanese or Chinese dump to the fallback. - **The plan API's `source` is `llm`** only when the greeting wrote the stored tasks. A completed dump whose greeting fell back to the generic starters gets the role set (owned from the dump's team) and reads as `fallback` with no "heard" line. - **Pending plan.** While the pipeline runs, `GET /plan` answers `ready: false` with the role plan, unstored, and `progress` (tasks written, team chosen). A voice take still transcribing counts as pending. At 30 s the step calls `POST /plan/settle`, which stores that role plan as the take's tasks and team unless a plan is already stored; see "The plan step's wait". - **Reads that store** are built from reads that raise, so a failed template or wizard read fails the request (the step keeps polling) instead of storing a degraded list for good. - **Ids.** Ranked rows keep their original position as `id`, so a task keeps the id Home may already have handed out. Each owned task stores `expert_name`, so the Otto toast can name an expert who has left. - **Hire while pending (no paywall only).** A hire that arrives before the pipeline finishes settles the plan first, so the team hired is the role team whose tasks Home then lists; with the paywall the Stripe round trip makes this window practically zero. - **Archived experts.** A repeat plan hire can bring back an expert the user archived, but only on an onboarding retake whose new plan names them again. - **Merged with dev's design-system lint (#16402, #16429).** The new rules fail any violation in a file the baseline doesn't list, so this branch's new components now use the primitives' own looks (`ef8ccdb6`). Visible differences from the screenshots: avatars keep the design system's own shape (rounded-xl) instead of a full circle, as dev's onboarding does; the row's remove button is the Button's `toggle` variant, quiet grey with no hover fill; the sending spinner is `LoadingSpinner` (16px); the two 48px CTAs set their label at 15px (`textSize="body-md"`) instead of 16px; the step card's corner is 24px instead of 28px; the Start here pill is 11px with `tracking-eyebrow`. The baseline only goes down (the task list 13 → 1, the block 9 → 3, the deleted card's entry gone). Dev's role-draft test (#16398) now steps from the brain dump straight to the plan step. - **Commit trailer.** AGENTS.md asks for a `Co-authored-by` trailer naming the model and the platform; the commits use the session's trailer with the platform added. ### The plan step's wait Found testing the preview: the step waited for the whole brain-dump pipeline. That was the Sonnet greeting with adaptive thinking on, the team and integration picks, the memory extraction, the Jev ranking, then the dump's `completed` status. Past the old 20 s give-up, the step showed the role plan while the hire and Home used the pipeline's, so the team named on the step could differ from the one hired. Fixed in this PR: - **Less to wait for.** The greeting call answers without thinking first (`intro.py`, reasoning off like the pipeline's other calls, `max_tokens` 4,000). The team job asks for the six experts the plan reads (`plan.MAX_OWNER_CANDIDATES`), not sixteen. Only the greeting, team and plan jobs end the step's wait. The provider picks and the memory extraction run beside them and write their own columns; a failed understanding write now costs the context, not the take. - **Progress, not skeletons.** The pending answer carries `progress`. The step shows it as a checklist where the heard card will be: "Turning your brain dump into tasks", "Finding experts who fit your work", "Choosing who takes each task". It polls every second (was 2.5 s) and announces "Your plan is ready" from a live region outside the busy skeleton. - **A plan the step gives up on is the plan.** The step gives up at 30 s (was 20 s). It then calls `POST /plan/settle`, which stores the role plan it showed as the take's tasks (rows with ids and states) and team. The greeting still writes its text, but leaves tasks already stored alone (`db.store_greeting`, conditional on an empty list). The team job leaves a stored team alone (`db.replace_team`, a conditional write on `updatedAt`, because a reset take holds a JSON null and an older row a SQL null). The plan job finds nothing to rank. So the trial hires, and Home lists, what the step showed. The cut-off moved to 30 s because giving up now costs the plan the brain dump would have written. - **Back and re-record.** The remembered plan (for a Stripe cancel return and the preparing screen's label) is forgotten whenever a new take or skip is sent. - **Measuring it.** The preview sends nothing to PostHog, so the new wait isn't measured yet. The backend now logs each leg ("Brain dump greeting for user … took Ns", "Brain dump team for user … took Ns", "Brain dump plan ready for user … Ns after the pipeline started"). `plan_viewed`'s `wait_ms` and `gave_up` give the same in PostHog after release. `gave_up` now means the 30 s wait ran out, and `plan_source` says whose plan was shown. - **Tests.** BE: `service_test.py` (the plan is ready before the provider picks and before the understanding is stored, a failed understanding write costs the context, tasks and team stored while the jobs ran stand, a lost team write is retried), `routes_test.py` (pending progress, settling mid-pipeline, settling a ready plan changes nothing, settling a take still being transcribed reads as recorded, hiring while the jobs run settles first, and the end-to-end "a plan the step gave up on is what the trial hires and Home lists"), `suggested_tasks_db_test.py` (the conditional writes against Postgres), `intro_test.py` (reasoning off). FE: `usePlan.test.tsx` (1 s polls, settles at 30 s, falls back when the settle fails, waits for a skip in flight, keeps a plan that lands during the settle), `PlanPanel.test.tsx` (the checklist and the announcement). ### Not run here, and notes - **Stripe test mode and a real `posthog.debug()` walk.** This environment has no backend stack or Stripe account. The flow was walked in `next dev` with every call mocked (Stripe Checkout intercepted to the success return), with PostHog captured in the page. The Playwright smoke and journeys suites ran in CI and passed with the updated helper. - **CI: two `experts_db_test` failures from skills-catalog `main`.** Catalog PR #24 (30 Oct 15:50 UTC) changed day-one rows and roles, which broke `test_wave_three_experts_are_skills_only_with_ordered_packs` and `test_roster_day_one_promises_match_work_the_expert_can_do_on_request` on every branch ([details](https://github.com/Significant-Gravitas/AutoGPT/pull/16308#issuecomment-6527907044)). Dev fixed them in #16443, merged here in `8d81826a`. - **Out of scope, seen in passing.** `home_viewed` does not fire on the first Home visit, because `HomeRecap` mounts only once the greeting has retired; the same on dev. The app sidebar logs a hydration mismatch on some loads (its mobile/desktop branch differs between server and client); the dev captures show it too. - **`POST /api/experts` stays open while paywalled.** Its entry in `paywall_routes_test.py` dates from when onboarding hired the team before the plan picker. Onboarding no longer calls it; whether it should now check the paywall is a separate decision, so I only updated the comment. - **The greeting fell back on the preview.** In the product manager's test, Home showed the generic greeting ("Thanks for talking me through your work — I've got it…"). That means the greeting call failed twice, and the plan then took the Product/PM starter set instead of tasks written from the brain dump. The owner fix above makes that set the team's work, but the tasks still come from the role, not from what the user said. The cause is not visible from here: no model keys in this environment. The preview backend logs it as `Brain dump greeting generation failed (attempt N): <ErrorType>: <message>`, now with the type. - **`Other` and the team of three.** Every plan now starts with three experts (`d6ea5524`). For Other, or no role, the user's pain points lead, then the experts the Other starter set needs (Quinn, Maya, Nadia), so its research, post, summary and watch tasks have owners; its first task, finding the work to hand off, stays Otto's. On a thin roster, any seat still empty is filled from general skills and never with a role that names no skill, so a filler may own no task. ### Edge cases Each of the handoff's 16 cases, with the tests that cover it. Paths: **BE** `backend/api/features/onboarding_dump/`, **OB** `frontend/src/app/(no-navbar)/onboarding/`, **PP** `OB/steps/SubscriptionStep/components/PlanPanel/__tests__/`, **ST** `frontend/src/app/(platform)/copilot/components/SuggestedTasks/__tests__/`. | # | Case | Covered by | | --- | --- | --- | | 1 | Skipped brain dump | BE `routes_test.py::test_a_skipped_dump_gets_the_role_plan_stored_on_first_read` (fallback, `skipped_dump`, Founder team, Home gets the same titles and ids); PP `PlanPanel.test.tsx` "speaks to the role for a skipped dump" | | 2 | Dump still generating, 30 s give-up settles the plan, no swap | PP `usePlan.test.tsx` "settles the plan at 30 s and shows the plan the server settled on", "keeps the settled plan when a poll in flight at 30 s lands later", "keeps a plan that lands while the settle is on its way"; BE `test_the_plan_is_pending_while_the_pipeline_runs` (a pending read stores nothing), `test_a_plan_the_step_gave_up_on_is_what_the_trial_hires_and_home_lists`. A *failed* dump answers at once with the role plan (`test_a_failed_take_plans_and_hires_from_the_role_not_the_team_it_stored`). | | 3 | Jev unset, timeout, error, cut state | BE `plan_test.py` (no key, timeout or raise, slow call, failed/empty/cut call, oversized state all return None; a timeout or raise logs one warning), `plan_job_test.py::test_without_jev_the_greeting_order_is_kept_with_hinted_owners`. An unset key warns once at import, like the copilot gate. | | 4 | Zero experts | BE `plan_owners_test.py::test_no_recommended_experts_leaves_every_task_with_otto`, `routes_test.py::test_a_plan_with_nobody_on_its_team_hires_nobody`; PP "leaves the tasks with Otto when nobody joins"; OB `usePreparingStep.test.ts` "says it is setting up the team when the plan has nobody to hire" | | 5 | Fewer / more than three | PP "shows fewer cards when the plan has fewer tasks"; ST `suggested-tasks.test.tsx` "keeps the rows after the first three under a closed '2 more tasks' row that opens on click" | | 6 | Owner outside the capped team | BE `plan_owners_test.py::test_a_team_of_four_owners_keeps_three_and_nulls_the_fourth`, `plan_job_test.py::test_a_jev_owner_left_off_the_capped_team_goes_to_otto` | | 7 | Back and re-record | PP `usePlan.test.tsx` "does not show a plan cached before the mount while the refetch is pending", "reads the plan only once a skip still on its way has landed"; BE `service_test.py::test_a_new_take_never_serves_the_previous_takes_answers` | | 8 | Stripe cancel return | PP "shows the remembered plan at once on a checkout return", "does not report a restored plan as a second view"; OB `PlanTrialStep.paywall.test.tsx` "reports checkout_abandoned when Stripe sends the user back without paying"; `usePreparingStep.test.ts` "never hires before the trial is confirmed"; BE `routes_test.py::test_a_user_still_behind_the_paywall_hires_nobody` | | 9 | No trial offer | OB `PlanTrialStep.paywall.test.tsx` "offers one Pro checkout beside the plan…"; PP "names the moment the team joins" ("joins your team when you subscribe") | | 10 | Hire fails for one or all | BE `test_a_failed_hire_is_reported_and_the_rest_still_join`; OB "never blocks Home on a failed hire, and reports it", "moves on after the ceiling when the hire never answers"; ST "shows each row's owner… and Otto… for an expert not on the team" | | 11 | Expert fired between visits | BE `suggested_tasks_test.py::test_each_owned_task_names_its_hired_expert`, `test_a_started_task_stays_with_the_expert_its_chat_is_with_once_fired`; ST "gives Otto a task whose expert has left the team, says so, and shows it in progress with Otto", "keeps naming the expert a started task went to after that expert is fired". The toast fires when such a task is started, once (the row then becomes a link). | | 12 | Old progress with `hire` | OB `progress.test.ts` / `page.test.tsx` "resumes a draft saved on the retired hire step at the plan step"; `store.test.ts` "never lays out the retired hire step" | | 13 | Old rows without owner or why | BE `suggested_tasks_db_test.py::test_prompts_stored_before_owners_existed_still_rehydrate`, `intro_test.py::test_a_prompt_keeps_its_why_and_one_without_still_parses`; ST "shows a task's why under its title, and nothing there when it has none" | | 14 | Role `Other` | BE `plan_owners_test.py::test_other_and_unknown_roles_take_the_other_set`; PP "keeps the line role-free for Other". See the note on `Other` above for the team. | | 15 | Motion, keyboard, viewports, phones, collapsed list | PP "offers nothing to click or focus"; ST `SuggestedTasksBlock.test.tsx` "still hides and shows the list when the user prefers less motion", "keeps the list folded for the same user after a reload…", `helpers.test.ts` "never collapses the list for a signed-out visitor…". By reading the code and in the screenshots: tab order on the step is Back, Start my free trial, Log out (DOM order; the progress bar has nothing focusable); 1280 × 720 and 1366 × 768 fit without scrolling and phones keep the sticky trial button (screenshots); rows stack under `sm` with the x always visible on touch; the collapse is a per-user key that sign-out does not clear. | | 16 | Expert limit mid-hire | BE `test_the_expert_limit_stops_the_hire_and_reports_the_rest`, `test_the_expert_limit_on_the_second_hire_keeps_the_first_and_reports_the_rest`; Home's Otto fallback is case 10's path | Screen reader and keyboard: rows announce "Start with Maria: {title}" and, once sent, "Open the chat with Maria: {title}" (ST); the toggle reports expanded and collapsed (ST); the walkthrough's Start here step lands on the row with the pill (`first-run-tour.test.tsx` "starts on a task's own button, never its Remove button"); focus after Remove goes to the next row, its Remove while locked, or the composer when the list goes (ST "keyboard and screen readers", `SuggestedTasksBlock.test.tsx` "focus after Remove"). ### Ticket lines Resolves SECRT-3201 Resolves SECRT-3132 Resolves SECRT-3131 Progresses SECRT-3134 (Start here and ranking shipped; account requirements per task still open) Progresses SECRT-3142 (owner per task shipped; Otto proposing delegation in chat still open) Related SECRT-3211 (what happens to a task after its chat finishes), out of scope Related SECRT-3212 (integrations on suggested tasks), out of scope Related SECRT-3225 (the Langfuse why line, before release) and SECRT-3257 (the PostHog insights, after release) ### Agents and large language models used - Claude Code (model reported in the commit trailers) ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [ ] I have tested my changes according to the test plan (all but a Stripe test-mode run, below): - [x] Backend: `poetry run pytest backend/api/features/onboarding_dump backend/api/features/experts backend/api/paywall_routes_test.py` plus dev's `onboarding_role` and `onboarding_wizard` tests: 1,172 passed, 29 skipped after the dev merge. `export-api-schema` matches the committed `openapi.json`. isort, black and pyright clean on every changed file. - [x] Frontend: `pnpm lint` and `pnpm types` exit 0 on the final head (the only warnings left in files this PR touches predate it); `pnpm test:unit`: 12,834 passed + 4 expected failures after the dev merge, including the design-system baseline tests. React Compiler coverage and bailout tests pass with no new bailouts. - [x] Each commit checked on its own in a clean worktree (types and the suites it touches). - [x] Screenshots of every changed screen, desktop and phone, plus 1280 × 720, 1366 × 768 and 320 × 640 for the plan step (linked above). Mocked `next dev`; light theme only. - [x] Both onboarding paths walked to Home with every PostHog `capture()` recorded, twice each (lists above). - [x] Two review passes with independent verification of each finding; the confirmed ones are fixed in `3782dfc4` (a started task's owner), the give-up fix in `130b92ad`, and `6eda537b` (focus after Remove). - [x] Playwright smoke and journeys suites: passed in CI with the updated `completeOnboardingWizard`. - [ ] Stripe test-mode run: not possible in this environment (no backend stack, no Stripe). #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes (two new optional model settings, commented out; the Jev key is the existing `TYPESAFE_API_KEY`) - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**): none No data migration is required: the new fields are optional, and rows written before this change read as Otto-owned with no `why`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01G58r45FBHQQQwJGu1SgbxC --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Krzysztof Czerwinski <[email protected]> |
||
|
|
9cb76f9026 |
docs(platform): Changelog v0.11.0 (#16651)
Resolves SECRT-3392. **Merge after #16315** (v0.10.0): this PR is stacked on its branch, and GitHub retargets it to `dev` once #16315 merges. ### Why / What / How **Why:** v0.11.0 shipped on November 4. The login page's What's new panel (#16326) and the in-app What's new card both read the changelog, so they show v0.10.0 as the latest until this lands. **What:** a standalone entry for v0.11.0, `docs/platform/changelog/november-4-2026.md`, listed in `docs/platform/SUMMARY.md` and at the top of the table in `docs/platform/changelog/README.md`. The What's new card shows the first three highlights: "Connect tools from the chat", "The login page shows what's new", "A friendlier first email". **How:** - Frontend changes people can see, written from each merged PR's description. 28 links, all merged PRs in v0.11.0. - Security work is never included in the changelog (a standing rule), so none of this release's security PRs are described or linked. - Flags checked in PostHog (project "AutoGPT Platform (PROD)") on 2026-11-05. Left out: background subtasks (#15733), behind `copilot-background-subtasks`, which is employees only. The running `trial-checkout-no-charge-copy` experiment is unchanged and nothing here depends on it. Spend approvals (#16352, #16303) are in because `expert-spend-approval` was retired as on for everyone. - Also left out: the single Microsoft connection (#16268), since what runs on it today is the Microsoft 365 Copilot chat connection, which we keep out of the changelog with the other bring-your-own subscriptions; the GitHub star prompt for self-hosted installs (#16527); the payment and proxy settings for self-hosters (#16459, #16522); the LaunchDarkly and Langfuse removals (#15853, #16026); the design-system and Tailwind work that changes no pixels; bot-only changes, admin, analytics and test work. - Copy follows the user-facing rule in `AGENTS.md` (#15502). No em dashes. **Heroes:** - `v0110-connect-hero.gif`: filmed on production in a signed-in session. `@hubspot` typed in the Home chat box (never sent), Connect an integration, then the Connect a service dialog and its Popular section. The Home recap behind the first frame is blurred because it holds internal notes. - `v0110-login-hero.gif`: filmed headlessly on the live login page, then a click through to signup. - `v0110-confirm-email-hero.png`: the after-render from #16279, cropped above the "Button not working?" box so no verification link appears. ### Changes 🏗️ - `docs/platform/changelog/november-4-2026.md`: the new entry - `docs/platform/.gitbook/assets/`: the three heroes above - `docs/platform/SUMMARY.md`: the entry under What's New - `docs/platform/changelog/README.md`: the row at the top of the table ### Agents and large language models used - Claude Code with Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] `python -m unittest discover -s .github/workflows/scripts -p "test_*.py"` passes - [x] Every PR link in the entry is a merged PR in v0.11.0, and each is in the release tag - [x] No security items, no em dashes, no "agent", "graph", "AutoPilot" or "Copilot" in the copy - [x] Flags and experiments checked against production PostHog on 2026-11-05 #### For configuration changes: No configuration changes. --- _Generated by [Claude Code](https://claude.ai/code/session_018TYcprig4Ui3qGu5smRtmN)_ --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Bentlybro <[email protected]> |
||
|
Toran Bruce RichardsandClaude Opus 5.5
|
30a2ba1221 |
feat(blocks): add Treg blocks for people, company and web data (#16393)
### Why / What / How **Why:** [Treg](https://treg.to) is "OpenRouter for agent tools": one token reaches 4,000+ API endpoints across ~120 providers (Apollo, Hunter, People Data Labs, DataForSEO, Exa, scrapers and more), billed per call at the provider's price from a prepaid balance, with no provider signup. Its headline use is people search and lead enrichment ([launch post](https://x.com/treg_ai/status/2114674262179188755)). AutoGPT only lists Treg as an MCP server today, so workflows can't use it as typed steps. Closes OPEN-3790. **What:** A `treg` provider (API key, `TREG_API_KEY`) and 21 blocks: - **People and contacts:** Search People, Find Work Email, Find Personal Email, Find Phone Number, Verify Email, Enrich Person, Find LinkedIn From Email - **Companies:** Search Companies, Enrich Company, Find Similar Companies, Find Company Jobs - **LinkedIn:** Get LinkedIn Profile, Get LinkedIn Company - **Web:** Web Search (web, news, Google, Google News, research papers, YouTube), Google Maps Search, Extract Web Page, Extract Structured Data, Map Website - **Catalog:** Search Catalog and Get Tool (free: find a tool, its inputs, price and every alternative provider), and Call Tool (call any catalog tool, or one your Treg team registered) **How:** The typed blocks call Treg's *routed* tools (`treg.<capability>`), which try providers cheapest first (your team's own keys first) and fall through on a miss. A shared `TregBlockBase` sends each call through the SSRF-safe `Requests` wrapper and records the charge Treg reports. - **Misses are not errors.** Every routed block emits `found` (false when nobody had an answer), plus `served_by`, `providers_tried`, `cost_usd`, `call_id`, Treg's `advice` and the provider's `raw` answer. Typed fields are emitted only when present and of the declared type, so a provider's `"500+"` never fails a run. - **Spending is capped.** Every paid block has `max_cost_usd` (default $0.50, max $25; Call Tool defaults to $1, Treg's own per-call default), sent as `X-Treg-Route-Max-Cost`, and `try_more_providers` to stop after the first provider. Calls are never retried, because a repeated paid call can bill again. A run that names nobody (e.g. a name without a domain) fails before any call. - **Found is not confirmed.** Find Work Email and Find Personal Email check the mailbox in the same call by default (`X-Treg-Route-Verify`), and `verified` is true only when that check says `valid`; a catch-all comes back as `catch_all`. - **Billing:** `X-Treg-Cost-Micro` (or `X-Treg-Reserved-Micro` for a call still pending) is recorded as `provider_cost` before any error is raised, Treg blocks run on the user's own Treg key only, which the platform doesn't bill. The platform's shared `treg-default` key is refused, because `treg` isn't in `RUNNABLE_SDK_DEFAULT_PROVIDERS` (the 1 Nov rule for vendors that may keep per-account state). The `COST_USD` base cost (150 credits/USD) would only apply if Treg is added to that list after checking it's safe. - **Call Tool safety:** tool references that could leave `/call/` (`..`, encoded dots or slashes, query or fragment) are refused before sending. Caller `X-Treg-*` headers other than the route options are refused, case-insensitively, so the spending cap can't be overridden. On the platform's key only catalog ids are accepted, since that key's Treg team is shared by every user. Redirects are off so the token can't be forwarded. - **Errors** carry Treg's code and message (e.g. `insufficient_balance` with the top-up link, or a hint to check the key on a 401). ### Changes 🏗️ - New `backend/blocks/treg/` package: provider config, HTTP client, shared base and schemas, 21 blocks in 9 modules, unit tests - `.env.default`: adds blank `TREG_API_KEY=` - `mcp_catalog.json`: the existing `mcp_treg` entry gets `"provider": "treg"`, so a Treg MCP connection files under the same service as the new API-key credential (the field exists for exactly this, as on `mcp_apollo`) - Generated block docs under `docs/integrations/block-integrations/treg/` with the manual sections written; README and SUMMARY updated by the generator - No frontend change: `frontend/public/integrations/treg.png` already exists **Screenshots.** After: searching "Treg" in the builder lists the new blocks (captured by the review bot's sandbox on `6ec1ad2`):  Before: no before screenshot. On `dev` the builder has no Treg blocks; Treg is only an MCP server in the MCP catalog. This PR changes no frontend code, and the blocks render from their schemas like every other SDK block. Found and left out of scope: - Polling an asynchronous generation (video, image) to completion. Call Tool returns the task descriptor as `async_task`. - Not yet run with a real Treg key (one is being set up); see the test plan. ### Agents and large language models used Claude Code with Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] `test_available_blocks` and the other per-block registry tests pass for all 21 blocks (mocked answers) - [x] `backend/blocks/treg/test_treg_*.py` (86 tests): request path, body and spending headers per block; misses as `found: false` and still billed; charges recorded before a failed call raises; pending calls billed at the reservation; nothing sent when no subject is given; email `verified` only on a `valid` check; each Web Search source routed correctly; Call Tool method, async task, refused bad tools, refused Treg headers in any case, platform key limited to catalog ids, failure inside a 2xx; LinkedIn references sent as URLs only for LinkedIn hosts; off-type and non-string values left out; error messages for each error-body shape; tool-reference validation; type coercion - [x] Client asserted: `X-Treg-Token` only, one attempt, no redirects, timeout, query flattening, reserved headers dropped - [x] Live against treg.to: Search Catalog and Get Tool return real tools, prices, success rates and 40 alternative providers; a routed call and Call Tool with a fake key return `HTTP 401 ... invalid token` - [x] `generate_block_docs.py --check` in sync; black, isort, ruff and pyright clean on the package - [x] `backend/copilot/capabilities` tests pass (Otto's capability search still finds the experts for "someone to run my LinkedIn") - [ ] Run with a real Treg key #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]> |
||
|
John AbabsehandClaude
|
0865d245d2 |
docs(platform): Changelog v0.10.0 (#16315)
Resolves SECRT-3216. Rebuilt on `dev` after #16020 (v0.9.0) merged. **Merge before the v0.11.0 changelog PR**, which is stacked on this one. ### Why / What / How **Why:** v0.10.0 shipped on October 27. Once #16020 merges, the public changelog and the in-app What's new card stop at v0.9.0. **What:** a standalone entry for v0.10.0, `docs/platform/changelog/october-27-2026.md`, listed in `docs/platform/SUMMARY.md` and at the top of the table in `docs/platform/changelog/README.md`. The What's new card shows the first three highlights: "A brain dump that helps you answer", "A calmer trial welcome", "A guided first run". **How:** - Leads with the brain dump step (#16252), then the new trial welcome (#16036, kept short since it changes again soon), then the frontend improvements from Abhi's PRs: the first-run walkthrough (#16107), hire and start in one click (#16101), the usage and credits explainer (#16095) and the Airtable base picker (#16094). 39 links, all merged PRs in v0.10.0. - Security work is never included in the changelog (a standing rule), so none of this release's security PRs are described or linked. - Flags checked in PostHog (project "AutoGPT Platform (PROD)") on 2026-10-27. Nothing listed depends on a flag that is off for the public. Left out: the "no charge today" line on trial checkout (#16050, #16143), a running 50/50 experiment (`trial-checkout-no-charge-copy`); the approval preview for Airtable changes in #16094, which only shows with `copilot-auto-mode` (employees only). - Also left out: the Google Map in chat (#15939), which only renders when a Maps key is set in production and that isn't confirmed yet; picking an Expert or mode from the Discord and Telegram bots (#15863, not in the web app, and the smart mode is plan-restricted); External API v2 (#12206, developer-facing); design-system and React Compiler refactors that change no pixels; admin, analytics, test and self-hosting work. - Copy follows the user-facing rule in `AGENTS.md` (#15502). No em dashes. **Heroes** (all six in place): - From production, filmed in a signed-in session: the first-run walkthrough (`v0100-first-run-hero.gif`, Show me around, with the Home recap behind the dim blurred because it holds internal notes) and the usage explainer (`v0100-usage-explainer-hero.gif`, the wallet, then How it works). - From the screenshots in the PRs, because production only shows these screens to a new account or mid-task: the brain dump (`v0100-brain-dump-hero.gif`, from the renders in #16252), the trial welcome (`v0100-trial-welcome-hero.png`, #16036), the hire card before and after Hire and start (`v0100-hire-handoff-hero.png`, #16101) and the Airtable base picker (`v0100-airtable-picker-hero.png`, #16094). ### Changes 🏗️ - `docs/platform/changelog/october-27-2026.md`: the new entry - `docs/platform/.gitbook/assets/`: the six heroes above - `docs/platform/SUMMARY.md`: the entry under What's New - `docs/platform/changelog/README.md`: the row at the top of the table ### Agents and large language models used - Claude Code with Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] `python -m unittest discover -s .github/workflows/scripts -p "test_*.py"` passes - [x] Every PR link in the entry is a merged PR in v0.10.0 - [x] No security items, no em dashes, no "agent", "graph", "AutoPilot" or "Copilot" in the copy - [x] Flags and experiments checked against production PostHog on 2026-10-27 - [x] All six heroes in place #### For configuration changes: No configuration changes. https://claude.ai/code/session_018TYcprig4Ui3qGu5smRtmN --- _Generated by [Claude Code](https://claude.ai/code)_ Co-authored-by: Claude <[email protected]> |
||
|
Toran Bruce RichardsandClaude Opus 5.5
|
70d0b65e5e |
feat(blocks): bring the Exa integration up to date with Exa's current API (#16395)
### Why / What / How
**Why:** Our Exa integration had fallen behind Exa's API, and part of it
was broken.
- All four Exa Research blocks fail on every run. Exa retired
`/research/v1` on 2026-04-01, and it now answers `410 RESEARCH_RETIRED`.
- Exa's newer products (the Agent API and Search Monitors) had no blocks
at all.
- The Search, Contents and Answer blocks offered search types and
categories Exa has since replaced, and lacked its current options.
- The Websets webhook trigger has never worked. It registered webhooks
at a URL that returns 404, sent Exa the wrong event names, and read the
wrong payload fields.
Linear: OPEN-3789.
**What:** Every current Exa endpoint relevant to workflows now has a
block. Existing blocks expose Exa's current options. Saved workflows
keep working: no block ID, input name, output name or enum value was
removed or renamed.
**How:** The new endpoints go through one small HTTP client
(`_exa_rest.py`) instead of `exa-py`. The installed SDK rejects values
Exa now returns (`pending`, `canceled`, `failed`), drops response fields
we need (`output`, `warnings`, `entities`), and checks options against
an allow-list that lags behind the API. Errors carry Exa's own message
and tag.
### Changes 🏗️
**Broken blocks fixed**
- The four Research blocks run again, on Exa Agent (`/agent/runs`), with
the same IDs, inputs and outputs.
- Model maps to effort: fast→low, standard→medium, pro→high.
- Statuses map back to the old vocabulary.
- `cost_pages` and `cost_reasoning_tokens` are still emitted, as 0,
because Exa Agent doesn't report them.
- The Websets trigger now works and is enabled:
- correct URL (`/websets/v0/webhooks`)
- Exa's own event names
- Exa's payload fields
- the `webset_id` filter is actually applied
- five new `monitor.*` events, off by default
**New blocks (30)**
- **Exa Agent (8):** run (waits by default), get, wait, list, cancel,
stop, delete, list events.
- Covers all seven efforts with their prices, output schemas with
field-level citations, input and exclusion records, follow-up runs, Exa
Connect data sources, and cost and duration budgets.
- **Search Monitors (10):** create, get, list, update, delete, batch,
trigger now, list runs, get run.
- Plus a **Search Monitor Trigger** that starts a workflow on each
delivery and checks Exa's signature once the monitor's secret is pasted
in.
- **Websets (12):**
- webhook create/get/update/delete/list
- webhook delivery attempts
- event list/get
- Websets monitor runs list/get
- import update
- team concurrency info
**Existing blocks brought up to date**
- **Search:**
- Types: `auto`, `lite`, `deep-lite`, `deep`. The legacy `fast`,
`instant`, `deep-reasoning`, `neural` and `keyword` are kept, labelled
with their replacements.
- Categories: `people` and `publication` added. `research paper` is sent
as `publication` and `linkedin profile` as `people`, per Exa's
changelog.
- New options: structured output (`output_schema` → `output_content` /
`output_grounding`), `system_prompt`, `additional_queries`, `objective`,
`user_identifier`.
- Outputs Exa's `warnings` and `notices`.
- Checks lite's parameter allow-list before calling, instead of
surfacing a bare 400.
- Crawl-date filters are no longer sent; Exa ignores them.
- **Contents settings** (Search, Find Similar, Contents):
- New: `max_age_hours`, `snapshot_as_of`, text verbosity and section
filters, highlights `max_characters`, `rich_image_links`.
- Legacy `livecrawl` and `num_sentences` are translated with Exa's
documented mappings. Deprecated params are no longer sent by default.
- **Answer:** `model` (exa / exa-pro / exa-fast / exa-research),
structured answers (`answer_structured`), `system_prompt`,
`user_location`, `user_identifier`, and cost output and billing.
- **Find Similar:** marked deprecated by Exa (pointing to Search), with
`exclude_source_domain` and `category` added.
- **Websets audit**, for example:
- the preview request body was wrong;
- canceled searches waited until timeout because the code checked
`"cancelled"`;
- imports were sent under the wrong key;
- added title, recall, maxPeoplePerCompany, exclude sources, the `url`
enrichment format, and the list/item filters.
- **Labels:** Search Type and Category now show readable labels instead
of `ExaSearchTypes` and a lowercase `category`, as do the new blocks'
enum and object fields.
**Billing and safety**
- **Blocks that keep or spend something in the Exa account need the
user's own Exa key.** Everyone on the platform's Exa key shares one Exa
account, so Websets, their searches, items, enrichments, monitors and
imports, Exa Agent and Research runs, Search Monitors, and the Websets
webhook and event blocks refuse the platform's shared key ("Use Credits
for Exa search" and `exa-default`) with a message telling the user to
add their own key. Batch Search Monitors could otherwise delete every
user's monitors, and a Websets webhook would forward everyone's events.
Search, Answer, Contents, Find Similar, Code Context and a plain Preview
Webset still run on the platform key and are billed from Exa's reported
cost.
- **Exa Agent and Research runs therefore only happen on the user's own
key.** Exa bills those runs to the user's Exa account and the platform
charges no credits for them (the Exa base cost only applies to the
platform's keys). The blocks still report the run's Exa cost once per
user per run: several blocks can see the same finished run (Exa Agent,
Get, Wait, Cancel, Stop and the Research blocks), and a Redis `SET NX`
claim keeps that reported figure from being counted twice. If Redis
can't be reached within 5s, the figure is recorded anyway.
- **Preview Webset with "include sample items"** runs a real Websets
search, and Exa's preview response carries no cost to bill, so that
option needs the user's own key. A plain preview stays allowed on the
platform key.
- **Up-front refusal.** Blocks that refuse the platform key declare it
(`refuses_platform_credentials`), so graph validation rejects a platform
credential on them before the graph runs or is scheduled, with the same
message used for refused SDK defaults, and the builder no longer
auto-selects the platform credential for them. The check inside `run()`
stays as the backstop.
- **Long waits fit the executor cap.** Exa Agent / Wait for Exa Agent
Run (up to 3 h), Create Research / Wait For Research (1 h) and the
Websets wait blocks (30 min) set `execution_timeout_seconds` to their
input's ceiling plus 5 minutes, so their own timeout output fires
instead of the executor cancelling `run()`. Run-starting blocks yield
the run ID before waiting.
- Exa REST calls send writes and deletes once (no automatic retry of
POST/PATCH/DELETE), retry reads a bounded number of times, and drop the
`x-api-key` header on a cross-origin redirect.
- IDs are URL-escaped in every path.
- The Agent API's nested `{"error": {...}}` responses are shown as
readable messages.
**Follow-up fixes in this PR (A/B/C)**
- **A. Preview Webset sample items** ran a real search on the shared key
with nothing billed. That option now refuses the platform key; a plain
preview doesn't.
- **B. AutoPilot couldn't run any Exa block for users without their own
key** (pre-existing on `dev`). The credential matcher counted the SDK
`exa-default` credential as the user's own, so a watched chat picked it
and the credentials manager then refused it, and a user with their own
key was asked to choose between it and `exa-default`. An SDK default
that isn't runnable for its provider (not in
`RUNNABLE_SDK_DEFAULT_PROVIDERS`) now never matches a credential field,
so the "Use Credits" system credential is used. Runnable SDK defaults
(e.g. Tavily) are unchanged.
- **C. Nothing told the system which blocks refuse the shared key**, so
graphs with them saved, scheduled and ran and only failed at the node.
The new `refuses_platform_credentials` block marker is read by graph
validation and exposed on the block's credentials field schema so the
builder doesn't auto-select a system credential for such a block. Exa's
stateful blocks and SmartLead's campaign blocks set it; a test keeps the
marker and the run-time refusal in agreement.
- Removed the Exa-only `_platform_key` guard: every block using it
already refused the shared key in `run()` first, so it never fired.
**Docs:** `docs/integrations/block-integrations/exa/` was regenerated,
with hand-written "how it works" sections for every new block and
updated sections for the changed ones.
**Behaviour changes worth knowing**
- Search now forwards all of its content settings. Before, it silently
dropped livecrawl, subpages, extras and context.
- Invalid inputs now fail early with a clear message instead of an Exa
400. Examples: a non-ISO country code, lite with unsupported options,
more than 5 webset criteria, `refresh` Websets monitors (removed by
Exa).
- `Webset.status` is a plain string now, so new Exa statuses don't crash
runs.
- Saved Highlights settings with only a query now get Exa's adaptive
length instead of 1 sentence. Saved graphs that stored the full default
object behave the same.
**Found and left out of scope**
- Exa's webhook signatures carry a timestamp that is not checked for
replay. Exa documents the check as optional and doesn't say whether
retried deliveries are re-signed, so a fixed window could drop
legitimate retries. This predates the PR; the new trigger uses the same
check.
- Create Research is still declared a read (`effect=READ`) although it
starts a paid run, as it was on `dev`; Exa Agent now declares
`EXTERNAL_JUDGED`.
- Most Websets item/enrichment/import-create/export calls still use
`exa-py`, whose models match the spec there.
### Screenshots (local stack: `dev` at
|
||
|
Toran Bruce RichardsandClaude Opus 5.5
|
78749c15b4 |
feat(blocks/firecrawl): add Firecrawl Alexandria data tool blocks (#16561)
### Why / What / How
**Why:** Firecrawl launched Alexandria, pitched as "headless Clay for
agents": search 250M+ people and 26M+ companies, get work emails, job
and company data, plus thousands of other structured-data providers
(jobs, finance and filings, public records, places, retail), all on the
same Firecrawl API key. Our Firecrawl blocks only cover scrape, search,
crawl, map and extract, so neither graphs nor AutoPilot can reach it.
**What:** Two new Firecrawl blocks, following Firecrawl's own
discover-then-run flow (the same one its MCP server and CLI use):
- **Firecrawl Find Data Tools**: free catalogue search. Describe the
data you need ("sales leaders at Series B software companies") or name
providers, capabilities or categories; get back matching tools with
their provider and capability IDs, price in Firecrawl credits and input
contract.
- **Firecrawl Run Data Tool**: runs one tool (provider + capability +
options) and returns its records and the credits it cost.
**How:** Both call `POST https://api.firecrawl.dev/v2/scrape` with an
`alexandria` body
([docs](https://docs.firecrawl.dev/features/alexandria)); discovery is
the built-in `firecrawl/find-tools` capability. Plain HTTP through
`Requests` because the locked `firecrawl-py` (4.42) predates Alexandria;
this avoids a dependency bump. Each run sends one `x-request-id` across
retries so Firecrawl runs a retried paid call once.
Billing reuses the Firecrawl provider's existing `COST_USD` rate:
`provider_cost` is the `creditsCost` Firecrawl reports × $0.001, the
same credit value the other Firecrawl blocks use. Cost is recorded
before a failed tool raises, since a failed call can still be charged. A
`THIRD_PARTY_DATA_TERMS_REQUIRED` refusal names the Firecrawl dashboard
link where an org admin accepts that provider's terms.
### Changes 🏗️
- `backend/blocks/firecrawl/alexandria.py`: the two blocks and the
shared Alexandria call.
- `backend/blocks/firecrawl/alexandria_test.py`: request shape, timeout
cap, terms and other refusals, billing (including on a failed tool),
catalogue option building.
- Generated block docs, with the manual sections filled in.
Not changed: the existing Firecrawl blocks, and the Firecrawl Agent
`exchange` option (we have no Firecrawl Agent block).
### Agents and large language models used
Claude Code with Claude Opus 5.5
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] `test_block.py` passes for both blocks (mocked run, effect, IDs)
- [x] `alexandria_test.py` passes (9 tests)
- [x] ruff, isort, black, pyright clean on the new files;
`generate_block_docs.py --check` in sync
- [x] AutoPilot's capability index returns both blocks for "find sales
leaders at Series B software companies", "find work email for a person",
"enrich company from domain" and "people search by job title"
- [ ] Live call against Firecrawl with a real key (not run: no
Alexandria-enabled key available locally; the keyless API refuses
provider tools)
**Before this works on the platform key:** the team behind our
`FIRECRAWL_API_KEY` needs Alexandria access, and an org admin must
accept the terms of any provider that requires them (Firecrawl dashboard
→ Settings → Data sources).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
|
||
|
Nicholas TindleandClaude Mythos 5.1
|
ea4a36ff27 |
fix(platform): generate the Web Push (VAPID) key pair per install (#16416)
### Why / What / How **Why.** `autogpt_platform/backend/.env.default` has shipped a working Web Push (VAPID) private/public key pair since #12723, and `frontend/.env.default` the same public key. The file is public, so every self-host that kept the defaults signs push notifications with a key anyone can read: whoever holds a user's push endpoint can notify that user as the site. It is the same class as the retired JWT and Fernet defaults, lower impact, and it slipped past the `check-env-defaults` guard because the pair was never on its list. **What.** The pair is blank in both `.env.default` files and generated per install by `make init-env`, the single-container first boot regenerates a published pair, and the backend refuses to sign with the published key while still starting. **How.** `fill-env` fills `VAPID_PRIVATE_KEY` and `VAPID_PUBLIC_KEY` as one, from a single generated P-256 key, and treats the published pair as blank (like a service password: nothing stored depends on it, so replacing it costs only the browsers' subscriptions). A pair with one side set is refused rather than half-filled, since the public key is derived from the private one, and so is a pair whose public key is not the private key's, in `make init-env` as on the container's boot. A 32-byte private key outside the curve's scalar range is refused as well (no public key derives from it), while one that is not a raw scalar is left to the backend, which also reads a DER key. A pair with the published key on one side and another key on the other is refused on the container's first boot rather than half-replaced (the dropped half would fail the next boot's changed-value check), and `make init-env` keeps a pair that holds a `$NAME` reference, which Compose fills in when the stack starts. A `backend/.env` from before the pair existed gets it appended, the way a later-declared service password is. `make init-env` runs under the system interpreter, which has no `cryptography`, so generation falls back to interpreter-only P-256 arithmetic; that path is checked against the RFC 6979 A.2.5 vector and against `cryptography` where it is installed. The backend's secrets guard warns on the published private key (by SHA-256 digest of the canonical Base64url spelling of the bytes it decodes to, since py-vapid reads a padded, wrapped or standard-alphabet key as the same key a decoder ignores the last character's unused bits, and a key of another length is read as DER, so such a key is reduced to its scalar first; the value is not reproduced) and `push_sender` skips sending with it, saying why once per process. The frontend takes `NEXT_PUBLIC_VAPID_PUBLIC_KEY` over the key it fetches from `GET /api/push/vapid-key`, so `fill-env` clears a published value from `frontend/.env` and otherwise leaves it alone. A single container started with the published pair in its environment replaces it on first boot and accepts that same environment on every later boot, and one that persisted the published pair under an earlier release replaces it on its next boot the same way. ### Changes 🏗️ - `backend/.env.default`, `frontend/.env.default`: VAPID pair blank; comments say how it is generated. - `single-container/runtime_config.py`: `VAPID_KEY_SETTINGS`, `PUBLISHED_VAPID_DIGESTS`, pair-wise fill in `fill_env_blanks`, missing-line backfill of the pair, clearing of a published `NEXT_PUBLIC_VAPID_PUBLIC_KEY`, published-pair regeneration in `_configured_or_generated_vapid` with `_verify_environment_matches` passing over it, replacement of a persisted published pair on the existing-config path, canonical-spelling digest, interpreter-only P-256 fallback, backend pair added to `BLANK_IN_ENV_DEFAULT`. - `backend/util/secrets_guard.py`: `vapid_private_key_is_published` (canonical spelling, a DER key reduced to its scalar), warning on the published key, hint that says to recreate the services. - `backend/data/push_sender.py`: refuses to send with the published key, warning once per process. - `Makefile`, `docs/platform/getting-started.md`: upgrade note. - Tests: fill, keep, refuse half-set, refuse a pair that is not one, a scalar outside the curve order and a half-published pair, keep a pair with a Compose reference, a DER encoding of the published key, replace published, append to an older file, clear the frontend's published copy, padded and wrapped spellings, environment regeneration, two boots on the published environment, a persisted published pair replaced, RFC vector, agreement with `cryptography`, no-`cryptography` path (blocked on every import name, scalar draw patched so a `cryptography` pair fails it); guard warning and spellings; push sender refusal warned once; guard and generator digests cross-checked. Configuration changes: `VAPID_PRIVATE_KEY` and `VAPID_PUBLIC_KEY` are now blank in `.env.default` and generated by `make init-env`; `NEXT_PUBLIC_VAPID_PUBLIC_KEY` is blank in the frontend's and cleared from an existing `frontend/.env` when it holds the published key. An install that still has the published pair sends no pushes until `make init-env` replaces it and the services are recreated; browsers subscribed under the old pair re-subscribe by turning notifications off and on. ### Agents and large language models used Claude Code with Claude Mythos 5.1. ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] `single-container/tests/test_runtime_config.py`: all 80 pass on Linux (WSL, Python 3.10, `cryptography` installed, so the no-`cryptography` test's blocking is exercised for real); on this Windows machine the remaining failures are the pre-existing `os.fchmod`/file-mode ones that also fail on untouched `dev` - [x] `check-env-defaults --root autogpt_platform` passes on the real files - [x] `backend/util/secrets_guard_test.py` + `backend/data/push_sender_test.py`: 50 passed - [x] black, isort, ruff, pyright clean on the touched files #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Mythos 5.1 <[email protected]> |
||
|
Reinier van der LeerandClaude Opus 4.8
|
4c8601afc3 |
feat(platform): Support input nodes alongside trigger nodes (#11220)
This PR lets a triggered preset carry the agent's own inputs beside its trigger config, so an agent can have both a webhook-trigger node and input nodes instead of choosing one. The v2 External API presents the two as separate fields. - Resolves [OPEN-2547: Support input blocks alongside trigger blocks](https://linear.app/autogpt/issue/OPEN-2547/support-input-blocks-alongside-trigger-blocks) ### Why Previously a triggered preset stored *only* the trigger config as its `inputs`, and the run modal fell back from the graph's input schema to the trigger's config schema — so there was nowhere to put a webhook-triggered agent's regular graph inputs, and the copilot's building prompt told Otto never to add an input node beside a trigger. That meant you couldn't configure regular graph inputs on a webhook-triggered agent. ### What - A triggered preset stores the agent's inputs and its trigger's settings side by side. A webhook delivery runs the agent with those inputs, and the trigger node gets its settings plus the event's payload. - Setting up a trigger takes the agent's inputs: `constant_inputs` on the internal `/presets/setup-trigger` route and the copilot `setup_agent_webhook_trigger` tool, `inputs` on the v2 `POST /library/presets/setup-trigger`. - Editing a trigger edits either half alone: the copilot `update_preset` tool and the v2 `PATCH /library/presets/{id}` take `trigger_config` beside `inputs`. - The v2 preset responses carry the agent's inputs in `inputs` and the trigger's settings in a new `trigger_config`, with its secret settings listed in `hidden_inputs` as before. - Existing triggered presets are converted to the new shape at deploy and on each REST boot. - A preset attached to a webhook runs only when its trigger fires: running it directly answers 400 on both execute routes. ### How - A preset's `inputs` now hold the regular graph inputs **plus** the trigger node's config nested under a per-node key `_node_input_mask_{node_id}`. The name matches the executor's existing `nodes_input_masks` concept. - `TriggeredPresetSetupRequest` gains a `constant_inputs` field (regular graph input values). `setup_triggered_preset` wraps the trigger config under the per-node key and stores `constant_inputs` alongside it. - On a webhook delivery, `_execute_webhook_preset_trigger` separates the two again: regular inputs are passed as graph `inputs`, while the trigger config + webhook payload become the trigger node's `nodes_input_masks` entry. The event-type filter is checked against the **unwrapped** trigger config. - `update_triggered_preset` extracts the trigger config from the same per-node key when re-registering the webhook. Delivery, the PATCH route and the copilot `update_preset` tool split a preset's inputs through one helper, `split_trigger_inputs`, which reads a preset with no mask as all trigger config. - The v2 routes translate between the stored shape and the two fields. A reader without the preset's graph splits it with `split_stored_trigger_inputs` (`library/model.py`), which takes the object under the mask-prefixed key as the trigger config and reads an attached preset with no mask as all trigger config. A PATCH rebuilds the stored shape from `inputs` and `trigger_config`, each left out keeping its saved value, and puts back the hidden secret settings a new `trigger_config` leaves out. `trigger_config` on a preset with no trigger is a 400. A new `trigger_config` on an active trigger needs `RUN_AGENT`, as new `inputs` do, because it re-registers the webhook; the MCP `update_preset` tool asks for it on the same terms. - The setup/update/delete logic lives in shared helpers (`api/features/library/triggers.py`) reused by the `/presets` routes, the copilot preset tools, and a DatabaseManager RPC endpoint, so all paths produce/consume the same input shape. - A data migration (`migrate_preset_trigger_params`) converts the bulk of existing triggered presets at deploy time. Its trigger-block list is fixed when the migration is written and an applied migration cannot re-run, so the authoritative conversion is `migrate_flat_triggered_preset_inputs()`, which runs on each REST boot and derives the block set from `get_webhook_block_ids()`. It also reaches presets whose webhook was detached, telling them from a run-template preset on the same trigger-bearing graph by comparing the preset's input names against the trigger block's config schema. The query carries the half of that test SQL can express — attached, or holding an input some trigger block declares — so a run-template preset is selected only when one of its inputs is named like a trigger field, which costs it a graph load on every boot; otherwise the converged database costs one selective query and no per-preset work; on the first boot after the migration it is bounded by `asyncio.wait_for(..., timeout=30)`, and since it fetches in id-ordered pages of 100 and each preset commits on its own, a timeout defers the remainder to the next boot rather than delaying the port. Every replica runs it, so a conversion skips when its delete finds the flat rows already gone: the first replica's delete holds them until commit, so a second that read the same preset deletes none and rolls back instead of adding a second mask row. Until then, a preset that is still flat is read with its inputs as the trigger config, as before the mask existed: a delivery runs with them, and an edit saves them nested under the mask. ### Changes 🏗️ - Add `constant_inputs` to `TriggeredPresetSetupRequest`; wrap the trigger config under `_node_input_mask_{node_id}` and store regular inputs alongside it - Update webhook execution to split the node input mask from regular graph inputs, and check the event filter against the unwrapped trigger config (fixes event-filtered triggers such as GitHub raising "Event filter is not configured on trigger") - Add a Prisma data migration to convert existing triggered presets - Tests for the wrap/extract helpers and the webhook execution path - Derive the trigger-block set from the registry rather than a literal list; wrap the preset version bump and mask re-key in one transaction; validate updated inputs in `update_triggered_preset` the way setup does; give the copilot `update_preset` tool a separate `trigger_config`, so graph inputs stop landing inside the trigger config; and stop the UI reporting success on a trigger edit it silently dropped - Let the PATCH route, the copilot `update_preset` tool and the library's trigger view edit an attached preset the boot backfill has not converted yet, instead of refusing it for missing its mask; the edit stores it nested. The trigger view reads such a preset as all trigger config, as the backend does - Log the backfill's designed timeout as a warning without a traceback - Reconfigure the trigger only for a preset that is attached or holds the mask, so editing a run-template preset on a graph that also has a trigger node is a plain update - Store the trigger config merged with the new credentials when a trigger preset's credentials change, so the preset no longer keeps executing with the credentials the user replaced (the stored mask overrides `preset.credentials` at execution) - Read a triggered preset's webhook secret (Generic Webhook's `secret_token`, All Quiet's `signing_secret`) from inside the mask through `preset_trigger_configs`, which returns every object under a mask-prefixed key because a graph input may use the prefix too, so a preset set up with a secret still refuses unsigned deliveries - Validate a trigger preset's inputs at setup and reconfigure as delivery does, except for credentials and the not-yet-received payload. The check ran as a dry run, which also skips the schema check, so a trigger config delivery refuses (a GitHub event filter `{"opened": "yes"}`) registered its webhook and then failed every event. `validate_and_construct_node_execution_input` gains a `trigger_setup` flag for this. It leaves absent credentials and the trigger's placeholder `payload` unchecked, so a typed payload such as Compass's `TranscriptionDataModel` doesn't refuse setup; delivery still validates the real payload. Each block keeps its own input check: `get_mismatch_error` takes the same `exclude_fields` as `validate_data`, so a block's override still runs in place of the base schema check (MCP's and AgentExecutor's leave their dynamic inputs to their targets) - Let a preset whose agent has no credentials update its inputs: `db.update_preset` refused non-empty `inputs` beside an empty `credentials` map, so editing a Generic Webhook trigger returned 400 (also broken on `dev`) - Remove the two remaining places the old either/or rule was still stated: the copilot's `agent_generation_guide.md` (so Otto builds the combined shape and passes `constant_inputs` at trigger setup) and `validate_io_blocks`'s error text - Apply the split/merge helpers on the library page's Templates tab, where a triggered preset whose webhook was detached is filed: its stored trigger config renders again, and "New agent task" no longer seeds the mask key into `constant_inputs`. The split is memoised, so a parent re-render no longer resets an open run modal's form - Trim 303 chars of restatement from both copilot tools' descriptions, so the two new parameters (`constant_inputs`, `trigger_config`) cost 178 net rather than 481. The trim is a saving, not what keeps the PR under - Raise both tool-schema ceilings for the two new parameters: the registry's `_CHAR_BUDGET` 77,810 → 78,204 and the largest session's `_SESSION_WIRE_BUDGET` 70,108 → 70,526 - Give the v2 preset routes `trigger_config` beside `inputs`, on the response and on PATCH, and `inputs` on setup-trigger; update the External API authentication guide to match - Ask for `RUN_AGENT` when the MCP `update_preset` tool is given a new `trigger_config`, as for new `inputs` - Label `trigger_config`, not `constant_inputs`, "Trigger settings" in `setup_agent_webhook_trigger`'s schema - Find a preset's trigger config by its prefix and an object value in the copilot `update_preset` tool and the library's trigger views, as the backend does, so a graph input named with the prefix stays one; trigger setup and reconfigure refuse a graph input that would itself read as a trigger config (a prefixed name holding an object), so a triggered preset holds exactly one - Re-key only the trigger config row when a preset moves to a new version, picked by the same rule, so a graph input named with the prefix keeps its name - Refuse graph inputs in the copilot `update_preset` tool on a trigger the boot backfill has not nested yet, with the trigger view's message, until its trigger settings are saved once - Check what the preset writer stores before a webhook is registered, which #16483 (a count) and #16485 (a 256 KiB byte bound, and the name and description lengths) check against the flat config: the graph inputs, the nested trigger config as one row, and the credentials. The slot #16483 reserves for the webhook `payload` is the trigger node's, not a preset row, and the executor validation this branch runs at setup and reconfigure already counts it there - Refuse a direct run of a preset attached to a webhook on the internal and v2 `/presets/{id}/execute` routes, with a 400 saying it runs when its trigger fires, as the copilot's `run_agent` already did. Its trigger node gets a payload only from a delivery, so such a run could never start, and it is now refused before the stored trigger config would be passed on as the agent's inputs > The two parameters add 178 characters to the registry and 164 to the largest session (measured against `dev` at |
||
|
Toran Bruce RichardsandClaude Opus 5.5
|
6574f3a118 |
feat(backend): warn canceled trials 3 days and 1 day before access ends (#16506)
### Why / What / How **Why:** A trialist who cancels gets the cancellation confirmation, then nothing until "Your AutoGPT trial has ended", when access is already gone. Stripe does fire `customer.subscription.trial_will_end` for canceled trials (17 of the last 40 live events), but the ending reminder dropped them as "cancellation is scheduled". Closes [SECRT-3324](https://linear.app/autogpt/issue/SECRT-3324). **What:** Two new trial notices for trials set not to renew, each with **Resume my trial** as its one action: | Kind | When | Subject | |---|---|---| | `expiring` | 3 days before the end (Stripe's `trial_will_end`) | Your AutoGPT trial ends in 3 days | | `last_call` | 18 to 25 hours before the end (new hourly pass) | Your AutoGPT access ends tomorrow | **How:** Both are ordinary `TRIAL_UPDATE` kinds, so they reuse the trial notice pipeline: the ownership and checkout-attempt checks, a dedupe claim per trial end date, the freshness check at delivery, and the billing sender, kit and footer. When `trial_will_end` arrives for a cancel-pending trial, it now becomes `expiring` instead of being dropped. The scheduler runs `send_trial_last_calls` at :15 every hour, in the database manager like the Stripe tier sweep. The pass lists running, unconverted, cancel-pending trials whose end falls in the window, and calls `notify_trial(..., "last_call")` for each. If one trial fails, the pass logs it and carries on. These are service messages: like the other trial notices, they go out even to people who opted out of product news. So they stick to facts about the person's own account, with no "Last chance" framing or urgency styling. Persuasion belongs to the MailerLite win-back sequence. ### Journey (7-day trial, canceled on day 2) Day 2: cancellation confirmed → day 4: `expiring` → day 6: `last_call` → day 7: trial ended. - `ending` ("you'll be charged") and `expiring`/`last_call` are mutually exclusive on `cancel_at_period_end`, so nobody gets both kinds for one trial end. - Each new notice goes at most once per trial end. Cancel → resume → cancel again does not resend it. - Neither goes to someone who canceled in the last day (Stripe's `canceled_at`): they have just had the cancellation notice, with the same date and the same way back. - A resume, a conversion or another change before delivery suppresses a queued notice, as for the other trial notices. - No MailerLite group change: the trial group already dropped them when they canceled. ### Before / after **Before:** a canceled trialist got no email at either point. The 3-day `trial_will_end` event was logged as `Dropped trial ending notice for user …: cancellation is scheduled`, and there was no 1-day email at all. **After** (rendered from this branch's templates with fixture data, Pro at the live trial price of $50.00 / month): | 3 days before | Last day | |---|---| | <img src="https://raw.githubusercontent.com/Significant-Gravitas/AutoGPT/2c7016e13dd73d56199e35138b1453044404fd4b/docs/engineering/email-review/secrt-3324/expiring-phone.png" width="300" alt="Your AutoGPT trial ends in 3 days"> | <img src="https://raw.githubusercontent.com/Significant-Gravitas/AutoGPT/2c7016e13dd73d56199e35138b1453044404fd4b/docs/engineering/email-review/secrt-3324/last_call-phone.png" width="300" alt="Your AutoGPT access ends tomorrow"> | Desktop: [3 days before](https://raw.githubusercontent.com/Significant-Gravitas/AutoGPT/2c7016e13dd73d56199e35138b1453044404fd4b/docs/engineering/email-review/secrt-3324/expiring-desktop.png) · [last day](https://raw.githubusercontent.com/Significant-Gravitas/AutoGPT/2c7016e13dd73d56199e35138b1453044404fd4b/docs/engineering/email-review/secrt-3324/last_call-desktop.png) Layout: a short lede, then the plan panel (plan, access end, auto-renew off), then two option cards, "If you resume" (with the price) and "If you do nothing", then the button with the cancel-again and tax note. All text meets WCAG AA contrast: labels and notes #575072 on #F5F3FC (6.8:1) and on white (7.5:1), body #2D2A3E on #F5F3FC (12.6:1), button text white on #6144DF (6.2:1). The plain-text part carries the same content in the same order. ### Found and left alone - **MailerLite overlap:** the live "Canceled trials resurrection" automation sends "Trial cancelled, but you can still use it" on trial day 5 or 6 to canceled trialists in the marketing audience. Once this ships, it lands between the 3-day and last-day emails. This PR doesn't change it; that's a separate decision. - In the plain-text version of the other trial emails, the CTA link runs into the "You received this…" line. Only the two new kinds are laid out cleanly here. ### Changes 🏗️ - `notifications/trial.py`: the `expiring` and `last_call` kinds, their windows, the recent-cancel rule, and `send_trial_last_calls`. - `data/subscription_trial.py` + `db_manager.py`: `list_trials_ending_unrenewed`, and `send_trial_last_calls` exposed over RPC. - `executor/scheduler.py`: the hourly `send_trial_last_calls` job. - Trial templates (HTML, text, subject): the layout above. - PostHog `trial_expiring` and `trial_last_call` events, sent on enqueue like the other trial events and pinned in `posthog_events_test`. The tracking plan and the email design doc are updated. - Tests cover rendering (including plain-text layout and no urgency styling), which reminder applies, the windows, the recent-cancel skip, claim keys, converted trials, the pass's error isolation, and job registration. `trial_audience_cancel_test`'s "no ending reminder while pending" test now expects the `expiring` notice, since this ticket changes that behaviour on purpose. ### Agents and large language models used Claude Code with Claude Opus 5.5. ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Trial, design-system, approval-copy, freshness and PostHog event tests pass locally (258), plus the scheduler registration test - [x] Rendered both emails at phone and desktop width and checked contrast - [ ] CI green - [ ] After deploy: a canceled trial with under 4 days left gets the 3-day email on its next `trial_will_end`, and the last-day email on the hourly pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Nicholas TindleandClaude Fable 5.1
|
f5f05f638c |
feat(platform): make background subtasks real work that returns to the chat (#16582)
### Why / What / How [SECRT-2813](https://linear.app/autogpt/issue/SECRT-2813), second increment. #15733 woke Otto after a background subtask for a bounded, tool-free summary: the child had ten minutes, the follow-up two, delivery a ten-minute window, and the result was pasted into the wake prompt. Nick's call on review: time is not a budget, and the wake should return to the main thread rather than deliver a summary. This makes background work real. **What changes for the user:** a background subtask runs until it is done, and when it finishes, fails or is stopped, Otto carries on in the original chat with everything it had, as soon as that chat is idle, however long that takes. No deadline, no delivery window, no expiry notice. **How:** - The child is spawned like a foreground subtask: the launching turn's tools minus the descent denials, its budget tree, cost attribution, credential pins, route and scope, and it may hand work on within the tree's depth. It keeps no deadline of its own; only a deadline the launching turn already had is enforced (`background_timeout` returns `None` otherwise). - The wake runs as a continuation of the launching turn under that turn's own envelope (same tools, spawn rights and spend ledger), as an unattended turn. It is governed by the user's allowance and spend limits the way every turn of that chat is, not by the subtask's node, so a spent subtask tree no longer blocks the chat from carrying on. - The wake message names the subtask and its outcome and points at `get_sub_session_result`. What the subtask produced stays in its own chat and reaches the model as a tool result, the way a foreground subtask's does, instead of as text inside a user message. The `result` column and the row-selection query go away; the UI keeps the short labeled notice. - Delivery waits for an idle parent indefinitely; the "follow-up window expired" notice is gone. Recovery still records a delivered continuation from Redis and records an interrupted child as failed when its chat went idle without an outcome. A child (or continuation) whose worker died is released by the sweep with the orphan rule the chat routes already apply: still running its claimed turn, no live stream, untouched for thirty seconds. That is a liveness check, not a work deadline, and it is now shared as `ORPHAN_RUNNING_RESET_SECONDS`. - A cancelled turn, a chat's or a child's, cancels the background work it launched and suppresses those wakes, so stopping a chat reaches grandchildren now that a child may delegate. - When the child's turn ends, the record notes which row holds its answer (`resultSequence`, nullable, on this PR's migration, backfilled for any task completed before it): every row after the launch is the turn's, a clarification it drained mid-turn included, when the worker still holds the turn or when the chat records no later turn (`_active_turn_id`); only once a later turn ran does recovery stop at the next user row. `get_sub_session_result` asks the record for every sub (one indexed read) and, for a background child, returns that turn's answer and tool calls, so a later turn in the child's chat is never handed back as the subtask's result, with the child's workspace files listed as for a foreground sub but only those written in the turn's window, from its launch to the next user message after its answer (a browser screenshot is saved for the child by another tool, so the turn's tool calls alone would miss it, while a later turn's files stay out); until the outcome is recorded it reports the child as still running rather than reading a partial transcript as a result. - A child still queued past a deadline it inherited is closed as timed out by the sweep (pickup would refuse it anyway); a child with no inherited deadline waits as long as it takes. - A launching turn whose allow-list leaves out `get_sub_session_result` is refused background work at launch (its continuation could not read the result), and the child session starts in the approval mode the parent's gate enforces, as the foreground path already does. - Cancelled or refused queued children give their node back; the continuation never used it. - `GET /sessions/{id}` reports `background_pending: bool` instead of `background_pending_until`; the tool output drops `background_expires_at`; the card polls a background subtask to its outcome rather than giving up at the foreground cap. - Migration: `deadline` becomes nullable and `result` is dropped (`20261104020000_background_subtasks_no_clock`). The flag is off and no rows exist, so the mixed-worker window carries no tasks. ### Changes 🏗️ - `background_task.py`, `background_subtasks.py`, `background_execution.py`, `background_delivery.py`, `background_task_db.py`, `turn_queue.py`, `db_manager.py`, `routes.py`, `run_sub_session.py`, `models.py`, `schema.prisma` + migration: the points above. Net about 530 lines smaller. - Frontend: `SubSessionLive`, `AgentCards`, `ChainRowView`, `backgroundSubtasks.ts`, the session mock helper and the regenerated `openapi.json`. - `docs/engineering/background-subtask-delivery.md` rewritten for the new behaviour; the harness investigation section is unchanged. ### Agents and large language models used Claude Fable 5.1 (Claude Code, in T3 Code). ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description. - [x] I have made and executed a test plan: - [x] Review round (Sentry, fourth): a child whose record waited for its own handoffs ends the way its last continuation ended, so a failed continuation reports a failed subtask rather than a completed one (the nested-handoff test runs both endings). - [x] Review round (CodeRabbit, fifth pass): the result tool reports a promotion refusal (a spent tree, say) from the child's chat ahead of the failed record the sweep writes (`test_result_tool_reports_why_a_background_child_was_refused`); a completed child without an answer row still bounds its file listing by its turn's window, and a record without a window lists nothing rather than everything; the window is applied in the workspace query itself, ahead of the 50-file manifest limit (`created_after`/`created_before` on `list_workspace_files` and `WorkspaceManager.list_files`, with a real-database test); and the sweep's orphan release leaves a chat alone while a worker still refreshes its session lock, whatever became of the stream metadata (`locked` case of the dead-worker test). - [x] Review round (Sentry, third): a child that hands work off itself no longer completes, and wakes the original chat, when its own turn ends with that work still out; its record waits for the last of that work to come back to it (`has_open_handoffs`), the sweep leaves such a child alone meanwhile, and the record completes with the child's latest answer when its last continuation ends (`test_a_child_that_handed_work_off_wakes_the_parent_once_that_work_is_back`). - [x] Review bot's second-round optional advice folded in: a real-database test for the sweep releasing a continuation whose worker died (`test_sweep_recovers_a_continuation_whose_worker_died`), and `_reconcile` split into settling the child's outcome and acting on it. - [x] Review round (CodeRabbit, fourth pass): the answer query reads the chat's active turn in the same statement as the rows (one snapshot, no window for a turn admitted in between); a recorded background result lists the child's workspace files from the authoritative listing, restricted to its turn's window so a later turn's stay out (`test_result_tool_lists_a_background_childs_workspace_files`, `test_the_sub_manifest_can_be_restricted_to_a_window`); the sweep re-arms the ledger key of every tree with open background work so its spend and node counts survive the wait, reopening being the fallback for a key lost anyway (`test_sweep_keeps_an_open_handoffs_tree_alive`); the migration states its conservative legacy rule instead of claiming to match recovery. - [x] Review bot's optional advice folded in: the sweep's orphan release fences on the task it already holds instead of re-reading it, the result tool's refusal branch has a test per terminal outcome, and the orphan threshold's rationale sits with its constant. - [x] Review round (CodeRabbit, third pass): the migration backfills `resultSequence` before dropping `result` (no such rows exist anywhere, the flag was never created, but the migration is now correct on its own); recovery uses the chat's recorded turn id to own the rows after a drained clarification unless a later turn ran (`test_answer_is_bounded_by_the_turn_the_worker_still_holds`, three endings). - [x] Review round (Sentry, second): a nested handoff can outlive the tree ledger's turn-lifetime key, so promotion and pickup of background work reopen a closed tree (`reopen_closed_tree`); a spent tree still refuses a nested continuation, which is the spend budget acting (`test_nested_continuation_reopens_an_expired_tree_but_not_a_spent_one`). - [x] Review round (CodeRabbit, second pass): the tool consults the record for every sub instead of the loaded message page, reports an unrecorded background child as running, and carries the recorded turn's tool calls so `sub_tool_call_count` and the file manifest are scoped to that turn (real-database tests). Its first finding, an empty intersection in `CopilotPermissions.merged_with_parent` reading back as allow-all, is pre-existing and untouched by this PR; flagged for its own fix. - [x] Review round (Sentry): a clarification drained mid-turn no longer hides the child's final answer; the answer row is recorded at completion with the worker-held bound, recovery keeps the conservative bound (`test_answer_is_bounded_by_the_turn_the_worker_still_holds`). - [x] Review round (GitHub Copilot's five findings): two overlap Capy's (dead-worker recovery, result-tool permission); a child still queued past an inherited deadline is now closed as timed out (new real-database test); the continuation-node refund is not reproducible (the continuation row is an answer row, so the queue never treats it as a spawned child); the six-hour staleness cutoff is the platform's bound on every turn, left as is. - [x] Review round (CodeRabbit's two findings): a launch without the result tool in the allow-list is refused, and the child inherits the parent's enforced approval mode; both with real-database tests. - [x] Review round (Capy's three findings, each with a real-database test): the sweep recovers a child whose worker died and leaves a live one alone; a cancelled child turn cancels and suppresses the grandchild it launched; the result tool returns the child's own answer when its chat has moved on. - [x] Real-Postgres/Redis handoff suite rewritten for the new rules: the wake runs under the launching turn's envelope and names the child instead of carrying its text; a spent subtask tree does not stop the chat; a cancelled queued child gives its node back; the child may delegate; the session state reports `pending`; recovery keeps a delivered continuation. The deadline, delivery-window and expiry-recovery tests are removed, and the inherited-deadline tests stay (`test_worker_enforces_an_inherited_deadline_even_during_silent_work`, the worker-clock test, the claimed-admission-failure test). - [x] In the Linux test image against a private Postgres/Redis/RabbitMQ stack: the background handoff, turn-queue, executor, session-waiter, sub-session, chat-route, scheduler and stream-registry suites pass (918 in the first run, the three old-envelope assertions then rewritten and the handoff suite re-run green). - [x] pyright 0 errors on the touched backend files; black/isort/ruff clean; private Prisma client regenerated for the schema change; OpenAPI export matches the committed `openapi.json`. - [x] Frontend: `tsc --noEmit` clean; eslint 0 errors (the four warnings are pre-existing); oxlint clean; the copilot lifecycle, ToolChain and helper Vitest suites pass, including the new cases that a background card keeps polling past the foreground cap and that the page polls while `background_pending` holds. - [ ] Smoke-test with a live provider: the review bot's sandbox cannot run Otto (revoked token); the behaviour is covered by the real-database handoff suite. #### For configuration changes: - [x] `.env.default` is already compatible. - [x] `docker-compose.yml` is already compatible. - [x] Configuration changes and rollout: apply the migration, deploy compatible API, database-manager, scheduler and workers, keep the flag off until then. --------- Co-authored-by: Claude Fable 5.1 <[email protected]> |
||
|
Nicholas Tindle
|
aef0f991a8 |
feat(backend): add Tencent Hy4 to the LLM catalog (#16605)
Tencent released the official Hy4 on OpenRouter today (2026-11-04), succeeding Hy4 Preview, which stays in the catalog. - `tencent/hy4` via OpenRouter: 1,048,576 context, 65,536 max output, tools and reasoning supported - Live price $0.417/$1.2505 per 1M, $0.021/1M cached input; credits at the standard 1.5x margin (62.55/187.575, cache read 3.15), flat tier 2 - New `LLMModel.TENCENT_HY4`, pinned-rate test, and the block docs model lists - `check_openrouter_prices.py` reports MATCH for the new entry |
||
|
Abhimanyu YadavandClaude Opus 5.5
|
d981a641e3 |
refactor(frontend): remove the Tailwind palette; roles are the only colours (SECRT-3097 C4) (#16612)
### Why / What / How **Why:** SECRT-3097 follow-up, PR C of the theming plan (item C4). PRs A (#16511), B (#16513) and D (#16556) moved every app colour onto the role tokens, but Tailwind's palette was still compiled from `tailwind.v3-preset.ts`, so `bg-red-500` still worked and the "one colour system" goal was held only by lint. This PR removes the palette and locks the door. **What:** colour systems 4 → 1. The roles in `src/app/tokens/theme.css` (one `:root` role block, mapped by `globals.css`'s `@theme inline`) are now the only colours Tailwind knows. `tailwind.v3-preset.ts` is gone, `--color-*: initial` resets Tailwind's colours, and four colour lint rules are at error with a baseline of 0. **How:** - **Preset deleted, nothing else moves.** The preset was more than a palette: it carried all of Tailwind 3's theme (shadows, blur, breakpoints, z-index…) that keeps every class at its Tailwind 3 value. Those keys move verbatim into `tailwind.config.ts` as `TAILWIND_3_THEME`; only `colors` (and `APP_COLORS`/`withAppColors`) is dropped. - **Proved on the compiled CSS.** I compiled `globals.css` with the app's PostCSS config on `dev` and on this branch and diffed them rule by rule (then again on the two production builds' CSS bundles). Every removed rule either has an identical replacement or is listed under "Visual changes" below. - **Screenshots** of every page the scripted harness reaches (`scripts/screenshot-pages.ts`, threshold 0) at 1440×900 and 390×844, `dev` vs this branch. ### Changes 🏗️ **Palette removal** - Delete `tailwind.v3-preset.ts` (1,390 lines). Its non-colour Tailwind 3 theme moves into `tailwind.config.ts` unchanged; `presets` is gone. - `theme.css` starts with `@theme { --color-*: initial; }`, before any colour is registered, so a palette class generates nothing. - `bg-message-table-header` read `theme(colors.white)`/`theme(colors.zinc.50)`; now `surface`/`surface-muted` (same values). **Last palette uses, snapped or kept exact** - **User-picked palettes** (all already in `ds-colour-exempt.json`): the expert colour picker (`ColorStep/helpers.ts`), folder artwork (`FolderIcon.tsx`) and sticky notes (`StickyNoteBlock.tsx`). 225 palette classes become the exact same hex as arbitrary values (`bg-rose-300` → `bg-[#fda4af]`), so their CSS is byte-identical. The expert's saved id (`rose-300`) is unchanged. - **Two new roles** for status text on the always-dark fills (toast, terminal, tooltip), which replaced the last three per-class exemptions: - `success-inverse` 110 231 183 (emerald-300, same value): Text/Icon `success-light` tone. - `error-inverse` 242 105 105 (red-400, same value): terminal exit code, and the library card's error tooltip (was red-300, see below). - **Streamdown** (vendored chat markdown) still writes seven palette classes in its mermaid error box and image hover veil. A block at the end of `theme.css` draws them inside `:where([data-streamdown] *)` only, with the palette's values (a role where one is identical). Its seven shadcn alias names stay as before (role aliases, for its markup only, as the plan allows). - `KitSearchSkeleton` shimmer read `theme(colors.zinc.*)`; now `rgb(var(--color-surface-hover|surface-muted))` (same values). - `MultiSelect`: `scrollbar-thumb-muted-foreground` (a shadcn alias the rules did not see) → `scrollbar-thumb-foreground-tertiary` (same colour). - `.storybook/preview.tsx`: `bg-background` → `bg-canvas` (same colour). - The tour backdrop shader defaulted to `var(--color-sky-700)`, a variable that never existed at runtime, so the canvas fell back to black. It now reads `foreground-base` and turns a role triplet into `rgb()`. - `scrollbar-track|thumb|corner-*` read `transparent` from the palette (`scrollbar-track-transparent`, 17 uses); they now accept it as a keyword. The compiled-CSS diff caught this one. **Lint (all `error`, baseline 0, stories included, off only for tests and `__generated__`)** - `design-system/no-raw-palette-classes` (renamed from `no-palette-colours`). - `design-system/no-shadcn-alias-colours` (renamed from `no-alias-colours`). - `design-system/no-hex-colours` (unchanged; it now leaves a hex that is a class's whole arbitrary value to the rule below, so it isn't reported twice). - **New `design-system/no-arbitrary-colour`:** a colour class whose arbitrary value is a colour literal (`bg-[#..]`, `text-[rgb(..)]`, `border-[color:hsl(..)]/40`, `[color:#fff]`), with the matching role in the message. It reads `ds-colour-exempt.json`. A role through `var()` is allowed. Colours inside shadows and gradients are not in scope. The wrapped shadcn `no-arbitrary-values` now leaves arbitrary colours to it. - The colour rules (and the alias codemod) now also read `scrollbar-thumb|track|corner-*` classes. - No suppression comments used the old rule names. `design-system-baseline.json` is unchanged (it had no colour entries). **Tests** - New `src/components/styles/__tests__/roles.test.ts`: every role in `color-roles.ts` has its value in the one `:root` block of `theme.css` and an `@theme inline` entry. It compiles the real stylesheet: `bg-<role>` exists for every role, and `bg-red-500`, `bg-zinc-100`, `bg-white`, `bg-black` generate nothing. It also checks the reset comes before any colour is registered, the preset file is gone and the config has no `colors`/`presets`. - `tailwind3-compat.test.ts`: its palette fixtures become roles/arbitrary hex (same assertions), plus a "generates nothing" check for palette classes and a check for `scrollbar-track-transparent`. - `design-system-contract.test.ts`: "keeps the raw palette frozen" (it imported `APP_COLORS`) becomes "has no raw palette". - `design-system.test.ts`: renamed rules, `no-arbitrary-colour` cases, a scrollbar alias case. **Docs:** `docs/engineering/design-system.md`, `frontend/AGENTS.md`, `CONTRIBUTING.md`, and the comments in `globals.css`/`theme.css`. **Size:** +1,612 / −1,708 (3,320 lines, over the ~3,000 guide; kept as one PR as asked). About 2,320 of those are the preset: 1,390 deleted, 930 of them moved into `tailwind.config.ts` unchanged. **Counts (success criteria)** | Measure | Before | After | |---|---|---| | Colour systems | roles + compiled palette (+ alias names for streamdown) | roles only (streamdown's alias names point to roles) | | `tailwind.v3-preset.ts` | 1,390 lines | gone | | Palette rules in the compiled stylesheet | 198 | 0 (streamdown's 7 classes are drawn by a scoped block in `theme.css`) | | Palette uses in `src/` (outside test fixtures and the two stories that show raw classes on purpose, as iframe sample HTML and a "don't" code example) | 228 classes (225 in the three user-picked palettes, 3 on dark surfaces) + 7 palette references (`theme(colors.*)` ×3, `var(--color-sky-700)` ×2, and 2 in `tailwind.config.ts`) | 0 | | Colour entries in `design-system-baseline.json` | 0 | 0 | | Colour lint rules at error | 3 | 4 | ### Visual changes No layout or text moves. One small colour change is visible, so this PR is labelled `pixel-change`. There are **no bigger visual changes**. **Small (slight shade shifts)** 1. **Library, workflow card error tooltip** (hover a card whose last run failed): error text #F69999 → #F26969 (red-300 → red-400, now `error-inverse`, the same red the terminal's exit code uses). The page captures don't reach this hover state, so this is the real tooltip classes rendered with each build's CSS: <img src="https://raw.githubusercontent.com/Abhi1992002/AutoGPT/2dd2fa7e08182cdb2c11559eefe01a7b423598d8/c4/library-tooltip-error.png" width="700"> 2. **Tour (`/tour/chat`) backdrop dots:** dot colour #000000 → #141414 (`foreground-base`), drawn at ≤ 14% opacity. The darkest dot pixel goes from 231 to 232 (out of 255): not visible. Zoomed, left before, right after: <img src="https://raw.githubusercontent.com/Abhi1992002/AutoGPT/2dd2fa7e08182cdb2c11559eefe01a7b423598d8/c4/tour-dots-zoom.png" width="500"> **Unchanged by construction (identical compiled CSS), not reachable by the scripted capture:** the expert colour picker on `/raise` (later step), sticky notes in the builder, toast/terminal green text, terminal exit code, streamdown's mermaid error box and image veil, and the scrollbars. **Page screenshots** (`dev` vs this branch, both production builds with `NEXT_PUBLIC_PW_TEST=true`, signed-in user with one library folder; 32 captures, 25 identical at threshold 0). The other 7 are noise that has nothing to do with this PR: comparing two captures of this same branch gives the same diffs. | Page | 1440×900 | 390×844 | Changed px (1440 / 390) | Why | |---|---|---|---|---| | `/build` | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__build__desktop.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__build__desktop.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__build__desktop.png) | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__build__mobile.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__build__mobile.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__build__mobile.png) | 0 / 0 | identical | | `/copilot` | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__copilot__desktop.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__copilot__desktop.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__copilot__desktop.png) | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__copilot__mobile.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__copilot__mobile.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__copilot__mobile.png) | 756 / 686 | Otto avatar only: it is randomised per load (same diff, 783 px, between two captures of this branch) | | `/home` | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__home__desktop.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__home__desktop.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__home__desktop.png) | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__home__mobile.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__home__mobile.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__home__mobile.png) | 0 / 714 | Otto avatar only (randomised per load, as above) | | `/library` | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__library__desktop.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__library__desktop.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__library__desktop.png) | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__library__mobile.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__library__mobile.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__library__mobile.png) | 0 / 0 | identical | | `/login` | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__login__desktop.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__login__desktop.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__login__desktop.png) | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__login__mobile.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__login__mobile.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__login__mobile.png) | 0 / 0 | identical | | `/marketplace` | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__marketplace__desktop.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__marketplace__desktop.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__marketplace__desktop.png) | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__marketplace__mobile.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__marketplace__mobile.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__marketplace__mobile.png) | 0 / 0 | identical | | `/raise` | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__raise__desktop.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__raise__desktop.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__raise__desktop.png) | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__raise__mobile.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__raise__mobile.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__raise__mobile.png) | 41909 / 1957 | live animated shader; never holds still on either build | | `/reset-password` | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__reset-password__desktop.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__reset-password__desktop.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__reset-password__desktop.png) | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__reset-password__mobile.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__reset-password__mobile.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__reset-password__mobile.png) | 0 / 0 | identical | | `/settings/account` | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__settings_account__desktop.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__settings_account__desktop.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__settings_account__desktop.png) | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__settings_account__mobile.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__settings_account__mobile.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__settings_account__mobile.png) | 0 / 0 | identical | | `/settings/api-keys` | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__settings_api-keys__desktop.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__settings_api-keys__desktop.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__settings_api-keys__desktop.png) | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__settings_api-keys__mobile.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__settings_api-keys__mobile.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__settings_api-keys__mobile.png) | 0 / 0 | identical | | `/settings/integrations` | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__settings_integrations__desktop.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__settings_integrations__desktop.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__settings_integrations__desktop.png) | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__settings_integrations__mobile.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__settings_integrations__mobile.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__settings_integrations__mobile.png) | 0 / 0 | identical | | `/settings/profile` | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__settings_profile__desktop.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__settings_profile__desktop.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__settings_profile__desktop.png) | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__settings_profile__mobile.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__settings_profile__mobile.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__settings_profile__mobile.png) | 0 / 0 | identical | | `/signup` | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__signup__desktop.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__signup__desktop.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__signup__desktop.png) | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__signup__mobile.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__signup__mobile.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__signup__mobile.png) | 0 / 0 | identical | | `/team` | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__team__desktop.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__team__desktop.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__team__desktop.png) | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__team__mobile.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__team__mobile.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__team__mobile.png) | 0 / 0 | identical | | `/team/autopilot` | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__team_autopilot__desktop.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__team_autopilot__desktop.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__team_autopilot__desktop.png) | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__team_autopilot__mobile.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__team_autopilot__mobile.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__team_autopilot__mobile.png) | 0 / 0 | identical | | `/tour/chat` | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__tour_chat__desktop.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__tour_chat__desktop.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__tour_chat__desktop.png) | [before](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/before/app__tour_chat__mobile.png) · [after](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/after/app__tour_chat__mobile.png) · [diff](https://github.com/Abhi1992002/AutoGPT/blob/evidence/ds-theming-c-palette/c4/diff/app__tour_chat__mobile.png) | 8000 / 984 | backdrop dots are placed with `Math.random()` (7,242 px between two captures of this branch); dot colour itself moves ≤ 1/255 | Library with the folder artwork (exact hex now), before / after: <img src="https://raw.githubusercontent.com/Abhi1992002/AutoGPT/2dd2fa7e08182cdb2c11559eefe01a7b423598d8/c4/before/app__library__desktop.png" width="420"> <img src="https://raw.githubusercontent.com/Abhi1992002/AutoGPT/2dd2fa7e08182cdb2c11559eefe01a7b423598d8/c4/after/app__library__desktop.png" width="420"> ### Agents and large language models used - Claude Code with Claude Opus 5.5 (`claude-opus-5-5`) ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] `pnpm format`, `pnpm lint` (design-system rules at error), `pnpm types`: pass - [x] `pnpm test:unit`: 13,149 passed. 5 tests in unrelated files (admin users, copilot mode selector, onboarding intro card, suggested tasks, team autopilot) timed out while the machine was overloaded; all 5 files pass on a re-run (75/75) - [x] Compiled `globals.css` on `dev` and on this branch and diffed every rule (and again on both production builds' CSS bundles): only the changes listed above - [x] Before/after screenshots of 16 pages at 1440 and 390 with `scripts/screenshot-pages.ts`, compared at threshold 0 - [x] `bg-red-500`, `text-zinc-600`, `bg-white`, `bg-black` compile to nothing (`roles.test.ts`, `tailwind3-compat.test.ts`) #### For frontend changes (design system, see `docs/engineering/design-system.md`): - [x] `pnpm lint` passes, including the `design-system/*` rules; I did not raise any count in `scripts/eslint-rules/design-system-baseline.json` - [x] No `dark:`/`light:` classes, raw palette classes (`bg-zinc-200`) or hex/`rgb()` colours (`bg-[#fff]`); colours use role tokens (exempt user-picked palettes keep exact hex through `ds-colour-exempt.json`) - [x] No imports from `src/components/__legacy__`, `legacy/` paths or `@/components/ui`, and no `LegacyButton` / `legacy-*` Button variants - [x] Copy uses the `Text` atom with a `variant`/`tone`; primitives get a `variant`/`size` (no copy or markup changes here) - [x] Before/after screenshots at 1440 and 390 px are attached - [x] User-facing copy says workflow and Otto, never agent or graph (no copy changes) #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**): `tailwind.config.ts` and `.oxlintrc.json` only 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Reinier van der LeerandClaude Opus 5.5
|
3a043ddc3c |
fix(backend): a run's output no longer deletes an earlier output with the same file name (#16343)
When two workflow runs store an output file under the same name, the second is now stored beside the first as `output (1).mp4`. Before, it deleted the first, and the first run's link went dead. Resolves #16341 ### Why / What / How **Why:** A block that returns a file (a video from a Replicate model, a generated image) stores it in the user's workspace at `/<filename>`. When a later run produced a file with the same name, it deleted the earlier run's file: the earlier output link answered 404 and the result was gone. Some models give every output the same name (`fictions-ai/autocaption` always returns `output.mp4`), so each run destroyed the last one's result. The same happened inside one chat, where two block runs saving one name under `/sessions/<id>/` replaced each other. **What:** A stored block output never replaces an existing file. When its name is taken, it is stored beside the existing file as `output (1).mp4`, `output (2).mp4` and so on, the way a browser names a second download, and every earlier `workspace://` reference keeps resolving to its own bytes. Writes that ask to overwrite are unchanged: an upload with `overwrite`, the `write_workspace_file` tool's `overwrite`, a same-named re-upload in a chat, and Otto's own working files. **How:** `store_media_file` called `WorkspaceManager.write_file(overwrite=True)`, which replaced the file already at the path, retiring its row and deleting its blob. It now passes a new `rename_on_conflict=True` instead. `write_file` looks the path up once. When the path is taken, one query lists the numbered `name (` siblings and the write takes the next number. If a concurrent write takes that number first, the index refuses the insert and the write moves on to the next one (two retries, as `overwrite` has). A renamed write replaces nothing, so it hands the insert's storage-quota transaction no file to retire. The number goes on the name as well as the path, because the Files page and downloads show the name. A folder per run would have left several rows all reading `output.mp4`. Each run's output now stays until someone deletes it, so it counts against the storage quota; before, the overwrite freed the previous copy. ### Changes 🏗️ - `backend/util/workspace.py`: `write_file(rename_on_conflict=...)`, and `_free_path`, which picks the next free `name (n).ext`. - `backend/util/file.py`: `store_media_file` stores block outputs with `rename_on_conflict=True` instead of `overwrite=True`. - `backend/util/file_db_test.py` (new): three runs store `output.mp4` through the real `store_media_file` and the database, and each keeps its own bytes, as `output.mp4`, `output (1).mp4` and `output (2).mp4`; and a run whose free name another write takes between the look and the insert lands at the next name, through the storage-quota transaction, while the other write's file stays live. - `backend/util/workspace_test.py`: a write that loses the free name to a concurrent write takes the next number and deletes nothing, and one whose path is free keeps the caller's file name. - `docs/engineering/workspace-media-architecture.md`: the new parameter and the rule. ### Verified On dev's code `file_db_test.py` fails because the first run's file id no longer resolves (`FileNotFoundError`); with this change it passes. Five mutations each turn a test red: `overwrite=True` restored in `store_media_file`; the rename dropped from the lost-race retry, after which the real-database test shows the other write's file retired; the free-name search always answering `(1)`; a renamed write looking up a file to replace; and a free path renaming the file anyway. Every test module that imports `backend.util.workspace` or `backend.util.file`, or names `WorkspaceManager` or `store_media_file`, passes (51 modules behind one stack-starter test: 1,530 passed, 12 xfailed), apart from `api/features/store/skill_db_test.py`, whose guard refuses this machine's shared database and which runs on a fake workspace manager; `util/architecture*_test.py` (92) and `blocks/test/test_block.py` (3,738, 84 skipped) pass too. Pyright reports nothing on the changed files. I did not run a workflow end to end on a live stack. ### Agents and large language models used - Claude Code with Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Three runs storing one filename keep three files, each readable by its own id (real database) - [x] A write that loses the free name to a concurrent write takes the next number and deletes nothing - [x] Workspace, file, architecture and block suites pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]> |
||
|
Reinier van der LeerandClaude Opus 5.5
|
7c9bd21884 |
docs(platform): tell v1 migrators to create a new API key, since keys can't be edited (#16304)
The v1 → v2 migration guide now tells readers to create a new API key with the v2 permissions they need, instead of adding those permissions to their existing key, which the app cannot do. Follow-up to #12206. ### Why / What / How **Why:** Someone migrating an integration from v1 follows the guide's checklist, opens the API keys page to add `RUN_AGENT` and `READ_RUN` to their key, and finds no way to do it. Both API-key pages (`/settings/api-keys`, and the old `/profile/api-keys`) only list, create and revoke keys. `PUT /api/api-keys/{key_id}/permissions` exists in the backend, but nothing in the frontend calls it. **What:** Checklist step 1 now says to create a new key with the v2 permissions the calls need, plus `IDENTITY` (which `GET /me` requires), check it with `GET /me`, keep the old key for the calls still on v1, and delete it once nothing uses it. The "Permissions" bullet under "What changes" no longer implies the old key can take new permissions. **How:** The step links to "Create an API key" on the authentication page, which already says a key's permissions can't be changed. ### Changes 🏗️ - `docs/platform/integrating/migrate-from-v1.md`: checklist step 1 and the "Permissions" bullet. ### Agents and large language models used - Claude Code with Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Docs only; nothing to execute. **Verified** on `origin/dev` `e2316d0328` by reading the source. No file in `autogpt_platform/frontend/src` references the `UpdateKeyPermissions` operation (the generated client is not tracked, so nothing can import its hook), and nothing calls the legacy `BackendAPI.updateAPIKeyPermissions` method (`src/lib/autogpt-server-api/client.ts:442`). The settings and profile API-key pages use only `useGetV1ListUserApiKeys`, `usePostV1CreateNewApiKey` and `useDeleteV1RevokeApiKey`. The create dialog offers every v2 permission: `CreateAPIKeyForm.tsx:71` renders `PermissionsCheckboxGroup`, whose groups (`schema.ts:36`) put every permission that `src/lib/api-key-scopes.ts` marks `v2` under "API v2". `GET /me` requires `IDENTITY` (`backend/api/external/v2/identity.py:26`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Abhimanyu YadavandClaude Opus 5.5
|
b3d680f628 |
refactor(frontend): collapse colours to one role token set (SECRT-3154) (#16511)
### Why / What / How **Why.** The app named colours four ways: 47 role tokens, 27 shadcn alias names (`bg-background`, `text-muted-foreground`, `border-border`, …, 545 uses), the Tailwind palette (`text-orange-600`, `bg-gray-100`, ~400 uses) and hex literals (~900). Changing the theme could not change the whole app, and a dark theme would have to be written four times. This is PR A of the SECRT-3097 follow-up (SECRT-3154, plan items C1+C2+C3): one colour system, the roles. **What.** Every colour in `src/app`, `src/components` and `src/lib` is now a role class (`bg-surface`, `text-orange-foreground`, `ring-sky/30`) or, in a string, `rgb(var(--color-<role>))`. The raw values live in one `:root` block in `src/app/tokens/theme.css`, registered by `globals.css`'s `@theme inline` (the T3 Code layout: a later theme is a block of values, no component edits). The alias names are gone from the theme, and three lint rules at error with a baseline of 0 keep the other systems out. Artwork, logos, chart series and user-picked palettes are listed in `ds-colour-exempt.json` with a reason. **How.** - **Aliases (C1):** `scripts/ds/alias-to-role.ts` renamed each alias class to the role it already aliased (545 renames, 182 files), keeping variants, `!` and opacity, plus `var(--color-<alias>)` in inline styles. Because each alias rendered its role, this moves no pixel: `scripts/check-token-equivalence.ts --base <dev worktree>`, extended with the 27 alias→role pairs, reports **123 pairs, 0 failing; 1,785 opacity swatches, 0 differing pixels**. streamdown's prebuilt markup (scanned from `node_modules`) still uses seven shadcn names, so those seven are defined in `theme.css`, each aliasing a role; app code cannot use them (`no-alias-colours`). - **Palette and hex (C2+C3):** `scripts/ds/palette-to-role.ts` applied the reviewed table `scripts/ds/palette-map.json` (396 swaps): each palette step names its role as text, fill and line, and says whether the colour stays the same (`none`, 217 swaps), shifts a shade within its hue (`slight`, 176) or changes hue (`bigger`, 3). Hex in UI code became role variables by hand (xyflow edge arrow, tutorial icons, drag previews, the block-menu drag image, a banner hover, loader keyframes, chat-table glows, shadows). - **New roles, only where repeated uses needed a colour no role had:** `orange`, `orange-foreground`, `yellow`, `yellow-foreground`, `sky`, `sky-foreground`, `indigo`, `pink`, `pink-foreground` (usage meters, data types, category accents, admin diagnostics, run badges), and `scrim` (the black behind dialogs and in shadows, which must stay dark in any theme). Pale surfaces and borders of a hue are the role at an opacity rather than more roles: `bg-orange/10` is exactly orange-50, `bg-pink/10` exactly pink-50, and T3 derives its `-surface`s the same way, so a dark theme gets them for free. Everything else snapped to an existing role (rose → error, sky-50 → info-surface, yellow-50 → warning-surface, teal → success/sky). - **Lint:** `design-system/no-alias-colours`, `no-palette-colours` and `no-hex-colours` (error, all of `src/**` including stories, ratcheted at 0). Each error names the replacement (the aliased role, or the `palette-map.json` row). `ds-colour-exempt.json` is the only escape hatch: `scripts/__tests__/colour-exempt.test.ts` fails a suppression comment for these rules, an entry that matches no file, an allowed value the file no longer uses, and any hex in a stylesheet other than `theme.css`. `no-raw-colors` reads the same list and leaves palette and hex to the new rules, so nothing is reported twice. 120 `no-raw-colors` suppressions written for "this hue has no role" are gone. - Dead code found on the way: the builder's `hexColor` for output handles was passed to `<Icon color>`, which the handle's `text-handle` class always overrode; it is removed (no pixel change). ### Changes 🏗️ - `src/app/tokens/theme.css` (new): the one `:root` block of colour roles (57, 10 new), the `.dark` smooth-ring override that already existed, and streamdown's seven alias names. - `src/app/globals.css`: imports the token file; `@theme inline` registers the roles only (27 alias registrations removed); remaining raw colours (default ring colour, glows, loader keyframes, shadow) now read roles. - 300 call-site files: alias and palette classes swapped for roles (mechanical; see the scripts). No Text/Icon/Button atom internals changed beyond colour-token swaps (`compact-variants.ts`; the shared `styles/tones.ts` from #16513 keeps its `success-light` emerald, now exempt there). - `scripts/ds/` (new): `alias-to-role.ts`, `palette-to-role.ts`, `palette-map.json` (the reviewed table), `colour-exempt.ts`, `colour-utilities.ts`. - `ds-colour-exempt.json` (new): 31 entries, each with a reason. Whole files: the token file, the vendored Tailwind 3 reset, Notion avatar artwork, empty-state illustrations, the AutoGPT logo, the glass orb, confetti, the dither shader, Otto's avatar aura and greeting orb, the memory-graph canvas, recharts charts (SVG attributes, where CSS variables do not resolve), the React-artifact iframe document, the MCP OAuth callback page, the tour's scripted documents (support-ticket numbers that read as hex), sample artifacts in two stories, and three user-picked palettes (expert colour picker, folder colours, sticky notes). Single values: chart series in the expert overview and execution analytics, the avatar fallback palette, the onboarding cover's baked-in backdrop, a spinner mask, the code block's dark fallback, `text-emerald-300` / `text-red-300` / `text-red-400` on always-dark surfaces (the error and success roles drop below contrast there until the dark theme gives them dark values), and the overview story's "don't" examples. - `scripts/eslint-rules/design-system.mts`, `.oxlintrc.json`: the three rules; role advice now reads `theme.css`; `no-raw-colors` defers palette/hex and honours the exempt list. Baseline: `styles/tones.ts` `no-raw-colors` 1 → 0, nothing raised. - Tests: `colour-exempt.test.ts` (new); patch-coverage tests for the changed JSON renderer, usage bar, execution-path badge and builder type colours; `design-system.test.ts` (rule cases), `theme-aliases.test.ts` (only streamdown's names remain, each renders its role; dropped names generate nothing), `design-system-contract.test.ts` (reads `theme.css`), `tailwind3-compat.test.ts` (ring default reads `--color-info`), `edgeStore.test.ts` (marker colour). No test deleted. - `src/components/styles/color-roles.ts` + Tokens/Colors story: the new Accents group and `scrim`; the "shadcn names" story is removed with the names. - `scripts/check-token-equivalence.ts`: the alias→role pairs, so it proves the rename against any base. - Docs: `docs/engineering/design-system.md` (Tokens section) and `frontend/AGENTS.md` (Taste → Colour). - Not in this PR: `tailwind.v3-preset.ts` is untouched (PR C deletes it; streamdown's red classes and the exempt user palettes still need it until then), no dark values, `forcedTheme` unchanged, no theme switcher, no Storybook toggle. CODEOWNERS is unchanged (it is admin-owned); `src/app/tokens/`, `ds-colour-exempt.json` and `scripts/ds/` could join the design-system block there. - Size: 315 files, +2,932/−1,320 (4,252 lines), over the ~3,000 guide; 776 of those lines are the reviewed `palette-map.json` table and most of the rest are one-token swaps. Kept as one PR as asked. ### Visual changes Refreshed after merging `dev` |
||
|
|
8fa5a699b2 |
feat(platform): wake Otto after opt-in background subtasks (#15733)
### Why / What / How [SECRT-2813](https://linear.app/autogpt/issue/SECRT-2813): background helpers were lost when the parent turn closed its SDK client. #15127 made native helpers foreground-only; that safeguard remains in place. This adds explicit `run_sub_session(background=true)`. A fresh subtask runs through the durable queue after Otto finishes its initial reply. Completion, failure, cancellation, or timeout wakes the original chat for one **budgeted, tool-free summary**. Foreground remains the default. A durable handoff record tracks acceptance, outcome, and follow-up admission. Stable IDs and execution claims prevent duplicate model runs; scheduler reconciliation recovers missed dispatch and delivery. Both stages retain the original budget tree, billing context, credentials, owner, and organization/team/expert scope. ### Changes 🏗️ - Add `copilot-background-subtasks`, **off by default**, and an additive `ChatBackgroundTask` migration. No new service or environment variable. - Bound the child to 10 minutes including queue time and its follow-up to 2 minutes, without extending an inherited deadline. Recheck the original budget and permissions at promotion and pickup, including block restrictions. Child and summary share one admitted background node. Existing spend limits meter settled usage; they **do not reserve concurrent spend or guarantee zero overshoot**. - Show pending status, a subtask link, and Stop. The session endpoint reports `background_outcome` for a child chat and `background_pending_until` for a parent chat from the task record, so cards and the parent's late-reply polling do not depend on the launch or acceptance rows staying within the 50-row transcript page. Cards poll until that outcome lands, with unknown status at expiry if needed. Both chat runtimes refresh for late replies; missing message history during session refresh safely disables polling, and unrelated workflow-result or skipped-run notices do not stop a pending follow-up refresh. Synthetic turns have a readable label and stay out of user-message analytics. - Child Stop reports cancellation; stopping the launching parent turn also suppresses its automatic wakes. Stopping a queued follow-up atomically saves one cancellation notice and releases the session, including when later user messages have been persisted. Saved execution identity and atomic, scoped cleanup protect successor turns even after Redis loss. Deleted handoffs release only their own queued turn; admission clears the previous turn ID before a successor can be dispatched. Execution identity remains separate from a shared analytics trace. - Recover delivery by saved wake ID. Busy chats defer delivery and receive a terminal notice if the delivery window expires. Recovery preserves a follow-up when queue cancellation loses to promotion or newer work. A connection change before delivery produces one static notice at the next idle opportunity; scope changes still suppress delivery. The scheduler holds its single-job slot until reconciliation finishes, preventing overlapping sweeps. - Recover completed child work, and a delivered follow-up, after a failed outcome write only from a matching owner/claimed-turn success record; preserve committed cancellation and reject missing or mismatched evidence. The worker that still holds a child's turn takes the answer after a clarification it drained mid-turn; recovery stops at the next user row. Distinguish cancellation/deadline/failure notices, and invalidate the parent chat cache after notice commit. A handled execution deadline finishes normally at the queue boundary; operation and cleanup errors still propagate. Duplicate pickup leaves the active turn intact. Admission refusals close with the actual reason; unreadable background limits retain the node for its summary, and background refusals omit approval-answer copy. - Fence dispatch cleanup by turn ID; admission clears the recorded turn identity, so one admission records one turn and a dispatcher that stalled past orphan recovery refuses itself instead of overwriting its successor. A scheduler stop names the turn it expires, and the worker ignores a stop whose turn is no longer running. A new HTTP send that loses admission returns a retryable 409 before saving the message; a racing double send of the same text still rides the first send's turn, as dev's #15918 made it. Scheduled one-shots retry busy chats within existing limits; recurring schedules keep their next tick. Preserve dev's scheduled-chat failure tracking and tracing. - Document harness findings, lifecycle, budgets, and rollout in [the design note](docs/engineering/background-subtask-delivery.md). Rollout: apply the migration, deploy compatible API/database-manager/scheduler/workers, then enable the flag for test accounts. Keep it off while workers are mixed. Disabling it prevents new opt-ins while accepted work drains. Review focus: durable handoff and cancellation races, original scope/budget preservation, and the pending/Stop/follow-up experience. ### Agents and large language models used Codex with gpt-6-astra, running in T3 Code. Claude Fable 5.1 (Claude Code, in T3 Code) for the dev merge, its validation and the review rounds after 2026-11-03. ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description. - [x] I have made and executed a test plan: - [x] Reproduce native background loss using the locked Claude SDK (0.2.161), installed CLI (2.1.303), fake credentials, and a local scripted Anthropic server; verify the existing foreground regression. - [x] Exercise 74 real Postgres/Redis handoff scenarios: parent finishes first, single follow-up, duplicate pickup, busy parent, budgets, Stop, timeout, lost metadata, owner/route rejection, and successor protection. Reproduce expiry/promotion and cascade-deletion races before their fixes, including missing Redis metadata and the admission-before-dispatch window. All 396 focused handoff/queue/background/admission/chat-route cases pass, including 74 real-database scenarios. Queued Stop reproduces the lost cancellation reason before its fix; new controls cover later pending messages, ownership, promotion, notice-write rollback, idempotency, and cache visibility. Three existing real-database concurrent-cap/status-CAS cases also pass. - [x] Pass 93 queue/background cases and 234 scheduler/async-bridge/reset cases. Reproduce malformed-envelope budget refunds, misleading background refusal copy, and a scheduler job returning while its coroutine still runs; verify their fixes and normal-child/approval/credential-sweep controls. - [x] Pass the focused worker/dispatch/scheduled-watch suites, 150 tracing/tool/sub-session cases, and 5 worker execution-context cases. Negative controls fail when promotion admission, worker admission, completion persistence, timeout enforcement, or retained summary budget is removed. - [x] Reproduce a handled deadline escaping the worker as `TimeoutError`; pass 32 timeout, Stop, shutdown, and manager cases. The manager acknowledges handled deadlines and user Stop, while real operation and cleanup timeouts remain rejected. - [x] Pass all 119 selected background lifecycle/worker cases on the latest patch, including 87 real Postgres/Redis scenarios. This includes the strengthened timeout outcome/session-release/single-notice assertions. Reproduce both route-change silence and successful work becoming failed/timed_out after an outcome-write failure; cover one notice, cache visibility, busy-parent deferral, scope rejection, matching completion recovery, and wrong-owner/turn, missing/failed/unclaimed record and cancellation controls. - [x] Reproduce both schema-size CI failures, then pass all 333 schema/sub-session/background cases on the branch and its prospective merge with dev `c31ec68f7d`. Trim only tool prose: registry 77,149 < 77,176; session wire 69,654 < 69,788. Existing budget limits are unchanged. - [x] Pass frontend card/history/chat-lifecycle coverage (82 cases) and notification/helper/chat-lifecycle coverage (102 cases), including late delivery in both runtimes, idle-child recovery, expiry, and declined Stop feedback. Reproduce the missing-history polling crash from frontend CI, then pass all 34 helper/lifecycle/kickoff cases, including omitted/null history with successful sends and replies in both runtimes. Reproduce six missed-refresh cases when standalone workflow-result or scheduled-run notices arrive before the follow-up; pass all 82 polling/lifecycle/history/kickoff cases with coverage after sharing the existing standalone-message classification. - [x] Validate the SQL migration transactionally; export OpenAPI and regenerate the frontend client. Use a temporary Prisma client and disposable, task-owned Postgres/Redis services. Cleanup is verified: no task-owned Postgres/Redis services remain after the latest successful run. - [x] Run frontend format/lint/types, focused backend pyright, Black/isort, and repository pre-commit checks including Ruff and secret scans. - [x] Smoke-test with a live provider and all deployed services: the `/dev-review` bot's local run on `1f0a1bc` (2026-10-23) sent a background request, saw the approval card ("Run in background: Yes"), got the parent's immediate reply, and received the follow-up turn "Background subtask completed. Otto is following up." / "17 + 25 = 42" after the parent turn closed, with the `ChatBackgroundTask` row at `outcome=completed`, `followupState=finished`. - [x] Merge dev (542 commits, 11 conflicting files) and re-validate on the merge: every PR backend module imports; pyright 0 errors on the 38 touched backend files; black/isort/ruff clean; the OpenAPI export matches the committed `openapi.json`. In the Linux test image against a private Postgres/Redis/RabbitMQ stack: background handoff, turn-queue, session-waiter, sub-session, executor utils/tracing/manager/processor, chat routes, scheduler and tool-base suites pass (batches of 308, 501 and the patched files). Frontend: `tsc --noEmit` clean, 944 copilot lifecycle/ToolChain/helper tests pass, prettier/eslint/oxlint clean on the touched files. - [x] Review round on the merge (GitHub Copilot's six findings, each with a test): session-level background state on `GET /sessions/{id}` with a DB test and a route test; turn-bound scheduler cancel with a manager test; the one-admission-one-record CAS, follow-up success recovery and worker-held result selection with real-DB tests. Re-ran the handoff, executor, route, queue, scheduler and sub-session suites in the Linux image, plus tsc, eslint and the copilot Vitest suites. - [x] Two dev-drift fixes on the merge, each with a test: dev's #15918 expects an identical double send on a refreshed slot to ride the first turn (the PR refused it with a 409), so `schedule_chat_turn` now collapses only an identical trailing text and refuses anything else before saving; and dev's `HandoffStatus` plus the React Compiler lint rules needed `useSubSessionEffectiveStatus` to return `string | null` and the poll cap to be keyed state instead of a setState inside the effect. Local limitations: the full repository suite was not run. Full backend types report four unrelated Windows/Prisma errors, and full-file Ruff findings in queue/processor code are unchanged. Focused types and pre-commit Ruff pass. Direct checks replace the local isort wrapper's unsupported Poetry option and the type hook that runs from the repository root. Spend overshoot from concurrent settled usage remains an existing budget limitation. #### For configuration changes: - [x] `.env.default` is already compatible. - [x] `docker-compose.yml` is already compatible. - [x] Configuration changes and rollout are listed above. --------- Co-authored-by: gpt-6-astra (Codex) <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> |
||
|
Abhimanyu YadavandClaude Opus 5.5
|
aba0ba606d |
refactor(frontend): one text-size scale; Tailwind's sizes reset (SECRT-3152) (#16556)
### Why / What / How
**Why:** SECRT-3097 follow-up, PR D of the theming plan (item T1,
SECRT-3152). The app had two type scales: Tailwind's
`text-xs`…`text-4xl` (about 1,050 uses) and the design-system steps the
`Text` atom uses (`caption`, `body`, `lead`, `heading`…). Sizes from the
first can't follow a theme or a redesign, and the two drifted (18px,
24px and 30px headings sat next to 20px, 22px and 28px ones).
**What:** the design-system scale is now the only text-size scale. Its
values live in one token file, Tailwind's own sizes are reset so
`text-sm` renders nothing, and a lint rule fails any raw or arbitrary
size at error with a baseline of 0.
**How:**
- **Tokens, T3 Code style.** `src/app/tokens/typography.css` holds one
`:root` block with every step's size and line height. `globals.css`'s
`@theme inline reference` maps each step to its `text-<step>` utility
(`--text-body: var(--text-body)`), and `--text-*: initial` drops
Tailwind's defaults. A redesign or a theme changes values in that one
file. This is a separate file from PR A's `tokens/theme.css`, so the two
PRs don't collide.
- **Reviewed mapping + codemod.** `scripts/ds/text-size-scale.ts` holds
the scale and the mapping table below. It also has a codemod that
rewrites class names inside string literals and template text only (via
the TypeScript AST, so comments are untouched). It keeps variant
prefixes and `!`. It rewrote 1,049 classes in 390 files.
- **Two new steps.** I added two steps so no size moves more than 4px
(except one emoji):
- `heading-md`, 24/32: Tailwind's `2xl`. Snapping it to `heading-sm`
22/24 would have cut every such line box by 8px.
- `hero`, 64/1: the tour hero. The scale topped out at 44px.
- **Elements kept as they are.** Converting ~1,000 raw `<p>`/`<span>`s
to `<Text>` would also force `font-sans`, `font-normal` and a default
colour onto copy that today inherits them from its parent. That is a
pixel risk unrelated to size. So the codemod swaps the size class in
place, and the Text variants that were named after Tailwind's scale were
renamed or merged (below).
| Tailwind | Step | Size / line height | Change |
|---|---|---|---|
| `text-xs` | `text-caption` | 12/16 → 12/18 | line +2px |
| `text-sm` | `text-body` | 14/20 → 14/22 | line +2px |
| `text-base` | `text-body-lg` | 16/24 → 16/26 | line +2px |
| `text-lg` | `text-lead` | 18/28 → 20/28 | size +2px |
| `text-xl` | `text-lead` | 20/28 | none |
| `text-2xl` | `text-heading-md` (new) | 24/32 | none |
| `text-3xl` | `text-heading` | 30/36 → 28/40 | size −2px, line +4px |
| `text-4xl` | `text-title` | 36/40 → 32/40 | size −4px |
| `text-5xl`…`9xl` | `text-display` / `text-hero` (new) | | only `7xl`
was used: 72 → 64px |
Text with an explicit `leading-*` keeps it, so its line height does not
move.
### Changes 🏗️
**Scale and tokens**
- New `src/app/tokens/typography.css`: 14 steps, tiny (10px) to hero
(64px).
- `globals.css`:
- `@import`s the token file;
- maps the steps in `@theme`;
- resets Tailwind's `--text-*`.
- `tw-merge.ts` knows `heading-md` and `hero`.
**Call sites:** 1,049 raw sizes → scale classes (390 files, stories and
tests included), plus these hand edits:
- **Text variants named after Tailwind's scale:**
- `lg-semibold` and `lg-medium` were now identical to `lead-semibold`
and `lead-medium`, so they were merged into them.
- `2xl`, `2xl-bold` and `2xl-bold-tracking-tight` were renamed to
`heading-md`, `heading-md-bold` and `heading-md-bold-tracking-tight`.
- **New `hero` Text variant:** `TourHero` uses it instead of a
suppressed `text-[4rem] leading-none` restyle.
- **`FileIllustration`:** the 5px/6px code glyphs (`{`, `:`, `</`) in
the file thumbnails are artwork far below the scale's 10px floor. They
are now drawn as SVG text in the exact box the text took (measured:
advance width × 1.5em, baseline at 1.1em), so the odd sizes are gone
without changing the thumbnails.
- **Dialog title:** dropped `text-md`, a class that never existed and
generated nothing.
- **Resolved TODOs:** removed the `TODO(tokens): SECRT-3152` comments
this PR settles (7 files).
**Streamdown:** the chat's markdown renderer, a dependency, writes
Tailwind's size names in its own markup (headings, table cells, inline
code, code-block header, download menus).
- `globals.css` gives those classes the same steps inside
`:where([data-streamdown], [data-streamdown] *)`.
- It weighs one class and sits before the utilities, so sizes the app
passes in still win, as before.
**Lint**
- New `design-system/no-raw-text-size`, at `error` for `src/**` (stories
and tests included, since a raw size renders nothing there either). It
is off only for `__generated__`.
- It reports named Tailwind sizes with any variant prefix, and arbitrary
lengths (`text-[13px]`, `text-[length:..]`, `text-[calc(..)]`). The
message names the step to use.
- The baseline has 0 entries for it.
**Tests and docs**
- `scripts/__tests__/type-scale.test.ts` compiles the real stylesheet
and checks:
- every step's utility reads its token;
- the token file holds exactly the scale;
- Tailwind's sizes generate nothing;
- the Streamdown shim uses the same steps.
- Rule tests in `design-system.test.ts`. `tailwind3-compat` and
`tw-merge` tests are updated to the new names.
- `docs/engineering/design-system.md`, Typography: the scale, the
mapping and the rule. The token stories show the new scale.
**Counts**
| Measure | Before | After |
|---|---|---|
| Type scales | 2 | 1 |
| Odd/arbitrary sizes | 10 distinct (5, 6, 10–16px, 4rem) | 0 |
| Raw `text-<size>` uses in code | ~1,050 | 0 (lint error, baseline 0) |
**Size:** +1,753 / −1,235 lines in 399 files, about 3,000 changed lines.
It stays one PR as agreed; the bulk is one-token class swaps from the
codemod.
### Visual changes
**Method**
- Playwright against `pnpm dev` of `dev` (
|
||
|
Abhimanyu YadavandClaude Opus 5.5
|
2ce23c2b44 |
refactor(frontend): Text, Icon and Button variants instead of call-site restyles (#16513)
### Why / What / How
**Why:** SECRT-3097 follow-up, PR B of the theming plan. Text, Icon and
Button were restyled at the call site about 1,700 times (colour, weight,
padding, radius and motion classes in `className`). Every one of those
is a look the primitive doesn't know about, so it can't follow a theme,
and copies of the same look drift apart.
**What:** the looks those call sites were reaching for are now props on
the atoms. The call sites use the props, and `className` is left for
layout. `design-system/no-restyle` now also checks Text, Icon and Button
inside `src/components/{atoms,molecules,organisms}`, where it used to be
off. The baseline is whatever is left.
**How:** the props are cva variants that use role tokens only, and the
colour values are shared by all three atoms
(`src/components/styles/tones.ts`). Call sites were moved in two ways:
- **A codemod** (not committed) that rewrote a site only when the merged
class set (twMerge, with alias colours resolved to their role and
size/line-height compared as computed values) came out identical.
- **Hand edits** for the rest, under one rule: the same rendering, or a
change of at most 4px or a slight shade shift. Every such change is
listed below.
No colour tokens were renamed or removed (PR A's job), and the type
scale is untouched (PR D's job).
### Changes 🏗️
**New props**
- **Text:** `weight`, `mono`, `tabular`, `italic`, `decoration`,
`transform`, `tracking` (token steps only), `align` and
`motion="colors"`. The existing `tone` gains `base`, `emphasis-soft`,
`inverse`, `brand`, `brand-strong`, `accent`, `error-strong`,
`warning-bright`, `info` and `info-strong`.
- **Icon:** `tone` (same values as Text), `spin`, and `motion`
(`transform` | `colors` | `opacity`). Icon only cn-merges `className`
when one of these props is set, so existing class strings pass through
untouched.
- **Button:**
- variants `ghost-muted`, `ghost-subtle`, `ghost-danger`,
`ghost-warning`, `brand`, `warning`, `success` (the "copied" state) and
`tray` (the chat composer's icon buttons);
- sizes `row` (32px), `inline` / `inline-xs` (no box, for a link inside
copy) and `icon-2xs` (24px);
- a `shape` prop (`pill` / `rounded` / `soft`) and a `tone` prop for
text colour;
- `compact-ghost` gains an `aria-pressed` fill.
- Stories (Text `Modifiers`; new Icon stories; Button `QuietVariants`,
`RowAndInlineSizes`, `Shapes`) and unit tests for every new prop.
`docs/engineering/design-system.md` is updated.
**Call sites:** 380 files. Besides the prop swaps:
- Box styling that sat on a `Text` (padding, fill, border) moved to a
wrapping element.
- Buttons that only fade in on row hover moved the reveal to a wrapping
`flex` span. `focus-visible` became `has-[:focus-visible]`, which
behaves the same.
- `DateTimeInput`'s trigger is now a plain `<button>` with the exact
classes it rendered before, like `DateInput`'s.
- `MessageBranchPage` no longer uses `ButtonGroupText`, which had no
other callers and was removed.
**Lint**
- `.oxlintrc.json` puts the Text/Icon/Button contracts into the
atoms/molecules/organisms override and lets every other component
through there.
- The Text contract also accepts `gap-*` and `text-ellipsis` (layout and
truncation).
- The baseline was regenerated:
- New entries are only those 20 newly-checked component files.
- The emerald-300 raw colour moved with the tones from `Text/helpers.ts`
to `styles/tones.ts`.
- No other count went up.
**Restyle counts:** Text/Icon/Button classes reported by `no-restyle`,
on the same tightened scope, counting suppressed ones too.
| | Before | After |
|---|---|---|
| Text | 585 | 80 |
| Icon | 515 | 43 |
| Button | 592 | 182 |
| **Total class uses** | **1,692** | **305** |
| Call sites | ~880 | ~150 |
The plan's target was under 150. The call-site count meets it; the
class-use count does not. What's left is mostly bespoke one-offs:
- split buttons;
- segmented controls;
- gradient text;
- responsive heading sizes;
- line-height overrides with no variant (PR D will fold those into the
type scale);
- group-hover colours whose state React doesn't know;
- raw palette colours (PR A).
**Size:** about 5,800 changed lines across 393 files, over the ~3,000
the plan expected. It stays one PR as agreed; the bulk is one-token prop
swaps.
### Visual changes
**Method**
- Playwright, against production builds of `dev` (
|
||
|
Krzysztof CzerwinskiandClaude Opus 5.5
|
9e2a2c2846 |
feat(backend): remove Langfuse tracing of AutoPilot conversations (#16026)
### Why / What / How **Why:** AI observability moved to PostHog. Since the 23 Oct release, PostHog shows each AutoPilot reply as one trace with its tool steps (#15604, #15605) and every model call (OpenRouter Broadcast), so Langfuse tracing is a second copy of every conversation at another vendor. Product (Toran, 22 Oct) approved removing Langfuse entirely; this PR removes the tracing only. Prompt management stays in Langfuse until [SECRT-3022](https://linear.app/autogpt/issue/SECRT-3022) moves the prompts to PostHog. **What:** No path sends anything to Langfuse except the two prompt fetches. Ticket: [SECRT-2711](https://linear.app/autogpt/issue/SECRT-2711). **How:** The tracing code is deleted rather than switched off. The prompt fetches go through one Langfuse client built with `tracing_enabled=False` (`backend/util/langfuse_prompts.py`), and nothing else in the backend builds a Langfuse client or calls `langfuse.get_client()`, so the SDK never sets up a tracer or span exporter, whatever `LANGFUSE_TRACING_ENABLED` says. ### Changes 🏗️ Removed (tracing): - Baseline path: the `langfuse.openai` wrapper (main and aux clients are plain `openai.AsyncOpenAI` again), `propagate_attributes` and the OTel cost attributes on the turn span. - SDK path: `_setup_langfuse_otel` (LangSmith's Claude Agent SDK hook → OTLP → Langfuse), the Langfuse parent span per turn, and the `get_user_tier` lookup that only fed its metadata. - `copilot/sdk/trace_export.py` and its `opentelemetry_otlp_credential_provider` plugin entry in `pyproject.toml`; the `langsmith` dependency (lock drops `langsmith`, `uuid-utils`, `xxhash`). - `copilot/inference/trace.py`, the Langfuse trace around background LLM calls (briefing, dream, learning review, consult, style eval). The calls and their cost records are unchanged; `InferenceContext.trace_id` and the `langfuse_trace_id` cost-log metadata go with it. - The `openrouter-cost-reconcile` event written onto the Langfuse trace (the cost record itself is unchanged). - Ratings: no Langfuse score any more. Ratings are still saved in Postgres and sent to PostHog as `$ai_metric`, unchanged. `MessageFeedbackResponse.langfuse_target` is removed (the frontend never read it; openapi.json regenerated). - `scripts/replay_session_trace.py`, which replayed Langfuse traces. - The `langfuse_tracing_environment` setting. `ChatMessage.langfuseTraceId` and `ChatMessageFeedback.langfuseTraceId`: nothing reads them now, so they are no longer written. The columns stay; SECRT-3022 can drop them with the rest of Langfuse. Google data redaction (#15490): the Langfuse `mask` and the LangSmith `hide_inputs`/`hide_outputs` hooks are removed with the tracing. The rest of `copilot/google_data_redaction.py` stays, because other code still calls it: the Google call/output checks and remembering (`tools/base.py`, `gate/held.py`, `sdk/tool_adapter.py`), `GOOGLE_DATA_METADATA_KEY` (`tools/workspace_files.py`), `holds_google_data` and `tracing_google_data()`. Frontend replay masking is untouched. Kept as is: - Prompt fetching: `_fetch_langfuse_prompt` with the bounded-stale cache (`langfuse_prompt_name`, `langfuse_prompt_cache_ttl`) and the onboarding greeting prompt (`BRAIN_DUMP_GREETING_PROMPT_NAME`). - Our PostHog traces and spans (`ai_spans.py`, #15604/#15605) and the OpenRouter Broadcast attribution (`llm_attribution.py`, `x-session-id`/`x-user-id` headers). Docs: `.env.default` says the Langfuse keys are for prompts only; `docs/engineering/tracking-plan.md` and code comments no longer mention a Langfuse score or trace. Config: - Still used: `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST` (prompt fetches only). - No longer used, safe to remove from deployment secrets/config later: `LANGFUSE_TRACING_ENVIRONMENT`, and any `LANGSMITH_*` or `OTEL_*` variables (`OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_RESOURCE_ATTRIBUTES`, `OTEL_PYTHON_EXPORTER_OTLP_HTTP_TRACES_CREDENTIAL_PROVIDER`) if they were set by hand. The code used to set those at startup itself. Outside this repo, not done here: - Turn off the Langfuse destination in OpenRouter's Broadcast settings, if it is on (PostHog stays). - Decide whether to delete the traces already in Langfuse or set a retention period. ### Agents and large language models used - Claude Code — Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] New `backend/util/langfuse_prompts_test.py`: the prompt client is built from settings with tracing off, sets up no tracer even with `LANGFUSE_TRACING_ENABLED=true`, and is built once - [x] Unit tests for changed modules pass locally (feedback routes and service, prompt cache, onboarding greeting, Google data redaction, inference record/context, briefing, dream orchestrator, consult, OpenRouter cost, SDK service helpers and retry scenarios, db/model stamps); service-backed tests left to CI - [x] ruff/isort/black and pyright on changed files; frontend `pnpm types`, prettier and the `AssistantMessageActions` test - [ ] After deploy to dev: AutoPilot replies still use the Langfuse system prompt, and no new traces appear in Langfuse #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Krzysztof CzerwinskiandClaude Opus 5.5
|
3c06724a3e |
refactor(platform): remove LaunchDarkly and read every flag from PostHog (#15853)
### Why / What / How **Why:** LaunchDarkly gets users' email, email domain, role and signup date, without consent. Dev and prod have served every flag from PostHog since 16 Oct (infra #437/#439), and #15583 made PostHog the code default, so LaunchDarkly no longer answers any flag. Removing it closes gap 3 of the data-flow audit and drops a vendor. The team decided on 23 Sep to remove it entirely, experiments included. Ticket: [SECRT-2708](https://linear.app/autogpt/issue/SECRT-2708). **What:** Both LaunchDarkly SDKs go, along with the `launchdarkly` and `dual` flag backends and the switch between them, on the backend and the frontend. Flags behave exactly as they do today: the backend asks PostHog with the same person properties, and the browser reads `/api/feature-flags` as it already does everywhere. **How:** The backend's LaunchDarkly `Context` was also the user context PostHog's person properties were built from. It is now a small dataclass with the same fields, cache and degraded-lookup rules. The backend sends the same properties: `role`, `email`, `email_domain`, `created_at`, the per-request `country`, `environment` and `is_anonymous`. ### Changes 🏗️ **Backend** - Removed the `launchdarkly-server-sdk` dependency; `poetry.lock` regenerated with Poetry 2.5.1. The `launchdarkly` extra on `sentry-sdk` was already gone (#14805). - `backend/util/feature_flag`: - Every read goes to PostHog. - Removed: LaunchDarkly init/shutdown, `get_client`, the dual shadow reads and mismatch log, and `mock_flag_variation`. - Settings: removed `FeatureFlagBackend`, `feature_flag_backend` and `launch_darkly_sdk_key`. - Infra #452 (merged 23 Oct) already removed `LAUNCH_DARKLY_SDK_KEY` and the flag-backend setting from prod. If any environment still sets either variable, `Settings` allows extra keys, so it is ignored and boot is unaffected. A test covers this. - Deleted `scripts/feature_flag_sync.py` and `scripts/sync_feature_flags_to_posthog.py` with their tests. They only copied LaunchDarkly flags into PostHog. - `country` (#14799) already reaches PostHog as the person property `country`, per read and never stored. - It stays `country`, not `$geoip_country_code`: server-side reads get no GeoIP, and the trial-offer flag in PostHog targets `country`. - An anonymous stand-in context (a failed user lookup) still sends no country, as before. - `.env.default`: removed `LAUNCH_DARKLY_SDK_KEY` and `FEATURE_FLAG_BACKEND`. - LaunchDarkly wording in comments and docstrings now says PostHog or feature flag. - Two messages Otto reads now say "feature flag" instead of "LaunchDarkly flag". - Three OpenAPI descriptions changed; `openapi.json` was regenerated. **Frontend** - Removed `launchdarkly-react-client-sdk` (lockfile regenerated), `LDProvider` and the LaunchDarkly context building, including the device context from #14288. - Removed the `NEXT_PUBLIC_FEATURE_FLAG_BACKEND` switch with its `launchdarkly`, `dual` and client-side `posthog` paths. Every environment runs `server`, so `useFlagSource` now always reads `/api/feature-flags`. - Removed `environment.areFeatureFlagsEnabled` / `getLaunchDarklyClientId`, and the `setPersonPropertiesForFlags` call, which only ran in the client-side `posthog`/`dual` modes. - Retired the `feature_flag_mismatched` event, which only the dual backend sent. - Experiments: `useExperiment.ts` and `useReportAssignment.ts` stay on PostHog (the LaunchDarkly experiment bridge was already deleted). The browser's assignment `source` is now only `"posthog"`. - `.env.default`: removed `NEXT_PUBLIC_LAUNCHDARKLY_ENABLED`, `NEXT_PUBLIC_LAUNCHDARKLY_CLIENT_ID` and `NEXT_PUBLIC_FEATURE_FLAG_BACKEND`. - Tests for the LaunchDarkly, dual and client-side PostHog paths are removed. Tests for the remaining behaviour now mock the flag seam instead of the LaunchDarkly SDK. **Docs and image** - Updated `docs/engineering/feature-flags.md`, `activation-metrics.md`, `tracking-plan.md`, `docs/platform/contributing/managing-llm-models.md`, `frontend/CONTRIBUTING.md` and the threat model. - Single-container image: removed `NEXT_PUBLIC_LAUNCHDARKLY_ENABLED` and the step that deleted `ldclient`'s bundled test certificates. **Kept on purpose** - The backend `AssignmentSource` still accepts `"launchdarkly"`, so existing `experiment_assignment` rows recorded with it stay valid. - The stored `routingSource` value `"ld"` is unchanged, because it is a stored value. **Config changes** - Removed env vars: `LAUNCH_DARKLY_SDK_KEY`, `FEATURE_FLAG_BACKEND` (backend); `NEXT_PUBLIC_LAUNCHDARKLY_ENABLED`, `NEXT_PUBLIC_LAUNCHDARKLY_CLIENT_ID`, `NEXT_PUBLIC_FEATURE_FLAG_BACKEND` (frontend). - Leftover values are ignored. Infra #452 already removed the LaunchDarkly key and flag-backend setting from prod. Any remaining `FEATURE_FLAG_BACKEND` or `NEXT_PUBLIC_FEATURE_FLAG_BACKEND` values in infra or Vercel can be dropped whenever convenient. **Rollback:** revert this PR and [infra #452](https://github.com/Significant-Gravitas/AutoGPT_cloud_infrastructure/pull/452), so `LAUNCH_DARKLY_SDK_KEY` and the Vercel LaunchDarkly vars come back. Then set `FEATURE_FLAG_BACKEND=launchdarkly` (backend) and `NEXT_PUBLIC_FEATURE_FLAG_BACKEND=launchdarkly` (frontend, needs a rebuild). LaunchDarkly's flags must still match PostHog's, because LaunchDarkly hasn't served since 16 Oct. Reverting this PR alone keeps PostHog serving, since it is already the code default. ### Agents and large language models used - Claude Code — Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Backend flag unit tests (`backend/util/feature_flag/`, `api/features/feature_flags/routes_test.py`, `data/subscription_trial_config_test.py`, `util/metrics_test.py`) pass locally, including real PostHog clients evaluating the country rule, the employee cohort and the visitor split in-process - [x] Frontend `pnpm lint`, `pnpm types` and `pnpm test:unit` pass (flag, experiment, PostHog provider and affected page tests) - [x] `openapi.json` regenerated; `poetry.lock` and `pnpm-lock.yaml` regenerated - [ ] CI green - [ ] After merge and deploy: flags on dev and prod still match today's (e.g. `/api/feature-flags` for an employee and for a signed-out visitor) #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Bently
|
b9033c3f25 |
feat(backend): add Gemini 3.5 Flash, 3.5 Flash Lite and 3.1 Flash Lite to the LLM catalog (#15087)
### Why / What / How
**Why:** Google has announced Gemini 4 Argon, and we want the platform
ready for the Gemini lineup. Our catalog was missing three current
Gemini models that OpenRouter already serves. Blocks couldn't select
them, because an id without an `LLMModel` value isn't valid.
**What:** This adds Gemini 3.5 Flash, Gemini 3.5 Flash Lite and Gemini
3.1 Flash Lite (the non-preview release) as `open_router` models. Each
one gets a catalog entry, an `LLMModel` enum line and a pricing test,
the same pattern as #14943, #14697 and #14422.
**How:** Context window, max output and prices are OpenRouter's live
values from `GET /api/v1/models` on 2026-09-30. Credit rates are USD per
1M × 150, the same conversion as `_USD_PER_1M_DIVISOR`. OpenRouter
models are charged from the `x-total-cost` header, so the per-1M rates
here only drive the builder's "$X in / $Y out" label.
| Model id | Context | Max output | $ in / out per 1M | Cached input |
Credits in / out per 1M | Credits per run | Price tier |
|---|---|---|---|---|---|---|---|
| `google/gemini-3.5-flash` | 1,048,576 | 65,536 | 1.50 / 9.00 | 0.15 |
225 / 1350 | 4 | 2 |
| `google/gemini-3.5-flash-lite` | 1,048,576 | 65,536 | 0.30 / 2.50 |
0.03 | 45 / 375 | 1 | 1 |
| `google/gemini-3.1-flash-lite` | 1,048,576 | 65,536 | 0.25 / 1.50 |
0.025 | 37.5 / 225 | 1 | 1 |
All three set `supports_tools`, `supports_json_output` and
`supports_reasoning`, because OpenRouter lists `tools`,
`response_format` and `reasoning` for each of them.
**Not added:**
- **Gemini 4 Argon.** It isn't on OpenRouter yet. It's missing from
`/api/v1/models` and from `?q=argon`, and every plausible id
(`google/gemini-4-argon`, `google/gemini-argon`, `google/gemini-4`, and
preview variants) returns 404 on `/endpoints`. A catalog entry will
follow when OpenRouter lists it.
- **Gemini 3.6 Flash and 3.7 Flash.** Gemini 3.8 Flash replaces both at
the same price.
- **Other variants.** The `:batch` variants, the
`~google/gemini-*-latest` aliases, `3.1-pro-preview-customtools`, and
the image models (Nano Banana).
**Deprecation warning** (found while comparing against OpenRouter; this
PR doesn't change it):
- `google/gemini-2.0-flash-001` and `google/gemini-2.0-flash-lite-001`
have no OpenRouter endpoints left. `scripts/check_openrouter_prices.py`
reports both as `NO_ENDPOINTS`.
- OpenRouter shows `expiration_date: 2026-10-20` for
`google/gemini-2.5-flash`, `google/gemini-2.5-flash-lite` and
`google/gemini-2.5-pro`.
- `google/gemini-2.5-flash-lite` is the default model for the simulator
(`executor/simulator.py`) and for AutoPilot's simulation mode
(`copilot/config.py`). Both need a new default before 2026-10-20.
**Related:** #14386 adds a separate route, a direct Google Gemini
provider. This PR doesn't overlap with it or change it.
### Changes 🏗️
- `backend/data/llm_registry/catalog.py`: new catalog entries for
`google/gemini-3.5-flash`, `google/gemini-3.5-flash-lite` and
`google/gemini-3.1-flash-lite`.
- `backend/data/llm_registry/llm_models.py`: `GEMINI_3_1_FLASH_LITE`,
`GEMINI_3_5_FLASH` and `GEMINI_3_5_FLASH_LITE` enum values.
- `backend/data/llm_registry/catalog_test.py`: a pricing test per model
that checks run credits, per-1M rates, max output, context window and
price tier.
- `docs/integrations/block-integrations/llm.md`: the new ids added to
the model lists in enum order.
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Before, on `dev` (
|
||
|
Reinier van der Leer
|
6c85f3b1ee | Merge branch 'master' into dev |
