1 Commits

Include renames

Author	SHA1	Message	Date
4 people	b7b75494f4	feat(backend/api): External API v2 (#12206) ... Adds v2 of the platform's public API at `/api/v2`: REST endpoints for a user's graphs, library agents, folders, presets and webhook triggers, runs and their human-in-the-loop reviews, schedules, workspace files, credits, credentials and marketplace submissions, plus an MCP server exposing AutoPilot's tools, all callable with an API key or an OAuth token and gated by per-resource scopes. - Resolves #11678 - Partly based on #12284 ### Why - v1 can create and execute a graph, execute a block, read the marketplace and manage integration credentials, and nothing else. - An app signing users in through OAuth (#11617) cannot list their agents, run a saved preset, set up a webhook trigger, follow, stop or share a run, answer a review, schedule a run, or read their files and spend through v1, so it cannot use the platform as its backend. ### What New surface, in `backend.api.external.v2` (too much to review whole; read `mcp_server.py`, `mcp_calls.py`, `mcp_tenancy.py` and `library/presets.py`, and scan `models.py`): - Resources: `/graphs`, `/library/agents`, `/library/folders`, `/library/presets`, `/runs`, `/runs/reviews`, `/schedules`, `/files`, `/credits`, `/integrations/credentials`, `/marketplace`, `/blocks`, `/search`, `/me` - MCP server at `/api/v2/mcp`, with its RFC 9728 metadata at `/.well-known/oauth-protected-resource/api/v2/mcp`: AutoPilot tools that opt in through `allow_external_use`; `tools/list` shows only the tools the caller's scopes allow; a refused or failed call returns an `isError` result - 13 new `APIKeyPermission` scopes, one per resource and verb (`schema.prisma`, migration `20261008140000_add_v2_api_key_permissions`; `READ_STORE` was already on `dev`), each described on the OAuth consent screen and in the API key dialog - v1 stays at `/external-api/v1`, deprecated: it stops working on 2027-02-01, after its last day of service on 2027-01-31 - v2 guides under `docs/platform/integrating/`, and the v2 OpenAPI spec published to GitBook from `master` (`docs-gitbook-openapi.yml`) Who can call what: - Every route resolves its credential to one organization and team; a row in another organization is a 404, over REST and MCP alike (`mcp_tenancy.py` checks every id argument of every exposed tool before it runs). - A banned account's API key or OAuth token is refused on every path: `X-API-Key`, an API key sent as a Bearer token (how MCP clients send one), an OAuth token, and the MCP server's own token check. - A graph someone else owns, a colleague's or a marketplace agent's, is read as its shared copy, as on the internal routes: no owner ids, credential references or secret inputs. - A colleague's run or graph in the caller's organization is readable but not writable: `DELETE /runs/{id}`, `POST /runs/{id}/stop`, `PUT /graphs/{id}` and `PUT /graphs/{id}/versions/active` answer 403 before acting. - An account without an active plan gets a 402 (`payment_required`) from every route that starts work, stores files or calls out, and from every MCP tool call. Reads, billing, deletes, stopping a run, sharing and organising stay open, as on the internal API since #15676. - MCP tool calls run outside the action approval gate: an MCP session is an automation's, so nobody is asked to approve a card, and the credential's scopes are the authorization, as in REST. A run an MCP call starts is tagged an API run, as v1's are, so its reviews come back through `/runs/reviews` rather than a chat that was never saved. Runs, presets and triggers: - `POST /library/agents/{id}/runs` and `POST /library/presets/{id}/execute` take an `Idempotency-Key`: a repeat returns the run the first request started, a repeat while that one is in flight is a 409, and the key reused on another agent, preset or body is a 422. - Both answer 422 for an input the graph does not have, or a required one omitted or `null`, with one `details.errors` entry per input (for a preset, unless it is a trigger, whose inputs are its block's settings); `POST /schedules` does the same, and schedules the active version when the request names none. - Presets: list, get, create, update, delete and execute. A preset must name a graph in the key's organization. One created for a graph that a single hired expert installed belongs to that expert, as on the internal routes, and so do triggers and schedules. Executing one runs for the expert it belongs to (409 while that expert is paused) and lands in the preset's organization. - Webhook triggers: `POST /library/presets/setup-trigger` registers the webhook and returns `webhook_url`, where a hand-configured service sends its events. New trigger `inputs` or `credentials_inputs` re-register the webhook, and whichever a PATCH leaves out keeps its saved value. Deleting a trigger removes its webhook once nothing else uses it. - A trigger's secret settings (a generic webhook's `secret_token`, All Quiet's `signing_secret`) are returned only when the trigger is created. Later reads leave them out of `inputs` and list them in `hidden_inputs`, and an update that leaves them out keeps them. - An active trigger starts runs, so arming one needs `RUN_AGENT` as well as `WRITE_LIBRARY`: `setup-trigger`; a PATCH that sets `is_active` to true, or gives a trigger that stays active new inputs or credentials (which re-register its webhook); and creating an active preset of an agent with a webhook trigger. Over MCP, `update_preset` needs it for `is_active: true`, and for new `inputs` unless they come with `is_active: false`, since the tool can't see the graph before it runs. - Over MCP: `list_presets`, `update_preset`, `delete_preset`, `list_agent_triggers` and `setup_agent_webhook_trigger`, scoped to the key's organization as above. Everything else: - `POST /schedules` creates a schedule for the `graph_id` in its body. - `POST /files/upload` writes to the workspace `GET /files` lists, and returns the file's id and a `workspace://` URI. `GET /files` lists the user's files, not Otto's working files or skill packages, as the Files page does. - Every response carries `X-RateLimit-Limit`, `-Remaining` and `-Reset`, and a 429 adds `Retry-After` (none when Redis is unreachable). An API key is counted per client and key head before it is hashed, and a request is verified once; a limiter that fails open logs only its own name, never the bucket. - A v2 request body is capped at 10 MB, MCP included; an upload gets its file cap plus 1 MB of framing, and a marketplace media upload past 10 MB gets a 413 as soon as it passes the cap. - `POST /runs/{id}/reviews` takes `auto_approve_future`, and answers 404 for an unknown review and 409 for a run that is not awaiting review. - One spelling per concept: `inputs`, `credentials_inputs`, amounts in `_cents`, durations in `_seconds`, times as ISO datetimes; one `ErrorResponse` body on every error. MCP results and 500s carry no internal exception text. - `POST /graphs` and `PUT /graphs/{id}` validate a version's credentials only when it is saved active. Activating a version moves the graph's webhook presets onto it only if the credential also has `RUN_AGENT`. - The served `openapi.json` lists only the declared `servers` (`root_path_in_servers=False`), not the mount path FastAPI would put first. Changes to existing code: > [!NOTE] > For an overview of all changes to the internal API, check out [`autogpt_platform/frontend/src/app/api/openapi.json`](https://github.com/Significant-Gravitas/AutoGPT/pull/12206/changes#diff-a1bb1783d6023599f9aaba5414bfd0b49637b61dad8da09c7bed1b2ee01ba852) - The external middleware resolves every credential in one `resolve_auth_info`, verified at most once per request, and refuses a banned owner's on each form; `require_api_key` and `require_access_token` are gone. - The internal upload route and the internal review route call the services v2 uses: `workspace/service.py` and `graph_executions/review/service.py`. The review service carries #15564's step-card handling and #15062's chat-queue wake. - `workspace/service.py`'s size-capped reader is public (`read_within_size_limit`), and refuses before reading when the multipart parser already counted the size. - The copilot's `run_agent` checks inputs with `unknown_inputs` and `missing_inputs` (`data/graph.py`), which v2 shares; a required input sent as `null` now counts as missing there too. - `db.update_preset` stores a preset's inputs and credentials in the same rows, and now rewrites them whenever both are given, either of them possibly `{}`. Before, it refused new inputs beside empty credentials, which the v2 PATCH and the copilot's `update_preset` tool both send for a preset that needs none, and it silently kept the old inputs when every input was cleared. `update_triggered_preset` prunes a webhook it registered for an update that then fails, unless it is the one the trigger already had. - The copilot's preset and trigger tools scope their listings, and the library entry a trigger setup creates, to an External API caller's organization; in a chat they behave as before. - `UserTransaction` is renamed `UserCreditTransactionAdminView` (admin credit routes and the admin spending page). - The Redis fixed-window rate limiter is extracted from `search/rate_limit.py` into `api/utils/rate_limit.py`, which v2 shares; #15703's public store-search limiter sits beside it. Its Redis round trip is bounded at 0.25 s and fails open past it, because every v2 request waits on several of these checks. - The store cache's helpers are public (`get_cached_store_agents` and the rest), for v1 and v2. - New setting `TRUSTED_PROXY_COUNT` (default 3), commented in `backend/.env.default`. - The API key dialog lists permissions in three groups, "API v2", "API v1 and v2" and "API v1 only (stops working 2027-02-01)", each with the consent screen's description, and a key's info dialog marks its v1-only permissions. The descriptions and each permission's API version live in `lib/api-key-scopes.ts`, which the consent page reads too; both are typed `Record<APIKeyPermission, ...>`, so a new permission fails tsc until it is described and classified. - `oauth-tool` offers every permission, v2's included, read from the enum. - Library data layer: `list_library_agents` takes `published` and `favorite` filters, and matches `published` by the graph's live, approved listing; `get_library_agent` fetches the marketplace listing with its active version and creator profile in one query; `add_store_agent_to_library` drops unneeded queries. - HITL data layer: `get_reviews` with optional filters; `get_pending_reviews_for_user` and `get_pending_reviews_for_execution` wrap it; `graph_runs_only` leaves out AutoPilot chat reviews. - `get_graph_all_versions` takes `include_subgraphs` and `offset`; `get_active_graph_version` returns a graph's active version number, and `get_graph(version=None)` keeps returning the newest. - `delete_store_submission_or_raise` raises why a submission couldn't be deleted; `delete_store_submission` wraps it and keeps answering true or false. - `GraphSettings.from_graph()` and `update_library_agent_version_and_settings` are removed (replaced by `GraphSettings()` and `update_agent_version_in_library`). - `PATCH /api/store/profile` accepts partial updates; `Profile` is replaced by `ProfileUpdateRequest`. `PATCH /library/agents/{id}` with `graph_version` points an entry only at the caller's own version in the key's organization, or at another user's version approved in the marketplace. - The consent page's Deny button does nothing until the app's info has loaded, which happens only once the backend has accepted the `redirect_uri`; the rest of that fix reached `dev` with #15057. Two tests pin the app-info endpoint's `redirect_uri` check. - `add_auth_responses_to_openapi` works with any auth scheme; exception handler setup is extracted from `rest_api` into `api/utils/exceptions.py:add_exception_handlers`; `export-api-schema` exports the external APIs' OpenAPI schemas too. - `# pragma: allowlist secret` on existing lines the secret scanner flagged; fast `pre-commit` checks run before slow ones. ### How - v2 is its own FastAPI app, mounted on the REST app at `/api/v2` beside the internal routes; v1 stays under `/external-api` (`backend.api.external.fastapi_app`). Each has its own `/docs`, and neither serves metrics, which the REST process serves on its own port. `rest_api_test.py` keeps v2 and the internal API apart: no internal route may start with `/api/v<n>`, and every v2 path sent to the host app must reach the v2 mount rather than an internal route with a path parameter. - Every v2 route depends on `require_permission(<scope>)`, which resolves the API key or OAuth token to a `TenantContext` (user, organization, team). A route that needs a plan depends on `require_paid_permission(<scope>)` instead, which adds the internal API's `enforce_payment_paywall`. v2's own `paywall_routes_test.py` fails on a route that neither checks nor sits on its open list. - An MCP call is held to the credential before the tool runs (`mcp_calls.py`): unknown arguments are refused, an argument that does more than the tool's own permission needs the matching one (`external_permissions`), ids are checked against the organization, and a tool that bills the platform runs the paywall and usage-cap checks. - The MCP server is served through `MCPMount`, which the platform app's lifespan runs (a mounted app's lifespan never runs); it answers 503 before startup. - The preset routes call the internal API's trigger helpers (`library/triggers.py`), so webhook registration and cleanup are one implementation. The hidden trigger settings are read from the trigger blocks' own inputs marked `secret`, so a new trigger block is covered without a list to maintain. - The anonymous rate-limit bucket keys on the `X-Forwarded-For` entry our own proxies appended, `TRUSTED_PROXY_COUNT` (default 3: Cloudflare, then Google's load balancer) places from the right; `0` uses the socket peer. - Idempotency keys live in Redis, held per user, organization and team, where the run lands. A key that can't be claimed is a retryable 503; an in-flight key holds for 24 hours, so a request that dies after its run started never lets a retry start a second one. - `contract_shape_test.py` checks the published OpenAPI document, so a later model cannot reintroduce a retired spelling or an amount without its unit. ### Known gaps - `pause_schedule` and `resume_schedule` are not exposed over MCP pending a decision on exposing them under `WRITE_SCHEDULE`. - MCP OAuth authorization-server discovery (RFC 8414) is not published: without dynamic client registration, and with a token endpoint that takes JSON rather than a form, it would advertise a flow a conformant client cannot complete. - `/search` still counts the user's library agents in other organizations in `total_count`; it needs the organization in the shared hybrid-search query. - One valid key presented from many addresses still costs a hash per request, up to 300 a minute per address. - The internal API and v1 have no request body cap; this PR caps v2 only. - v2's requests are counted under the REST API's metrics (`autogpt_rest_api_*`, handlers `/api/v2/...`), not the external API's, so the dashboard's External API panels show v1 only until a separate infra change adds v2's handlers to them. That change also routes `/api/v2` on `dev-api.agpt.co`, which routes only `/external-api` today. ### Verified On `7b1b7cf192` I ran `backend/api/external/`, the library, review, search and store suites, all of `backend/copilot/tools/`, `architecture_test.py`, both paywall route tests and `test_block.py`: 8071 passed, 84 skipped. What did not pass is this machine's, not this branch's: the store skill suites refuse a database that already holds listings (62 errors at setup), the browser tests need Chromium (5), `test_ensure_content_embedding_blocks` fails on clean `dev` here, and `skill_file_routes_test.py` passes alone (9 of 9). pyright is clean over the backend, and the frontend type-checks and its auth page tests pass (45). The commits after it (two test-only fixes, the rate-limit warning, the file listing, the shared graph copy and the MCP run's tagging) were checked with `backend/api/external/`, `copilot/gate/`, the graph routes, `run_agent`'s tests and the executor's. Each new guard turned its test red when removed: the ban check on each of the four credential paths, the hidden trigger settings on read and on update, the `RUN_AGENT` requirement over REST and MCP, the rate-limit warning naming its bucket, Otto's files in the listing, a non-owner's graph read whole, and an MCP session made interactive, which parks an approval. None of this ran against a live stack. On `20e9247c22`, which moves v2 to `/api/v2`, groups the dialog's permissions and moves v1's end date: `backend/api/external/`, both paywall route tests, `rest_api_test.py`, `public_metrics_test.py` and `architecture_test.py` (472 passed), `test_block.py` (3714 passed, 84 skipped), and the API key, consent and profile frontend suites (135 passed), with tsc clean. The two route guards each went red on a mutation (an internal `/api/v3` route; an internal `/api/{x}/library/agents`, which answered `GET /api/v2/library/agents`), as did the dialog tests and the tsc check on the version map. A REST server started from this tree served `/api/v2/openapi.json` and `/api/v2/docs`, answered `/api/v2/me` and `/api/v2/mcp/` with 401s pointing at the new discovery document, served that document, still served v1 at `/external-api/v1`, and answered 404 at `/external-api/v2`. On `f4db212b6b`, which fixes the preset update: `backend/api/features/library`, `backend/api/external/`, the copilot's preset tool tests, the webhook ingress tests and `architecture_test.py` (680 passed), and `test_block.py` (3714 passed, 84 skipped). The two PATCH tests drive the real `db.update_preset` and went red with the old guard; the prune went red when removed and when it pruned the trigger's own webhook; and the tests added for a listing's and an update's withheld trigger secrets and a trigger's run each went red when that code was reverted. On `61dd619efb`, which merges `dev` at `ea588f7375` (#15057's redirect checks and client-id encoding, #16175's invitation-only membership): the consent page conflicted, and keeps the shared descriptions module, the Deny guard, `dev`'s `encodeURIComponent` on the app-info request and `dev`'s test file. The internal spec equals a fresh export. `backend/api/external/`, the library, OAuth, orgs, paywall, `rest_api` and public-metrics tests, the copilot's preset tool tests and `architecture_test.py` passed (860), as did `test_block.py` (3714 passed, 84 skipped) and the consent, setup-wizard, proxy and API key frontend suites (324), with tsc clean. On `76ea8cf9e9` through `0a63dac8cf`, which close the remaining review findings, each fix has a test that goes red without it, through the real route, or the real `db.update_preset` where the bug sat between layers: credentials-only and clear-all PATCHes; a limiter whose Redis never answers; an idempotency key reused under another team; expert attribution on presets, triggers and schedules; a key holding only `WRITE_LIBRARY` refused when it arms an active trigger by PATCH or creates an active trigger preset, and the MCP equivalent; a failed attach pruning its webhook; an MCP tool failure returning no internal text; and a team-pinned key stopping once its membership ends. On `0a63dac8cf`, `backend/api/external/`, the library suites, the copilot's preset and trigger tool tests, both paywall route tests and `architecture_test.py` passed (703), as did `test_block.py` (3714 passed, 84 skipped). ### Agents and large language models used - Claude Code with Claude Opus 5.5 (implementation, merges, tests, docs) - T3 Code delegated reviewers for #15376's security review rounds: Claude Mythos 5.1 and Claude Fable 5.1 (Claude), GPT Daybreak Blue (Codex) ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] The `backend/api/external/` package and the library, graphs, review, workspace and copilot tool suites, on this head - [x] Every v2 route's scope and tenant check (`permissions_test.py`, `tenancy_test.py`), and every MCP tool's id arguments (`mcp_tenancy_test.py`) - [x] The published OpenAPI contract (`contract_test.py`, `contract_shape_test.py`) #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: Nicholas Tindle <[email protected]> Co-authored-by: Nicholas Tindle <[email protected]>	2026-10-26 00:14:28 +00:00