mirror of
https://github.com/agno-agi/agno.git
synced 2026-10-12 02:10:54 +00:00
KaustubhandClaude Fable 5.1
1069000f7c
## Summary Before: <img width="537" height="251" alt="image" src="https://github.com/user-attachments/assets/d72630df-d7a8-4b64-b49b-c7adee008f5f" /> After: <img width="523" height="532" alt="image" src="https://github.com/user-attachments/assets/dcb94cbe-6acc-47be-b215-f156099d90e9" /> External agents (`ClaudeAgent`, `CodexAgent`) now report run and session metrics the same way native agents do, so their runs and sessions show tokens and cost in AgentOS: on the run output, on the streamed `RunCompleted` event, on the session, in the sessions list token column, and in the daily `/metrics` aggregation. Before this, `RunOutput.metrics` was always empty for external agents and their sessions were invisible to the metrics page. **Base (`agents/base.py`)** - `ExternalRunResult.metrics` for non-streaming adapters, and an `ExternalRunMetricsEvent` that streaming adapters yield; the base consumes it and does not forward it to clients. - A per-run timer in the base sets `duration` and, when streaming, `time_to_first_token`, so every external agent measures these the same way. Timing is wall clock from the adapter's point of view (it includes process startup), matching how native agents measure it. - `_build_metrics(totals, models, additional_metrics)` assembles `RunMetrics` from a framework's turn totals and its per-model entries. Adapters only parse their SDK's usage object. Totals are taken from the framework's own figures rather than summed from the model entries, because Claude Code counts its helper model separately from the main conversation. - `_apersist_run_in_session` accumulates completed runs into `session_data["session_metrics"]` (same `SessionMetrics.accumulate_from_run` path as native agents), before the worker-owned write so background runs are covered. The persistence fallback re-reads the session, so a failed write cannot double count. **Claude (`agents/claude/agent.py`)** - Maps the SDK `ResultMessage`: `usage` for tokens, `total_cost_usd` for cost, `model_usage` for one `ModelMetrics` per model (Claude Code's Haiku helper model shows up as its own entry with its own cost), `num_turns` and `duration_api_ms` into `additional_metrics`. - Like the native Anthropic model, `input_tokens` excludes the cached prefix; cached tokens are in `cache_read_tokens` and `cache_write_tokens`. Cache reads count every API call inside the turn, so a turn with one tool call reports roughly two reads of the ~15k harness prefix. **Codex (`agents/codex/agent.py`)** - Codex reports usage after every model request as `thread/tokenUsage/updated`, with `last` (that request) and `total` (the thread so far). A turn with tool calls makes several requests, so the turn's usage is the growth of the thread total over the turn: the final `total` minus the `total` before the turn (the first report's `total` minus its own `last`). Reading only `last`, or the SDK's `TurnResult.usage` which is just the latest report, counts a single request (review finding from @ysolanky, replicated: a two-tool turn recorded 14972 of 44739). - The non-streaming path now consumes the notification stream itself (`_acollect_turn`, same rules as the SDK's `run()` for items and the final response) so it sees every usage report; the streaming path keeps the first and last report. A repeated report does not double count. Like the OpenAI API, `input_tokens` includes the cached prefix. Codex reports no cost, so cost stays unset. **Cookbooks** - `cookbook/frameworks/claude-agent-sdk/metrics.py` and `cookbook/frameworks/codex/codex_metrics.py`: run metrics for a plain and a streamed turn, then session totals. - `cookbook/frameworks/claude-agent-sdk/metrics_agentos.py` and `cookbook/frameworks/codex/codex_metrics_agentos.py`: serve the agent through AgentOS; `--verify` drives two runs over HTTP and prints the run metrics, the session totals, the sessions list column and the `/metrics` aggregation. - READMEs explain the conventions (cache token handling, why Claude cache reads look large) and TEST_LOGs record the live runs. Builds on the metrics mapping sketched in #9027 for Claude; that PR's `adapter_state` side-channel is replaced by the result field and event above, and nothing from it is rebased. ## Type of change - [ ] Bug fix - [x] New feature - [ ] Breaking change - [ ] Improvement - [ ] Model update - [ ] Other: --- ## Checklist - [x] Code complies with style guidelines - [x] Ran format/validation scripts (`./scripts/format.sh` and `./scripts/validate.sh`) - [x] Self-review completed - [x] Documentation updated (comments, docstrings) - [x] Examples and guides: Relevant cookbook examples have been included or updated (if applicable) - [x] Tested in clean environment - [x] Tests added/updated (if applicable) ### Duplicate and AI-Generated PR Check - [x] I have searched existing [open pull requests](https://github.com/agno-agi/agno/pulls) and confirmed that no other PR already addresses this issue - [x] If a similar PR exists, I have explained below why this PR is a better approach - [x] Check if this PR was entirely AI-generated (by Copilot, Claude Code, Cursor, etc.) --- ## Additional Notes **Verified against the SDKs on the same runs.** The raw SDK result was captured inside the adapter's own runs and compared field by field with what Agno recorded. - Claude: input, output, total, cache read, cache write, cost, turn count, API duration and every per-model entry identical on a plain turn and on a streamed tool turn. Session cost equals the sum of the SDK cost figures to seven decimals. - Codex: input, cached input, output, reasoning and total identical to the SDK's usage reports on single-request turns, and on a two-tool turn (three requests) the recorded usage equals the thread total growth on both paths: streamed 44725 of 44725, non-streamed 43289 of 43289. Session total equals the SDK's own running thread total, so nothing is double counted. - The only intentional difference is `duration`: the SDKs report their own turn time, Agno reports wall clock including process startup. The SDK figure is still available for Claude as `additional_metrics["api_duration"]`. **Verified in AgentOS** through the test client and the UI: run output has `metrics`, the streamed `RunCompleted` has `metrics`, `GET /sessions/{id}` returns the totals with per-model details, `GET /sessions` shows the token column, and `GET /metrics` counts the runs and tokens for the day. **Unit tests**: eleven new tests across `test_claude.py` and `test_codex.py`, including three-request turns on both paths where `last` differs from the per-turn delta, a resumed thread with earlier usage, and a repeated usage report; the non-streaming fakes now drive the notification stream against the existing fake SDKs: run metrics from the result message or usage, streamed metrics on `RunCompleted` without a forwarded event, session accumulation across runs with per-model details, and duration reported for runs without usage. Agents and run suites pass; ruff and mypy clean. **Why the Codex numbers look large**: a one-line question costs ~15k input tokens because Codex's prefix is its base instructions (~3.5k), a developer message with skills, permissions and plugin instructions (~3.1k), environment context, and the schemas for built-in tools, configured MCP servers and plugins. The metrics report it as billed. Claude's `total_tokens` looks small by comparison because cache tokens are kept in their own fields, following the native Anthropic model. --------- Co-authored-by: Claude Fable 5.1 <[email protected]>
Build, run, and manage agent platforms.
Introduction
Agno is a framework and runtime for agent platforms. Build agents, run them as a service, manage your platform using a web UI.
- Build your agent platform using the Agno SDK.
- Run your agent platform using the AgentOS runtime.
- Manage everything using the AgentOS UI.
Agno allows you to own your agent stack. Maintain control of your data, memory, and security posture (JWT-based RBAC), and turn your agent platform into a learning loop with simulations and usage data.
Get started
Hand this prompt to your coding agent (Claude Code, Cursor, Codex):
Help me set up my agent platform.
Clone https://github.com/agno-agi/agentos-railway into a folder called
agent-platform, cd in, read the README, and follow the get started guide.
Your coding agent will set up your agent platform and run it locally using Docker, giving you a REST API for serving your agents, a Postgres database for storing your data and traces, an MCP server, and a control plane.
Deploying somewhere else? Use the same prompt but point it to a different repo. The starter templates are identical except for the deploy scripts: swap agentos-railway for agentos-docker, agentos-aws, agentos-gcp, agentos-azure, agentos-fly, agentos-render, agentos-modal, or agentos-helm.
Prefer to code by hand?
Features
- Production API. 50+ endpoints with SSE and websockets to build a product on top.
- Storage. Store sessions, memory, knowledge, and traces in your own database.
- 100+ integrations. Connect to GitHub, Slack, Postgres, and more using pre-built toolkits.
- Context Providers. Access live data from Slack, Drive, wikis, MCP, and custom sources.
- Human approval. Pause runs for user confirmation. Block tools that require admin approval.
- Observability. Monitor with OpenTelemetry tracing, run history, and audit logs.
- Security. Get JWT-based RBAC and multi-user, multi-tenant isolation out of the box.
- Interfaces. Expose your agents via Slack, Telegram, WhatsApp, Discord, AG-UI, A2A.
- Scheduling. Cron-based scheduling and background jobs with no external infrastructure.
- Deploy anywhere. Run on any cloud platform that runs containers. Docker, Railway, AWS, GCP.
Use Agno with your coding agent
Two options:
- Recommended: Add Agno docs as an MCP server. Add docs.agno.com/mcp to your favourite coding agent.
- Add Agno docs as an indexed source. In Cursor: Settings → Indexing & Docs → Add
https://docs.agno.com/llms-full.txt. Also works in VSCode, Windsurf, and similar tools.
Read the full guide here.
Community
- (https://x.com/AgnoAgi): follow for releases and demos
- Newsletter: monthly updates on what's shipping
Contributing
See the contributing guide.
License
Agno is distributed under the Apache-2.0 license.
Telemetry
Agno sends a telemetry event per agent run so we know which model providers to prioritize. Prompts, messages, and outputs are never sent. Disable by setting AGNO_TELEMETRY=false.