mirror of
https://github.com/agno-agi/agno.git
synced 2026-10-11 22:09:58 +00:00
KaustubhandClaude Opus 5.5
18413f7f60
## Summary Large Mintlify sites sync as `partial` today even though every documentation page indexes. A `partial` result skips pruning, so pages removed from the site stay searchable indefinitely. docs.langchain.com reproduces it. Four causes, all fixed here: 1. **Links to non-page files fail discovery.** LangChain's `llms.txt` has an "OpenAPI Specs" section: - `https://api.host.langchain.com/openapi.json` is off-host, which marks discovery incomplete. - Two same-site `.json` specs are treated as pages, fetched as `…openapi.json.md`, and 404 as page failures. Discovery now skips links to files that cannot be pages (specs, data, media, archives: `.json`, `.yaml`, `.xml`, `.csv`, `.pdf`, images, audio and video, archives, fonts) wherever they point. `.js` and `.css` are excluded, because pages can be named like `/guides/node.js`. **Off-host page links still block pruning:** `test_collisions_and_foreign_destinations_cannot_prune` is unchanged. 2. **Depth-first traversal exceeds `max_depth` on indexes the root lists directly.** Mintlify roots list every `/_llms/` index directly, and each index also links its children. Depth-first traversal follows the first index's chain down to depth 4+ and flags the run, even though the same indexes are reachable at depth 1. Discovery is now breadth-first, so each index is reached at its shallowest depth. `max_depth` and `max_indexes` are unchanged. 3. **Mintlify inlines component code into page Markdown.** Pages carry top-level MDX statements such as `export const PatternEmbed = ({...}) => {...}`. On LangChain's Deep Agents overview that's 48 KB of a 77 KB page. The code was chunked and embedded as content. The `fumadocs` and `mintlify` profiles now drop top-level `import`/`export` statements. `strip_esm=False` keeps them, and the `markdown` profile stays verbatim. The matching is conservative: - A statement must start a block (first line or after a blank line) and match ESM syntax, so a wrapped prose line beginning "export the file…" is kept. - Statements inside code fences are kept. - Multi-line statements end when brackets balance outside strings and comments. Quotes reset at line ends, so an apostrophe in embedded JSX text can't swallow the page. - A statement that never closes is kept rather than risk dropping the rest of the page. 4. **Listed pages that redirect elsewhere fail as pages.** A live sync of docs.langchain.com ended `partial` with 38 failed pages, and every one was a listed page that redirects somewhere else: - 24 to other hosts (reference.langchain.com, GitHub, academy.langchain.com, status and marketing pages): `SyncFailed`. - 4 to a `#section` of another page: `invalid_source_url`. - 10 to a same-site URL without `.md`, which serves HTML. That HTML was fetched and embedded as page text until the provider rejected it at its 8,192-token input limit. A listed `.md` page that redirects off-origin, to a section or to a non-Markdown URL now raises `PageMoved`. The page is **skipped, not failed**: it isn't retried and is never stored, and it doesn't make the run `partial`. If an older sync stored it, it's pruned. `SyncReport.skipped` and `skipped_paths` (up to 20) report skipped pages, and each is logged with its target. A move to another `.md` URL on the same origin is still followed. Index (`llms.txt`) redirects behave as before, and foreign index redirects still fail. A `.md` page answered with `text/html` now raises `PageNotMarkdown`, a failed page, instead of being embedded. The same sync surfaced a normalizer gap, also fixed: component tags with `>` or `<` inside quoted attributes (`<Tab title="From Fleet > Integrations">`, `<ResponseField type="Record<string, string>">`) were left in page text. They now render, and `ParamField` is named by its `path`/`query`/`body`/`header` attribute. ## Results on live corpora Discovery and profile output were measured without a database or model calls. The full sync ran on a docs-agent template stack with this branch mounted: | | Before | After | |---|---|---| | docs.langchain.com discovery | 1,668 entries incl. 2 `.json` specs, `complete=False` | 1,666 pages, `complete=True` | | docs.langchain.com `mintlify` output | inlined component code indexed | 1.7 MB removed across 55 of 1,666 pages; all 29 headings and every prose line kept on the Deep Agents page; the cut ends at the component's closing `};` | | docs.langchain.com full sync (Postgres + pgvector, `mintlify`) | `partial` in 1,147 s: 1,628 updated, **38 failed**, pruning skipped | `completed` in 243 s: 0 failed, 38 skipped, 1,628 pages stored, no alias stored | | docs.agno.com discovery | 3,913 pages, complete | unchanged | | docs.agno.com `fumadocs` output | — | byte-identical with and without `strip_esm` on all 3,913 pages, so existing indexes need no `index_version` bump | ## Type of change - [x] Bug fix - [ ] New feature - [ ] Breaking change - [x] Improvement - [ ] Model update - [ ] Other: --- ## Checklist - [x] Code complies with style guidelines - [x] Ran format/validation scripts (`./scripts/format.sh` and `./scripts/validate.sh`) - [x] Self-review completed - [x] Documentation updated (comments, docstrings) - [x] Examples and guides: Relevant cookbook examples have been included or updated (if applicable) - [ ] Tested in clean environment - [x] Tests added/updated (if applicable) ### Duplicate and AI-Generated PR Check - [x] I have searched existing [open pull requests](https://github.com/agno-agi/agno/pulls) and confirmed that no other PR already addresses this issue - [ ] If a similar PR exists, I have explained below why this PR is a better approach - [x] Check if this PR was entirely AI-generated (by Copilot, Claude Code, Cursor, etc.) --- ## Additional Notes **Tests** - New unit tests: - Non-page links (same-site, off-host, mixed case, a `node.js` page) are skipped without marking discovery incomplete. - A five-deep `/_llms/` chain listed directly by the root completes. - ESM stripping on a Mintlify-shaped page, for both profiles. - The keep cases: disabled, unclosed, verbatim profile. - Redirects: aliases to another host, to a `#section` or to a non-`.md` URL raise `PageMoved` on the first request, with no retry. A move to another `.md` URL is followed, `llms.txt` redirects still are, and a `.md` page served as HTML raises `PageNotMarkdown`. - MDX: quoted attribute values containing `<` and `>`, and `ParamField` location names. - New integration test: an alias is skipped, not failed. The run stays `completed`, and the alias's copy stored by an earlier sync is pruned. - `tests/unit/knowledge` and `tests/unit/reader`: same results as `main`, plus the 4 new passing tests. Both trees share 35 failures from optional dependencies (docling, PDF image) missing in the local venv. - `tests/unit/knowledge/test_page_source_reliability.py` and `tests/unit/reader/test_mdx_normalize.py`: 53 passed. - `tests/integration/knowledge/test_page_storage.py` against PostgreSQL 18 + pgvector: 160 passed. The CLI test needs this worktree on `PYTHONPATH`, because it runs a subprocess. **Validation** - `./scripts/format.sh`: also reformats five unrelated files on current `main`. I reverted those to keep this diff scoped. - `./scripts/validate.sh`: ruff passes. mypy reports the same 54 errors as `main`, none in the changed files. **Cookbook** - `27_public_pages/README.md` documents breadth-first nested indexes, skipped non-page links, `strip_esm` and skipped redirect aliases (`SyncReport.skipped`). - TEST_LOG has entries for the discovery change and for the live LangChain sync before and after the redirect fix. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Build, run, and manage agent platforms.
Introduction
Agno is a framework and runtime for agent platforms. Build agents, run them as a service, manage your platform using a web UI.
- Build your agent platform using the Agno SDK.
- Run your agent platform using the AgentOS runtime.
- Manage everything using the AgentOS UI.
Agno allows you to own your agent stack. Maintain control of your data, memory, and security posture (JWT-based RBAC), and turn your agent platform into a learning loop with simulations and usage data.
Get started
Hand this prompt to your coding agent (Claude Code, Cursor, Codex):
Help me set up my agent platform.
Clone https://github.com/agno-agi/agentos-railway into a folder called
agent-platform, cd in, read the README, and follow the get started guide.
Your coding agent will set up your agent platform and run it locally using Docker, giving you a REST API for serving your agents, a Postgres database for storing your data and traces, an MCP server, and a control plane.
Deploying somewhere else? Use the same prompt but point it to a different repo. The starter templates are identical except for the deploy scripts: swap agentos-railway for agentos-docker, agentos-aws, agentos-gcp, agentos-azure, agentos-fly, agentos-render, agentos-modal, or agentos-helm.
Prefer to code by hand?
Features
- Production API. 50+ endpoints with SSE and websockets to build a product on top.
- Storage. Store sessions, memory, knowledge, and traces in your own database.
- 100+ integrations. Connect to GitHub, Slack, Postgres, and more using pre-built toolkits.
- Context Providers. Access live data from Slack, Drive, wikis, MCP, and custom sources.
- Human approval. Pause runs for user confirmation. Block tools that require admin approval.
- Observability. Monitor with OpenTelemetry tracing, run history, and audit logs.
- Security. Get JWT-based RBAC and multi-user, multi-tenant isolation out of the box.
- Interfaces. Expose your agents via Slack, Telegram, WhatsApp, Discord, AG-UI, A2A.
- Scheduling. Cron-based scheduling and background jobs with no external infrastructure.
- Deploy anywhere. Run on any cloud platform that runs containers. Docker, Railway, AWS, GCP.
Use Agno with your coding agent
Two options:
- Recommended: Add Agno docs as an MCP server. Add docs.agno.com/mcp to your favourite coding agent.
- Add Agno docs as an indexed source. In Cursor: Settings → Indexing & Docs → Add
https://docs.agno.com/llms-full.txt. Also works in VSCode, Windsurf, and similar tools.
Read the full guide here.
Community
- (https://x.com/AgnoAgi): follow for releases and demos
- Newsletter: monthly updates on what's shipping
Contributing
See the contributing guide.
License
Agno is distributed under the Apache-2.0 license.
Telemetry
Agno sends a telemetry event per agent run so we know which model providers to prioritize. Prompts, messages, and outputs are never sent. Disable by setting AGNO_TELEMETRY=false.