mirror of
https://github.com/langchain-ai/docs.git
synced 2026-08-27 02:41:59 -04:00
Lauren Hirata Singh
5b4af66dff
## Why Follow-up to #5504. The custom `llms.txt` that PR added was 286,685 characters, which fails the AFDocs `llms-txt-size` check. This is my regression: I chose a single large file because I could not verify at the time whether the checker followed nested indexes. It does — the check output and the AFDocs reference both specify the nested shape. ## LLMS TXT Size — fail → pass **Authoritative thresholds** (from the [AFDocs check reference](https://afdocs.dev/checks/content-discoverability)): | Result | Condition | |---|---| | Pass | Under **50,000** characters | | Warn | 50,000–100,000 | | Fail | Over 100,000 | Note the pass bar is 50 K, not 100 K. **Size is not the only cost.** An oversized index triggers the AFDocs [index truncation coefficient](https://afdocs.dev/agent-score-calculation#index-truncation-coefficient), which discounts `llms-txt-valid`, `llms-txt-links-resolve`, `llms-txt-links-markdown`, and `llms-txt-coverage` in proportion to the visible fraction. At 286,685 characters that is roughly **0.35** — four *passing* checks were contributing about a third of their weight. Fixing size restores them to 1.0, so this is worth more than the single failing check. ### What changed `_generate_llms_txt` in `pipeline/core/builder.py` now emits a root index plus section indexes. Root: **16,449 characters** (was 286,685), listing 9 section indexes and the 117 pages belonging to no large section. | File | chars | entries | |---|---:|---:| | `/llms.txt` (root) | 16,449 | 117 inline + 9 section links | | `/oss/python/llms.txt` | 39,983 | 236 | | `/oss/python/llms-2.txt` | 40,101 | 255 | | `/oss/python/llms-3.txt` | 5,310 | 47 | | `/oss/javascript/llms.txt` | 39,957 | 236 | | `/oss/javascript/llms-2.txt` | 15,778 | 121 | | `/langsmith/llms.txt` | 39,999 | 271 | | `/langsmith/llms-2.txt` | 30,851 | 219 | | `/langsmith/smith-api/llms.txt` | 40,018 | 369 | | `/langsmith/smith-api/llms-2.txt` | 20,214 | 167 | Section files target 40 K rather than 50 K so they do not drift over the threshold as pages are added between splits. ### Three constraints this is built around - **Coverage must not regress.** AFDocs's coverage walker descends *exactly one level* into linked `.txt` files, and treats `.txt` files found at that depth as "omitted subtrees" whose pages leave the coverage denominator. Every section index is therefore one hop from the root and contains **zero** further `.txt` links, so all 2,038 pages stay directly verified. Coverage is currently ~100% and stays there. - **Link sampling must still see pages.** A root containing only `.txt` links could starve `llms-txt-links-markdown` and `llms-txt-links-resolve`. The root keeps 117 real `.md` links (the small sections, inlined rather than split out) so both checks sample real pages from the canonical file. - **Section paths are derived, not hardcoded.** Each section's directory comes from the deepest directory shared by its pages. An earlier hardcoded map would have inlined the 60 K OpenAPI group into the root, defeating the split. ### Verification - 2,038 `.md` entries across root and sections, **0 duplicates, 0 pages lost** - Every file under 50,000 characters - 0 nested `.txt` links inside section files - 207 tests pass (1 new, covering root size, one-hop depth, no deeper nesting, and exact page conservation) - `make lint` clean, `make broken-links` clean ## Also in this PR ### LLMS Full Size — split into per-language corpora The combined corpus was 15,050,368 characters. Roughly half was the Python and TypeScript renders of the same documentation, so the build now emits three corpora, with the root pointing at the other two: | File | chars | |---|---:| | `/llms-full.txt` | 6,246,717 | | `/oss/python/llms-full.txt` | 6,801,843 | | `/oss/javascript/llms-full.txt` | 4,655,598 | Root is **down 58.5%** from what the check currently sees. **This requires expanding snippets ourselves.** Mintlify expands snippet imports at render, so a corpus assembled from the raw build tree would silently lose content from the 308 pages that import one. Verified page by page against the published corpus: expansion lands 8–10% *under* Mintlify's for snippet-heavy pages (`deepagents/customization` 139,256 vs 152,958; `langchain/agents` 64,279 vs 67,990), and only 7 of 1,525 shared pages come out more than 30% larger — all pages that have genuinely grown since that snapshot. The combined total exceeds Mintlify's because this covers every built page (2,038) rather than only the sitemap subset (1,586). The extra pages are integration pages that are reachable but not in navigation, which `llms.txt` already indexes; excluding them here would make the two files disagree about what exists. Both custom files open with the site title as an H1, which Mintlify requires of a custom `llms.txt` or `llms-full.txt` and which `LLMS Full Valid` checks. **Two caveats worth weighing at review.** This replaces a Mintlify-maintained, auto-updating artifact with a generator in this repo, so snippet-expansion correctness is now our problem. And the expected range is still undocumented by both AFDocs and Mintlify — 6.2 MB is a large reduction, but whether it clears the bar is unverified until this deploys. ### Correction to #5504 That PR's description claims the base64 removal roughly halved `llms-full.txt`. **That was inferred, not measured, and it is wrong** — the live file contains zero base64 and is unchanged in size, so Mintlify never included data URIs there. The base64 work was still worth doing: it cut the served `google_imagen` page from 6.69 MB to 7.6 KB, confirmed live. I have not edited the merged PR body; say the word and I will. ### Markdown Content Parity — warning, no change recommended Not a content defect. Regular pages differ by 0.7–3.9%, all UI chrome ("skip to main content", "was this page helpful"). The only substantial gaps are OpenAPI pages, where the **markdown is richer than the HTML** (175 words vs 55) because the html-only tokens are interactive API-playground widgets with no markdown equivalent. Per the check's own guidance, set `--parity-pass-threshold` and `--parity-warn-threshold` to `0` for informational mode. Trimming markdown to match HTML would make things worse for agents. ### Content Structure and Authentication — skipped, not failures Both score 100. `Tabbed Content Serialization` is skipped because no tabbed content was found in the sample; `Auth Alternative Access` because all pages are public. Nothing to fix. ## Keeping it correct Generating our own indexes removes the staleness problem outright. They rebuild from source on every `make build`, `build/` is gitignored so nothing can rot in the tree, `publish.yml` builds before deploying, and new pages are picked up by walking the build tree with no registration step. That is strictly fresher than what it replaces: Mintlify's `llms-full.txt` was serving with `age: 25126` against a 24-hour cache and did not contain pages that were already live. What it does introduce is **correctness drift**, which nothing would have noticed. Two guards for the two ways it can happen. **Build-time validation** (`_validate_llms_indexes`) fails the build when the emitted files break an invariant agents depend on: - root `llms.txt` over 50,000 characters - any section index over the same threshold - a section index linking to further `.txt` files, which would push those pages out of the coverage denominator - a page listed in two indexes, or a page count that disagrees with what was built This runs wherever `make build` runs, so CI already covers it through the link-checking job. Five unit tests exercise each failure mode — a validator that cannot fail is not a guard. **A weekly workflow** (`.github/workflows/check-llms-urls.yml`) covers what no local check can. The API reference URLs are derived by reproducing Mintlify's slug rules, which is not a published contract, and a change on their side only surfaces against the deployed site. `scripts/check_llms_urls.py` samples those URLs, reports any that stop resolving, and names the two functions to fix. This is the guard that would have caught the 47 fabricated URLs in #5504. Worth noting: running the checker before shipping it caught a flaw in the checker itself. Concurrent HEAD requests reported two healthy pages as broken, so it now retries and falls back to GET. It is stable across repeated runs at different seeds. ## Review notes The one thing I could not verify locally is that Mintlify serves nested `.txt` files from the build directory. Strong indirect evidence: `/langsmith/langsmith-platform-openapi.json` returns 200 today, so nested static assets are served, and nested `llms.txt` paths are not on Mintlify's reserved-path list. **Worth confirming after deploy** that `https://docs.langchain.com/oss/python/llms.txt` returns 200 — if it does not, coverage would fall back to the root's 117 links and the split should be reverted. ## AI disclosure Authored with Claude Code (Claude Opus 5). Thresholds were read from the AFDocs published check reference rather than assumed, and page conservation was verified programmatically across all ten generated files. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>