## Why
Coverage fell from ~100% to **57%** after #5514. This is my regression,
and the cause is a Mintlify behavior I assumed rather than tested.
**Mintlify serves the exact filename `llms.txt` at any path, and 404s on
anything else.** The numbered variants my split produced were never
reachable:
| URL | status |
|---|---|
| `/oss/python/llms.txt` | 200 |
| `/oss/python/llms-2.txt` | **404** |
| `/oss/python/llms-3.txt` | **404** |
| `/langsmith/smith-api/llms.txt` | 200 |
| `/langsmith/smith-api/llms-2.txt` | **404** |
Only 4 of 9 section indexes were reachable. That left 1,229 visible
links — 1,112 across the four served files plus 117 inline in the root —
which is exactly what the audit reported: 883 sitemap pages plus 346
links not in the sitemap. The other **802 pages did not exist as far as
any agent was concerned**.
Confirmed the rule is filename-based, not path-based:
`.well-known/security.txt` is in the build and 404s, while
`/oss/python/llms.txt` byte-matches the file my build produced, so
Mintlify is serving my static file by name.
## What changed
Sections now split **by directory** rather than by filename. Every index
is written as `<prefix>/llms.txt`.
An oversized section sheds its heaviest child directories into their own
indexes until the remainder fits, instead of giving every child its own
file — a naive recursive split produced 109 files, most of them a single
page. Descriptions are also dropped from section indexes, which roughly
halves each entry; the root keeps them, since it has room.
| | before | after |
|---|---|---|
| root | 16,449 chars | 21,656 chars |
| section indexes | 9 (5 unreachable) | 56 (all reachable) |
| largest section | 40,101 | 42,263 |
| pages indexed | 2,038 (1,229 visible) | 2,038 |
All 56 are named `llms.txt`, none exceed 50,000 characters, and each is
one hop from the root so the coverage walker still reaches everything.
## The real gap
Every guard I added in #5514 checked **local files**, and nothing
checked they were **served**. The build-time validator passed happily
while a third of the index was unreachable in production.
`scripts/check_llms_urls.py` now verifies each section index returns 200
before it samples page URLs. Run against production today it correctly
reports 52 of 56 unserved, because the new paths are not deployed yet —
that is the check working. A unit test asserts no index is ever named
`llms-N.txt`.
## Still open: LLMS Full Size (a Mintlify routing issue, not a repo one)
The custom `llms-full.txt` **is** being picked up — Mintlify just serves
it on the wrong route. Probing all four endpoints:
| Route | Bytes | Whose |
|---|---:|---|
| `/llms.txt` | 16,449 | ours |
| `/.well-known/llms.txt` | 99,940 | Mintlify's, **truncated** |
| `/llms-full.txt` | 15,101,956 | Mintlify's |
| `/.well-known/llms-full.txt` | 6,246,717 | ours |
The override is applied to exactly one route per file, and they are
crossed. Verified by content rather than size:
`/.well-known/llms-full.txt` opens with `# Docs by LangChain`, while
`/.well-known/llms.txt` still ends with `_Note: this index was truncated
to stay under 100,000 characters; 569 pages and 3 OpenAPI specs
omitted._`. Neither `.well-known` file exists in our build (it contains
only `security.txt`), so both are served by Mintlify's own routing.
Two consequences:
1. **`LLMS Full Size` will keep failing** while the check reads
`/llms-full.txt`, because that route serves Mintlify's 15.1 MB file
regardless of the custom one. Nothing in this repo changes that.
2. **`/.well-known/llms.txt` serves a truncated index** that omits 569
pages, so any agent following the `.well-known` convention gets the
pre-fix file even though the root is correct.
This looks like a Mintlify bug and needs a support ticket, not a code
change here. Their documentation does not describe how custom files
interact with the `.well-known` mirrors. Separately, **no target size
for `llms-full.txt` is documented** by AFDocs or Mintlify, so even with
the routing fixed I cannot say whether 6.2 MB clears the bar.
## Other checks
- **Markdown Content Parity** (warning): unchanged diagnosis, not a
content defect. The only substantive gap is OpenAPI pages, where the
markdown is *richer* than the HTML. Set both parity thresholds to 0 for
informational mode.
- **LLMS TXT Directive Html** (1 of 15 pages), **Page Size Html** (1
page at 59K markdown from 1,195K HTML, 98% boilerplate), **Redirect
Behavior** (1 cross-host redirect): all single-page warnings in
Mintlify-controlled chrome or intentional `reference.langchain.com`
redirects. No repo change proposed.
- **Content Structure** and **Authentication**: skipped, not failures.
Both score 100.
## Validation
- 215 tests pass (1 new)
- `make lint` clean, `make broken-links` clean
- 56 section indexes, 0 named `llms-N.txt`, 0 over 50,000 chars
- 2,038 pages indexed, verified no duplicates and none lost
## AI disclosure
Authored with Claude Code (Claude Opus 5). The serving rule was
established by probing production rather than inferred, after the
previous PR shipped on an untested assumption about exactly this.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Why
Follow-up to #5504. The custom `llms.txt` that PR added was 286,685
characters, which fails the AFDocs `llms-txt-size` check. This is my
regression: I chose a single large file because I could not verify at
the time whether the checker followed nested indexes. It does — the
check output and the AFDocs reference both specify the nested shape.
## LLMS TXT Size — fail → pass
**Authoritative thresholds** (from the [AFDocs check
reference](https://afdocs.dev/checks/content-discoverability)):
| Result | Condition |
|---|---|
| Pass | Under **50,000** characters |
| Warn | 50,000–100,000 |
| Fail | Over 100,000 |
Note the pass bar is 50 K, not 100 K.
**Size is not the only cost.** An oversized index triggers the AFDocs
[index truncation
coefficient](https://afdocs.dev/agent-score-calculation#index-truncation-coefficient),
which discounts `llms-txt-valid`, `llms-txt-links-resolve`,
`llms-txt-links-markdown`, and `llms-txt-coverage` in proportion to the
visible fraction. At 286,685 characters that is roughly **0.35** — four
*passing* checks were contributing about a third of their weight. Fixing
size restores them to 1.0, so this is worth more than the single failing
check.
### What changed
`_generate_llms_txt` in `pipeline/core/builder.py` now emits a root
index plus section indexes.
Root: **16,449 characters** (was 286,685), listing 9 section indexes and
the 117 pages belonging to no large section.
| File | chars | entries |
|---|---:|---:|
| `/llms.txt` (root) | 16,449 | 117 inline + 9 section links |
| `/oss/python/llms.txt` | 39,983 | 236 |
| `/oss/python/llms-2.txt` | 40,101 | 255 |
| `/oss/python/llms-3.txt` | 5,310 | 47 |
| `/oss/javascript/llms.txt` | 39,957 | 236 |
| `/oss/javascript/llms-2.txt` | 15,778 | 121 |
| `/langsmith/llms.txt` | 39,999 | 271 |
| `/langsmith/llms-2.txt` | 30,851 | 219 |
| `/langsmith/smith-api/llms.txt` | 40,018 | 369 |
| `/langsmith/smith-api/llms-2.txt` | 20,214 | 167 |
Section files target 40 K rather than 50 K so they do not drift over the
threshold as pages are added between splits.
### Three constraints this is built around
- **Coverage must not regress.** AFDocs's coverage walker descends
*exactly one level* into linked `.txt` files, and treats `.txt` files
found at that depth as "omitted subtrees" whose pages leave the coverage
denominator. Every section index is therefore one hop from the root and
contains **zero** further `.txt` links, so all 2,038 pages stay directly
verified. Coverage is currently ~100% and stays there.
- **Link sampling must still see pages.** A root containing only `.txt`
links could starve `llms-txt-links-markdown` and
`llms-txt-links-resolve`. The root keeps 117 real `.md` links (the small
sections, inlined rather than split out) so both checks sample real
pages from the canonical file.
- **Section paths are derived, not hardcoded.** Each section's directory
comes from the deepest directory shared by its pages. An earlier
hardcoded map would have inlined the 60 K OpenAPI group into the root,
defeating the split.
### Verification
- 2,038 `.md` entries across root and sections, **0 duplicates, 0 pages
lost**
- Every file under 50,000 characters
- 0 nested `.txt` links inside section files
- 207 tests pass (1 new, covering root size, one-hop depth, no deeper
nesting, and exact page conservation)
- `make lint` clean, `make broken-links` clean
## Also in this PR
### LLMS Full Size — split into per-language corpora
The combined corpus was 15,050,368 characters. Roughly half was the
Python and TypeScript renders of the same documentation, so the build
now emits three corpora, with the root pointing at the other two:
| File | chars |
|---|---:|
| `/llms-full.txt` | 6,246,717 |
| `/oss/python/llms-full.txt` | 6,801,843 |
| `/oss/javascript/llms-full.txt` | 4,655,598 |
Root is **down 58.5%** from what the check currently sees.
**This requires expanding snippets ourselves.** Mintlify expands snippet
imports at render, so a corpus assembled from the raw build tree would
silently lose content from the 308 pages that import one. Verified page
by page against the published corpus: expansion lands 8–10% *under*
Mintlify's for snippet-heavy pages (`deepagents/customization` 139,256
vs 152,958; `langchain/agents` 64,279 vs 67,990), and only 7 of 1,525
shared pages come out more than 30% larger — all pages that have
genuinely grown since that snapshot.
The combined total exceeds Mintlify's because this covers every built
page (2,038) rather than only the sitemap subset (1,586). The extra
pages are integration pages that are reachable but not in navigation,
which `llms.txt` already indexes; excluding them here would make the two
files disagree about what exists.
Both custom files open with the site title as an H1, which Mintlify
requires of a custom `llms.txt` or `llms-full.txt` and which `LLMS Full
Valid` checks.
**Two caveats worth weighing at review.** This replaces a
Mintlify-maintained, auto-updating artifact with a generator in this
repo, so snippet-expansion correctness is now our problem. And the
expected range is still undocumented by both AFDocs and Mintlify — 6.2
MB is a large reduction, but whether it clears the bar is unverified
until this deploys.
### Correction to #5504
That PR's description claims the base64 removal roughly halved
`llms-full.txt`. **That was inferred, not measured, and it is wrong** —
the live file contains zero base64 and is unchanged in size, so Mintlify
never included data URIs there. The base64 work was still worth doing:
it cut the served `google_imagen` page from 6.69 MB to 7.6 KB, confirmed
live. I have not edited the merged PR body; say the word and I will.
### Markdown Content Parity — warning, no change recommended
Not a content defect. Regular pages differ by 0.7–3.9%, all UI chrome
("skip to main content", "was this page helpful"). The only substantial
gaps are OpenAPI pages, where the **markdown is richer than the HTML**
(175 words vs 55) because the html-only tokens are interactive
API-playground widgets with no markdown equivalent. Per the check's own
guidance, set `--parity-pass-threshold` and `--parity-warn-threshold` to
`0` for informational mode. Trimming markdown to match HTML would make
things worse for agents.
### Content Structure and Authentication — skipped, not failures
Both score 100. `Tabbed Content Serialization` is skipped because no
tabbed content was found in the sample; `Auth Alternative Access`
because all pages are public. Nothing to fix.
## Keeping it correct
Generating our own indexes removes the staleness problem outright. They
rebuild from source on every `make build`, `build/` is gitignored so
nothing can rot in the tree, `publish.yml` builds before deploying, and
new pages are picked up by walking the build tree with no registration
step. That is strictly fresher than what it replaces: Mintlify's
`llms-full.txt` was serving with `age: 25126` against a 24-hour cache
and did not contain pages that were already live.
What it does introduce is **correctness drift**, which nothing would
have noticed. Two guards for the two ways it can happen.
**Build-time validation** (`_validate_llms_indexes`) fails the build
when the emitted files break an invariant agents depend on:
- root `llms.txt` over 50,000 characters
- any section index over the same threshold
- a section index linking to further `.txt` files, which would push
those pages out of the coverage denominator
- a page listed in two indexes, or a page count that disagrees with what
was built
This runs wherever `make build` runs, so CI already covers it through
the link-checking job. Five unit tests exercise each failure mode — a
validator that cannot fail is not a guard.
**A weekly workflow** (`.github/workflows/check-llms-urls.yml`) covers
what no local check can. The API reference URLs are derived by
reproducing Mintlify's slug rules, which is not a published contract,
and a change on their side only surfaces against the deployed site.
`scripts/check_llms_urls.py` samples those URLs, reports any that stop
resolving, and names the two functions to fix. This is the guard that
would have caught the 47 fabricated URLs in #5504.
Worth noting: running the checker before shipping it caught a flaw in
the checker itself. Concurrent HEAD requests reported two healthy pages
as broken, so it now retries and falls back to GET. It is stable across
repeated runs at different seeds.
## Review notes
The one thing I could not verify locally is that Mintlify serves nested
`.txt` files from the build directory. Strong indirect evidence:
`/langsmith/langsmith-platform-openapi.json` returns 200 today, so
nested static assets are served, and nested `llms.txt` paths are not on
Mintlify's reserved-path list. **Worth confirming after deploy** that
`https://docs.langchain.com/oss/python/llms.txt` returns 200 — if it
does not, coverage would fall back to the root's 117 links and the split
should be reverted.
## AI disclosure
Authored with Claude Code (Claude Opus 5). Thresholds were read from the
AFDocs published check reference rather than assumed, and page
conservation was verified programmatically across all ten generated
files.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Lauren Hirata Singh