mirror of
https://github.com/langchain-ai/docs.git
synced 2026-08-24 12:23:02 -04:00
Lauren Hirata Singh
2276580e29
## Why Coverage fell from ~100% to **57%** after #5514. This is my regression, and the cause is a Mintlify behavior I assumed rather than tested. **Mintlify serves the exact filename `llms.txt` at any path, and 404s on anything else.** The numbered variants my split produced were never reachable: | URL | status | |---|---| | `/oss/python/llms.txt` | 200 | | `/oss/python/llms-2.txt` | **404** | | `/oss/python/llms-3.txt` | **404** | | `/langsmith/smith-api/llms.txt` | 200 | | `/langsmith/smith-api/llms-2.txt` | **404** | Only 4 of 9 section indexes were reachable. That left 1,229 visible links — 1,112 across the four served files plus 117 inline in the root — which is exactly what the audit reported: 883 sitemap pages plus 346 links not in the sitemap. The other **802 pages did not exist as far as any agent was concerned**. Confirmed the rule is filename-based, not path-based: `.well-known/security.txt` is in the build and 404s, while `/oss/python/llms.txt` byte-matches the file my build produced, so Mintlify is serving my static file by name. ## What changed Sections now split **by directory** rather than by filename. Every index is written as `<prefix>/llms.txt`. An oversized section sheds its heaviest child directories into their own indexes until the remainder fits, instead of giving every child its own file — a naive recursive split produced 109 files, most of them a single page. Descriptions are also dropped from section indexes, which roughly halves each entry; the root keeps them, since it has room. | | before | after | |---|---|---| | root | 16,449 chars | 21,656 chars | | section indexes | 9 (5 unreachable) | 56 (all reachable) | | largest section | 40,101 | 42,263 | | pages indexed | 2,038 (1,229 visible) | 2,038 | All 56 are named `llms.txt`, none exceed 50,000 characters, and each is one hop from the root so the coverage walker still reaches everything. ## The real gap Every guard I added in #5514 checked **local files**, and nothing checked they were **served**. The build-time validator passed happily while a third of the index was unreachable in production. `scripts/check_llms_urls.py` now verifies each section index returns 200 before it samples page URLs. Run against production today it correctly reports 52 of 56 unserved, because the new paths are not deployed yet — that is the check working. A unit test asserts no index is ever named `llms-N.txt`. ## Still open: LLMS Full Size (a Mintlify routing issue, not a repo one) The custom `llms-full.txt` **is** being picked up — Mintlify just serves it on the wrong route. Probing all four endpoints: | Route | Bytes | Whose | |---|---:|---| | `/llms.txt` | 16,449 | ours | | `/.well-known/llms.txt` | 99,940 | Mintlify's, **truncated** | | `/llms-full.txt` | 15,101,956 | Mintlify's | | `/.well-known/llms-full.txt` | 6,246,717 | ours | The override is applied to exactly one route per file, and they are crossed. Verified by content rather than size: `/.well-known/llms-full.txt` opens with `# Docs by LangChain`, while `/.well-known/llms.txt` still ends with `_Note: this index was truncated to stay under 100,000 characters; 569 pages and 3 OpenAPI specs omitted._`. Neither `.well-known` file exists in our build (it contains only `security.txt`), so both are served by Mintlify's own routing. Two consequences: 1. **`LLMS Full Size` will keep failing** while the check reads `/llms-full.txt`, because that route serves Mintlify's 15.1 MB file regardless of the custom one. Nothing in this repo changes that. 2. **`/.well-known/llms.txt` serves a truncated index** that omits 569 pages, so any agent following the `.well-known` convention gets the pre-fix file even though the root is correct. This looks like a Mintlify bug and needs a support ticket, not a code change here. Their documentation does not describe how custom files interact with the `.well-known` mirrors. Separately, **no target size for `llms-full.txt` is documented** by AFDocs or Mintlify, so even with the routing fixed I cannot say whether 6.2 MB clears the bar. ## Other checks - **Markdown Content Parity** (warning): unchanged diagnosis, not a content defect. The only substantive gap is OpenAPI pages, where the markdown is *richer* than the HTML. Set both parity thresholds to 0 for informational mode. - **LLMS TXT Directive Html** (1 of 15 pages), **Page Size Html** (1 page at 59K markdown from 1,195K HTML, 98% boilerplate), **Redirect Behavior** (1 cross-host redirect): all single-page warnings in Mintlify-controlled chrome or intentional `reference.langchain.com` redirects. No repo change proposed. - **Content Structure** and **Authentication**: skipped, not failures. Both score 100. ## Validation - 215 tests pass (1 new) - `make lint` clean, `make broken-links` clean - 56 section indexes, 0 named `llms-N.txt`, 0 over 50,000 chars - 2,038 pages indexed, verified no duplicates and none lost ## AI disclosure Authored with Claude Code (Claude Opus 5). The serving rule was established by probing production rather than inferred, after the previous PR shipped on an untested assumption about exactly this. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>