mirror of
https://github.com/paperclipai/paperclip-docs.git
synced 2026-11-07 06:16:07 -10:00
Aron PrinsandClaude Opus 5
29beedeb6a
## Why Google has indexed effectively nothing on `docs.paperclip.ing`. Search Console over 3 months: **8 of 192 URLs** have ever drawn an impression, **681 of 975 impressions (70%) go to `/guides/agent-developer/comments-and-communication`** — a legacy URL that 301s and isn't in the sitemap. Strip that, the homepage, and the team's own `site:` queries, and the canonical docs corpus earned **~7 organic impressions in 90 days**. It isn't an eligibility problem. Pages are fetchable, self-canonical, `index, follow`, and identical under a Googlebot UA. It's the internal link graph. ## What was actually broken **1. Every relative markdown link 404'd.** Authors write `./glossary.md`, but pages are served from trailing-slash route directories, so links resolved one level too deep: ``` page: /guides/welcome/key-concepts/ href: ./glossary.md → /guides/welcome/key-concepts/glossary.md 404 ( /guides/welcome/glossary.md would be 200 ) ``` Crawled all 192 live pages: **all 742 unique `.md` targets return 404**. That's **1,283 of 1,306 in-body internal links — 98% — broken**, leaving **0.1 working in-body links per page**. It went unnoticed because `app.js:postProcessInternalLinks` repairs these at runtime. Crawlers index the pre-render. **2. The sidebar wasn't server-rendered.** Interior pages shipped `<div id="sb-sections"></div>`, filled by JS. So the raw HTML offered the logo, prev, next — and ~7 dead links. No link equity reached deep pages. Search Console classified 98% of crawling as refresh, 2% as discovery. Net effect: a crawler saw every page as a dead end that was 98% broken links. Crawl demand collapsed — 14 crawled of 184, 15% of crawl responses 404. ## Changes - **Rewrite markdown links to canonical routes at build time**, and fail the build on any link no nav page publishes. (Passed first run with zero unresolved — every link had a valid destination, just emitted in a 404ing form.) - **Server-render the sidebar and breadcrumb** into every interior route from the manifest `app.js` already uses. `app.js` detects `data-server-rendered` and keeps the DOM rather than re-rendering identical markup. The mobile drawer stays client-built — it's `inert`/`aria-hidden` until a JS toggle opens it, so rendering it duplicated 192 links/page (~41KB) for no crawl benefit. - **Intercept canonical route links in articles** so in-page navigation stays a soft transition instead of a full reload. - **Omit sitemap `<lastmod>` when git history is untrustworthy.** Reproduced the real mechanism: in a `--depth=1` clone, `git log -1` returns the single fetched commit's date for *every* file. Two files that differ locally (`2026-08-18` / `2026-07-08`) both come back `2026-08-25` shallow. That is exactly the live `192 × 2026-08-20`. Cloudflare Pages shallow-clones by default. - **Ship `redirects.json`** — `app.js` fetches it at boot and it 404'd. - **Canonicalise the footer's `/reference/api/overview`** link, which spent a 301 on all 193 pages. - **Add `scripts/verify-crawlable-links.mjs` + CI**, gating on raw `.md` hrefs, missing targets, redirecting targets, relative hrefs, an unrendered sidebar, and uniform sitemap dates. The last two were both caught by the new verifier on its first run. ## Result | | Before (production) | After (generated bundle) | |---|---|---| | Broken in-body links | 1,283 (98%) | **0** | | Working in-body links | 23 | **1,501** | | Crawlable nav links/page | 0 | **192** | | Distinct sitemap dates | 1 | **24** (0 on shallow CI) | `Crawlable-link contract passed: 41078 internal links across 194 pages resolve in one hop.` ## Verified Built and driven in a real browser at `localhost:4321`: - Raw HTML (crawler / no-JS view): 192 inline sidebar links, breadcrumb rendered, **0 local `.md` hrefs**, single H1. Remaining empty containers are JS-only UI (search modal, drawer, runtime status). - SPA intact, no reloads, no JS errors: in-article links, prev/next, sidebar links, mobile drawer (opens, 192 links, closes on navigate), search (8 results for "adapter"). Breadcrumb and active-link state follow correctly throughout. - Subpath build for gh-pages emits correct `/paperclip-docs/...` hrefs. - Homepage unchanged: 15 landing cards, 192 directory links. - Shallow-clone build confirmed to warn and emit zero `<lastmod>`. ## Notes for review - **This can't ship until the Cloudflare GitHub App is unsuspended.** Everything here is verified against a local build only. - **CI will be red on one pre-existing, unrelated failure**: `docs:test:skill-source-blocks` on `docs/reference/skills/optional/content/simplified-english.md` (missing its skill-source block, from release #95). Confirmed failing on clean `main`. Left in the aggregate rather than weakening the suite — fixing it needs the real `SKILL.md` from the parent repo, which is sync-docs work. - **Production will now ship no `lastmod` at all** on Cloudflare's shallow clone. Correct versus lying. To get real dates back, set the Pages build command to `git fetch --unshallow || true; npm run docs:build` — documented in the generated `DEPLOY.md`. - Page weight 67KB → 109KB from the inlined nav. Highly repetitive, compresses well, but it is a real increase. - Weak/duplicate titles were deliberately skipped, per the plan's own ranking (secondary). - Plan item 4 (contextual links from high-authority surfaces to 20–25 priority pages) is a `paperclip-website` change plus editorial judgment, so it's out of scope here. Worth noting that fixing the internal graph recovers most of what that item was meant to buy — the dominant authority failure was internal, not off-site. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
Paperclip Docs
Source for the Paperclip documentation site — user guides, API and CLI reference, adapter docs, and deployment notes.
Read the docs: https://docs.paperclip.ing/
What's inside
- Guides — Getting started, core concepts, and day-to-day usage.
- API reference — Control-plane endpoints and payload schemas.
- CLI reference —
paperclipcommand surface. - Adapters — Plugin authoring and the integrations catalogue.
- Deploy — Self-hosting, upgrades, and operational guidance.
The site is a single static shell that renders the Markdown files in docs/ directly — no framework, no build step required to read the raw pages.
Build and deployment
Build a release bundle locally with:
npm run docs:build
Cloudflare Pages is connected directly to paperclipai/paperclip-docs through GitHub. There is no normal Wrangler publish step for this repo:
- Pushes to
maindeploy production athttps://docs.paperclip.ing/. - Pushes to other branches create Cloudflare Pages preview/canary deployments.
- Canary URLs are created by Cloudflare for each deployment, for example
https://92b9a99c.paperclip-docs-74t.pages.dev. Use the URL shown in the Cloudflare Pages deployment row or GitHub deployment/check for the pushed branch and commit.
That auto-deploy is not guaranteed — it has silently failed to fire on a push to main before. After anything reader-facing lands, check the live site rather than the merge, and republish by hand if it's stale:
npm run docs:build
npx wrangler pages deploy .site --project-name paperclip-docs --branch main
Contributing
Spotted a typo, a broken link, or something that could be clearer? There are two easy paths, both linked from the footer of every docs page:
- Suggest an edit — opens the underlying Markdown file in GitHub's web editor. Make the change and submit a PR without leaving the browser.
- Report an issue — opens a prefilled Docs feedback issue with the page URL already attached.
For larger changes, fork the repo and open a PR against main. Screenshots live under docs/user-guides/screenshots/{light,dark}/ and should be provided in both themes when replacing UI captures.
Community
License
Copyright (c) 2026 Paperclip Community.
The Paperclip software is open source — see the main paperclip repository for its license.
The documentation in this repository is licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0). You may read and share it for non-commercial purposes with attribution. Commercial use of any kind requires prior written permission from the copyright holders. See LICENSE for the full text.