paperclipai/paperclip-docs
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip-docs.git synced 2026-11-07 06:16:07 -10:00
Code Issues Packages Projects Releases Wiki Activity
Files
codex/github-low-trust-docs
paperclip-docs/site/index.html
T
Aron PrinsandClaude Opus 5 29beedeb6a seo: make the docs link graph crawlable (#113) 
## Why

Google has indexed effectively nothing on `docs.paperclip.ing`. Search
Console over 3 months: **8 of 192 URLs** have ever drawn an impression,
**681 of 975 impressions (70%) go to
`/guides/agent-developer/comments-and-communication`** — a legacy URL
that 301s and isn't in the sitemap. Strip that, the homepage, and the
team's own `site:` queries, and the canonical docs corpus earned **~7
organic impressions in 90 days**.

It isn't an eligibility problem. Pages are fetchable, self-canonical,
`index, follow`, and identical under a Googlebot UA. It's the internal
link graph.

## What was actually broken

**1. Every relative markdown link 404'd.** Authors write
`./glossary.md`, but pages are served from trailing-slash route
directories, so links resolved one level too deep:

```
page: /guides/welcome/key-concepts/   href: ./glossary.md
  → /guides/welcome/key-concepts/glossary.md   404
  ( /guides/welcome/glossary.md would be 200 )
```

Crawled all 192 live pages: **all 742 unique `.md` targets return 404**.
That's **1,283 of 1,306 in-body internal links — 98% — broken**, leaving
**0.1 working in-body links per page**.

It went unnoticed because `app.js:postProcessInternalLinks` repairs
these at runtime. Crawlers index the pre-render.

**2. The sidebar wasn't server-rendered.** Interior pages shipped `<div
id="sb-sections"></div>`, filled by JS. So the raw HTML offered the
logo, prev, next — and ~7 dead links. No link equity reached deep pages.
Search Console classified 98% of crawling as refresh, 2% as discovery.

Net effect: a crawler saw every page as a dead end that was 98% broken
links. Crawl demand collapsed — 14 crawled of 184, 15% of crawl
responses 404.

## Changes

- **Rewrite markdown links to canonical routes at build time**, and fail
the build on any link no nav page publishes. (Passed first run with zero
unresolved — every link had a valid destination, just emitted in a
404ing form.)
- **Server-render the sidebar and breadcrumb** into every interior route
from the manifest `app.js` already uses. `app.js` detects
`data-server-rendered` and keeps the DOM rather than re-rendering
identical markup. The mobile drawer stays client-built — it's
`inert`/`aria-hidden` until a JS toggle opens it, so rendering it
duplicated 192 links/page (~41KB) for no crawl benefit.
- **Intercept canonical route links in articles** so in-page navigation
stays a soft transition instead of a full reload.
- **Omit sitemap `<lastmod>` when git history is untrustworthy.**
Reproduced the real mechanism: in a `--depth=1` clone, `git log -1`
returns the single fetched commit's date for *every* file. Two files
that differ locally (`2026-08-18` / `2026-07-08`) both come back
`2026-08-25` shallow. That is exactly the live `192 × 2026-08-20`.
Cloudflare Pages shallow-clones by default.
- **Ship `redirects.json`** — `app.js` fetches it at boot and it 404'd.
- **Canonicalise the footer's `/reference/api/overview`** link, which
spent a 301 on all 193 pages.
- **Add `scripts/verify-crawlable-links.mjs` + CI**, gating on raw `.md`
hrefs, missing targets, redirecting targets, relative hrefs, an
unrendered sidebar, and uniform sitemap dates.

The last two were both caught by the new verifier on its first run.

## Result

| | Before (production) | After (generated bundle) |
|---|---|---|
| Broken in-body links | 1,283 (98%) | **0** |
| Working in-body links | 23 | **1,501** |
| Crawlable nav links/page | 0 | **192** |
| Distinct sitemap dates | 1 | **24** (0 on shallow CI) |

`Crawlable-link contract passed: 41078 internal links across 194 pages
resolve in one hop.`

## Verified

Built and driven in a real browser at `localhost:4321`:

- Raw HTML (crawler / no-JS view): 192 inline sidebar links, breadcrumb
rendered, **0 local `.md` hrefs**, single H1. Remaining empty containers
are JS-only UI (search modal, drawer, runtime status).
- SPA intact, no reloads, no JS errors: in-article links, prev/next,
sidebar links, mobile drawer (opens, 192 links, closes on navigate),
search (8 results for "adapter"). Breadcrumb and active-link state
follow correctly throughout.
- Subpath build for gh-pages emits correct `/paperclip-docs/...` hrefs.
- Homepage unchanged: 15 landing cards, 192 directory links.
- Shallow-clone build confirmed to warn and emit zero `<lastmod>`.

## Notes for review

- **This can't ship until the Cloudflare GitHub App is unsuspended.**
Everything here is verified against a local build only.
- **CI will be red on one pre-existing, unrelated failure**:
`docs:test:skill-source-blocks` on
`docs/reference/skills/optional/content/simplified-english.md` (missing
its skill-source block, from release #95). Confirmed failing on clean
`main`. Left in the aggregate rather than weakening the suite — fixing
it needs the real `SKILL.md` from the parent repo, which is sync-docs
work.
- **Production will now ship no `lastmod` at all** on Cloudflare's
shallow clone. Correct versus lying. To get real dates back, set the
Pages build command to `git fetch --unshallow || true; npm run
docs:build` — documented in the generated `DEPLOY.md`.
- Page weight 67KB → 109KB from the inlined nav. Highly repetitive,
compresses well, but it is a real increase.
- Weak/duplicate titles were deliberately skipped, per the plan's own
ranking (secondary).
- Plan item 4 (contextual links from high-authority surfaces to 20–25
priority pages) is a `paperclip-website` change plus editorial judgment,
so it's out of scope here. Worth noting that fixing the internal graph
recovers most of what that item was meant to buy — the dominant
authority failure was internal, not off-site.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
2026-08-25 21:57:07 +02:00

206 lines
13 KiB
HTML
Raw Permalink Blame History

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Paperclip Docs</title>
<meta name="description" data-seo-managed content="Guides, references, and walkthroughs for running Paperclip, an AI company operating system for agent teams, governance, budgets, and workflows." />
<meta name="robots" data-seo-managed content="index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1" />
<link rel="canonical" data-seo-managed href="https://docs.paperclip.ing/" />
<meta property="og:type" data-seo-managed content="website" />
<meta property="og:site_name" data-seo-managed content="Paperclip Docs" />
<meta property="og:title" data-seo-managed content="Paperclip Docs" />
<meta property="og:description" data-seo-managed content="Guides, references, and walkthroughs for running Paperclip, an AI company operating system for agent teams, governance, budgets, and workflows." />
<meta property="og:url" data-seo-managed content="https://docs.paperclip.ing/" />
<meta name="twitter:card" data-seo-managed content="summary" />
<meta name="twitter:title" data-seo-managed content="Paperclip Docs" />
<meta name="twitter:description" data-seo-managed content="Guides, references, and walkthroughs for running Paperclip, an AI company operating system for agent teams, governance, budgets, and workflows." />
<script>
/* Apply theme before first paint (dark by default) to avoid a flash. */
(function () {
var saved = localStorage.getItem('pc-guides-theme') || 'dark';
if (saved === 'dark') document.documentElement.setAttribute('data-theme', 'dark');
})();
</script>
<link rel="icon" type="image/svg+xml" href="data:image/svg+xml;utf8,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 24 24' fill='none' stroke='%23111' stroke-width='1.5' stroke-linecap='round' stroke-linejoin='round'><path d='M21.44 11.05l-9.19 9.19a6 6 0 0 1-8.49-8.49l9.19-9.19a4 4 0 0 1 5.66 5.66l-9.2 9.19a2 2 0 0 1-2.83-2.83l8.49-8.48'/></svg>" />
<link rel="stylesheet" href="styles.css" />
</head>
<body>
<header id="header" class="navbar">
<div class="navbar-inner">
<div class="navbar-left">
<button class="icon-btn navbar-hamburger" id="hamburger" aria-label="Open menu" aria-expanded="false">
<svg viewBox="0 0 18 18" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" aria-hidden="true">
<path d="M3 5h12M3 9h12M3 13h12"/>
</svg>
</button>
<button class="navbar-search-btn navbar-search-mobile js-search-open" aria-label="Search docs" aria-haspopup="dialog">
<svg width="14" height="14" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.75" stroke-linecap="round" aria-hidden="true">
<circle cx="7" cy="7" r="5"/><path d="m11 11 3 3"/>
</svg>
</button>
<a href="https://paperclip.ing/blog" class="navbar-link">Blog</a>
<a href="/" class="navbar-link" data-nav="home">Docs</a>
<a href="https://github.com/paperclipai/paperclip" class="navbar-link navbar-link--icon" target="_blank" rel="noopener" aria-label="GitHub">
<svg width="16" height="16" viewBox="0 0 24 24" fill="currentColor" aria-hidden="true"><path d="M12 0C5.37 0 0 5.37 0 12c0 5.31 3.435 9.795 8.205 11.385.6.105.825-.255.825-.57 0-.285-.015-1.23-.015-2.235-3.015.555-3.795-.735-4.035-1.41-.135-.345-.72-1.41-1.23-1.695-.42-.225-1.02-.78-.015-.795.945-.015 1.62.87 1.845 1.23 1.08 1.815 2.805 1.305 3.495.99.105-.78.42-1.305.765-1.605-2.67-.3-5.46-1.335-5.46-5.925 0-1.305.465-2.385 1.23-3.225-.12-.3-.54-1.53.12-3.18 0 0 1.005-.315 3.3 1.23.96-.27 1.98-.405 3-.405s2.04.135 3 .405c2.295-1.56 3.3-1.23 3.3-1.23.66 1.65.24 2.88.12 3.18.765.84 1.23 1.905 1.23 3.225 0 4.605-2.805 5.625-5.475 5.925.435.375.81 1.095.81 2.22 0 1.605-.015 2.895-.015 3.3 0 .315.225.69.825.57A12.02 12.02 0 0 0 24 12c0-6.63-5.37-12-12-12z"/></svg>
GitHub
</a>
</div>
<div class="navbar-logo">
<a id="logo" class="navbar-logo-link" href="https://paperclip.ing">
<svg width="22" height="22" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">
<path d="m16 6-8.414 8.586a2 2 0 0 0 2.829 2.829l8.414-8.586a4 4 0 1 0-5.657-5.657l-8.379 8.551a6 6 0 1 0 8.485 8.485l8.379-8.551"/>
</svg>
<span>Paperclip</span>
</a>
<a class="navbar-logo-link navbar-logo-sub" href="https://docs.paperclip.ing">Docs</a>
</div>
<div class="navbar-right">
<button id="search-trigger" class="navbar-search-btn js-search-open" aria-label="Search docs" aria-haspopup="dialog">
<svg width="14" height="14" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.75" stroke-linecap="round" aria-hidden="true">
<circle cx="7" cy="7" r="5"/><path d="m11 11 3 3"/>
</svg>
<span class="navbar-search-label">Search</span>
<kbd class="navbar-search-kbd">⌘K</kbd>
</button>
<a href="https://github.com/paperclipai/paperclip" class="navbar-link navbar-link--icon navbar-github-mobile" target="_blank" rel="noopener" aria-label="GitHub">
<svg width="18" height="18" viewBox="0 0 24 24" fill="currentColor" aria-hidden="true"><path d="M12 0C5.37 0 0 5.37 0 12c0 5.31 3.435 9.795 8.205 11.385.6.105.825-.255.825-.57 0-.285-.015-1.23-.015-2.235-3.015.555-3.795-.735-4.035-1.41-.135-.345-.72-1.41-1.23-1.695-.42-.225-1.02-.78-.015-.795.945-.015 1.62.87 1.845 1.23 1.08 1.815 2.805 1.305 3.495.99.105-.78.42-1.305.765-1.605-2.67-.3-5.46-1.335-5.46-5.925 0-1.305.465-2.385 1.23-3.225-.12-.3-.54-1.53.12-3.18 0 0 1.005-.315 3.3 1.23.96-.27 1.98-.405 3-.405s2.04.135 3 .405c2.295-1.56 3.3-1.23 3.3-1.23.66 1.65.24 2.88.12 3.18.765.84 1.23 1.905 1.23 3.225 0 4.605-2.805 5.625-5.475 5.925.435.375.81 1.095.81 2.22 0 1.605-.015 2.895-.015 3.3 0 .315.225.69.825.57A12.02 12.02 0 0 0 24 12c0-6.63-5.37-12-12-12z"/></svg>
</a>
<a href="https://github.com/paperclipai/paperclip" class="navbar-cta" target="_blank" rel="noopener">
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><polygon points="12 2 15.09 8.26 22 9.27 17 14.14 18.18 21.02 12 17.77 5.82 21.02 7 14.14 2 9.27 8.91 8.26 12 2"/></svg>
Star<span class="navbar-cta-count" data-star-count hidden></span>
</a>
</div>
</div>
</header>
<!-- ─── Search modal (⌘K) ──────────────────────────────────── -->
<div id="search-modal" role="dialog" aria-modal="true" aria-label="Search" hidden>
<div id="search-modal-backdrop"></div>
<div id="search-modal-panel">
<div id="search-wrap">
<svg id="search-icon" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" aria-hidden="true">
<circle cx="7" cy="7" r="5"/><path d="m11 11 3 3"/>
</svg>
<input id="search-input" type="search" placeholder="Search the docs…" autocomplete="off" spellcheck="false" aria-label="Search guides" aria-haspopup="listbox" />
<kbd id="search-kbd">esc</kbd>
<div id="search-results" role="listbox" aria-label="Search results"></div>
</div>
</div>
</div>
<!-- ─── Floating theme toggle (bottom-right) ───────────────── -->
<button class="theme-toggle" id="theme-toggle" aria-label="Toggle light/dark mode">
<span class="theme-toggle-icon theme-toggle-sun" aria-hidden="true">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<circle cx="12" cy="12" r="5"/><line x1="12" y1="1" x2="12" y2="3"/><line x1="12" y1="21" x2="12" y2="23"/><line x1="4.22" y1="4.22" x2="5.64" y2="5.64"/><line x1="18.36" y1="18.36" x2="19.78" y2="19.78"/><line x1="1" y1="12" x2="3" y2="12"/><line x1="21" y1="12" x2="23" y2="12"/><line x1="4.22" y1="19.78" x2="5.64" y2="18.36"/><line x1="18.36" y1="5.64" x2="19.78" y2="4.22"/>
</svg>
</span>
<span class="theme-toggle-icon theme-toggle-moon" aria-hidden="true">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path d="M21 12.79A9 9 0 1 1 11.21 3 7 7 0 0 0 21 12.79z"/>
</svg>
</span>
</button>
<div id="drawer-backdrop"></div>
<aside id="drawer" aria-label="Mobile navigation" aria-hidden="true" inert>
<div id="drawer-head">
<span class="label">Documentation</span>
<button class="icon-btn" id="drawer-close" aria-label="Close menu">
<svg viewBox="0 0 18 18" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round"><path d="m4 4 10 10M14 4 4 14"/></svg>
</button>
</div>
<a class="sb-back" href="/" data-nav="home">
<svg viewBox="0 0 12 12" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round"><path d="M8 2 4 6l4 4"/></svg>
All docs
</a>
<div id="drawer-sections"></div>
</aside>
<!-- ─── Landing page ─────────────────────────────────────── -->
<section id="landing">
<div id="landing-hero">
<div id="landing-eyebrow">Documentation</div>
<h1 id="landing-title">Everything you need to <em>run Paperclip.</em></h1>
<p id="landing-desc">Guides, references, and walkthroughs for the people running AI agents at work. Start at the quickstart, or jump anywhere below.</p>
</div>
<div class="card-grid" id="landing-cards"></div>
</section>
<!-- ─── Article view ─────────────────────────────────────── -->
<div id="article-view">
<div id="body">
<nav id="sidebar" aria-label="Documentation">
<a class="sb-back" href="/" data-nav="home">
<svg viewBox="0 0 12 12" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round"><path d="M8 2 4 6l4 4"/></svg>
All docs
</a>
<div id="sb-sections"></div>
</nav>
<main id="content">
<div id="breadcrumb" aria-label="Breadcrumb"></div>
<!-- Empty on every served document. Loading and error text is written here
by app.js only during a real client-side transition, so raw HTML never
claims a page is loading or broken. -->
<div id="runtime-status" role="status" aria-live="polite" hidden></div>
<article id="article" style="display:none"></article>
<div id="page-nav" style="display:none"></div>
</main>
<aside id="toc-rail" aria-label="On this page">
<div class="toc-rail-inner">
<div class="toc-rail-title">On this page</div>
<nav id="toc-rail-links" aria-label="Page sections"></nav>
</div>
</aside>
</div>
</div>
<footer class="footer" data-theme="dark">
<div class="footer-wordmark-row footer-container">
<span class="footer-wordmark">Paperclip</span>
</div>
<div class="footer-columns footer-container">
<div class="footer-col">
<div class="footer-col-heading">Product</div>
<a href="https://paperclip.ing/#get-started" target="_blank" rel="noopener">Get started</a>
<a href="/guides/getting-started/five-minute-path/" data-nav="link">Quickstart</a>
</div>
<div class="footer-col">
<div class="footer-col-heading">Platform</div>
<a href="/guides/org/agents/" data-nav="link">Agents</a>
<a href="https://paperclip.ing/#governance-section" target="_blank" rel="noopener">Governance</a>
<a href="/guides/org/adapters/" data-nav="link">Integrations</a>
</div>
<div class="footer-col">
<div class="footer-col-heading">Developers</div>
<a href="/" data-nav="home">Documentation</a>
<a href="/reference/api/overview/" data-nav="link">API reference</a>
<a href="https://github.com/paperclipai/paperclip" target="_blank" rel="noopener">GitHub</a>
</div>
<div class="footer-col">
<div class="footer-col-heading">Resources</div>
<a href="https://paperclip.ing/blog" target="_blank" rel="noopener">Blog</a>
<a href="https://paperclip.ing/changelog" target="_blank" rel="noopener">Changelog</a>
<a href="https://github.com/paperclipai/paperclip/blob/main/CONTRIBUTING.md" target="_blank" rel="noopener">Contributing</a>
<a href="https://github.com/paperclipai/paperclip/blob/main/LICENSE" target="_blank" rel="noopener">License</a>
<a href="https://discord.gg/m4HZY7xNG3" target="_blank" rel="noopener">Discord</a>
</div>
</div>
<div class="footer-bottom footer-container">
<span class="footer-copy">&copy; 2026 Paperclip. Open source under MIT.</span>
<div class="footer-bottom-links">
<a href="https://paperclip.ing/tos#privacy" target="_blank" rel="noopener">Privacy</a>
<a href="https://paperclip.ing/tos" target="_blank" rel="noopener">Terms</a>
</div>
</div>
</footer>
<script src="app.js" defer></script>
</body>
</html>
Reference in new issue View Git Blame Copy Permalink