paperclipai/paperclip-docs
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip-docs.git synced 2026-11-07 06:16:07 -10:00
Code Issues Packages Projects Releases Wiki Activity
266 Commits 57 Branches 11 Tags
codex/github-low-trust-docs
Commit Graph
6 Commits
Author SHA1 Message Date
Aron PrinsandClaude Opus 5 4644dda3a3 test: guard sitemap lastmod against shallow clones and metadata sweeps (#119) 
Picks up the regression test from #104 (@cryppadotta) — closed as
superseded, but its fixture design was the best part of it and worth
landing on its own.

## Why this gap mattered

The sitemap `lastmod` bug shipped twice, and **neither failure was
detectable by any check in `docs:test`**:

| | Symptom | Why nothing caught it |
|---|---|---|
| #117 | 0 dates in production for weeks | Shallow checkout doesn't make
`git log` *fail* — it succeeds, returning one date per file |
| #118 | 192 concurrent `git fetch --unshallow` fighting over
`.git/shallow.lock`, dateless sitemap | Reproduces only on a shallow
clone — never locally, never in CI |

Both are Cloudflare-only failure modes. The only detector we had was
reading the live sitemap, which is how this whole thread started.

## What landed

`scripts/verify-sitemap-lastmod-history.mjs`, wired into `docs:test`.
Three cases against **real depth-1 clones** — the fixture design and
`--no-local` insight are @cryppadotta's:

1. Full history publishes real per-document dates
2. Shallow clone that can reach origin recovers history
3. Shallow clone with unreachable origin omits dates rather than lying

Two changes on top of the original:

**Overlay the working-tree builder into the clone.** The clone carries
committed HEAD, so without this the shallow cases silently test the last
commit instead of the change being made. No-op in CI; the difference is
real locally, and it is what makes the check usable while iterating.

**Assert no single date covers >75% of pages.** The existing "more than
one distinct date" assertion passes at *2* distinct dates — precisely
the state #118 fixed, where 190 of 193 pages carried the date of a
frontmatter-only sweep. Bounding the largest cluster catches that class
without pinning a date that legitimate edits will move.

## Verified by breaking it

A check that has never failed proves nothing, so I reverted each fix in
turn:

**Revert body-aware dating (#118):**
```
  ok  lastmod values vary across documents (2 distinct) rather than one build date
FAIL  full history: no single date dominates (2026-08-25 covers 190/193, 98% <= 75%)
```
Note the pre-existing assertion passing right above the failure — that
is the gap this closes.

**Revert the single-flight memo (#118):**
```
FAIL  unshallowed build dates every route (2/193)
FAIL  unshallowed build produces varied dates (1 distinct)
```
The `.git/shallow.lock` race, caught by Case 2 and nothing else.

Restored, clean run is green: 193/193 dated, 31 distinct, largest
cluster 28%.

## Cost

**~55s** added to `docs:test` for three builds (`docs-test` was 32s).
Steep for a test suite, and worth flagging rather than burying — but the
alternative detector for this failure class is noticing the live sitemap
is wrong.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
 2026-08-29 11:10:35 +02:00
Aron PrinsandClaude Opus 5 325c9eb819 seo: hand-write titles and descriptions for all 192 pages (#115) 
## Why

The build derived `<title>` from the **sidebar label** and the
description from the **first paragraph clipped to 220 characters**. Both
are the wrong source. A sidebar label is written for a 240px-wide nav
that supplies its own context; a search result has none.

On production that produced:

- **7 pages titled "Overview"**, and 32 pages sharing a title with
another page
- **109 of 193 descriptions cut off mid-word**, and 174 over 160
characters (where Google truncates)

```
/guides/welcome/key-concepts/
  BEFORE (220):  ...and if you misunderstand them early, the product wi
```

Near-identical titles are the weak-differentiation signal behind
"crawled — currently not indexed", which is the exact symptom on this
property.

## What changed

`seo_title` and `seo_description` frontmatter on **every one of the 192
pages**, written by hand against each page's real content and optimised
for the click — lead with what the reader can do, name the concrete
things covered, no keyword stuffing, no repeating the title.

| | Before | After |
|---|---|---|
| Duplicate titles | 32 pages | **0** |
| Duplicate descriptions | — | **0** |
| Descriptions cut mid-word | 109 | **0** |
| Descriptions over 160 chars | 174 | **0** |
| Title length (incl. suffix) | up to "Overview" | **26–60** |
| Description length | 31–240 | **122–158** |

**No prose was rewritten.** I measured content overlap between the
colliding pairs first — only 17–23%, so these were always distinct pages
wearing the same label, not duplicate content. The `docs/` diff is **564
insertions, zero deletions**.

### Examples

```
BEFORE — all seven identical:  "Overview | Paperclip Docs"
AFTER:  "Adapters Overview"      "API Overview"        "CLI Overview"
        "Deployment Overview"    "Bundled Skills"      "Optional Skills"
        "Experimental Features"

BEFORE — both identical:  "Issues | Paperclip Docs"
AFTER:  "Issues: Creating and Tracking Work"   /guides/day-to-day/issues/
        "Issues API"                           /reference/api/issues/
```

```
/guides/getting-started/five-minute-path/
BEFORE (220):  The shortest route from a running Paperclip instance to an agent that has
               just completed work for you. After Paperclip is installed, the path takes
               about 5 minutes: create a company, hire a CEO agent, approve its first s
AFTER  (131):  Install Paperclip, create a company, hire a CEO agent, and approve its
               first strategy — a working AI company in about five minutes.
```

## Plumbing

- **`site/build-release.mjs`** prefers the authored fields; the derived
path stays only as a fallback the CI gate prevents ever reaching.
`<title>`, `<meta name="description">`, and the `TechArticle` JSON-LD
all read from the same source.
- **`site/app.js`** uses the same fields, so a client-side transition
cannot leave a different title on screen than the crawler was served.
Verified in-browser: navigating `/guides/day-to-day/issues/` →
`/reference/api/issues/` swaps to exactly the server-rendered values, no
JS errors.
- **`scripts/verify-seo-metadata.mjs`** (new) enforces presence,
uniqueness, length, complete-sentence endings, and the
frontmatter-safety rules. Wired into `docs:test` and CI as its own step.

I negative-tested the gate rather than assuming it works — it correctly
fails on a duplicate title, a mid-word description, and a missing field,
and passes again once restored.

## Automating it going forward

`skills/sync-docs/SKILL.md` now:

- gains an **"Authored SEO metadata"** section: why the fields exist,
the enforced rules as a table, and how to actually write them (title
names the subject not the nav position; description writes for the
click; never generate mechanically from the first paragraph — *that is
the fallback the gate exists to prevent*).
- requires both fields in the **page-authoring instruction** given to
every subagent, with guidance on when to update an existing description
versus leave it.
- runs `docs:test:seo-metadata` and `docs:test:crawlable-links` as
**fail-loud gates in Phase 7** before commit.
- documents all six build gates in the files/gates section.

So once this lands, a new page cannot ship without hand-written
metadata, and a title collision cannot reach production.

## Verified

All six verifiers pass:

```
Static route verification passed
Asset fingerprint verification passed
Hierarchical Skills nav verification passed
Skill source block verification passed (11 skill pages, 1 control page)
Crawlable-link contract passed: 40886 internal links across 194 pages resolve in one hop
Authored-SEO contract passed: 192 pages carry a unique, hand-written seo_title and seo_description
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
 2026-08-25 22:51:42 +02:00
Aron PrinsandClaude Opus 5 29beedeb6a seo: make the docs link graph crawlable (#113) 
## Why

Google has indexed effectively nothing on `docs.paperclip.ing`. Search
Console over 3 months: **8 of 192 URLs** have ever drawn an impression,
**681 of 975 impressions (70%) go to
`/guides/agent-developer/comments-and-communication`** — a legacy URL
that 301s and isn't in the sitemap. Strip that, the homepage, and the
team's own `site:` queries, and the canonical docs corpus earned **~7
organic impressions in 90 days**.

It isn't an eligibility problem. Pages are fetchable, self-canonical,
`index, follow`, and identical under a Googlebot UA. It's the internal
link graph.

## What was actually broken

**1. Every relative markdown link 404'd.** Authors write
`./glossary.md`, but pages are served from trailing-slash route
directories, so links resolved one level too deep:

```
page: /guides/welcome/key-concepts/   href: ./glossary.md
  → /guides/welcome/key-concepts/glossary.md   404
  ( /guides/welcome/glossary.md would be 200 )
```

Crawled all 192 live pages: **all 742 unique `.md` targets return 404**.
That's **1,283 of 1,306 in-body internal links — 98% — broken**, leaving
**0.1 working in-body links per page**.

It went unnoticed because `app.js:postProcessInternalLinks` repairs
these at runtime. Crawlers index the pre-render.

**2. The sidebar wasn't server-rendered.** Interior pages shipped `<div
id="sb-sections"></div>`, filled by JS. So the raw HTML offered the
logo, prev, next — and ~7 dead links. No link equity reached deep pages.
Search Console classified 98% of crawling as refresh, 2% as discovery.

Net effect: a crawler saw every page as a dead end that was 98% broken
links. Crawl demand collapsed — 14 crawled of 184, 15% of crawl
responses 404.

## Changes

- **Rewrite markdown links to canonical routes at build time**, and fail
the build on any link no nav page publishes. (Passed first run with zero
unresolved — every link had a valid destination, just emitted in a
404ing form.)
- **Server-render the sidebar and breadcrumb** into every interior route
from the manifest `app.js` already uses. `app.js` detects
`data-server-rendered` and keeps the DOM rather than re-rendering
identical markup. The mobile drawer stays client-built — it's
`inert`/`aria-hidden` until a JS toggle opens it, so rendering it
duplicated 192 links/page (~41KB) for no crawl benefit.
- **Intercept canonical route links in articles** so in-page navigation
stays a soft transition instead of a full reload.
- **Omit sitemap `<lastmod>` when git history is untrustworthy.**
Reproduced the real mechanism: in a `--depth=1` clone, `git log -1`
returns the single fetched commit's date for *every* file. Two files
that differ locally (`2026-08-18` / `2026-07-08`) both come back
`2026-08-25` shallow. That is exactly the live `192 × 2026-08-20`.
Cloudflare Pages shallow-clones by default.
- **Ship `redirects.json`** — `app.js` fetches it at boot and it 404'd.
- **Canonicalise the footer's `/reference/api/overview`** link, which
spent a 301 on all 193 pages.
- **Add `scripts/verify-crawlable-links.mjs` + CI**, gating on raw `.md`
hrefs, missing targets, redirecting targets, relative hrefs, an
unrendered sidebar, and uniform sitemap dates.

The last two were both caught by the new verifier on its first run.

## Result

| | Before (production) | After (generated bundle) |
|---|---|---|
| Broken in-body links | 1,283 (98%) | **0** |
| Working in-body links | 23 | **1,501** |
| Crawlable nav links/page | 0 | **192** |
| Distinct sitemap dates | 1 | **24** (0 on shallow CI) |

`Crawlable-link contract passed: 41078 internal links across 194 pages
resolve in one hop.`

## Verified

Built and driven in a real browser at `localhost:4321`:

- Raw HTML (crawler / no-JS view): 192 inline sidebar links, breadcrumb
rendered, **0 local `.md` hrefs**, single H1. Remaining empty containers
are JS-only UI (search modal, drawer, runtime status).
- SPA intact, no reloads, no JS errors: in-article links, prev/next,
sidebar links, mobile drawer (opens, 192 links, closes on navigate),
search (8 results for "adapter"). Breadcrumb and active-link state
follow correctly throughout.
- Subpath build for gh-pages emits correct `/paperclip-docs/...` hrefs.
- Homepage unchanged: 15 landing cards, 192 directory links.
- Shallow-clone build confirmed to warn and emit zero `<lastmod>`.

## Notes for review

- **This can't ship until the Cloudflare GitHub App is unsuspended.**
Everything here is verified against a local build only.
- **CI will be red on one pre-existing, unrelated failure**:
`docs:test:skill-source-blocks` on
`docs/reference/skills/optional/content/simplified-english.md` (missing
its skill-source block, from release #95). Confirmed failing on clean
`main`. Left in the aggregate rather than weakening the suite — fixing
it needs the real `SKILL.md` from the parent repo, which is sync-docs
work.
- **Production will now ship no `lastmod` at all** on Cloudflare's
shallow clone. Correct versus lying. To get real dates back, set the
Pages build command to `git fetch --unshallow || true; npm run
docs:build` — documented in the generated `DEPLOY.md`.
- Page weight 67KB → 109KB from the inlined nav. Highly repetitive,
compresses well, but it is a real increase.
- Weak/duplicate titles were deliberately skipped, per the plan's own
ranking (secondary).
- Plan item 4 (contextual links from high-authority surfaces to 20–25
priority pages) is a `paperclip-website` change plus editorial judgment,
so it's out of scope here. Worth noting that fixing the internal graph
recovers most of what that item was meant to buy — the dominant
authority failure was internal, not off-site.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
 2026-08-25 21:57:07 +02:00
Aron PrinsandClaude Opus 4.7 4ef86508a2 Restore GitHub issue templates 
Bring back the support, bug, docs-feedback, and config templates that
were removed in 5b23678 so issue intake has structure again.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
 2026-04-17 20:23:02 +02:00
Aron Prins 5b2367875c (chore): cleanup 2026-04-17 20:17:49 +02:00
Aron Prins 3a852ec7e9 Initial docs site import 2026-04-17 13:54:39 +02:00