Commit Graph
13 Commits
Author SHA1 Message Date
Aron PrinsandClaude Opus 5 c593c23233 seo: date sitemap entries by body changes, not frontmatter sweeps (#118)
Follow-up to #117. That PR got `<lastmod>` publishing at all; this one
makes the dates worth resubmitting.

## The problem

**190 of 193 pages read `2026-08-25`** — the date of #115, which added
`seo_title`/`seo_description` to all 192 pages. Here is everything it
changed in a representative file:

```diff
+---
+seo_title: "Key Concepts: Companies, Agents, Tasks"
+seo_description: The seven ideas behind Paperclip — company, agent, task...
+---
+
 # Key Concepts
```

Not a word of prose. `lastmod` is meant to carry the date a page's
**content** last changed, so resubmitting this hands Search Console the
"whole site changed at once" signal that `dropUniformLastmod` and the
shallow-checkout guard both exist to prevent. It cleared
`dropUniformLastmod` only because 3 pages happened to differ.

## The fix

Walk each file's history and report the newest commit that changed the
**rendered body** — comparing `parseFrontmatter().body` across
consecutive commits, so frontmatter-only commits are skipped. Reusing
the build's own parser means the body compared is exactly the body the
site renders.

| | Before | After |
|---|---|---|
| Distinct dates | **2** | **31** |
| Largest single-date cluster | 190 pages | 54 pages |
| Range | 2 days | 2026-04-17 → 2026-08-27 |

Top of the new distribution:

| Date | Pages |
|---|---|
| 2026-08-18 | 54 |
| 2026-06-04 | 21 |
| 2026-08-25 | 17 |
| 2026-07-08 | 16 |
| 2026-07-20 | 15 |
| 2026-04-22 | 12 |

Spot-checked both directions:

- `key-concepts.md` — last touched by the frontmatter sweep, now dates
to **2026-07-08**, the commit that changed "Claude Local" to "Claude
Code" in the body.
- `day-to-day/issues.md` — **keeps 2026-08-25**, because release commit
`6c4519a` that day added a real "Who may resolve a card" section. Its
frontmatter-only `paperclip_version` bump in the same release was
correctly skipped.

## Two supporting fixes

**Concurrency.** ~600 git round-trips in series cost 40s. Resolving them
in parallel: **11s**.

**A race the concurrency exposed** — and the reason this is worth
reading closely. `hasTrustworthyGitHistory()` memoised its *resolved
value*, so once all 192 pages asked at once, every one of them passed
the guard before the first resolved:

```
Could not deepen the shallow checkout: fatal: Unable to create '.git/shallow.lock': File exists.
Still shallow: omitting sitemap <lastmod> rather than dating every page alike.
   × 192
```

192 concurrent `git fetch --unshallow` calls fighting over the lock, and
**a sitemap with no dates at all**. This only reproduces on a shallow
clone — which is to say only on Cloudflare Pages, never locally or in
`docs-test`. Fixed by memoising the promise (single-flight).

## Verification

| Scenario | Result |
|---|---|
| Full clone | 193 dates, 31 distinct, 11s |
| `git clone --depth 1` (reproduces CF) | one unshallow, **193 dates, 31
distinct** |
| Shallow + unreachable remote | no dates, exit 0 |
| `npm run docs:test` (6 suites) | pass |
| `npm run sync:test` | 39 pass, 0 fail (+1 new) |

New unit test covers the `--follow --name-status` rename parsing — 55
docs files have been renamed, and reading the wrong field silently drops
their pre-rename history.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
2026-08-29 10:55:32 +02:00
Aron PrinsandClaude Opus 4.8 8a858777f2 sync-docs: tag-accuracy gate + release-channel-aware skill (#111)
Makes /sync-docs correct under the canary→nightly→beta→stable channel model. Adds the release-mode tag-accuracy gate (scripts/sync/check-tag-accuracy.mjs) that verifies every changed page against the stable tag and flags post-tag leaks; stable-CalVer guard + channel glossary, tag-scoped drift/verify, and Phase 5.7 in the skill; realign fix that keeps nightly's drafts; a release-channels watcher; the analysis/plan doc; and a hermetic test (sync:test 38/38).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 11:48:38 +02:00
Aron PrinsandClaude Opus 4.8 55de2cbb81 fix(sync): pin drift check to an immutable SHA, key cache by SHA (#62)
Ports the drift-checker fix to `main` (same change already merged to
`nightly` in #61). `check-drift.mjs` is byte-identical across branches,
so this cherry-pick applies cleanly.

## Problem

The drift checker printed the parent's **live** SHA in its header but
keyed its file cache by the moving name `drift-master` with no
invalidation — reusing days-old file contents while claiming to check
today's commit, silently hiding real drift.

## Fix

- Resolve the ref to a full commit SHA before any reads; cache key =
`drift-<sha>`, never `drift-master`.
- Fail closed if the ref can't be resolved.
- Mode-aware ref from `.sync-state.json` (release → `base_release_sha`,
nightly → `last_seen_parent_sha`) before falling back to the default
branch. `main` is release-mode, so its checks now pin to the release SHA
instead of moving `master`.
- Testable cache path (`PAPERCLIP_SYNC_CACHE_DIR` + fixture SHA stub);
+2 regression tests. `npm run sync:test`: 37 passed, 0 failed.

## Triage note

Running the fixed checker on `main`'s released docs against the release
tag `v2026.707.0` returns **0 drift** — released docs are correct about
the shipped product; this is a tooling fix only, no doc content changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 <[email protected]>
2026-07-15 13:36:41 +02:00
Aron Prins 7e25158fca Release docs for paperclip v2026.707.0 (#49)
Promotes v2026.707.0 docs from nightly: Custom sandbox images + Ramp skill (new), Work Timeline (#48) and Secret scopes (#47) content, screenshot capture targets, environments nav reconcile, sync-state bump to v2026.707.0, and accuracy fixes from adversarial verification. 0 drift.
2026-07-08 11:11:32 +02:00
Aron PrinsandClaude Opus 4.8 1bba6f6601 docs(access): document members, roles, permissions, invites & offboarding (#38)
Back-fills documentation for the human multi-user collaboration surface
(company members, invites, roles, permissions, offboarding). These
features are already shipped and live; this fills doc gaps and fixes one
live permission-key drift. Branched off `main` (not `nightly`) because
it's editorial back-fill of released features, intended to ship
independently of the nightly sync cadence.

## Tier 1 — correctness
- **Fix live permission drift** in `administration/company.md`: the
grant list and Owner/Admin role descriptions were missing
`environments:manage` and `tasks:manage_active_checkouts` (both present
in `packages/shared/src/constants.ts` `PERMISSION_KEYS`;
`environments:manage` is an Owner/Admin default in
`server/src/services/company-member-roles.ts`).
- **New** `administration/roles-and-permissions.md`: canonical 8-key
catalog (what each gates), role→default-grant matrix, additive
precedence rule, `instance_admin` layer, and current limitations.
- `maintenance/follow-ups.md`: note to teach the drift checker the
permission catalog so this can't silently drift again. (Sync tooling
itself untouched.)

## Tier 2 — how-tos
- `how-to/add-a-human-teammate.md` — invite → join request → approve →
set role/grants (UI + CLI).
- `how-to/offboard-a-member.md` — suspend vs archive, work reassignment,
membership-status-gates-access, instance-level cleanup.
- `how-to/enable-multi-user-login.md` — `local_trusted → authenticated`,
board-claim handoff, then invite people.

## Tier 3 — concept
- `guides/org/members-and-access.md` — shared human+agent principal
model, roles vs grants, human roles ≠ agent org chart, instance admin,
the member profile page, and the "left a project ≠ lost access"
clarification.

## Verification
- All 5 new pages registered in `site/content.json` (valid JSON, +5
pages).
- `sync:lint-links` and `sync:verify-nav` clean for these changes. The
single link the linter reports pre-exists on `main` in a skills file
untouched here; the 2 nav orphans are the intentional CLI redirect
anchors.
- All content verified against `paperclipai/paperclip@master` source
(permission keys, role defaults, CLI syntax, deployment modes).

## Follow-up after merge
Back-merge `main → nightly` so the next release merge stays clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
2026-07-01 18:11:25 +02:00
Aron Prins 17e046db5f Harden screenshot runner environment 2026-05-30 17:52:14 +02:00
Aron Prins 52542db923 Fix sync verification edge cases 2026-05-13 21:46:12 +02:00
Aron PrinsandClaude Opus 4.7 c071db51e4 Add post-edit code verification helper
Drift detection catches the coarse case (doc references X, parent
doesn't have X). Verification is finer-grained: doc claims specific
properties (flag names, env vars, route paths, schema field names,
config values) and we check each individual claim against parent
code, post-edit.

scripts/sync/verify-edit.mjs (1266 lines):
- Takes a doc file + parent ref. Extracts claims by kind:
  cli-command, cli-flag, env-var, rest-route,
  adapter-config-field, adapter-config-value, file-path.
- Uses structural cues (Config sections, code fences, tables) to
  avoid extracting from natural prose — guard against
  false-positives on phrasing like "use the `model` field".
- Per-claim verification:
  - cli-* → grep parent cli/src/** for command/flag literals
  - env-var → check .env.example, server/src/config.ts, packages/**/env.ts
  - rest-route → grep matching routes/<surface>.ts with {id}↔:id normalization
  - adapter-* → grep packages/adapters/<name>/** for literal field/value
  - file-path → gh contents API existence check
- Three output buckets: verified, unverified (with Levenshtein-≤3
  fuzzy "did you mean" suggestion), suspicious.
- Cached gh fetches under /tmp/paperclip-sync/verify-<ref-sha>/.
- Same fixture-loader pattern as compare-window/check-drift.

Tests: 6 new (CLI pass/fail, adapter field typo with suggest,
REST route normalization, env var pass, prose-guard
false-positive). 26/26 total passing.

Live test against test/sync-docs-catchup-run:cursor-cloud.md →
20/20 claims verified. The catchup agent's worry that adapter docs
might invent field names didn't materialize on cursor-cloud (every
config field is present in packages/adapters/cursor-cloud/**) — but
the tool would have caught it. acpx-local hasn't been verified yet;
worth running before the next merge decision.

Wired as sync:verify-edit. SKILL.md update to follow.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-05-13 13:54:46 +02:00
Aron PrinsandClaude Opus 4.7 16b910db88 Add directory-rename detection helper
GitHub's compare API marks individual file renames via status='renamed'
when git's similarity heuristic catches them, but DIRECTORY renames
(common case: packages/adapters/foo-local → foo-cloud) appear as a
bunch of add/remove pairs. Without aggregation, the adapters watcher
would treat the addition as a genuinely-new adapter and create a
duplicate doc page.

scripts/sync/detect-renames.mjs (252 lines):
- Input: a compare-window.mjs JSON file (or stdin).
- Detects directory renames under hard-coded watched prefixes:
  packages/adapters/, packages/plugins/sandbox-providers/,
  server/src/adapters/, skills/.
- Aggregates added/removed files into depth-2 dirs, requires ALL
  files in a candidate dir to share the add/remove status (mixed
  with 'modified' = the dir is alive, not a rename).
- Three matching signals, ranked by reliability:
  - file-content-overlap (≥50% suffix-path intersection) → high
  - shared-prefix (≥3 chars; ≥50% of longer leaf → high)
  - levenshtein (≤30% of shorter leaf) → medium
- Greedy 1:1 assignment when multiple pairs compete.
- Reports: renames, added_dirs_genuinely_new, removed_dirs_no_match.

Wired as sync:detect-renames in package.json.

Tests: 5 new (pure rename, shared-prefix, genuine new dir, genuine
removal, multi-candidate tiebreak). 20/20 total passing.

Live run on v2026.318.0...v2026.512.0: 0 renames detected
(correct — no dirs got renamed in that window; cursor-cloud is a
genuinely new sibling of cursor-local, not a rename). 7 dirs flagged
as genuinely new and verified.

A follow-up commit will wire this into the SKILL Phase 3 manifest
build so the adapter watcher consults it before creating doc pages.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-05-13 13:46:52 +02:00
Aron PrinsandClaude Opus 4.7 1922822817 Add YAML frontmatter support to the docs build pipeline
The SKILL says to add `paperclip_version: <tag>` frontmatter in
release mode but `site/build-release.mjs` didn't strip YAML —
`---` blocks would render as horizontal rules in the SPA. The
catchup wet run correctly refused to add the frontmatter for
this reason.

Now the build pipeline is frontmatter-aware:

- site/build-release.mjs: exported parseFrontmatter() (~30 line
  inline regex parser, no deps), copyMarkdownIntoRelease() that
  strips YAML on copy, frontmatter attached to each page in
  releaseNav before writing content.json. Guarded main() so test.mjs
  can import the module without triggering a build.
- site/app.js: stripFrontmatter() safety net in renderMarkdown plus
  a small "Documenting Paperclip <version>" footer block inside
  appendPageFeedback() that renders when page.frontmatter
  .paperclip_version is present.
- scripts/sync/test.mjs: 3 new cases (well-formed, none,
  malformed). 15/15 passing.

Parser intentionally minimal — scalar key:value lines only.
Swap in js-yaml later if frontmatter usage outgrows that.

Manual test: temporarily added frontmatter to claude-local.md,
built, confirmed YAML stripped from rendered body and version
recorded in content.json. Edit reverted.

A follow-up SKILL.md update will switch the release-mode voice
rule from "skip the frontmatter" to "add paperclip_version".

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-05-13 13:41:57 +02:00
Aron PrinsandClaude Opus 4.7 72f56f64ad Add drift detection — catches docs referencing removed parent surfaces
The wet-run found POST /api/companies/{companyId}/logo documented in
docs/reference/api/companies.md but absent from the current parent
server/src/routes/companies.ts. The cumulative-diff path is blind to
this — it sees what's new, not what's vanished. Drift is independent
of the diff window: it can exist regardless of when the last sync ran.

scripts/sync/check-drift.mjs (712 lines):
- Four drift classes:
  - parent-path-missing (high confidence): doc references to
    cli/src/..., packages/..., server/src/... paths that 404 in parent.
  - cli-command-missing/ambiguous (high/medium): documented
    `paperclipai <subcommand>` not found in parent cli source.
  - env-var-missing (high): table rows in environment-variables.md
    that don't appear in .env.example or watched config files.
  - rest-route-missing (medium): ## METHOD /api/... headers in
    docs/reference/api/*.md not registered in matching routes/*.ts.
    Normalisation handles {id} ↔ :id and prefix-stripping for
    relative-mounted routers.
- Cached gh contents under /tmp/paperclip-sync/drift-* (per-ref).
- Same fixture-loader pattern as compare-window.mjs for hermetic
  unit tests via PAPERCLIP_SYNC_FIXTURE_DIR.
- Never auto-resolves — always surfaces for human review.

SKILL.md changes:
- New Phase 1.5: runs check-drift against parent default branch
  HEAD, independent of the diff window. Drift records form a new
  manifest tier (not auto-merge, not pr, not context-only).
- Phase 4 dry-run summary includes drift counts.
- Phase 7 PR bodies get a "⚠ Drift" section.

Tests: 4 new (12/12 total passing). Live run against parent master
caught the motivating logo-route case plus 40 other genuine drift
candidates that should be triaged separately (env vars in untracked
packages, the goals-and-projects → goals.ts+projects.ts file-split
mismatch, etc.).

sync:check now chains lint-links + verify-nav + check-drift.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-05-13 11:55:47 +02:00
Aron PrinsandClaude Opus 4.7 0dded0ef75 Implement cumulative-diff bisection as a helper script
The wet-run hit the GitHub compare API's 300-file/250-commit cap on
the first call against the 318→512 window. The SKILL had the
algorithm in prose — that's load-bearing now, so it gets a real
implementation.

scripts/sync/compare-window.mjs (392 lines):
- Recursive midpoint SHA bisection. Truncation detected by
  files.length===300 OR total_commits>commits.length.
- SHA enumeration via compare's .commits[].sha (up to 250); falls
  back to paginated /commits for windows exceeding that.
- Leaf responses cached under /tmp/paperclip-sync/<a..b>.json so
  re-runs within a day are cheap.
- Status-merge rules when unioning: added+removed → drop;
  added+modified → modified; modified+removed → removed;
  renamed wins over modified.
- CLI: node scripts/sync/compare-window.mjs <A> <B> [--json] [--repo]
- Fixture-driven testability via PAPERCLIP_SYNC_FIXTURE_DIR env var
  so unit tests stay hermetic without HTTP mocks.

Test coverage (3 new unit + 1 live integration in test.mjs):
- No-truncation case (1 leaf, no recursion).
- Single split (2 leaves, status-merge verified).
- Status merge cancellation (added+removed → drop).
- Live: real 318→512 window returns 1,726 unique files across 22
  leaves with truncated_leaves=0 and the three sentinel files
  present.

SKILL.md Phase 2: replace the algorithm prose with a one-line
helper invocation. Status-merge rules stay in SKILL as contract.

8/8 tests passing.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-05-13 11:40:41 +02:00
Aron PrinsandClaude Opus 4.7 e09564a34d Add sync:test — unit tests for helper scripts
Self-contained test runner (no framework). Creates tmpdir fixtures
per test and asserts script behaviour via subprocess invocation.

Covers:
- lint-links: positive + 9 skip-case patterns (external URLs, schemes,
  anchors, absolute SPA routes, template placeholders).
- verify-nav: dangling + orphan classification, screenshots-dir exclusion,
  --strict mode.
- backfill-screenshots: idempotency (preserves entries with non-empty
  depends_on, scaffolds the rest).

4/4 passing on the current source. No source-script bugs found.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-05-13 10:47:45 +02:00