From 8a858777f2bf2f59ea13f5fb8bca0ed137106ce3 Mon Sep 17 00:00:00 2001 From: Aron Prins Date: Tue, 25 Aug 2026 11:48:38 +0200 Subject: [PATCH] sync-docs: tag-accuracy gate + release-channel-aware skill (#111) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Makes /sync-docs correct under the canary→nightly→beta→stable channel model. Adds the release-mode tag-accuracy gate (scripts/sync/check-tag-accuracy.mjs) that verifies every changed page against the stable tag and flags post-tag leaks; stable-CalVer guard + channel glossary, tag-scoped drift/verify, and Phase 5.7 in the skill; realign fix that keeps nightly's drafts; a release-channels watcher; the analysis/plan doc; and a hermetic test (sync:test 38/38). Co-Authored-By: Claude Opus 4.8 --- maintenance/release-channels-plan.md | 202 ++++++++++++++++++ scripts/sync/anchor-map.json | 14 ++ scripts/sync/check-tag-accuracy.mjs | 298 +++++++++++++++++++++++++++ scripts/sync/realign-nightly.mjs | 50 +++-- scripts/sync/test.mjs | 91 ++++++++ skills/sync-docs/SKILL.md | 53 ++++- 6 files changed, 683 insertions(+), 25 deletions(-) create mode 100644 maintenance/release-channels-plan.md create mode 100755 scripts/sync/check-tag-accuracy.mjs diff --git a/maintenance/release-channels-plan.md b/maintenance/release-channels-plan.md new file mode 100644 index 0000000..21807b3 --- /dev/null +++ b/maintenance/release-channels-plan.md @@ -0,0 +1,202 @@ +# Proposal: keep `/sync-docs` working under Paperclip's release-channel model + +**Status:** implemented on `feat/sync-docs-release-channels` · **Date:** 2026-08-25 + +> **Implementation landed with this document.** All five changes below are in this +> branch: the tag-accuracy gate (`scripts/sync/check-tag-accuracy.mjs`, §5.2), the +> stable-CalVer guard + channel glossary and tag-scoped drift/verify (§5.1, §5.3) +> and the new Phase 5.7 in `skills/sync-docs/SKILL.md`, the realign fix +> (`scripts/sync/realign-nightly.mjs`, §5.4), and the `release-channels` watcher in +> `scripts/sync/anchor-map.json` (§5.5). The gate was dry-run against the +> v2026.824.0 leak fixture until it flagged exactly Kimi + the reaper env var with +> zero false positives on in-tag pages, and against the corrected release (0 leaks). +**Trigger:** the v2026.824.0 release run shipped (and we caught) docs for features +that were **not in the release** — Kimi adapter, `PAPERCLIP_WORKSPACE_REAPER_COOLDOWN_DAYS`, +and a mission-less onboarding rewrite. All three exist on `master` but landed +**after** the stable tag was cut. This is now a *structural* problem, because +Paperclip has formalized a four-lane release model where `master` runs far ahead +of `stable`. + +The goal is explicit: **keep the single-trigger `/sync-docs` skill working as-is** +(one command auto-detects nightly vs release and handles everything), updated so +it stays correct under the new schedule. + +--- + +## 1. How releases work now (the channel model) + +From the parent's `doc/CHANNELS.md` and the release announcement: + +``` +canary → nightly → beta → stable +``` + +| Channel | What it is | Cadence | npm dist-tag | Docker tag | +|---|---|---|---|---| +| `canary` | every merge to `master` | many/day | `canary` | `:canary` | +| `nightly` | newest green `master` build, full smoke suite, published if green | nightly | `nightly` | `:nightly` | +| `beta` | a nightly a maintainer promotes behind an approval gate; re-smoked | on promotion | `beta` | `:beta` | +| `stable` | manually cut release; must soak ≥3 days as beta first | ~weekly | `latest` | `:latest` | + +Behavior change worth noting: **Docker `:latest` now tracks stable only** (it used +to move on every master merge). npm `@latest` has always meant stable. Each lane +promotion republishes the *exact source commit* of the prior lane, so a beta shares +its SHA with a nightly and a nightly with a canary; the version string dates the +promotion and carries the lane (`2026.MDD.P-beta.N`). + +--- + +## 2. What the skill assumes today + +`skills/sync-docs/SKILL.md` runs two modes off one trigger: + +- **nightly mode** → tracks parent **`master` HEAD**, targets our `nightly` branch, + deploys a Cloudflare preview. Cumulative diff base = `base_release_tag`, next = `master` HEAD. +- **release mode** → fires when `gh api …/releases/latest` returns a newer tag than + `base_release_tag`; targets `main` (→ docs.paperclip.ing) via a PR cut from `nightly`. + +Two assumptions are now broken: + +1. **"`nightly` ≈ what will be released."** Our `nightly` branch tracks `master` + HEAD (effectively the **canary** lane). Under the channel model, `master` sits + *many* commits ahead of `stable` — for v2026.824.0 the stable tag was **173 + commits behind** `master`, and even the merge-base was behind. So when release + mode cuts the release branch **from `nightly`**, it inherits every post-tag + `master` draft. Those become **leaks** against the stable release. + +2. **"drift/verification against `master` is enough."** `check-drift` runs + `--against ` (`master`). In release mode that is the *wrong* + reference: a page can be perfectly consistent with `master` and still document + something absent from the **stable tag**. That is exactly why drift returned + `0` on the run that leaked Kimi. + +--- + +## 3. What actually broke on v2026.824.0 (evidence) + +| Leak | Why it leaked | How we caught it | +|---|---|---| +| Kimi adapter page + nav + overview row | `packages/adapters/kimi-local` is **404 at the tag** (only on `master`, commit `dc5b070` post-tag) | manual per-feature check against the tag | +| `PAPERCLIP_WORKSPACE_REAPER_COOLDOWN_DAYS` | absent from `.env.example`/config at the tag | manual check | +| Onboarding "mission step is gone" rewrite | the tag's `OnboardingWizard.tsx` **still has** the mission step; the drop is a post-tag change | manual check | + +All three were drafted by nightly runs whose cited parent SHA was **ahead of the +tag**. They were verified out by an adversarial pass against the tag — a pass the +skill does not currently perform. Separately, the completeness pass found the +opposite failure: three *in-tag* surfaces (Claude setup-token login, adapter +device-login, issue file-resource availability) that nightly had **not** drafted +and which we had to author during the release. + +**Net:** on any given release, the release branch can be simultaneously *ahead* of +the tag (post-tag `master` leaks) and *behind* it (in-tag surfaces nightly skipped). + +--- + +## 4. Recon: can we just anchor to `beta`? + +The tempting fix is "point the preview branch at `beta` (the release candidate) +instead of `master`, so it already matches the imminent stable." Recon says **no, +not cleanly**: + +- **GitHub Releases and git tags exist for `stable` only.** All releases are + `prerelease=false` clean CalVer (`vYYYY.MDD.P`). The only prerelease git tags in + the repo are legacy pre-CalVer package tags (`paperclipai@0.3.x-canary.N`). +- **No git ref for beta/nightly/canary.** `git/ref/tags/{beta,nightly,canary}` → + 404. There is no branch or moving tag to diff against. +- **npm exposes no `gitHead`.** `npm view paperclipai@beta` returns the version + string but not the commit; the lane SHA lives in CI job summaries and Docker + `:sha-` tags — not something to diff a docs window against reliably. +- **Beta can lag stable.** Today: `beta = 2026.818.0-beta.1` while `stable = + 2026.824.0`. Stable was cut from an earlier beta and the beta lane hasn't + advanced past it, so "beta = what stable becomes" does **not** hold moment to + moment. Anchoring the preview to beta could show content *older* than live. + +Conclusion: we cannot make `beta` the diff anchor with the parent's current +infrastructure. Keep tracking `master` for the preview and **make release mode +tag-accurate** instead. + +--- + +## 5. Recommended plan (keeps the single trigger) + +Five focused changes to the skill and its helpers. Nothing changes about the +one-command UX or the nightly/release branch model. + +### 5.1 Pin the release target to the `stable` lane, defensively +- Keep `releases/latest` for mode detection (it already returns stable), but add + an explicit guard: the release target **must** match `^v?\d{4}\.\d{1,4}\.\d+$` + (clean CalVer). Any `-beta/-nightly/-canary` tag is **never** a release target. +- Add a one-paragraph glossary to the skill: "our `nightly` branch tracks parent + `master` HEAD (≈ the **canary** lane); the parent's **`nightly` channel** is a + different thing (a smoke-tested `master` build). Docs ship on **stable**." This + kills the naming collision without a disruptive branch rename. + +### 5.2 Add a release-mode **tag-accuracy gate** (the core fix — automates what we did by hand) +New phase, run in release mode after the manifest is built and before the PR is finalized: +1. Enumerate every doc page changed on `nightly` vs `main` (the content that would ship). +2. For each, extract its concrete surface claims (reuse `verify-edit.mjs`) and + **verify against the stable tag**, not `master`. +3. Any page whose underlying surface is **absent at the tag** = a post-tag leak. + **Auto-quarantine** it: revert that page (or the added section) to its `main` + state and list it in the PR body under **"⚠ Post-tag leaks removed."** +4. Symmetrically, surface in-tag surfaces that changed in the window but have **no** + doc touch (the "nightly skipped it" case) as **"⚠ Undocumented in-tag surfaces."** + +This makes the leak/gap detection we ran manually a standing, automatic part of +every release. + +### 5.3 Point drift + verification at the **tag** in release mode +- `check-drift.mjs` and the Phase 5.5 `verify-edit.mjs` calls should default + `--against` to the **resolved stable tag** in release mode (they use `master` + today). Nightly mode keeps using `master`. One-line change at each call site. + +### 5.4 Fix the realign so it doesn't strip valid `master` drafts from `nightly` +- Today `realign-nightly.mjs` fast-forwards `nightly` **onto the leak-stripped + release branch**, which *removes* the (legitimate) post-tag `master` drafts from + `nightly` (we saw Kimi/reaper/onboarding vanish from `nightly` after the + v2026.824.0 realign). It self-heals on the next nightly run, but it's wasteful + and surprising. +- Change realign to **re-anchor** `nightly` to `main`'s squash commit (merge + `origin/main` to restore ancestry) **without** resetting `nightly`'s content to + the release branch — i.e. preserve nightly's master-tracking drafts, only update + the base anchor. Verify `main` is an ancestor of `nightly` as today. + +### 5.5 Watch the channel surfaces so they never go stale +- Add an anchor-map watcher for `doc/CHANNELS.md` + `cli/src/commands/channels.ts` + → `docs/how-to/update-paperclip.md`, `docs/reference/cli/installation.md`, so + channel/tag changes (e.g. the `:latest` semantics change) always flow into the + docs. + +--- + +## 6. What deliberately stays the same +- **One trigger.** `/sync-docs` still auto-detects nightly vs release and does the + whole run. No new required flags. +- **Branch model.** `nightly` (preview) and `main` (live) unchanged; release still + PRs `nightly → main` and squash-merges. +- **Cumulative diffs, quarantine, reconciliation** — all retained. The tag-accuracy + gate is *additive*. +- **Preview stays bleeding-edge.** The `nightly` preview keeps showing `master` + content; that is correct for a contributor preview. Correctness is enforced at + **release** time, where it ships to end users. + +--- + +## 7. Rollout +1. Land this proposal (docs only, no behavior change). +2. Implement 5.1 + 5.3 (small, low-risk anchor/guard changes). +3. Implement 5.2 (the gate) — the highest-value change; dry-run it against the + v2026.824.0 window as a regression test (it must flag Kimi, reaper, onboarding). +4. Implement 5.4 (realign) and 5.5 (watcher). +5. Update `maintenance/runbook.md` and the skill's "Special cases" with the + tag-divergence scenario and the manual fallback. + +## 8. Open questions +- Should the preview branch eventually be **renamed** `canary` (accurate) or + `preview` (neutral) to end the collision with the parent `nightly` channel? Out + of scope here (Cloudflare + state migration); flagged for later. +- Do we want a lightweight **release-smoke parity** note in docs (that nightly/beta + are smoke-gated) — content, not skill, but worth a follow-up. +- If the parent ever publishes **git tags for beta** (`2026.MDD.P-beta.N`), revisit + §4: anchoring the preview to beta would then become feasible and would shrink the + leak surface at the source. diff --git a/scripts/sync/anchor-map.json b/scripts/sync/anchor-map.json index d8fc503..ab87b3d 100644 --- a/scripts/sync/anchor-map.json +++ b/scripts/sync/anchor-map.json @@ -149,6 +149,20 @@ "tier": "context-only", "detect": "Migrations themselves rarely need user-facing docs, but a sudden burst of migrations often correlates with a schema change watched elsewhere. Use this watcher's hits as context only — never produce edits directly. Deliberately NOT promoted above context-only: most migrations are columns, indexes, and renames that need no prose, and promoting this would put all of them in front of a human for triage. The gap that would justify revisiting is a migration changing user-visible behaviour WITHOUT touching a packages/**/schema/*.ts file the `schemas` watcher already catches — a data backfill that changes a default, a retention window, or a constraint that starts rejecting input that used to be accepted." }, + { + "name": "release-channels", + "parent_paths": [ + "doc/CHANNELS.md", + "cli/src/commands/channels.ts" + ], + "docs_paths": [ + "docs/how-to/update-paperclip.md", + "docs/reference/cli/installation.md" + ], + "tier": "pr", + "detect": "The release-channel model (canary → nightly → beta → stable), its npm dist-tags and Docker image tags, and the `paperclipai channels` command. `doc/CHANNELS.md` is the authoritative parent guide; `channels.ts` holds the RELEASE_CHANNELS descriptors and the version→channel regexes. Changes here (a new lane, a dist-tag rename, the `:latest`-tracks-stable semantics) must flow into the channels sections of update-paperclip.md and the CLI installation page. Note: channels.ts is also matched by the cli-commands watcher — dedupe by surface, don't emit twice.", + "channel_context": "Docs ship on the STABLE lane only. Do not describe beta/nightly/canary versions as the release; they are npm/Docker artifacts with no git tag. See maintenance/release-channels-plan.md." + }, { "name": "release-notes", "source": "github-releases", diff --git a/scripts/sync/check-tag-accuracy.mjs b/scripts/sync/check-tag-accuracy.mjs new file mode 100755 index 0000000..5123e83 --- /dev/null +++ b/scripts/sync/check-tag-accuracy.mjs @@ -0,0 +1,298 @@ +#!/usr/bin/env node +// check-tag-accuracy.mjs — release-mode gate that catches "post-tag leaks": +// doc pages that changed on the preview branch (head) versus the released +// baseline (base) but document a surface that is NOT present at the stable +// release tag. Under Paperclip's channel model our `nightly` preview tracks +// parent `master` (≈ canary), which runs far ahead of the `stable` tag, so a +// release branch cut from `nightly` inherits post-tag master drafts as leaks +// (see maintenance/release-channels-plan.md). +// +// It reuses verify-edit.mjs as the claim engine: for each changed doc page it +// verifies the page's concrete claims (file paths, env vars, CLI commands/flags, +// adapter config fields, REST routes) against the stable TAG rather than master. +// +// Classification per page: +// leak — has >=1 HIGH-confidence claim absent at the tag (file-path, +// env-var, cli-command, cli-flag). These are safe to auto-quarantine +// (revert the page/section to `base`) — a mechanical identifier the +// released code simply does not contain. +// review — a new page, or medium-confidence-only misses (rest-route, +// adapter-config-field), that a human must confirm against the tag. +// REST routes are medium because constant/prefix registration hides +// real matches (a known verify-edit false-negative). +// clean — every claim verifies at the tag. +// +// The gate is advisory: it always exits 0. It never edits docs; the skill's +// release phase decides what to quarantine from the `leaks` list and what to +// surface under review. +// +// Usage: +// node scripts/sync/check-tag-accuracy.mjs --tag \ +// [--base main] [--head HEAD] [--repo paperclipai/paperclip] [--json] + +import { execFileSync, spawnSync } from "node:child_process"; +import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, openSync, closeSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, dirname } from "node:path"; +import { fileURLToPath } from "node:url"; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const VERIFY_EDIT = join(HERE, "verify-edit.mjs"); + +// Claim kinds whose absence at the tag is a mechanical, high-confidence leak. +// REST routes and adapter-config fields are deliberately NOT here: route +// prefixing / constant registration and missing adapter doc-sources produce +// false negatives, so they route to `review` instead of auto-quarantine. +const HIGH_CONF_KINDS = new Set([ + "file-path", + "env-var", + "cli-command", + "cli-flag", +]); + +// Flags Commander registers automatically — never explicitly `.option()`-ed, so +// verify-edit reports them unverified. They are not leaks. +const BUILTIN_FLAGS = new Set(["--help", "-h", "--version", "-V"]); + +// A verify-edit claim is only a *product-surface* signal if it points at parent +// source. Markdown link targets (a page's own relative `.md` links) and +// Commander built-in flags are noise, not leaks. +function isSurfaceClaim(u) { + if (u.kind === "file-path" && /\.md$/i.test(u.value)) return false; + if (u.kind === "cli-flag" && BUILTIN_FLAGS.has(u.value)) return false; + return true; +} + +function parseArgs(argv) { + const out = { base: "main", head: "HEAD", repo: "paperclipai/paperclip", json: false }; + for (let i = 0; i < argv.length; i++) { + const a = argv[i]; + if (a === "--tag") out.tag = argv[++i]; + else if (a === "--base") out.base = argv[++i]; + else if (a === "--head") out.head = argv[++i]; + else if (a === "--repo") out.repo = argv[++i]; + else if (a === "--json") out.json = true; + else if (a === "-h" || a === "--help") out.help = true; + } + return out; +} + +function git(args) { + return execFileSync("git", args, { encoding: "utf8", maxBuffer: 64 * 1024 * 1024 }); +} + +// Changed doc pages at head vs base. A/M/R only — a deleted page cannot leak. +function changedDocs(base, head) { + const out = git([ + "diff", + "--name-status", + "--diff-filter=AMR", + `${base}`, + `${head}`, + "--", + "docs/**/*.md", + ]); + const files = []; + for (const line of out.split("\n")) { + if (!line.trim()) continue; + const parts = line.split("\t"); + const status = parts[0][0]; // A | M | R + const file = parts[parts.length - 1]; // R lines carry old\tnew; take new + if (file.endsWith(".md")) files.push({ file, status }); + } + return files; +} + +// Line numbers (in the HEAD version of `file`) that this release added or +// changed vs base. Used to scope claims on a MODIFIED page to what the release +// actually introduced — a pre-existing entry that happens not to verify at the +// tag is drift, not a release leak. +function addedLines(base, head, file) { + const set = new Set(); + let diff; + try { + diff = git(["diff", "--unified=0", `${base}`, `${head}`, "--", file]); + } catch { + return set; + } + let newLine = 0; + for (const line of diff.split("\n")) { + const h = line.match(/^@@ -\d+(?:,\d+)? \+(\d+)(?:,\d+)? @@/); + if (h) { + newLine = parseInt(h[1], 10); + continue; + } + if (line.startsWith("+++")) continue; + if (line.startsWith("+")) { + set.add(newLine); + newLine++; + } else if (!line.startsWith("-")) { + // context line (shouldn't appear with --unified=0, but be safe) + newLine++; + } + } + return set; +} + +function lineOf(location) { + const m = String(location).match(/:(\d+)$/); + return m ? parseInt(m[1], 10) : null; +} + +function showAtRef(ref, file) { + try { + return git(["show", `${ref}:${file}`]); + } catch { + return null; + } +} + +function runVerifyEdit(tmpDocPath, tag, repo, outFile) { + // Redirect the child's stdout to a real file rather than a pipe. verify-edit + // writes a large JSON blob and then exits, and a pipe would truncate that at + // the ~64KB OS buffer before it drains (the big adapter/env pages overflow it). + // A regular-file fd is a blocking write, so the full output lands intact. + const fd = openSync(outFile, "w"); + let r; + try { + r = spawnSync( + "node", + [VERIFY_EDIT, tmpDocPath, "--against", tag, "--repo", repo, "--json"], + { stdio: ["ignore", fd, "pipe"], encoding: "utf8" }, + ); + } finally { + closeSync(fd); + } + let stdout = ""; + try { + stdout = readFileSync(outFile, "utf8"); + } catch { + /* ignore */ + } + try { + return JSON.parse(stdout); + } catch { + const msg = (r?.stderr || r?.error?.message || "verify-edit produced no parseable JSON").split("\n")[0]; + return { error: msg, claims_extracted: 0, verified: 0, unverified: [], suspicious: [] }; + } +} + +function main() { + const args = parseArgs(process.argv.slice(2)); + if (args.help || !args.tag) { + process.stderr.write( + "usage: check-tag-accuracy.mjs --tag [--base main] [--head HEAD] [--repo owner/repo] [--json]\n", + ); + process.exit(args.tag ? 0 : 2); + } + + const changed = changedDocs(args.base, args.head); + const tmpRoot = mkdtempSync(join(tmpdir(), "tagacc-")); + + const leaks = []; + const review = []; + const clean = []; + const errors = []; + + try { + for (const { file, status } of changed) { + const content = showAtRef(args.head, file); + if (content === null) continue; // vanished at head — skip + // Materialise under a temp root preserving the docs/... suffix so + // verify-edit's path-based doc-kind sniffing still works. + const tmpPath = join(tmpRoot, file); + mkdirSync(dirname(tmpPath), { recursive: true }); + writeFileSync(tmpPath, content); + + const res = runVerifyEdit(tmpPath, args.tag, args.repo, join(tmpRoot, "verify-out.json")); + if (res.error) { + errors.push({ file, error: res.error }); + continue; + } + let unverified = (res.unverified || []).filter(isSurfaceClaim); + // For a modified page, judge only the claims this release added/changed — + // a pre-existing miss is drift, not a release leak. A new page (A) is all + // new content, so keep every claim. + if (status !== "A") { + const added = addedLines(args.base, args.head, file); + unverified = unverified.filter((u) => { + const ln = lineOf(u.location); + return ln !== null && added.has(ln); + }); + } + const high = unverified.filter((u) => HIGH_CONF_KINDS.has(u.kind)); + const med = unverified.filter((u) => !HIGH_CONF_KINDS.has(u.kind)); + + const claimsOf = (arr) => + arr.map((u) => ({ kind: u.kind, value: u.value, evidence: u.evidence })); + + if (high.length > 0) { + leaks.push({ file, status, claims: claimsOf(high) }); + } else if (status === "A" && (res.verified === 0 || med.length > 0)) { + review.push({ + file, + status, + reason: "new page whose surface could not be confirmed at the tag", + claims: claimsOf(med), + }); + } else if (med.length > 0) { + review.push({ + file, + status, + reason: "medium-confidence claims unverified at the tag (confirm not a leak)", + claims: claimsOf(med), + }); + } else { + clean.push(file); + } + } + } finally { + rmSync(tmpRoot, { recursive: true, force: true }); + } + + const result = { + tag: args.tag, + base: args.base, + head: args.head, + repo: args.repo, + files_checked: changed.length, + leaks, + review, + clean_count: clean.length, + errors, + }; + + if (args.json) { + process.stdout.write(JSON.stringify(result, null, 2) + "\n"); + return; + } + + const L = []; + L.push(`Tag-accuracy check — head '${args.head}' vs base '${args.base}' against tag ${args.tag}`); + L.push(`Doc pages changed: ${result.files_checked} | leaks: ${leaks.length} | review: ${review.length} | clean: ${clean.length}`); + if (leaks.length) { + L.push(""); + L.push("## ⚠ Post-tag leaks (auto-quarantine candidates — surface absent at the tag)"); + for (const l of leaks) { + L.push(`- ${l.file}`); + for (const c of l.claims) L.push(` [${c.kind}] ${c.value} — ${c.evidence}`); + } + } + if (review.length) { + L.push(""); + L.push("## ⚠ Review against the tag (medium-confidence / new page)"); + for (const r of review) { + L.push(`- ${r.file} — ${r.reason}`); + for (const c of r.claims) L.push(` [${c.kind}] ${c.value}`); + } + } + if (errors.length) { + L.push(""); + L.push("## Errors"); + for (const e of errors) L.push(`- ${e.file}: ${e.error}`); + } + if (!leaks.length && !review.length) L.push("OK — no post-tag leaks; every changed page verifies at the tag."); + process.stdout.write(L.join("\n") + "\n"); +} + +main(); diff --git a/scripts/sync/realign-nightly.mjs b/scripts/sync/realign-nightly.mjs index 13497db..0093081 100644 --- a/scripts/sync/realign-nightly.mjs +++ b/scripts/sync/realign-nightly.mjs @@ -7,9 +7,11 @@ // "merge main into nightly" sees pages both branches created as add/add // conflicts. This script heals that, and must run after EVERY release merge: // -// 1. fast-forward `nightly` onto the release branch (content == shipped) -// 2. merge `origin/main` into `nightly` (re-anchors the merge-base at the -// squash commit, so the next cycle's divergence starts fresh) +// merge `origin/main` into `nightly` with `-X ours` — this re-anchors the +// merge-base at the squash commit (so the next cycle's divergence starts +// fresh) while KEEPING nightly's content, including the post-tag `master` +// drafts that Phase 5.7 quarantined from the release branch. It does NOT +// fast-forward nightly onto the release tree (that would delete those drafts). // // Usage: node scripts/sync/realign-nightly.mjs [--push] // e.g. node scripts/sync/realign-nightly.mjs release/v2026.720.0 @@ -59,22 +61,36 @@ const startBranch = git("branch", "--show-current"); git("checkout", "nightly"); git("merge", "--ff-only", "origin/nightly"); -// Step 1: bring nightly's content to exactly what shipped. -const ff = tryGit("merge", "--ff-only", releaseBranch); -if (!ff.ok) { - // Something landed on nightly after the release branch forked. Same content - // guarantees a clean three-way merge for the release files themselves. - console.log("fast-forward not possible (nightly moved since the release forked) — doing a normal merge"); - const m = tryGit("merge", releaseBranch, "-m", `Merge ${releaseBranch} into nightly (post-release realign)`); - if (!m.ok) die(`merge of ${releaseBranch} into nightly conflicted:\n${m.out}\nResolve by hand, then run the main-merge step yourself.`); +// Re-anchor ancestry onto main's squash commit WITHOUT discarding nightly's +// post-tag drafts. +// +// We deliberately do NOT fast-forward nightly onto the release branch. Phase 5.7 +// quarantines post-tag leaks (Kimi, reaper, an onboarding rewrite, ...) from the +// *release* branch only — those drafts are still legitimate on `nightly` because +// they document `master` features that ship in a *future* stable release. +// Fast-forwarding nightly onto the release tree would delete them. +// +// Instead do one merge of origin/main into nightly with `-X ours`: it makes the +// squash commit a parent of nightly (restoring ancestry so the next nightly-mode +// main-merge no longer explodes into add/add conflicts), keeps nightly's own +// version of every page both branches created, still pulls in main-only hot-fixes +// on pages nightly did not touch, and preserves nightly-only drafts (Kimi et al.) +// untouched. +const anchor = tryGit( + "merge", + "-X", + "ours", + "origin/main", + "-m", + "Merge main into nightly (re-anchor after squash-merged release; keep nightly drafts)", +); +if (!anchor.ok) { + if (/Already up to date/i.test(anchor.out)) { + die("origin/main is already an ancestor of nightly — nothing to realign (was this already run?)"); + } + die(`merge of origin/main into nightly failed:\n${anchor.out}\nResolve by hand, keeping nightly's drafts, then re-run the post-check.`); } -// Step 2: merge main so the squash commit becomes an ancestor of nightly. -// This is what actually re-anchors merge-base(main, nightly) — do not skip it -// even though the trees already match. -const anchor = tryGit("merge", "origin/main", "-m", "Merge main into nightly (re-anchor after squash-merged release)"); -if (!anchor.ok) die(`merge of origin/main conflicted (unexpected — trees matched):\n${anchor.out}`); - const base = git("merge-base", "origin/main", "nightly"); const mainTip = git("rev-parse", "origin/main"); if (base !== mainTip) die(`post-check failed: merge-base(main, nightly) is ${base.slice(0, 9)}, expected main's tip ${mainTip.slice(0, 9)}`); diff --git a/scripts/sync/test.mjs b/scripts/sync/test.mjs index 4dc3f34..7e0e933 100644 --- a/scripts/sync/test.mjs +++ b/scripts/sync/test.mjs @@ -1257,6 +1257,97 @@ test("detect-renames: two candidates → file-overlap wins over shared-prefix", `from=${renForAdded.from}`); }); +// ---------------------------------------------------------------------------- +// check-tag-accuracy (hermetic end-to-end: real git fixture repo + stubbed +// parent via PAPERCLIP_SYNC_FIXTURE_DIR — no network) +// ---------------------------------------------------------------------------- + +const TAGACC_SCRIPT = join(SELF_DIR, "check-tag-accuracy.mjs"); + +// A file-path claim in a doc page: `/...ts` (verified iff the parent +// returns 200 at the tag). Using only file-path claims keeps verify-edit from +// reaching for the tree/env/route/adapter fetchers, so the whole run is stubbed +// by a handful of contents-*.json files. +function makeTagAccFixture() { + const dir = makeFixture(); + copyFileSync(TAGACC_SCRIPT, join(dir, "scripts/sync/check-tag-accuracy.mjs")); + copyFileSync(VERIFY_SCRIPT, join(dir, "scripts/sync/verify-edit.mjs")); + copyFileSync(join(SELF_DIR, "anchor-map.json"), join(dir, "scripts/sync/anchor-map.json")); + + const g = (...args) => { + const r = spawnSync("git", args, { cwd: dir, encoding: "utf8" }); + if (r.status !== 0) throw new Error(`git ${args.join(" ")} failed: ${r.stderr || r.stdout}`); + return r.stdout; + }; + g("init", "-q"); + g("config", "user.email", "t@example.com"); + g("config", "user.name", "t"); + g("config", "commit.gpgsign", "false"); + + // BASE: `existing.md` already links a pre-existing file path. + writeFile(dir, "docs/how-to/existing.md", "# Existing\n\nSee `packages/stale/pre.ts` for details.\n"); + g("add", "-A"); + g("commit", "-q", "-m", "base"); + + // HEAD: append a NEW file-path link to existing.md, add a leak page and a + // clean page. + writeFile( + dir, + "docs/how-to/existing.md", + "# Existing\n\nSee `packages/stale/pre.ts` for details.\n\nAlso see `packages/ghost/added.ts` now.\n", + ); + writeFile(dir, "docs/how-to/ghost-leak.md", "# Ghost\n\nRuns from `packages/ghost/src/index.ts`.\n"); + // real page: a .ts that DOES exist at the tag (200) plus a .md link that does + // not (404). The .md must be ignored (it's a doc link, not a product surface). + writeFile( + dir, + "docs/how-to/real-clean.md", + "# Real\n\nRuns from `packages/adapters/real/src/index.ts`. Config is `packages/adapters/real/config.md`.\n", + ); + g("add", "-A"); + g("commit", "-q", "-m", "head"); + + // Stubbed parent: only the real .ts resolves at the tag. Everything else 404s + // by omission. + const fixDir = join(dir, "_fixtures"); + mkdirSync(fixDir, { recursive: true }); + writeFileSync(join(fixDir, "repo.json"), JSON.stringify({ default_branch: "master" })); + writeFileSync( + join(fixDir, "contents-packages__adapters__real__src__index.ts-faketag.json"), + JSON.stringify({ content: "export {}\n" }), + ); + return { dir, fixDir }; +} + +test("check-tag-accuracy: flags a post-tag leak, keeps in-tag page clean, scopes to added lines", () => { + const { dir, fixDir } = makeTagAccFixture(); + const r = spawnSync( + process.execPath, + [join(dir, "scripts/sync/check-tag-accuracy.mjs"), "--tag", "faketag", "--base", "HEAD~1", "--head", "HEAD", "--json"], + { cwd: dir, encoding: "utf8", env: { ...process.env, PAPERCLIP_SYNC_FIXTURE_DIR: fixDir } }, + ); + assert(r.status === 0, `gate should always exit 0, got ${r.status}. stderr=${r.stderr}`); + const out = JSON.parse(r.stdout); + const leakFiles = out.leaks.map((l) => l.file); + + // The new page whose only surface is a tag-absent .ts is a leak. + assert(leakFiles.includes("docs/how-to/ghost-leak.md"), `ghost-leak.md should be a leak; leaks=${JSON.stringify(leakFiles)}`); + // The modified page is a leak because of its ADDED tag-absent .ts. + assert(leakFiles.includes("docs/how-to/existing.md"), `existing.md should be a leak; leaks=${JSON.stringify(leakFiles)}`); + // The page whose .ts exists at the tag (and whose only miss is a .md doc link) is NOT a leak. + assert(!leakFiles.includes("docs/how-to/real-clean.md"), `real-clean.md must not be a leak; leaks=${JSON.stringify(leakFiles)}`); + assert(!out.review.some((x) => x.file === "docs/how-to/real-clean.md"), "real-clean.md must not be in review (its .ts verifies; the .md link is ignored)"); + + // Added-line scoping: the pre-existing `pre.ts` (also 404) must NOT be flagged + // — only the added `added.ts`. + const ex = out.leaks.find((l) => l.file === "docs/how-to/existing.md"); + const vals = ex.claims.map((c) => c.value); + assert(vals.includes("packages/ghost/added.ts"), `existing.md leak should cite the added path; got ${JSON.stringify(vals)}`); + assert(!vals.includes("packages/stale/pre.ts"), `pre-existing path must be scoped out of the leak; got ${JSON.stringify(vals)}`); + + rmSync(dir, { recursive: true, force: true }); +}); + // ---------------------------------------------------------------------------- console.log(""); diff --git a/skills/sync-docs/SKILL.md b/skills/sync-docs/SKILL.md index 9f5eee8..ca78cca 100644 --- a/skills/sync-docs/SKILL.md +++ b/skills/sync-docs/SKILL.md @@ -39,7 +39,8 @@ The branch model is non-negotiable: end users on the latest *released* paperclip - `scripts/sync/check-drift.mjs` — Phase 1.5 drift detection (finds documented surfaces missing from parent). - `scripts/sync/detect-renames.mjs` — Phase 3 directory rename detection (distinguishes a renamed surface from a brand-new one). - `scripts/sync/verify-edit.mjs` — Phase 5.5 post-edit verification (checks that authored claims still match parent code). -- `scripts/sync/realign-nightly.mjs` — post-release realign (Phase 7): fast-forwards `nightly` onto the merged release branch and re-anchors the merge-base after the squash-merge severs ancestry. +- `scripts/sync/check-tag-accuracy.mjs` — **release mode only** (Phase 5.7): the tag-accuracy gate. Verifies every doc page changed on `nightly` vs `main` against the **stable release tag** and flags post-tag leaks — content nightly drafted from `master` that is not in the release. See `maintenance/release-channels-plan.md` for why this exists. +- `scripts/sync/realign-nightly.mjs` — post-release realign (Phase 7): re-anchors `nightly` onto the merged release squash commit (restoring ancestry) **without discarding** nightly's post-tag `master` drafts. ## Invocation @@ -65,10 +66,21 @@ The branch model is non-negotiable: end users on the latest *released* paperclip ### Phase 1 — Decide mode and target branch +> **Release channels — read this first.** Paperclip publishes on four lanes: +> `canary` (every merge to ``) → `nightly` (a green `master` build, +> smoke-tested nightly) → `beta` (a promoted nightly, soaks ≥3 days) → `stable` +> (the manually cut release). **Only `stable` has git tags and GitHub Releases** +> (clean CalVer `vYYYY.MDD.P`); beta/nightly/canary are npm/Docker artifacts with +> no git ref we can diff against. **Docs ship on `stable`.** Note the naming +> collision: *our* `nightly` branch tracks parent `` HEAD (≈ the +> **canary** lane), which runs far ahead of the stable tag — so a release branch +> cut from `nightly` inherits post-tag `master` drafts. Phase 5.7 is the gate that +> catches those. See `maintenance/release-channels-plan.md`. + 1. Read `.sync-state.json`. Note `branch_mode` and `base_release_tag`. -2. Fetch parent's latest release: `gh api repos/paperclipai/paperclip/releases/latest -q '.tag_name'`. +2. Fetch parent's latest **stable** release: `gh api repos/paperclipai/paperclip/releases/latest -q '.tag_name'`. `releases/latest` already excludes prereleases, but **guard defensively**: the release target MUST match `^v?\d{4}\.\d{1,4}\.\d+$` (clean CalVer). Any `-beta`/`-nightly`/`-canary` tag is never a release target — if the API ever returns one, treat the run as nightly mode and note it in the summary. 3. Auto-detect mode (unless overridden): - - If a new release tag exists AND `base_release_tag` is older → **release mode**. + - If a new stable release tag exists AND `base_release_tag` is older → **release mode**. - Otherwise → **nightly mode**. 4. Check out the right branch: - Release mode: ensure on `main`. Release mode is **self-sufficient** — it does not require the `nightly` branch to exist or to have drafts. If `nightly` exists with relevant drafts, they're used as a starting point; if not, the release run computes everything from scratch. @@ -110,10 +122,13 @@ Cache result under `/tmp/paperclip-sync//` so we don't refetch within a run Drift is the inverse of the cumulative diff: it's the set of things **we already document** that have since vanished or moved upstream. It exists regardless of when the last sync happened — a parent surface can disappear between two sync runs even if our diff window is empty. The wet-run that motivated this phase found `POST /api/companies/{companyId}/logo` documented but absent from current `server/src/routes/companies.ts`, with no sign of it in any diff window the sync had ever processed. -Run the drift checker against parent HEAD: +Run the drift checker against the reference for this mode. **Nightly mode → ``** (drift is about the live upstream). **Release mode → the stable release tag** — checking release-mode drift against `master` is blind to the tag divergence and will pass surfaces that exist on `master` but not in the release (that is how the Kimi leak slipped past a `0`-drift run). Use `` in release mode: ``` +# nightly mode node scripts/sync/check-drift.mjs --json --against +# release mode +node scripts/sync/check-drift.mjs --json --against ``` The script scans `docs/**` for four reference classes and verifies each one still exists in parent: @@ -246,10 +261,11 @@ If all pass: make the mechanical edit directly. Examples: append a row to `envir ### Phase 5.5 — Verify edits against parent code -For every file touched in Phase 5 (auto-merge or PR-tier): +For every file touched in Phase 5 (auto-merge or PR-tier). Verify against the reference for this mode — **nightly mode → ``**, **release mode → the stable ``** (verifying release edits against `master` would confirm claims that are true on `master` but absent from the release): ``` -node scripts/sync/verify-edit.mjs --against --json +node scripts/sync/verify-edit.mjs --against # nightly mode +node scripts/sync/verify-edit.mjs --against # release mode ``` Collect all `unverified` and `suspicious` records into a **Verification Report** for the run. @@ -261,6 +277,24 @@ Routing rules: - **PR tier edits (batched-release mode).** Never auto-merge if any unverified records exist. They flow into the release PR with the failed claims listed under `⚠ Verification Failures`. - **Suspicious records.** Logged and surfaced in the run summary / PR body, but never block. They're informational. +### Phase 5.7 — Tag-accuracy gate (release mode only) + +Phase 5.5 verifies the edits *this run* authored. This phase verifies **everything the release would ship** — including drafts nightly authored in earlier runs — against the stable tag. It exists because our `nightly` branch tracks parent `master` (≈ canary), which runs far ahead of the stable tag, so a release branch cut from `nightly` inherits post-tag `master` drafts as leaks (this is exactly how Kimi, `PAPERCLIP_WORKSPACE_REAPER_COOLDOWN_DAYS`, and a mission-less onboarding rewrite reached a release branch). Skip entirely in nightly mode. + +Run the gate over the pages the release changes vs the live baseline: + +``` +node scripts/sync/check-tag-accuracy.mjs --tag --base main --head --json +``` + +It reuses `verify-edit.mjs` per changed page and classifies each: + +- **`leaks`** — a page has ≥1 **high-confidence** claim (`file-path`, `env-var`, `cli-command`, `cli-flag`) that the release-tag code does not contain. These are mechanical identifiers the shipped code simply lacks. **Auto-quarantine** each: revert the page (or just the offending section) to its `main` state, and list it in the PR body under **`### ⚠ Post-tag leaks removed`**. Only added/changed lines are judged, so a pre-existing stale entry is left to the drift phase, not quarantined here. +- **`review`** — new pages, or medium-confidence-only misses (`rest-route`, `adapter-config-field`). REST routes are medium because constant/prefix registration hides real matches (a known `verify-edit` false negative — do **not** auto-quarantine them). Surface under the PR's `### ⚠ Verification Failures` / review notes for a human to confirm against the tag. +- **`clean`** — every added claim verifies at the tag. + +**Known limit — behavioural/prose leaks.** The gate only catches leaks that carry a checkable *identifier*. A pure prose/UI rewrite with no code identifier (the onboarding "mission step is gone" case) will read as `clean`. So release mode **also** keeps the adversarial nightly-draft check: spawn a small verification pass (per changed guide page, compare its user-visible claims against the tag's UI/flow) and treat a page describing behaviour absent at the tag as a leak — revert it to `main` and list it under `### ⚠ Post-tag leaks removed`. + ### Phase 6 — Screenshot staleness check Read `docs/user-guides/screenshots/registry.json`. For each entry: @@ -354,12 +388,15 @@ rather than shipping. node scripts/sync/realign-nightly.mjs release/v2026.X.Y --push ``` - It fast-forwards `nightly` onto the release branch, merges `origin/main` back in to re-anchor the merge-base at the squash commit, verifies main's tip is an ancestor of nightly, and pushes. Skipping this arms an add/add conflict trap that detonates at the next nightly run (see the special case below). Do not run any nightly-mode sync between the squash-merge and the realign. + It re-anchors `nightly` onto the release squash commit (restoring ancestry) and pushes. Skipping this arms an add/add conflict trap that detonates at the next nightly run (see the special case below). Do not run any nightly-mode sync between the squash-merge and the realign. + + **Do not let the realign discard nightly's post-tag drafts.** Because Phase 5.7 quarantines post-tag leaks *from the release branch only*, those drafts (Kimi, etc.) are still legitimate on `nightly` — they document `master` features that ship in a *future* stable release. The realign must re-anchor `nightly` to `main`'s squash commit **without** resetting nightly's content to the release branch. `realign-nightly.mjs` does this by merging `origin/main` into `nightly` (favouring nightly's content on conflict) rather than fast-forwarding nightly onto the release branch. If the drafts do get dropped, the next nightly run regenerates them from the cumulative window — wasteful but self-healing. PR body structured sections (each omitted if empty — never silently dropped): + - `### ⚠ Post-tag leaks removed` — every page Phase 5.7 quarantined, with the offending claim(s) and the note that the surface is absent at the release tag (present on `master`, shipping in a later release). Confirms the release documents only what it contains. - `### ⚠ Drift` — every drift candidate from Phase 1.5 grouped by `kind`, high-confidence first, medium-confidence prefixed with `Verify:`. Never auto-resolved — the PR explicitly asks the reviewer to act on each entry (update path, delete section, or confirm false positive). - - `### ⚠ Verification Failures` — every unverified record from Phase 5.5, with the doc location (file:line) and the helper's `suggest` field. Reviewer corrects before merge. + - `### ⚠ Verification Failures` — every unverified record from Phase 5.5 **and** every Phase 5.7 `review` entry, with the doc location (file:line) and the helper's `suggest` field. Reviewer corrects before merge. - `### ↻ Renames detected` — every directory rename from Phase 3, formatted ` → ` with the helper's `confidence` and `signal`. Confirms that no spurious new doc pages were created for a renamed surface. 5. Update `.sync-state.json`: ```json