sync-docs: tag-accuracy gate + release-channel-aware skill (#111)

Makes /sync-docs correct under the canary→nightly→beta→stable channel model. Adds the release-mode tag-accuracy gate (scripts/sync/check-tag-accuracy.mjs) that verifies every changed page against the stable tag and flags post-tag leaks; stable-CalVer guard + channel glossary, tag-scoped drift/verify, and Phase 5.7 in the skill; realign fix that keeps nightly's drafts; a release-channels watcher; the analysis/plan doc; and a hermetic test (sync:test 38/38).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Aron PrinsandClaude Opus 4.8 authored and GitHub committed 2026-08-25 11:48:38 +02:00
1 parent 6c4519af4a
commit 8a858777f2
6 files changed
+683 -25

No files matched your search

+202
View File
@@ -0,0 +1,202 @@
# Proposal: keep `/sync-docs` working under Paperclip's release-channel model
**Status:** implemented on `feat/sync-docs-release-channels` · **Date:** 2026-08-25
> **Implementation landed with this document.** All five changes below are in this
> branch: the tag-accuracy gate (`scripts/sync/check-tag-accuracy.mjs`, §5.2), the
> stable-CalVer guard + channel glossary and tag-scoped drift/verify (§5.1, §5.3)
> and the new Phase 5.7 in `skills/sync-docs/SKILL.md`, the realign fix
> (`scripts/sync/realign-nightly.mjs`, §5.4), and the `release-channels` watcher in
> `scripts/sync/anchor-map.json` (§5.5). The gate was dry-run against the
> v2026.824.0 leak fixture until it flagged exactly Kimi + the reaper env var with
> zero false positives on in-tag pages, and against the corrected release (0 leaks).
**Trigger:** the v2026.824.0 release run shipped (and we caught) docs for features
that were **not in the release** — Kimi adapter, `PAPERCLIP_WORKSPACE_REAPER_COOLDOWN_DAYS`,
and a mission-less onboarding rewrite. All three exist on `master` but landed
**after** the stable tag was cut. This is now a *structural* problem, because
Paperclip has formalized a four-lane release model where `master` runs far ahead
of `stable`.
The goal is explicit: **keep the single-trigger `/sync-docs` skill working as-is**
(one command auto-detects nightly vs release and handles everything), updated so
it stays correct under the new schedule.
---
## 1. How releases work now (the channel model)
From the parent's `doc/CHANNELS.md` and the release announcement:
```
canary → nightly → beta → stable
```
| Channel | What it is | Cadence | npm dist-tag | Docker tag |
|---|---|---|---|---|
| `canary` | every merge to `master` | many/day | `canary` | `:canary` |
| `nightly` | newest green `master` build, full smoke suite, published if green | nightly | `nightly` | `:nightly` |
| `beta` | a nightly a maintainer promotes behind an approval gate; re-smoked | on promotion | `beta` | `:beta` |
| `stable` | manually cut release; must soak ≥3 days as beta first | ~weekly | `latest` | `:latest` |
Behavior change worth noting: **Docker `:latest` now tracks stable only** (it used
to move on every master merge). npm `@latest` has always meant stable. Each lane
promotion republishes the *exact source commit* of the prior lane, so a beta shares
its SHA with a nightly and a nightly with a canary; the version string dates the
promotion and carries the lane (`2026.MDD.P-beta.N`).
---
## 2. What the skill assumes today
`skills/sync-docs/SKILL.md` runs two modes off one trigger:
- **nightly mode** → tracks parent **`master` HEAD**, targets our `nightly` branch,
deploys a Cloudflare preview. Cumulative diff base = `base_release_tag`, next = `master` HEAD.
- **release mode** → fires when `gh api …/releases/latest` returns a newer tag than
`base_release_tag`; targets `main` (→ docs.paperclip.ing) via a PR cut from `nightly`.
Two assumptions are now broken:
1. **"`nightly` ≈ what will be released."** Our `nightly` branch tracks `master`
HEAD (effectively the **canary** lane). Under the channel model, `master` sits
*many* commits ahead of `stable` — for v2026.824.0 the stable tag was **173
commits behind** `master`, and even the merge-base was behind. So when release
mode cuts the release branch **from `nightly`**, it inherits every post-tag
`master` draft. Those become **leaks** against the stable release.
2. **"drift/verification against `master` is enough."** `check-drift` runs
`--against <parent-default>` (`master`). In release mode that is the *wrong*
reference: a page can be perfectly consistent with `master` and still document
something absent from the **stable tag**. That is exactly why drift returned
`0` on the run that leaked Kimi.
---
## 3. What actually broke on v2026.824.0 (evidence)
| Leak | Why it leaked | How we caught it |
|---|---|---|
| Kimi adapter page + nav + overview row | `packages/adapters/kimi-local` is **404 at the tag** (only on `master`, commit `dc5b070` post-tag) | manual per-feature check against the tag |
| `PAPERCLIP_WORKSPACE_REAPER_COOLDOWN_DAYS` | absent from `.env.example`/config at the tag | manual check |
| Onboarding "mission step is gone" rewrite | the tag's `OnboardingWizard.tsx` **still has** the mission step; the drop is a post-tag change | manual check |
All three were drafted by nightly runs whose cited parent SHA was **ahead of the
tag**. They were verified out by an adversarial pass against the tag — a pass the
skill does not currently perform. Separately, the completeness pass found the
opposite failure: three *in-tag* surfaces (Claude setup-token login, adapter
device-login, issue file-resource availability) that nightly had **not** drafted
and which we had to author during the release.
**Net:** on any given release, the release branch can be simultaneously *ahead* of
the tag (post-tag `master` leaks) and *behind* it (in-tag surfaces nightly skipped).
---
## 4. Recon: can we just anchor to `beta`?
The tempting fix is "point the preview branch at `beta` (the release candidate)
instead of `master`, so it already matches the imminent stable." Recon says **no,
not cleanly**:
- **GitHub Releases and git tags exist for `stable` only.** All releases are
`prerelease=false` clean CalVer (`vYYYY.MDD.P`). The only prerelease git tags in
the repo are legacy pre-CalVer package tags (`[email protected]`).
- **No git ref for beta/nightly/canary.** `git/ref/tags/{beta,nightly,canary}` →
404. There is no branch or moving tag to diff against.
- **npm exposes no `gitHead`.** `npm view paperclipai@beta` returns the version
string but not the commit; the lane SHA lives in CI job summaries and Docker
`:sha-<short>` tags — not something to diff a docs window against reliably.
- **Beta can lag stable.** Today: `beta = 2026.818.0-beta.1` while `stable =
2026.824.0`. Stable was cut from an earlier beta and the beta lane hasn't
advanced past it, so "beta = what stable becomes" does **not** hold moment to
moment. Anchoring the preview to beta could show content *older* than live.
Conclusion: we cannot make `beta` the diff anchor with the parent's current
infrastructure. Keep tracking `master` for the preview and **make release mode
tag-accurate** instead.
---
## 5. Recommended plan (keeps the single trigger)
Five focused changes to the skill and its helpers. Nothing changes about the
one-command UX or the nightly/release branch model.
### 5.1 Pin the release target to the `stable` lane, defensively
- Keep `releases/latest` for mode detection (it already returns stable), but add
an explicit guard: the release target **must** match `^v?\d{4}\.\d{1,4}\.\d+$`
(clean CalVer). Any `-beta/-nightly/-canary` tag is **never** a release target.
- Add a one-paragraph glossary to the skill: "our `nightly` branch tracks parent
`master` HEAD (≈ the **canary** lane); the parent's **`nightly` channel** is a
different thing (a smoke-tested `master` build). Docs ship on **stable**." This
kills the naming collision without a disruptive branch rename.
### 5.2 Add a release-mode **tag-accuracy gate** (the core fix — automates what we did by hand)
New phase, run in release mode after the manifest is built and before the PR is finalized:
1. Enumerate every doc page changed on `nightly` vs `main` (the content that would ship).
2. For each, extract its concrete surface claims (reuse `verify-edit.mjs`) and
**verify against the stable tag**, not `master`.
3. Any page whose underlying surface is **absent at the tag** = a post-tag leak.
**Auto-quarantine** it: revert that page (or the added section) to its `main`
state and list it in the PR body under **"⚠ Post-tag leaks removed."**
4. Symmetrically, surface in-tag surfaces that changed in the window but have **no**
doc touch (the "nightly skipped it" case) as **"⚠ Undocumented in-tag surfaces."**
This makes the leak/gap detection we ran manually a standing, automatic part of
every release.
### 5.3 Point drift + verification at the **tag** in release mode
- `check-drift.mjs` and the Phase 5.5 `verify-edit.mjs` calls should default
`--against` to the **resolved stable tag** in release mode (they use `master`
today). Nightly mode keeps using `master`. One-line change at each call site.
### 5.4 Fix the realign so it doesn't strip valid `master` drafts from `nightly`
- Today `realign-nightly.mjs` fast-forwards `nightly` **onto the leak-stripped
release branch**, which *removes* the (legitimate) post-tag `master` drafts from
`nightly` (we saw Kimi/reaper/onboarding vanish from `nightly` after the
v2026.824.0 realign). It self-heals on the next nightly run, but it's wasteful
and surprising.
- Change realign to **re-anchor** `nightly` to `main`'s squash commit (merge
`origin/main` to restore ancestry) **without** resetting `nightly`'s content to
the release branch — i.e. preserve nightly's master-tracking drafts, only update
the base anchor. Verify `main` is an ancestor of `nightly` as today.
### 5.5 Watch the channel surfaces so they never go stale
- Add an anchor-map watcher for `doc/CHANNELS.md` + `cli/src/commands/channels.ts`
→ `docs/how-to/update-paperclip.md`, `docs/reference/cli/installation.md`, so
channel/tag changes (e.g. the `:latest` semantics change) always flow into the
docs.
---
## 6. What deliberately stays the same
- **One trigger.** `/sync-docs` still auto-detects nightly vs release and does the
whole run. No new required flags.
- **Branch model.** `nightly` (preview) and `main` (live) unchanged; release still
PRs `nightly → main` and squash-merges.
- **Cumulative diffs, quarantine, reconciliation** — all retained. The tag-accuracy
gate is *additive*.
- **Preview stays bleeding-edge.** The `nightly` preview keeps showing `master`
content; that is correct for a contributor preview. Correctness is enforced at
**release** time, where it ships to end users.
---
## 7. Rollout
1. Land this proposal (docs only, no behavior change).
2. Implement 5.1 + 5.3 (small, low-risk anchor/guard changes).
3. Implement 5.2 (the gate) — the highest-value change; dry-run it against the
v2026.824.0 window as a regression test (it must flag Kimi, reaper, onboarding).
4. Implement 5.4 (realign) and 5.5 (watcher).
5. Update `maintenance/runbook.md` and the skill's "Special cases" with the
tag-divergence scenario and the manual fallback.
## 8. Open questions
- Should the preview branch eventually be **renamed** `canary` (accurate) or
`preview` (neutral) to end the collision with the parent `nightly` channel? Out
of scope here (Cloudflare + state migration); flagged for later.
- Do we want a lightweight **release-smoke parity** note in docs (that nightly/beta
are smoke-gated) — content, not skill, but worth a follow-up.
- If the parent ever publishes **git tags for beta** (`2026.MDD.P-beta.N`), revisit
§4: anchoring the preview to beta would then become feasible and would shrink the
leak surface at the source.
+14
View File
@@ -149,6 +149,20 @@
"tier": "context-only",
"detect": "Migrations themselves rarely need user-facing docs, but a sudden burst of migrations often correlates with a schema change watched elsewhere. Use this watcher's hits as context only — never produce edits directly. Deliberately NOT promoted above context-only: most migrations are columns, indexes, and renames that need no prose, and promoting this would put all of them in front of a human for triage. The gap that would justify revisiting is a migration changing user-visible behaviour WITHOUT touching a packages/**/schema/*.ts file the `schemas` watcher already catches — a data backfill that changes a default, a retention window, or a constraint that starts rejecting input that used to be accepted."
},
{
"name": "release-channels",
"parent_paths": [
"doc/CHANNELS.md",
"cli/src/commands/channels.ts"
],
"docs_paths": [
"docs/how-to/update-paperclip.md",
"docs/reference/cli/installation.md"
],
"tier": "pr",
"detect": "The release-channel model (canary → nightly → beta → stable), its npm dist-tags and Docker image tags, and the `paperclipai channels` command. `doc/CHANNELS.md` is the authoritative parent guide; `channels.ts` holds the RELEASE_CHANNELS descriptors and the version→channel regexes. Changes here (a new lane, a dist-tag rename, the `:latest`-tracks-stable semantics) must flow into the channels sections of update-paperclip.md and the CLI installation page. Note: channels.ts is also matched by the cli-commands watcher — dedupe by surface, don't emit twice.",
"channel_context": "Docs ship on the STABLE lane only. Do not describe beta/nightly/canary versions as the release; they are npm/Docker artifacts with no git tag. See maintenance/release-channels-plan.md."
},
{
"name": "release-notes",
"source": "github-releases",
+298
View File
@@ -0,0 +1,298 @@
#!/usr/bin/env node
// check-tag-accuracy.mjs — release-mode gate that catches "post-tag leaks":
// doc pages that changed on the preview branch (head) versus the released
// baseline (base) but document a surface that is NOT present at the stable
// release tag. Under Paperclip's channel model our `nightly` preview tracks
// parent `master` (≈ canary), which runs far ahead of the `stable` tag, so a
// release branch cut from `nightly` inherits post-tag master drafts as leaks
// (see maintenance/release-channels-plan.md).
//
// It reuses verify-edit.mjs as the claim engine: for each changed doc page it
// verifies the page's concrete claims (file paths, env vars, CLI commands/flags,
// adapter config fields, REST routes) against the stable TAG rather than master.
//
// Classification per page:
// leak — has >=1 HIGH-confidence claim absent at the tag (file-path,
// env-var, cli-command, cli-flag). These are safe to auto-quarantine
// (revert the page/section to `base`) — a mechanical identifier the
// released code simply does not contain.
// review — a new page, or medium-confidence-only misses (rest-route,
// adapter-config-field), that a human must confirm against the tag.
// REST routes are medium because constant/prefix registration hides
// real matches (a known verify-edit false-negative).
// clean — every claim verifies at the tag.
//
// The gate is advisory: it always exits 0. It never edits docs; the skill's
// release phase decides what to quarantine from the `leaks` list and what to
// surface under review.
//
// Usage:
// node scripts/sync/check-tag-accuracy.mjs --tag <stable-tag> \
// [--base main] [--head HEAD] [--repo paperclipai/paperclip] [--json]
import { execFileSync, spawnSync } from "node:child_process";
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, openSync, closeSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join, dirname } from "node:path";
import { fileURLToPath } from "node:url";
const HERE = dirname(fileURLToPath(import.meta.url));
const VERIFY_EDIT = join(HERE, "verify-edit.mjs");
// Claim kinds whose absence at the tag is a mechanical, high-confidence leak.
// REST routes and adapter-config fields are deliberately NOT here: route
// prefixing / constant registration and missing adapter doc-sources produce
// false negatives, so they route to `review` instead of auto-quarantine.
const HIGH_CONF_KINDS = new Set([
"file-path",
"env-var",
"cli-command",
"cli-flag",
]);
// Flags Commander registers automatically — never explicitly `.option()`-ed, so
// verify-edit reports them unverified. They are not leaks.
const BUILTIN_FLAGS = new Set(["--help", "-h", "--version", "-V"]);
// A verify-edit claim is only a *product-surface* signal if it points at parent
// source. Markdown link targets (a page's own relative `.md` links) and
// Commander built-in flags are noise, not leaks.
function isSurfaceClaim(u) {
if (u.kind === "file-path" && /\.md$/i.test(u.value)) return false;
if (u.kind === "cli-flag" && BUILTIN_FLAGS.has(u.value)) return false;
return true;
}
function parseArgs(argv) {
const out = { base: "main", head: "HEAD", repo: "paperclipai/paperclip", json: false };
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a === "--tag") out.tag = argv[++i];
else if (a === "--base") out.base = argv[++i];
else if (a === "--head") out.head = argv[++i];
else if (a === "--repo") out.repo = argv[++i];
else if (a === "--json") out.json = true;
else if (a === "-h" || a === "--help") out.help = true;
}
return out;
}
function git(args) {
return execFileSync("git", args, { encoding: "utf8", maxBuffer: 64 * 1024 * 1024 });
}
// Changed doc pages at head vs base. A/M/R only — a deleted page cannot leak.
function changedDocs(base, head) {
const out = git([
"diff",
"--name-status",
"--diff-filter=AMR",
`${base}`,
`${head}`,
"--",
"docs/**/*.md",
]);
const files = [];
for (const line of out.split("\n")) {
if (!line.trim()) continue;
const parts = line.split("\t");
const status = parts[0][0]; // A | M | R
const file = parts[parts.length - 1]; // R lines carry old\tnew; take new
if (file.endsWith(".md")) files.push({ file, status });
}
return files;
}
// Line numbers (in the HEAD version of `file`) that this release added or
// changed vs base. Used to scope claims on a MODIFIED page to what the release
// actually introduced — a pre-existing entry that happens not to verify at the
// tag is drift, not a release leak.
function addedLines(base, head, file) {
const set = new Set();
let diff;
try {
diff = git(["diff", "--unified=0", `${base}`, `${head}`, "--", file]);
} catch {
return set;
}
let newLine = 0;
for (const line of diff.split("\n")) {
const h = line.match(/^@@ -\d+(?:,\d+)? \+(\d+)(?:,\d+)? @@/);
if (h) {
newLine = parseInt(h[1], 10);
continue;
}
if (line.startsWith("+++")) continue;
if (line.startsWith("+")) {
set.add(newLine);
newLine++;
} else if (!line.startsWith("-")) {
// context line (shouldn't appear with --unified=0, but be safe)
newLine++;
}
}
return set;
}
function lineOf(location) {
const m = String(location).match(/:(\d+)$/);
return m ? parseInt(m[1], 10) : null;
}
function showAtRef(ref, file) {
try {
return git(["show", `${ref}:${file}`]);
} catch {
return null;
}
}
function runVerifyEdit(tmpDocPath, tag, repo, outFile) {
// Redirect the child's stdout to a real file rather than a pipe. verify-edit
// writes a large JSON blob and then exits, and a pipe would truncate that at
// the ~64KB OS buffer before it drains (the big adapter/env pages overflow it).
// A regular-file fd is a blocking write, so the full output lands intact.
const fd = openSync(outFile, "w");
let r;
try {
r = spawnSync(
"node",
[VERIFY_EDIT, tmpDocPath, "--against", tag, "--repo", repo, "--json"],
{ stdio: ["ignore", fd, "pipe"], encoding: "utf8" },
);
} finally {
closeSync(fd);
}
let stdout = "";
try {
stdout = readFileSync(outFile, "utf8");
} catch {
/* ignore */
}
try {
return JSON.parse(stdout);
} catch {
const msg = (r?.stderr || r?.error?.message || "verify-edit produced no parseable JSON").split("\n")[0];
return { error: msg, claims_extracted: 0, verified: 0, unverified: [], suspicious: [] };
}
}
function main() {
const args = parseArgs(process.argv.slice(2));
if (args.help || !args.tag) {
process.stderr.write(
"usage: check-tag-accuracy.mjs --tag <stable-tag> [--base main] [--head HEAD] [--repo owner/repo] [--json]\n",
);
process.exit(args.tag ? 0 : 2);
}
const changed = changedDocs(args.base, args.head);
const tmpRoot = mkdtempSync(join(tmpdir(), "tagacc-"));
const leaks = [];
const review = [];
const clean = [];
const errors = [];
try {
for (const { file, status } of changed) {
const content = showAtRef(args.head, file);
if (content === null) continue; // vanished at head — skip
// Materialise under a temp root preserving the docs/... suffix so
// verify-edit's path-based doc-kind sniffing still works.
const tmpPath = join(tmpRoot, file);
mkdirSync(dirname(tmpPath), { recursive: true });
writeFileSync(tmpPath, content);
const res = runVerifyEdit(tmpPath, args.tag, args.repo, join(tmpRoot, "verify-out.json"));
if (res.error) {
errors.push({ file, error: res.error });
continue;
}
let unverified = (res.unverified || []).filter(isSurfaceClaim);
// For a modified page, judge only the claims this release added/changed —
// a pre-existing miss is drift, not a release leak. A new page (A) is all
// new content, so keep every claim.
if (status !== "A") {
const added = addedLines(args.base, args.head, file);
unverified = unverified.filter((u) => {
const ln = lineOf(u.location);
return ln !== null && added.has(ln);
});
}
const high = unverified.filter((u) => HIGH_CONF_KINDS.has(u.kind));
const med = unverified.filter((u) => !HIGH_CONF_KINDS.has(u.kind));
const claimsOf = (arr) =>
arr.map((u) => ({ kind: u.kind, value: u.value, evidence: u.evidence }));
if (high.length > 0) {
leaks.push({ file, status, claims: claimsOf(high) });
} else if (status === "A" && (res.verified === 0 || med.length > 0)) {
review.push({
file,
status,
reason: "new page whose surface could not be confirmed at the tag",
claims: claimsOf(med),
});
} else if (med.length > 0) {
review.push({
file,
status,
reason: "medium-confidence claims unverified at the tag (confirm not a leak)",
claims: claimsOf(med),
});
} else {
clean.push(file);
}
}
} finally {
rmSync(tmpRoot, { recursive: true, force: true });
}
const result = {
tag: args.tag,
base: args.base,
head: args.head,
repo: args.repo,
files_checked: changed.length,
leaks,
review,
clean_count: clean.length,
errors,
};
if (args.json) {
process.stdout.write(JSON.stringify(result, null, 2) + "\n");
return;
}
const L = [];
L.push(`Tag-accuracy check — head '${args.head}' vs base '${args.base}' against tag ${args.tag}`);
L.push(`Doc pages changed: ${result.files_checked} | leaks: ${leaks.length} | review: ${review.length} | clean: ${clean.length}`);
if (leaks.length) {
L.push("");
L.push("## ⚠ Post-tag leaks (auto-quarantine candidates — surface absent at the tag)");
for (const l of leaks) {
L.push(`- ${l.file}`);
for (const c of l.claims) L.push(` [${c.kind}] ${c.value} — ${c.evidence}`);
}
}
if (review.length) {
L.push("");
L.push("## ⚠ Review against the tag (medium-confidence / new page)");
for (const r of review) {
L.push(`- ${r.file} — ${r.reason}`);
for (const c of r.claims) L.push(` [${c.kind}] ${c.value}`);
}
}
if (errors.length) {
L.push("");
L.push("## Errors");
for (const e of errors) L.push(`- ${e.file}: ${e.error}`);
}
if (!leaks.length && !review.length) L.push("OK — no post-tag leaks; every changed page verifies at the tag.");
process.stdout.write(L.join("\n") + "\n");
}
main();
+33 -17
View File
@@ -7,9 +7,11 @@
// "merge main into nightly" sees pages both branches created as add/add
// conflicts. This script heals that, and must run after EVERY release merge:
//
// 1. fast-forward `nightly` onto the release branch (content == shipped)
// 2. merge `origin/main` into `nightly` (re-anchors the merge-base at the
// squash commit, so the next cycle's divergence starts fresh)
// merge `origin/main` into `nightly` with `-X ours` — this re-anchors the
// merge-base at the squash commit (so the next cycle's divergence starts
// fresh) while KEEPING nightly's content, including the post-tag `master`
// drafts that Phase 5.7 quarantined from the release branch. It does NOT
// fast-forward nightly onto the release tree (that would delete those drafts).
//
// Usage: node scripts/sync/realign-nightly.mjs <release-branch> [--push]
// e.g. node scripts/sync/realign-nightly.mjs release/v2026.720.0
@@ -59,22 +61,36 @@ const startBranch = git("branch", "--show-current");
git("checkout", "nightly");
git("merge", "--ff-only", "origin/nightly");
// Step 1: bring nightly's content to exactly what shipped.
const ff = tryGit("merge", "--ff-only", releaseBranch);
if (!ff.ok) {
// Something landed on nightly after the release branch forked. Same content
// guarantees a clean three-way merge for the release files themselves.
console.log("fast-forward not possible (nightly moved since the release forked) — doing a normal merge");
const m = tryGit("merge", releaseBranch, "-m", `Merge ${releaseBranch} into nightly (post-release realign)`);
if (!m.ok) die(`merge of ${releaseBranch} into nightly conflicted:\n${m.out}\nResolve by hand, then run the main-merge step yourself.`);
// Re-anchor ancestry onto main's squash commit WITHOUT discarding nightly's
// post-tag drafts.
//
// We deliberately do NOT fast-forward nightly onto the release branch. Phase 5.7
// quarantines post-tag leaks (Kimi, reaper, an onboarding rewrite, ...) from the
// *release* branch only — those drafts are still legitimate on `nightly` because
// they document `master` features that ship in a *future* stable release.
// Fast-forwarding nightly onto the release tree would delete them.
//
// Instead do one merge of origin/main into nightly with `-X ours`: it makes the
// squash commit a parent of nightly (restoring ancestry so the next nightly-mode
// main-merge no longer explodes into add/add conflicts), keeps nightly's own
// version of every page both branches created, still pulls in main-only hot-fixes
// on pages nightly did not touch, and preserves nightly-only drafts (Kimi et al.)
// untouched.
const anchor = tryGit(
"merge",
"-X",
"ours",
"origin/main",
"-m",
"Merge main into nightly (re-anchor after squash-merged release; keep nightly drafts)",
);
if (!anchor.ok) {
if (/Already up to date/i.test(anchor.out)) {
die("origin/main is already an ancestor of nightly — nothing to realign (was this already run?)");
}
die(`merge of origin/main into nightly failed:\n${anchor.out}\nResolve by hand, keeping nightly's drafts, then re-run the post-check.`);
}
// Step 2: merge main so the squash commit becomes an ancestor of nightly.
// This is what actually re-anchors merge-base(main, nightly) — do not skip it
// even though the trees already match.
const anchor = tryGit("merge", "origin/main", "-m", "Merge main into nightly (re-anchor after squash-merged release)");
if (!anchor.ok) die(`merge of origin/main conflicted (unexpected — trees matched):\n${anchor.out}`);
const base = git("merge-base", "origin/main", "nightly");
const mainTip = git("rev-parse", "origin/main");
if (base !== mainTip) die(`post-check failed: merge-base(main, nightly) is ${base.slice(0, 9)}, expected main's tip ${mainTip.slice(0, 9)}`);
+91
View File
@@ -1257,6 +1257,97 @@ test("detect-renames: two candidates → file-overlap wins over shared-prefix",
`from=${renForAdded.from}`);
});
// ----------------------------------------------------------------------------
// check-tag-accuracy (hermetic end-to-end: real git fixture repo + stubbed
// parent via PAPERCLIP_SYNC_FIXTURE_DIR — no network)
// ----------------------------------------------------------------------------
const TAGACC_SCRIPT = join(SELF_DIR, "check-tag-accuracy.mjs");
// A file-path claim in a doc page: `<prefix>/...ts` (verified iff the parent
// returns 200 at the tag). Using only file-path claims keeps verify-edit from
// reaching for the tree/env/route/adapter fetchers, so the whole run is stubbed
// by a handful of contents-*.json files.
function makeTagAccFixture() {
const dir = makeFixture();
copyFileSync(TAGACC_SCRIPT, join(dir, "scripts/sync/check-tag-accuracy.mjs"));
copyFileSync(VERIFY_SCRIPT, join(dir, "scripts/sync/verify-edit.mjs"));
copyFileSync(join(SELF_DIR, "anchor-map.json"), join(dir, "scripts/sync/anchor-map.json"));
const g = (...args) => {
const r = spawnSync("git", args, { cwd: dir, encoding: "utf8" });
if (r.status !== 0) throw new Error(`git ${args.join(" ")} failed: ${r.stderr || r.stdout}`);
return r.stdout;
};
g("init", "-q");
g("config", "user.email", "[email protected]");
g("config", "user.name", "t");
g("config", "commit.gpgsign", "false");
// BASE: `existing.md` already links a pre-existing file path.
writeFile(dir, "docs/how-to/existing.md", "# Existing\n\nSee `packages/stale/pre.ts` for details.\n");
g("add", "-A");
g("commit", "-q", "-m", "base");
// HEAD: append a NEW file-path link to existing.md, add a leak page and a
// clean page.
writeFile(
dir,
"docs/how-to/existing.md",
"# Existing\n\nSee `packages/stale/pre.ts` for details.\n\nAlso see `packages/ghost/added.ts` now.\n",
);
writeFile(dir, "docs/how-to/ghost-leak.md", "# Ghost\n\nRuns from `packages/ghost/src/index.ts`.\n");
// real page: a .ts that DOES exist at the tag (200) plus a .md link that does
// not (404). The .md must be ignored (it's a doc link, not a product surface).
writeFile(
dir,
"docs/how-to/real-clean.md",
"# Real\n\nRuns from `packages/adapters/real/src/index.ts`. Config is `packages/adapters/real/config.md`.\n",
);
g("add", "-A");
g("commit", "-q", "-m", "head");
// Stubbed parent: only the real .ts resolves at the tag. Everything else 404s
// by omission.
const fixDir = join(dir, "_fixtures");
mkdirSync(fixDir, { recursive: true });
writeFileSync(join(fixDir, "repo.json"), JSON.stringify({ default_branch: "master" }));
writeFileSync(
join(fixDir, "contents-packages__adapters__real__src__index.ts-faketag.json"),
JSON.stringify({ content: "export {}\n" }),
);
return { dir, fixDir };
}
test("check-tag-accuracy: flags a post-tag leak, keeps in-tag page clean, scopes to added lines", () => {
const { dir, fixDir } = makeTagAccFixture();
const r = spawnSync(
process.execPath,
[join(dir, "scripts/sync/check-tag-accuracy.mjs"), "--tag", "faketag", "--base", "HEAD~1", "--head", "HEAD", "--json"],
{ cwd: dir, encoding: "utf8", env: { ...process.env, PAPERCLIP_SYNC_FIXTURE_DIR: fixDir } },
);
assert(r.status === 0, `gate should always exit 0, got ${r.status}. stderr=${r.stderr}`);
const out = JSON.parse(r.stdout);
const leakFiles = out.leaks.map((l) => l.file);
// The new page whose only surface is a tag-absent .ts is a leak.
assert(leakFiles.includes("docs/how-to/ghost-leak.md"), `ghost-leak.md should be a leak; leaks=${JSON.stringify(leakFiles)}`);
// The modified page is a leak because of its ADDED tag-absent .ts.
assert(leakFiles.includes("docs/how-to/existing.md"), `existing.md should be a leak; leaks=${JSON.stringify(leakFiles)}`);
// The page whose .ts exists at the tag (and whose only miss is a .md doc link) is NOT a leak.
assert(!leakFiles.includes("docs/how-to/real-clean.md"), `real-clean.md must not be a leak; leaks=${JSON.stringify(leakFiles)}`);
assert(!out.review.some((x) => x.file === "docs/how-to/real-clean.md"), "real-clean.md must not be in review (its .ts verifies; the .md link is ignored)");
// Added-line scoping: the pre-existing `pre.ts` (also 404) must NOT be flagged
// — only the added `added.ts`.
const ex = out.leaks.find((l) => l.file === "docs/how-to/existing.md");
const vals = ex.claims.map((c) => c.value);
assert(vals.includes("packages/ghost/added.ts"), `existing.md leak should cite the added path; got ${JSON.stringify(vals)}`);
assert(!vals.includes("packages/stale/pre.ts"), `pre-existing path must be scoped out of the leak; got ${JSON.stringify(vals)}`);
rmSync(dir, { recursive: true, force: true });
});
// ----------------------------------------------------------------------------
console.log("");
+45 -8
View File
@@ -39,7 +39,8 @@ The branch model is non-negotiable: end users on the latest *released* paperclip
- `scripts/sync/check-drift.mjs` — Phase 1.5 drift detection (finds documented surfaces missing from parent).
- `scripts/sync/detect-renames.mjs` — Phase 3 directory rename detection (distinguishes a renamed surface from a brand-new one).
- `scripts/sync/verify-edit.mjs` — Phase 5.5 post-edit verification (checks that authored claims still match parent code).
- `scripts/sync/realign-nightly.mjs` — post-release realign (Phase 7): fast-forwards `nightly` onto the merged release branch and re-anchors the merge-base after the squash-merge severs ancestry.
- `scripts/sync/check-tag-accuracy.mjs` — **release mode only** (Phase 5.7): the tag-accuracy gate. Verifies every doc page changed on `nightly` vs `main` against the **stable release tag** and flags post-tag leaks — content nightly drafted from `master` that is not in the release. See `maintenance/release-channels-plan.md` for why this exists.
- `scripts/sync/realign-nightly.mjs` — post-release realign (Phase 7): re-anchors `nightly` onto the merged release squash commit (restoring ancestry) **without discarding** nightly's post-tag `master` drafts.
## Invocation
@@ -65,10 +66,21 @@ The branch model is non-negotiable: end users on the latest *released* paperclip
### Phase 1 — Decide mode and target branch
> **Release channels — read this first.** Paperclip publishes on four lanes:
> `canary` (every merge to `<parent-default>`) → `nightly` (a green `master` build,
> smoke-tested nightly) → `beta` (a promoted nightly, soaks ≥3 days) → `stable`
> (the manually cut release). **Only `stable` has git tags and GitHub Releases**
> (clean CalVer `vYYYY.MDD.P`); beta/nightly/canary are npm/Docker artifacts with
> no git ref we can diff against. **Docs ship on `stable`.** Note the naming
> collision: *our* `nightly` branch tracks parent `<parent-default>` HEAD (≈ the
> **canary** lane), which runs far ahead of the stable tag — so a release branch
> cut from `nightly` inherits post-tag `master` drafts. Phase 5.7 is the gate that
> catches those. See `maintenance/release-channels-plan.md`.
1. Read `.sync-state.json`. Note `branch_mode` and `base_release_tag`.
2. Fetch parent's latest release: `gh api repos/paperclipai/paperclip/releases/latest -q '.tag_name'`.
2. Fetch parent's latest **stable** release: `gh api repos/paperclipai/paperclip/releases/latest -q '.tag_name'`. `releases/latest` already excludes prereleases, but **guard defensively**: the release target MUST match `^v?\d{4}\.\d{1,4}\.\d+$` (clean CalVer). Any `-beta`/`-nightly`/`-canary` tag is never a release target — if the API ever returns one, treat the run as nightly mode and note it in the summary.
3. Auto-detect mode (unless overridden):
- If a new release tag exists AND `base_release_tag` is older → **release mode**.
- If a new stable release tag exists AND `base_release_tag` is older → **release mode**.
- Otherwise → **nightly mode**.
4. Check out the right branch:
- Release mode: ensure on `main`. Release mode is **self-sufficient** — it does not require the `nightly` branch to exist or to have drafts. If `nightly` exists with relevant drafts, they're used as a starting point; if not, the release run computes everything from scratch.
@@ -110,10 +122,13 @@ Cache result under `/tmp/paperclip-sync/<sha>/` so we don't refetch within a run
Drift is the inverse of the cumulative diff: it's the set of things **we already document** that have since vanished or moved upstream. It exists regardless of when the last sync happened — a parent surface can disappear between two sync runs even if our diff window is empty. The wet-run that motivated this phase found `POST /api/companies/{companyId}/logo` documented but absent from current `server/src/routes/companies.ts`, with no sign of it in any diff window the sync had ever processed.
Run the drift checker against parent HEAD:
Run the drift checker against the reference for this mode. **Nightly mode → `<parent-default>`** (drift is about the live upstream). **Release mode → the stable release tag** — checking release-mode drift against `master` is blind to the tag divergence and will pass surfaces that exist on `master` but not in the release (that is how the Kimi leak slipped past a `0`-drift run). Use `<release-tag>` in release mode:
```
# nightly mode
node scripts/sync/check-drift.mjs --json --against <parent-default>
# release mode
node scripts/sync/check-drift.mjs --json --against <release-tag>
```
The script scans `docs/**` for four reference classes and verifies each one still exists in parent:
@@ -246,10 +261,11 @@ If all pass: make the mechanical edit directly. Examples: append a row to `envir
### Phase 5.5 — Verify edits against parent code
For every file touched in Phase 5 (auto-merge or PR-tier):
For every file touched in Phase 5 (auto-merge or PR-tier). Verify against the reference for this mode — **nightly mode → `<parent-default>`**, **release mode → the stable `<release-tag>`** (verifying release edits against `master` would confirm claims that are true on `master` but absent from the release):
```
node scripts/sync/verify-edit.mjs <doc-path> --against <parent-default-or-tag> --json
node scripts/sync/verify-edit.mjs <doc-path> --against <parent-default> # nightly mode
node scripts/sync/verify-edit.mjs <doc-path> --against <release-tag> # release mode
```
Collect all `unverified` and `suspicious` records into a **Verification Report** for the run.
@@ -261,6 +277,24 @@ Routing rules:
- **PR tier edits (batched-release mode).** Never auto-merge if any unverified records exist. They flow into the release PR with the failed claims listed under `⚠ Verification Failures`.
- **Suspicious records.** Logged and surfaced in the run summary / PR body, but never block. They're informational.
### Phase 5.7 — Tag-accuracy gate (release mode only)
Phase 5.5 verifies the edits *this run* authored. This phase verifies **everything the release would ship** — including drafts nightly authored in earlier runs — against the stable tag. It exists because our `nightly` branch tracks parent `master` (≈ canary), which runs far ahead of the stable tag, so a release branch cut from `nightly` inherits post-tag `master` drafts as leaks (this is exactly how Kimi, `PAPERCLIP_WORKSPACE_REAPER_COOLDOWN_DAYS`, and a mission-less onboarding rewrite reached a release branch). Skip entirely in nightly mode.
Run the gate over the pages the release changes vs the live baseline:
```
node scripts/sync/check-tag-accuracy.mjs --tag <release-tag> --base main --head <release-branch-or-nightly> --json
```
It reuses `verify-edit.mjs` per changed page and classifies each:
- **`leaks`** — a page has ≥1 **high-confidence** claim (`file-path`, `env-var`, `cli-command`, `cli-flag`) that the release-tag code does not contain. These are mechanical identifiers the shipped code simply lacks. **Auto-quarantine** each: revert the page (or just the offending section) to its `main` state, and list it in the PR body under **`### ⚠ Post-tag leaks removed`**. Only added/changed lines are judged, so a pre-existing stale entry is left to the drift phase, not quarantined here.
- **`review`** — new pages, or medium-confidence-only misses (`rest-route`, `adapter-config-field`). REST routes are medium because constant/prefix registration hides real matches (a known `verify-edit` false negative — do **not** auto-quarantine them). Surface under the PR's `### ⚠ Verification Failures` / review notes for a human to confirm against the tag.
- **`clean`** — every added claim verifies at the tag.
**Known limit — behavioural/prose leaks.** The gate only catches leaks that carry a checkable *identifier*. A pure prose/UI rewrite with no code identifier (the onboarding "mission step is gone" case) will read as `clean`. So release mode **also** keeps the adversarial nightly-draft check: spawn a small verification pass (per changed guide page, compare its user-visible claims against the tag's UI/flow) and treat a page describing behaviour absent at the tag as a leak — revert it to `main` and list it under `### ⚠ Post-tag leaks removed`.
### Phase 6 — Screenshot staleness check
Read `docs/user-guides/screenshots/registry.json`. For each entry:
@@ -354,12 +388,15 @@ rather than shipping.
node scripts/sync/realign-nightly.mjs release/v2026.X.Y --push
```
It fast-forwards `nightly` onto the release branch, merges `origin/main` back in to re-anchor the merge-base at the squash commit, verifies main's tip is an ancestor of nightly, and pushes. Skipping this arms an add/add conflict trap that detonates at the next nightly run (see the special case below). Do not run any nightly-mode sync between the squash-merge and the realign.
It re-anchors `nightly` onto the release squash commit (restoring ancestry) and pushes. Skipping this arms an add/add conflict trap that detonates at the next nightly run (see the special case below). Do not run any nightly-mode sync between the squash-merge and the realign.
**Do not let the realign discard nightly's post-tag drafts.** Because Phase 5.7 quarantines post-tag leaks *from the release branch only*, those drafts (Kimi, etc.) are still legitimate on `nightly` — they document `master` features that ship in a *future* stable release. The realign must re-anchor `nightly` to `main`'s squash commit **without** resetting nightly's content to the release branch. `realign-nightly.mjs` does this by merging `origin/main` into `nightly` (favouring nightly's content on conflict) rather than fast-forwarding nightly onto the release branch. If the drafts do get dropped, the next nightly run regenerates them from the cumulative window — wasteful but self-healing.
PR body structured sections (each omitted if empty — never silently dropped):
- `### ⚠ Post-tag leaks removed` — every page Phase 5.7 quarantined, with the offending claim(s) and the note that the surface is absent at the release tag (present on `master`, shipping in a later release). Confirms the release documents only what it contains.
- `### ⚠ Drift` — every drift candidate from Phase 1.5 grouped by `kind`, high-confidence first, medium-confidence prefixed with `Verify:`. Never auto-resolved — the PR explicitly asks the reviewer to act on each entry (update path, delete section, or confirm false positive).
- `### ⚠ Verification Failures` — every unverified record from Phase 5.5, with the doc location (file:line) and the helper's `suggest` field. Reviewer corrects before merge.
- `### ⚠ Verification Failures` — every unverified record from Phase 5.5 **and** every Phase 5.7 `review` entry, with the doc location (file:line) and the helper's `suggest` field. Reviewer corrects before merge.
- `### ↻ Renames detected` — every directory rename from Phase 3, formatted `<from> → <to>` with the helper's `confidence` and `signal`. Confirms that no spurious new doc pages were created for a renamed surface.
5. Update `.sync-state.json`:
```json