mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-07 06:16:06 -10:00
## Thinking Path > - Paperclip coordinates agents that make and verify repository changes. > - Local verification currently defaults to the complete Vitest suite. > - Contributor instructions also require broad local checks at each PR handoff. > - CI already owns conservative affected-test selection and broad final-revision checks. > - This pull request adds local selection and pass reuse from those existing contracts. > - Agents can get focused feedback and use CI to close the remaining coverage gaps. ## Linked Issues or Issue Description **Subsystem affected** Developer verification scripts and contributor guidance. Related selection work: #18398 and #18464. **Problem or motivation** An agent that repeats an unchanged local verification step starts the complete suite again. The handoff guidance also duplicates broad CI coverage locally. **Proposed solution** Add `pnpm verify` for the actual working tree. Select affected Vitest specifications with the CI task policy and dependency graph. Reuse successful checks whose inputs are modeled. Run opaque checks fresh. Report exit 2 when broader verification is required. Update the contributor contract to accept applicable final-revision CI coverage. **Alternatives considered** Plain `vitest related` omits filesystem and opaque dependencies. Commit-only cache keys miss dirty files. Running full local suites at every handoff repeats CI work. **Roadmap alignment** This is repository development tooling. It does not change the product roadmap. ## What Changed - Add `pnpm verify`, `--explain`, `--base`, `--full`, and `--no-cache`. - Reuse CI selection and exact runner ownership for dirty, staged, committed, deleted, renamed, and untracked inputs. - Keep native server suites and serialized suites in their local owners. Keep six sequential chat shards. - Cache selected passes within one worktree. Hash source, fixtures, installed package contents, dependency bindings and installation metadata, dotenv files, child environment, toolchains, platform, and selection. - Run opaque consumers fresh. Preserve failed, interrupted, mutated-input, and partial-coverage outcomes. - Update AGENTS.md, CONTRIBUTING.md, and DEVELOPING.md guidance. Record the shared `verify` command as the sole exception to the package-script rule. Register the new tests in the existing CI contract-test command. - Add optional root/environment inputs to shared graph helpers. Existing CI defaults stay the same. ## Verification - Fourteen new integration tests pass locally across focused runs with real Vitest selection and execution; CI colors are covered. - 51 existing targeted selector/input, project-isolation, and stable-runner checks passed. Four relevant runner contracts also pass after the workspace change. - Repeated unchanged execution skips reusable tests. Source, environment, installed code, dependency bindings, and new untracked tests invalidate results. - Installed-code mutation with a restored modification time still invalidates a cached pass. Internal skill-directory links preserve reuse. - Workspace B/C tests prove discovery, reusable/fresh partition execution, pass reuse, and external-input failure. Chat selection produces exactly six single-worker partitions. - A deliberate unrelated failing test is omitted by narrow selection and caught by `--full`. - Red/green regressions prove fingerprints preserve file boundaries and reject edits made between diff discovery and snapshotting. - Opaque external-file changes still fail on a subsequent run. Source mutation and process interruption cannot publish reusable passes. - `pnpm verify --explain` works in the checkout and requests broad CI coverage for this tooling change. No duplicate full local suite was started. - [Final-revision CI](https://github.com/paperclipai/paperclip/actions/runs/42117291789) passed on `ed85d96ddff82c05c325948680b593b0f48c473d` (attempt 2). The scoped Canary retry preserved all passing test/build results. - [Greptile review](https://github.com/paperclipai/paperclip/pull/19345#issuecomment-6670050175) is 5/5; all four review threads are resolved. - The additional Paperclip review concluded neutral because its execution stopped; it produced no assessment. ## Risks - Pass reuse depends on conservative dependency classification. Opaque consumers, external database/live settings, and non-isolated projects retain fresh or broad coverage. - Local passes cover selected Vitest checks. Required CI, build/typecheck, release, browser, and platform acceptance remain separate obligations. - Cross-worktree caching and cross-agent scheduling are outside this change. - The initial installation scan reads dependency contents; stat-validated, worktree-local digest memoization avoids rereading unchanged contents. Subsequent scans still inspect file identity and nanosecond change/modification times. - No dependency lockfile or workflow changes. ## Model Used OpenAI Codex, `gpt-6.1-sol`, max reasoning, with local shell and file tools. Context window size is not exposed by this host. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>
100 lines
5.7 KiB
JavaScript
100 lines
5.7 KiB
JavaScript
#!/usr/bin/env node
|
|
import { realpathSync } from "node:fs";
|
|
import { fileURLToPath } from "node:url";
|
|
import { parseArgs } from "node:util";
|
|
import { createHash } from "node:crypto";
|
|
import { comparison, localEnvironment, protocolVersion, requestKey, snapshot } from "./local-verify/inputs.mjs";
|
|
import { planVerification } from "./local-verify/plan.mjs";
|
|
import { invalidatePass, mayReuseAttempt, readPass, readPlan, runPlan, writeRecord } from "./local-verify/run.mjs";
|
|
|
|
function describe(plan, cached) {
|
|
console.log(`[verify] ${plan.kind === "full" ? "Full Vitest suite" : plan.kind === "broad" ? "Broader verification required"
|
|
: `${plan.selectedFiles.length}${plan.totalFiles === undefined ? "" : `/${plan.totalFiles}`} Vitest test files selected`}${cached ? "; reusing a passing local result" : ""}`);
|
|
if (plan.reason) console.log(`[verify] ${plan.reason}`);
|
|
if (plan.retainedFiles) console.log(`[verify] ${plan.retainedFiles} test files retained conservatively`);
|
|
if (plan.requiredCI.length) console.log(`[verify] Required CI coverage: ${plan.requiredCI.join(", ")}`);
|
|
console.log("[verify] Local Vitest coverage does not replace required CI, typecheck/build, browser, release, or platform-specific checks.");
|
|
}
|
|
|
|
export async function verify({ root = process.cwd(), base, full = false, explain = false, noCache = false, env = localEnvironment() } = {}) {
|
|
root = realpathSync(root);
|
|
const before = snapshot(root, env), diff = comparison(root, base), key = requestKey(before, diff, full);
|
|
const canReuse = !noCache && before.cacheable && mayReuseAttempt(root, key);
|
|
const cached = canReuse ? readPass(root, key) : null;
|
|
console.log(`[verify] Base ${diff.base.slice(0, 12)}; ${diff.changed.length} changed paths, including local edits`);
|
|
if (cached) {
|
|
if (snapshot(root, env).hash !== before.hash) throw new Error("Inputs changed during cache lookup. Run verification again.");
|
|
describe(cached.plan, true);
|
|
console.log(`[verify] Passed result from ${cached.completedAt}; input fingerprint ${before.hash.slice(0, 12)}`);
|
|
if (explain) console.log(JSON.stringify({ ...diff, inputFingerprint: before.hash, cacheable: before.cacheable, plan: cached.plan }, null, 2));
|
|
return 0;
|
|
}
|
|
if (!explain) {
|
|
invalidatePass(root, key);
|
|
writeRecord(root, `attempt-${key}`, { version: protocolVersion, key, status: "running", plan: null });
|
|
}
|
|
const plan = (canReuse ? readPlan(root, key) : null)
|
|
?? await planVerification({ root, env, diff, full });
|
|
if (plan.manifest) plan.manifest.head = diff.head;
|
|
if (snapshot(root, env).hash !== before.hash) throw new Error("Inputs changed during selection. Run verification again.");
|
|
describe(plan, false);
|
|
if (explain) {
|
|
console.log(JSON.stringify({ ...diff, inputFingerprint: before.hash, cacheable: before.cacheable, plan }, null, 2));
|
|
return plan.kind === "broad" ? 2 : 0;
|
|
}
|
|
if (plan.kind === "broad") {
|
|
console.log("[verify] Run focused checks and use required CI for broad coverage, or explicitly run pnpm verify --full.");
|
|
writeRecord(root, `attempt-${key}`, { version: protocolVersion, key, status: "broad", plan });
|
|
return 2;
|
|
}
|
|
if (before.cacheable && !noCache && plan.kind === "selected") {
|
|
writeRecord(root, `plan-${key}`, { version: protocolVersion, key, status: "planned", plan });
|
|
}
|
|
invalidatePass(root, key);
|
|
const startedAt = new Date().toISOString();
|
|
let code = 0;
|
|
for (const job of plan.jobs) {
|
|
const jobKey = createHash("sha256").update(`${key}\0${job.id}`).digest("hex");
|
|
if (canReuse && job.cacheable && readPass(root, jobKey)) {
|
|
console.log(`[verify] Reusing passed ${job.id}`);
|
|
continue;
|
|
}
|
|
invalidatePass(root, jobKey);
|
|
code = await runPlan(root, env, { ...plan, jobs: [job] });
|
|
if (snapshot(root, env).hash !== before.hash) {
|
|
console.error("[verify] Inputs changed during verification. The result cannot be reused.");
|
|
code ||= 1;
|
|
}
|
|
if (code !== 0) break;
|
|
if (!noCache && before.cacheable && job.cacheable) writeRecord(root, jobKey, {
|
|
version: protocolVersion, key: jobKey, status: "passed", plan, jobId: job.id, completedAt: new Date().toISOString(),
|
|
});
|
|
}
|
|
const after = snapshot(root, env);
|
|
if (after.hash !== before.hash) {
|
|
console.error("[verify] Inputs changed during verification. The result cannot be reused.");
|
|
code ||= 1;
|
|
}
|
|
const record = { version: protocolVersion, key, status: code === 0 ? "passed" : "failed", startedAt,
|
|
completedAt: new Date().toISOString(), head: diff.head, inputFingerprint: before.hash, plan, exitCode: code };
|
|
writeRecord(root, "last-run", record);
|
|
writeRecord(root, `attempt-${key}`, record);
|
|
if (code === 0 && !noCache && before.cacheable && after.cacheable && plan.jobs.every(job => job.cacheable)) writeRecord(root, key, record);
|
|
console.log(`[verify] ${code === 0 ? "Local Vitest verification passed" : "Verification failed or interrupted"}${code === 0 && !before.cacheable ? "; external inputs disable pass reuse" : ""}`);
|
|
return code;
|
|
}
|
|
|
|
async function main() {
|
|
const { values } = parseArgs({ options: { base: { type: "string" }, full: { type: "boolean" },
|
|
explain: { type: "boolean" }, "no-cache": { type: "boolean" }, help: { type: "boolean" } } });
|
|
if (values.help) {
|
|
console.log("Usage: pnpm verify [--base REF] [--explain] [--full] [--no-cache]\nRuns affected local Vitest checks. Exit 2 means broader coverage is required; use CI or --full.");
|
|
return;
|
|
}
|
|
process.exitCode = await verify({ base: values.base, full: values.full, explain: values.explain, noCache: values["no-cache"] });
|
|
}
|
|
|
|
if (process.argv[1] && realpathSync(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
|
main().catch(error => { console.error(`[verify] ${error.message}`); process.exitCode = 1; });
|
|
}
|