mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-07 06:16:06 -10:00
DottaandClaude Opus 5.5
871a9b3180
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane owns task execution, recovery and Retry for every agent run > - Server restarts were the largest remaining source of stalled tasks in the staging soak campaign > - The soak could only observe restarts that staging happened to perform, so a failure could not be repeated > - This pull request adds a script harness that causes the restarts on purpose and grades each task on fixed rules > - The benefit is a repeatable restart failure with a replay command and an evidence bundle for each fixer ## Linked Issues or Issue Description **What existing behavior does this improve?** The restart test coverage. Today the runner-e2e restart cells test one fixed restart point per cell. They cannot run many tasks at once, cannot choose a random fault time, and cannot replay a failure. **Subsystem affected** Test tooling (`tests/restart-chaos/`). It reuses the runner-e2e process code in `tests/runner-e2e/`. Product code does not change. **Current behavior** Restart robustness is checked through a small set of fixed cells. A random restart during a real task has no defined pass rule. **Proposed behavior** A harness starts an isolated control plane and seeds a "Chaos Lab" company. It launches seeded workloads, injects K1, K2 or K4 faults at seeded times, and grades each task on rules R1 to R6 plus a deterministic task oracle. `--replay <wave>/<task>` repeats one task. **Reason and benefit** Fixers get a replay command and an evidence bundle for each failure signature. The first smoke wave already found three product signatures (dead Retry after a restart, a false over-budget stop after restarts, and an OpenCode turn killed mid-turn that ends as a provider failure). **Breaking changes** None. The change adds new test files, one entry in the test-project typecheck list and a gitignore entry. ## What Changed - Add `tests/restart-chaos/`: - `supervisor.ts` owns one isolated server. It reuses `replaceController`, the process-tree stopper, the environment isolation helpers, the config seed, the port reservation, the wrapper-owned Postgres and `server-entry.ts` from `tests/runner-e2e/`. It does not copy them. Faults: K1 (SIGKILL, immediate restart), K2 (SIGTERM, restart) and K4 (SIGKILL, 2 to 5 minute hold). Postgres and the controller runtime cache stay across a fault. - `seed.ts` creates the company, encrypted provider secrets, native Codex, ACPX Claude and OpenCode agents from the runner-e2e catalog, and the `chaos-sandbox` project. The project is a git repository with a Python CLI, one failing test and a four-minute `bench.sh`. - `scheduler.ts` makes a seeded plan of workload, adapter, fault, fire time, hold and nonce for each task. It saves the plan so that `--replay` can run one task again. - `observer.ts` polls each issue and its runs. When a task stops and offers Retry, it sends `POST /api/issues/:id/retry` one time, with the same body as `useTaskRetry.ts`. - `grader.ts` grades R1 to R6 as pure functions. `workloads.ts` defines workloads A1, B1, B5 and C1 and their oracles. - `evidence.ts` and `ledger.ts` write an evidence bundle for each failure, a JSONL ledger and `SUMMARY.md`. All results go to the gitignored `tests/restart-chaos/results/`. Known provider key values are redacted. - Unit tests: `scheduler.test.ts` (determinism, replay and CLI parsing) and `grader.test.ts` (R1 to R6 and the oracles on fixture timelines). These tests need no credentials. - Add `tests/restart-chaos` to the test-project typecheck list (`scripts/typecheck-test-projects.mjs`), with a `tsconfig.layers.json` like `tests/runner-e2e`. The harness runs through direct script invocations, not `package.json` scripts. - After the first Greptile review: - A failed fault or an unexpected server exit now gets one bounded recovery. If the recovery fails, the wave fails and does not hang. - Incomplete cleanup is now a harness error, and the harness keeps the instance. - After the C1 follow-up comment, the observer needs a fresh rest window before it grades the task. - `--replay` now fires the recorded faults that crossed the task, at the same offsets, including the faults that other tasks requested. - A failed workspace-operation read is now a sweep error. It cannot give a clean rest. - R2 now checks that workspace files from before the first fault still exist at rest. - B5 now needs the completion record that `bench.sh` writes for each task. - After the second round of reviews: - An idle `todo` or `in_progress` task with no live run and no named wait is no longer treated as resting. The observer keeps watching it until the task resumes or passes the R5 stall limit. - A failed injected fault now fails the wave with exit 2. - T-oracle commands run agent-editable code. They now run under a macOS Seatbelt sandbox with a scratch `HOME` and no network, and they cannot read the operator's home directory. On other platforms these commands are refused unless the operator explicitly allows them. - R4 now counts every active run. A wake comment may be served again only after a run that did not succeed. - Evidence redaction now covers every environment variable name that ends in `KEY`, `TOKEN`, `SECRET` or `PASSWORD`. - If the pre-fault snapshot fails, only R2 for that task is left unchecked. The wave continues. - B5 also needs bench progress that the harness observed while it polled. - Add `doc/plans/2026-11-06-control-plane-restart-chaos.md` and its campaign ledger. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 38 tests pass. The tests include observer tests with a fake API, a real `bench.sh` run, and a sandbox test that cannot read a canary secret. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - `server/scripts/typecheck-layers.test.mjs`: 45 tests pass, including "covers every project under tests/". - A supervisor check with no credentials passed. K1 was ready again in about 6 seconds. K2 with a 3-second hold was ready again in about 9 seconds. Company data stayed readable. - One paid smoke wave ran: `--env local --adapters codex,claude,opencode --workloads A1,B1,C1 --faults K1 --tasks 3 --concurrency 2 --seed 7`. The harness worked from start to end, and all three tasks crossed a restart. The 0/3 result comes from product failures, which the campaign ledger records. A scan found no provider key in the results. - To run a wave: ```sh pnpm --filter @paperclipai/paperclip-runner build:runner-binaries set -a; source ~/.secrets; set +a node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude,opencode --concurrency 4 --tasks 8 --faults K1,K2,K4 --seed 42 node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --replay <wave>/<task> ``` ## Risks - Low product risk. The change adds test tooling only and does not change product code. - A wave spends provider money. The company budget hard cap (`--budget-cents`, default 3000) is the only spend limit. The plan's per-wave and per-night refusal is not implemented yet. - Known gaps: - `--env daytona` is reserved. It fails with a TODO in `seed.ts`. The Daytona sandbox sweep for R6 is not implemented yet. - The project requests isolated git-worktree workspaces. A B1 task still ran in the shared fixture repository. Concurrent B1 and B5 tasks can share files. The oracles use marker files for each task. - Only K1, K2 and K4 at seeded random times are available. Boundary triggers (B1 to B16), other faults and the UI probe come in Phase 2. - The paid smoke did not exercise B5 or K4. - On macOS, B1 oracle commands need `python3` outside the operator's home directory, for example `/usr/bin` or Homebrew. On Linux they are refused until a sandbox is added. - The B5 proof is best-effort against a deliberately adversarial agent, because that agent can write to the workspace. - Startup phase timings come from polling `/api/ready`. The server records its own timings only as OpenTelemetry spans. - `pnpm verify` and the CI Vitest jobs do not select these unit tests. The root Vitest projects cover workspace packages only. CI typechecks the files, but the tests run only through the vitest command above. - After the smoke, rest detection and the review fixes changed. A recovery action that the board owns now counts as a rest, and R6 reports it. A paid wave did not run again after these changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code as an agent with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
107 lines
2.0 KiB
Plaintext
107 lines
2.0 KiB
Plaintext
node_modules
|
|
node_modules/
|
|
**/node_modules
|
|
**/node_modules/
|
|
dist/
|
|
dist-preview/
|
|
dist-flow-preview/
|
|
server/ui-dist/
|
|
server/dist-runtime/
|
|
packages/paperclip-runner/runner/target/
|
|
ui/storybook-static/
|
|
ui/storybook/generated/
|
|
ui/storybook/.storybook/preview.tsx
|
|
maps/generated/
|
|
.env
|
|
*.tsbuildinfo
|
|
drizzle/meta/
|
|
.vite/
|
|
coverage/
|
|
.DS_Store
|
|
._*
|
|
.AppleDouble
|
|
.LSOverride
|
|
data/
|
|
.paperclip/
|
|
.pnpm-store/
|
|
tmp-*
|
|
cli/tmp/
|
|
|
|
# Scratch/seed scripts (but not scripts/ dir)
|
|
check-*.mjs
|
|
!.github/scripts/check-*.mjs
|
|
!.github/scripts/tests/check-*.mjs
|
|
!scripts/check-*.mjs
|
|
new-agent*.json
|
|
newcompany.json
|
|
seed-*.mjs
|
|
server/check-*.mjs
|
|
server/seed-*.mjs
|
|
packages/db/seed-*.mjs
|
|
|
|
# npm publish build artifacts
|
|
cli/package.dev.json
|
|
|
|
# Build artifacts in src directories
|
|
server/src/**/*.js
|
|
server/src/**/*.js.map
|
|
server/src/**/*.d.ts
|
|
server/src/**/*.d.ts.map
|
|
tmp/
|
|
feedback-export-*
|
|
diagnostics/
|
|
|
|
# Editor / tool temp files
|
|
*.tmp
|
|
.vscode/
|
|
.claude/settings.local.json
|
|
.paperclip-local/
|
|
.paperclip-runtime/
|
|
/.idea/
|
|
/.agents/
|
|
|
|
# Personal agent instructions
|
|
/AGENTS.local.md
|
|
|
|
# Doc maintenance cursor
|
|
.doc-review-cursor
|
|
|
|
# Playwright
|
|
tests/e2e/test-results/
|
|
tests/e2e/playwright-report/
|
|
tests/runner-e2e/results/
|
|
tests/restart-chaos/results/
|
|
.env.runner-e2e.local
|
|
tests/release-smoke/test-results/
|
|
tests/release-smoke/playwright-report/
|
|
test-results/issue-detail-perf/
|
|
tests/storybook-visual/.cache/
|
|
tests/storybook-visual/.snapshots/
|
|
tests/storybook-visual/baseline-review/
|
|
tests/storybook-visual/test-results/
|
|
tests/storybook-visual/repository-skill-results/
|
|
tests/storybook-visual/playwright-report/
|
|
.superset/
|
|
.superpowers/
|
|
.claude/worktrees/
|
|
.herenow
|
|
.vercel/
|
|
.env.local
|
|
test-results/
|
|
|
|
# Retained local lifecycle behavior measurements
|
|
.lifecycle-baseline/
|
|
# Copilot assets are generated from pinned platform packages during installation.
|
|
/packages/paperclip-runner/provider-assets/
|
|
/server/provider-assets/
|
|
|
|
# Local native Runner test/session state
|
|
.paperclip-runner-prp/
|
|
|
|
# Opt-in CI timing artifacts
|
|
.ci-profile/
|
|
.ci-test-selection/
|
|
|
|
# Content-addressed CI task results
|
|
.turbo/
|