0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-07 06:16:06 -10:00

Files

T

DottaandClaude Opus 5.5 871a9b3180 test(restart-chaos): add Phase 1 control-plane restart chaos harness (#19477 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The control plane owns task execution, recovery and Retry for every
agent run
> - Server restarts were the largest remaining source of stalled tasks
in the staging soak campaign
> - The soak could only observe restarts that staging happened to
perform, so a failure could not be repeated
> - This pull request adds a script harness that causes the restarts on
purpose and grades each task on fixed rules
> - The benefit is a repeatable restart failure with a replay command
and an evidence bundle for each fixer

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The restart test coverage. Today the runner-e2e restart cells test one
fixed restart point per cell. They cannot run many tasks at once, cannot
choose a random fault time, and cannot replay a failure.

**Subsystem affected**
Test tooling (`tests/restart-chaos/`). It reuses the runner-e2e process
code in `tests/runner-e2e/`. Product code does not change.

**Current behavior**
Restart robustness is checked through a small set of fixed cells. A
random restart during a real task has no defined pass rule.

**Proposed behavior**
A harness starts an isolated control plane and seeds a "Chaos Lab"
company. It launches seeded workloads, injects K1, K2 or K4 faults at
seeded times, and grades each task on rules R1 to R6 plus a
deterministic task oracle. `--replay <wave>/<task>` repeats one task.

**Reason and benefit**
Fixers get a replay command and an evidence bundle for each failure
signature. The first smoke wave already found three product signatures
(dead Retry after a restart, a false over-budget stop after restarts,
and an OpenCode turn killed mid-turn that ends as a provider failure).

**Breaking changes**
None. The change adds new test files, one entry in the test-project
typecheck list and a gitignore entry.

## What Changed

- Add `tests/restart-chaos/`:
- `supervisor.ts` owns one isolated server. It reuses
`replaceController`, the process-tree stopper, the environment isolation
helpers, the config seed, the port reservation, the wrapper-owned
Postgres and `server-entry.ts` from `tests/runner-e2e/`. It does not
copy them. Faults: K1 (SIGKILL, immediate restart), K2 (SIGTERM,
restart) and K4 (SIGKILL, 2 to 5 minute hold). Postgres and the
controller runtime cache stay across a fault.
- `seed.ts` creates the company, encrypted provider secrets, native
Codex, ACPX Claude and OpenCode agents from the runner-e2e catalog, and
the `chaos-sandbox` project. The project is a git repository with a
Python CLI, one failing test and a four-minute `bench.sh`.
- `scheduler.ts` makes a seeded plan of workload, adapter, fault, fire
time, hold and nonce for each task. It saves the plan so that `--replay`
can run one task again.
- `observer.ts` polls each issue and its runs. When a task stops and
offers Retry, it sends `POST /api/issues/:id/retry` one time, with the
same body as `useTaskRetry.ts`.
- `grader.ts` grades R1 to R6 as pure functions. `workloads.ts` defines
workloads A1, B1, B5 and C1 and their oracles.
- `evidence.ts` and `ledger.ts` write an evidence bundle for each
failure, a JSONL ledger and `SUMMARY.md`. All results go to the
gitignored `tests/restart-chaos/results/`. Known provider key values are
redacted.
- Unit tests: `scheduler.test.ts` (determinism, replay and CLI parsing)
and `grader.test.ts` (R1 to R6 and the oracles on fixture timelines).
These tests need no credentials.
- Add `tests/restart-chaos` to the test-project typecheck list
(`scripts/typecheck-test-projects.mjs`), with a `tsconfig.layers.json`
like `tests/runner-e2e`. The harness runs through direct script
invocations, not `package.json` scripts.
- After the first Greptile review:
- A failed fault or an unexpected server exit now gets one bounded
recovery. If the recovery fails, the wave fails and does not hang.
- Incomplete cleanup is now a harness error, and the harness keeps the
instance.
- After the C1 follow-up comment, the observer needs a fresh rest window
before it grades the task.
- `--replay` now fires the recorded faults that crossed the task, at the
same offsets, including the faults that other tasks requested.
- A failed workspace-operation read is now a sweep error. It cannot give
a clean rest.
- R2 now checks that workspace files from before the first fault still
exist at rest.
- B5 now needs the completion record that `bench.sh` writes for each
task.
- After the second round of reviews:
- An idle `todo` or `in_progress` task with no live run and no named
wait is no longer treated as resting. The observer keeps watching it
until the task resumes or passes the R5 stall limit.
  - A failed injected fault now fails the wave with exit 2.
- T-oracle commands run agent-editable code. They now run under a macOS
Seatbelt sandbox with a scratch `HOME` and no network, and they cannot
read the operator's home directory. On other platforms these commands
are refused unless the operator explicitly allows them.
- R4 now counts every active run. A wake comment may be served again
only after a run that did not succeed.
- Evidence redaction now covers every environment variable name that
ends in `KEY`, `TOKEN`, `SECRET` or `PASSWORD`.
- If the pre-fault snapshot fails, only R2 for that task is left
unchecked. The wave continues.
- B5 also needs bench progress that the harness observed while it
polled.
- Add `doc/plans/2026-11-06-control-plane-restart-chaos.md` and its
campaign ledger.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
38 tests pass. The tests include observer tests with a fake API, a real
`bench.sh` run, and a sandbox test that cannot read a canary secret.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.
- `server/scripts/typecheck-layers.test.mjs`: 45 tests pass, including
"covers every project under tests/".
- A supervisor check with no credentials passed. K1 was ready again in
about 6 seconds. K2 with a 3-second hold was ready again in about 9
seconds. Company data stayed readable.
- One paid smoke wave ran: `--env local --adapters codex,claude,opencode
--workloads A1,B1,C1 --faults K1 --tasks 3 --concurrency 2 --seed 7`.
The harness worked from start to end, and all three tasks crossed a
restart. The 0/3 result comes from product failures, which the campaign
ledger records. A scan found no provider key in the results.
- To run a wave:

  ```sh
  pnpm --filter @paperclipai/paperclip-runner build:runner-binaries
  set -a; source ~/.secrets; set +a
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env
local --adapters codex,claude,opencode --concurrency 4 --tasks 8
--faults K1,K2,K4 --seed 42
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--replay <wave>/<task>
  ```

## Risks

- Low product risk. The change adds test tooling only and does not
change product code.
- A wave spends provider money. The company budget hard cap
(`--budget-cents`, default 3000) is the only spend limit. The plan's
per-wave and per-night refusal is not implemented yet.
- Known gaps:
- `--env daytona` is reserved. It fails with a TODO in `seed.ts`. The
Daytona sandbox sweep for R6 is not implemented yet.
- The project requests isolated git-worktree workspaces. A B1 task still
ran in the shared fixture repository. Concurrent B1 and B5 tasks can
share files. The oracles use marker files for each task.
- Only K1, K2 and K4 at seeded random times are available. Boundary
triggers (B1 to B16), other faults and the UI probe come in Phase 2.
  - The paid smoke did not exercise B5 or K4.
- On macOS, B1 oracle commands need `python3` outside the operator's
home directory, for example `/usr/bin` or Homebrew. On Linux they are
refused until a sandbox is added.
- The B5 proof is best-effort against a deliberately adversarial agent,
because that agent can write to the workspace.
- Startup phase timings come from polling `/api/ready`. The server
records its own timings only as OpenTelemetry spans.
- `pnpm verify` and the CI Vitest jobs do not select these unit tests.
The root Vitest projects cover workspace packages only. CI typechecks
the files, but the tests run only through the vitest command above.
- After the smoke, rest detection and the review fixes changed. A
recovery action that the board owns now counts as a rest, and R6 reports
it. A paid wave did not run again after these changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in
Claude Code as an agent with tool use (shell, file edits) and extended
thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>

2026-11-06 10:45:52 -06:00

4.9 KiB

Raw Blame History

Restart chaos ledger

Companion to 2026-11-06-control-plane-restart-chaos.md. One section per wave. Classes: P product · Q agent quality · T test/harness · E unrelated environment.

Phase 0 — 2026-11-06

Master: 6cafcaf89b. Host: macOS, local runner (Daytona image pending a CI build).

Check	Result
`controller-replacement-smoke.ts` (no agent)	PASS, 39 s, Postgres survived replacement
Local restart cells, first run	0/7 — T: worktree `node_modules` older than lockfile (UI missing `mdast-util-from-markdown`). Not counted.
Local restart cells, rerun	2/7 (below)
Daytona runner image	Blocked. Local build failed: stale Dockerfile lock pin, then the Docker VM ran out of disk. Moved to CI (`runner-full-stack-e2e.yml`, manual dispatch) after PR #19467.

Local baseline rerun (`local-2026-11-06T14-27-42-229Z`)

Cell	Result	First error	Fixer
stock-harness · Codex · continuity-restart	PASS	—	—
stock-harness · ACPX Claude · continuity-restart	PASS	—	—
everyday-workflows · Codex · recover-controller	FAIL (cleanup_failure)	`kill EPERM` during owned-tree cleanup at the restart	subagent: EPERM
everyday-workflows · ACPX Claude · recover-controller	FAIL	Timed out waiting for an active run with saved source	subagent: Claude recover
stock-harness · OpenCode · continuity-restart	FAIL	Timed out waiting for the chat turn to settle to waiting	subagent: OpenCode
local-session-integrity · OpenCode · question-restart-resume	FAIL	Timed out waiting for the pending user question	subagent: OpenCode
completion-updates · Codex · handoff-completion-restart	FAIL	Timed out waiting for the worker to wait while the originating chat is idle	subagent: handoff

Signatures found

ID	Class	Signature	Status
RC-01	T	Harness looked for saved source only at the workspace root; ACPX Claude saved `slugify-tool/slugify.py`, so the restart never fired	PR #19471 (open)
RC-02	P	ACPX turn active at restart: runner closes it with `acpx_active_turn_recovery_closed` and no error, the control plane reports a generic "Provider turn failed", no continuation, queued follow-up unconsumed (fails R1). Codex has had this handled since #15297; ACPX is excluded by `provider.kind === "codex"` and `driverKind === "codex_app_server"` gates	fixer running

| RC-03 | T | macOS killpg returns EPERM (not ESRCH) on a zombie-only group; the supervisor rethrew it and aborted the restart. Fixed cell passed 2/2, and native Codex recovery itself works | PR #19470 (open) | | RC-04 | P (latency) | Graceful shutdown always waits the full 30 s heartbeat finalizer drain when a native run is restart-suspended (server/src/index.ts ~L2032, drainActiveRunExecutions). Every graceful restart (K2) or deploy takes ≥30 s and, under the harness, escalates to SIGKILL | queued for a fixer |

| RC-05 | P | Dead Retry after restart (fails R3): Retry accepted but no run for >15 min, stuck at retryWait.reason=execution_recovery. After the next restart the Retry vanished (waiting_for_user, canRetry=false). OpenCode t01, Claude t02 | fixer running (with RC-06) | | RC-06 | P | False over-budget stop after restarts: a new issue was blocked "original recovery target is over budget" at 22¢ of 500¢, then a company hard_stop with pendingRunCount=1, reservationCents=0, unpricedUsagePolicy=block (recovery/service.ts ~L4826). Codex t03 never ran | fixer running (with RC-05) | | RC-02b | P | OpenCode mid-turn kill also ends as a provider failure (t01). Sent to the RC-02 fixer | with RC-02 |

Harness smoke `w-20261106T144559-s7` (seed 7, K1 ×3, local, concurrency 2)

0/3 pass. The harness worked: every failure is product-side (RC-02b, RC-05, RC-06). Branch worktree-agent-a826af5179b88689b @ e864139cec, not yet a PR.

Setup note: fresh worktrees need pnpm install --frozen-lockfile and a built paperclip-runnerd (pnpm --filter @paperclipai/paperclip-runner build) before native cells can run. The chaos harness must check this before starting a wave.

Classification of the remaining cells is pending each fixer's diagnosis. A timeout before the restart point counts against setup or the agent, not restart robustness.

Side findings

Nightly Runner Full-Stack E2E never builds a Daytona image. Its operating-overhead allowance (RUNNER_E2E_OPERATING_OVERHEAD_JSON) expired 2026-11-03, and the nightly sets needs_daytona=false anyway. PR #19467 (merged, 7e6d882558) names the error and removes the stale Dockerfile lock default. The nightly stays red until the allowance is renewed (Dotta's call).
Image build moved to CI: manual dispatch run 42240434011 (core-compatibility.runner-codex.daytona.message-marker), 2026-11-06 15:09Z.
tests/runner-e2e/workflow-security.test.ts has 5 local failures in pr-trusted.yml and chat-catalog-testops-case.yml. Not investigated.

4.9 KiB Raw Blame History Unescape Escape