mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-07 06:16:06 -10:00
master
2260
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
b5b972ceb1 |
fix(runner): stop provider tool processes that leave the owned process group (#19727)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The local runner (`paperclip-runnerd`) supervises each provider process, such as the ACPX sidecar for Claude, in its own process group > - Run teardown stops that process group, and nothing else > - Claude Code starts each Bash tool command in a new session (`setsid`). A command such as `nohup ./bench.sh & echo $!` leaves a job in that session, and the wrapper exits at once. The job is then a child of init and outside the owned group > - In a restart chaos storm, a run failed after a restart resume. Its tool job kept running beside the task's next run, in the same workspace, for more than four minutes > - This pull request makes the supervisor record every process that its provider starts, including processes that leave the group, and stop them all before teardown reports success > - The benefit is that a run's tools cannot outlive the run and cause duplicate side effects in the next run ## Linked Issues or Issue Description **What happened?** A restart chaos storm ran a local ACPX Claude task. After a restart resume, the agent ran `nohup ./bench.sh <nonce> > log 2>&1 & echo "started pid $!"` and then ended its turn without a result. The run failed with `native_finalization_missing`. The runner tore down the sidecar group, and the guardian killed the provider group. `bench.sh` was in the session that Claude Code made for the tool (pgid and sid = the exited wrapper, ppid = 1). It was never signalled. It kept forking `sleep 1` while the task's next run started and finished in the same workspace. **Steps to reproduce** 1. Run a local ACPX Claude task whose agent runs a Bash tool command such as `nohup ./bench.sh x > log 2>&1 & echo $!`. 2. End the run (any outcome, including failure after a restart resume). 3. Run `ps -axo pid,ppid,pgid,command | grep bench.sh`. The job is still running with ppid 1. **Agent adapter(s) involved** Claude Code (local ACPX). The gap is in the shared supervisor, so any provider whose tools leave the group is affected. **Root cause** `SupervisedProcess` (`packages/paperclip-runner/runner/crates/runner-core/src/process_supervisor.rs`) spawns with `process_group(0)` and stops only that group in `terminate_group`, `terminate_group_immediately`, `wait` and `Drop`. Its own comment said that a caller that must exclude escaped or re-parented descendants needs a separate lifetime fence. No caller had one. The ACPX sidecar transport (`acpx_sidecar_transport.rs`, `shutdown` and `poison`) and the ACPX provider guardian (`installation-integrity.ts`, which kills only its own group) therefore left every `setsid` tool job behind. The failure path after the resume was not special: the same gap applies to every teardown. **Expected behavior** When a run ends for any reason, every process that its provider or tools started stops, and teardown proves that they are gone. Identity comes from PID, group, session and start time, never from a command name. ## What Changed - Add the `paperclip-process-table` crate. It reads PID, parent, process group and start time for every process (libproc on macOS, `/proc` on Linux). `runner-core` forbids unsafe code, so the two checked libproc calls live in this small crate. - Add `process_lineage.rs` to `runner-core`. A sampler thread (every 50 ms) records each descendant of the supervised process by PID and start time. It also records each group and session that a descendant leads, with the leader's start time. A process belongs to the lineage when its parent belongs, or when it is in a recorded group or session and started no earlier than its leader. Groups and sessions that the supervised process only inherited (runnerd's session) are never recorded. A recorded ID whose PID now names a different process is dropped. - `SupervisedProcess` samples the lineage just before it signals, and after the leader is reaped it stops every live member: TERM, then KILL after the shutdown grace. It fails with an error until it proves that no member is alive (grace plus 5 s). This applies to `terminate_group`, `terminate_group_immediately`, `wait` and `Drop`. - If the lineage sampler cannot start, the supervisor stops the process it just started and reports the failure, so a provider never runs without tool cleanup. - Table reads and state updates happen under one lock, so an older read cannot erase members that a newer read recorded. - The process table reports processes that exist but cannot be read. A recorded member that cannot be read stays unresolved, and teardown fails until its exit is proven. - The Runner workspace now has two members. `scripts/runner-release-crate-cache.mjs` and `scripts/runner-debug-crate-cache.mjs` cache and mark current the process-table lib unit with the Runner crate (otherwise Cargo rebuilds the Runner crate after every restore). The debug contract accepts only this library-only member and still refuses any other workspace change. The workflow cache path lists and the debug clean step are updated to match. - `terminate_group_immediately` now accepts `EPERM` from the group signal when the leader has already exited. macOS returns `EPERM` for a group whose only member is the unreaped leader. The lineage pass then proves that no other member remains. ## Verification - `cargo test -p paperclip-runner-core --lib -- process_lineage tool_job immediate_group_fence` (macOS, local). - New real-process tests in `process_supervisor.rs`: a supervised shell starts a wrapper with `setsid` (Perl `POSIX::setsid`), the wrapper runs `nohup <loop> &`, writes the job PID and exits. The test asserts that the job is outside the owned group and re-parented, then calls `terminate_group`, `terminate_group_immediately` or drops the process, and asserts that the job is gone when the call returns. - Red: with the tracker disabled, all three fail with `tool job <pid> outlived its provider`. - Green: all three pass. - Unit tests in `process_lineage.rs` cover a re-parented job, a job that left its group but stayed in the recorded session, a session made after the first sighting, PID reuse, and no claim through the supervisor's inherited session. - `node --test scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/runner-debug-crate-cache.test.mjs` (new case: the process-table library is cached with the crate, and any other member still needs review). - `cargo test -p paperclip-process-table` (includes an exited process reported as gone), and `cargo test -p paperclip-runner-core --lib` (570 passed). `acpx_provider_backend::tests::unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend` fails sometimes under full-suite load. It fails at the same rate with the tracker disabled, so it is a flake that already existed. - Sampler cost: one table read takes about 0.4 ms for 1,200 processes on macOS (about 0.8% of one core for each supervised process). - The restart-chaos replay (`--replay w-20261107T023921-s7106/t67`) did not run. The local session was not permitted to load provider credentials. It is not proven against a live Claude run. ## Risks - Teardown now stops all the processes that a provider started, also intentional background jobs. This is the intended contract: nothing outlives the run. - A wrapper that lives less than one sample (50 ms) and orphans a job before the sampler sees it is not recorded. Claude Code's wrapper sources a shell snapshot and lives for seconds. On Linux, a child-subreaper would close this gap. That is not in this change. - When `runnerd` itself is killed with SIGKILL, this code does not run. The provider guardian still kills its own group, but escaped tool sessions are not stopped in that case. - Teardown can return a new error when it cannot prove the lineage stopped. Callers already handle `terminate_group` errors as provider shutdown failures. - The server does not yet gate the next run on the tool lineage of the prior run directly. It relies on the runner finishing its teardown before the runner exits. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
b2af45f0f1 |
fix(runner): let OpenAI API-key Codex agents generate images with their existing key (#19654)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - New users often connect a Codex agent with an OpenAI API key during
onboarding.
> - A user asked such an agent for a logo. The agent said that the
built-in image generator was not available and that its fallback needed
a separate API key. It then drew an SVG by hand.
> - #19516 removed `--disable image_generation`. That change is correct,
but it does not reach this path. The pinned Codex 0.160.0 shows its
`image_gen` tool only when Codex signs in with ChatGPT. The check is
`image_generation_available` in `codex-rs/core/src/tools/spec_plan.rs`.
An API key never gets the tool.
> - The runner keeps the provider key out of the agent's shell on
purpose, so the skill's `scripts/image_gen.py` fallback cannot run.
> - This pull request adds a Paperclip `generate_image` tool. The server
makes the image with the agent's existing OpenAI key. The key never goes
to the agent process or the shell.
> - The benefit is that API-key Codex agents can make images with no
second key. The spend goes to the agent's budget and the budget can stop
it.
## Linked Issues or Issue Description
Refs #19516
**What happened?**
An agent connected through onboarding with an OpenAI API key (Codex
harness, `gpt-5.6-sol`) could not generate images. The agent said: "The
image skill's built-in generator isn't available in this run, and its
fallback would require a separate API key." It then drew an SVG instead.
**Steps to reproduce**
1. Create a company with a Codex agent that uses an OpenAI API key.
2. Ask the agent for a raster image, for example a logo PNG.
3. The agent reports that image generation is not available. It does not
make the image.
**Expected behavior**
The agent makes the image with the OpenAI key that already powers it.
**Root cause**
The agent ran on the native runner's Codex app-server path, with
`auth.json = {"OPENAI_API_KEY": ...}`. Codex 0.160.0 removes
`image_gen.imagegen` from the model's tools unless
`current_auth_uses_codex_backend()` is true. That function is true only
for a ChatGPT sign-in.
I proved this with the real pinned binary. I gave it the runner's
isolated launch arguments and the API-key auth file, and pointed it at a
local Responses endpoint. The first model request offers only `exec`,
`wait`, `request_user_input` and the collaboration tools. #19516's test
checked only `codex features list`, so it could not see this gate.
## What Changed
- **New `server/src/services/native-runtime/generate-image-tool.ts`:**
- The tool schema and definition.
- `imageGenerationBindingForRun`. It applies only to native Codex runs
on a direct OpenAI `api_key` connection with no router selection.
- The OpenAI Images API call (`gpt-image-2`, `b64_json`, PNG check, 180
s timeout). The reply is read as a stream with a byte cap before it is
parsed. A decoded image larger than 20 MiB is refused.
- A reviewed rate card for `gpt-image-2`: text input $5, image input $8
and image output $30 per 1M tokens, from
developers.openai.com/api/docs/pricing, retrieved 2026-11-06.
- A create-only, symlink-confined local write.
- **`paperclip-runner-tool-authority.ts`:**
- Adds the `imageGeneration` binding.
- Advertises the tool only for standard mode with a workspace. A remote
target must also have an installer. Dot, Muse, Grok and native review do
not get the tool.
- Adds the `#generateImage` handler. Before any spend, it checks again:
the run and task, budget authority, `issue:mutate`, case handoff, and
the connection.
- Stores a claim-before-spend per-call receipt in
`resultJson.imageToolReceipts`. A retry or recovery replays the receipt
and does not pay again.
- Saves a `charged` receipt (usage, price, request ID and charge time)
as soon as OpenAI returns. A failed save, failed accounting write or
crash can then still settle the spend. A retry or recovery records the
cost once and never buys again.
- Records one `cost_events` row through `createCostEventInTransaction`.
The row is `metered_api` with `costStatus: "estimated"`, and the budget
is evaluated. The row is recorded even if saving fails.
- Checks the task's project budget, as well as the company and agent
budgets, before spending.
- Writes content-free `runner.image_generate` activity entries: started,
completed, failed and unconfirmed.
- **`ai-connections.ts`:** adds `selectImageGeneration`. It checks again
that the connection is an active, healthy OpenAI `api_key` connection
with no custom routing. It also checks the responsible user's
membership, the grant audience and the agent or company installation.
These are the same rules that runtime selection uses.
- **`native-session-executor.ts`:** gets the binding from the run's
admitted `contextSnapshot.aiConnection`. The value comes only from the
server, never from tool input.
- **`native-session-resume.ts`:** the native tool contract changes from
`v18` to `v19` and now lists `generate_image`. A persisted provider
thread cannot add tools when it resumes, so older threads must rotate.
- **`doc/runner-api-tools.md`:** a new section about image generation
for OpenAI API-key Codex agents.
- **#19516: kept, not reverted.** Its changes have no effect on the
API-key path. They are still correct for Codex agents that sign in with
ChatGPT, because `image_gen` works there and saves to
`CODEX_HOME/generated_images`. Those agents need the read grant that
#19516 added. On the API-key path, the new tool replaces the built-in
tool. The tool's description tells the agent to use it instead of
`image_gen` or `scripts/image_gen.py`.
## Verification
**Red on `master` (
|
||
|
DottaandClaude Opus 5.5
|
5eac099fbb |
fix(agents): batch independent follow-up questions into one card (#19634)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents ask users for input with the `request_human_input` tool or
the `ask_user_questions` interaction. Paperclip shows each call as one
question card.
> - In an onboarding walk, a Codex execution agent asked five
independent questions (name, menu and prices, contact, style, output).
It posted five cards, one after the other. Each card came 20-60 s after
the previous answer.
> - The tool description said: "For one question at a time, ask only the
next unanswered question and wait for its real answer." Agents read this
as a rule. The paperclip skill showed only a one-question example and
did not tell agents to batch.
> - No server cap exists. A question set accepts up to 64 questions.
> - This pull request changes the guidance. Agents now ask independent
inputs together in one card. They use a follow-up card only when a
question depends on an earlier answer. One-at-a-time stays available
when the user asks for it.
> - The benefit is fewer waits for the user: one card and one agent turn
instead of five.
## Linked Issues or Issue Description
**What happened?**
An execution agent asked a user five independent follow-up questions in
five separate question cards. Each card appeared only after the user
answered the previous card and the agent finished another turn (20-60 s
each).
**Expected behavior**
When an agent needs several independent inputs, it asks them together in
one multi-question card. The onboarding interview already does this ("1
of 4" pager).
**Steps to reproduce**
1. Complete onboarding on the Codex API path and hand off a task such as
"Create printable cake menu and logo".
2. Watch the execution agent's first turns.
3. See five sequential single-question cards.
**Paperclip version or commit**
master at
|
||
|
|
5a92d53f7d |
fix(adapter-utils): handle closed child stdin safely (#14057)
Handle closed child stdin without losing the child result. Keep each child stdin error handler scoped to that child. Expected EPIPE and stream-destroyed failures no longer crash the host. The child close event remains the source of its exit result. Verification: 2,024 tests passed and 12 live/platform cases skipped across 87 complete adapter-utils suites. The final 140-test server-utils suite passed. The regression reproduces the EPIPE crash on unchanged master. Final qualification: all applicable CI passed and exact-head Greptile Apex scored 5/5 with no unresolved findings. Co-Authored-By: Grzegorz Golaś <[email protected]> Co-Authored-By: Devin Foley <[email protected]> Co-Authored-By: Paperclip <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
2468b3f907 |
fix(native): tell a restart continuation that earlier processes may be gone (#19804)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a control-plane restart loses a turn, the native runtime sends a same-conversation continuation (#19501, #19699, #19771) > - A restart can also kill shell or background processes that earlier turns started > - In three restart storms, the continuation agent could not inspect its killed `bench.sh` and set the task to blocked > - The continuation prompt told the agent not to "blindly rerun an action whose outcome is unknown", so the agent did not re-run the command > - This pull request tells the agent that earlier processes may be gone, and to check the workspace outputs and re-run a command whose expected output is missing > - The benefit is that a task can finish after a restart instead of blocking on a process that no longer exists ## Linked Issues or Issue Description Refs #19501, #19699, #19771 **What happened?** Restart chaos workload B5 (Codex and Claude) failed this way three times: storm 6b t14, s7111 t16 and storm 10 t12. The agent started `bench.sh` in the background. A restart killed it. The restart continuation agent tried to find its tool session and got "process inspection is denied". It then blocked the task. The comment read: "Blocked by the Paperclip runtime restart: the benchmark session is unavailable…". It added that it "cannot verify whether the original run is still active or recover its output, so I have not rerun it". **Expected behavior** The agent knows it is in a resumed session after a restart. It checks the workspace for the command's outputs. If the expected output is missing, it re-runs the command and finishes the task. **Root cause** `packages/paperclip-runner/src/native-session-runtime.ts`, `restartContinuation`: the prompt did not say that processes from earlier turns may be gone. It told the agent to inspect each unfinished command's "current state" and not to "blindly rerun an action whose outcome is unknown". With the process gone, the agent could not inspect it, so it blocked. ## What Changed - The restart continuation prompt now says: - This is a resumed session after a Paperclip control-plane restart. - Shell commands and background processes from earlier turns may no longer be running, and their tool sessions may be gone. Do not block only because one cannot be found or inspected. - Check the workspace for the outputs or artifacts the command should have produced. If the expected output is present, use it. If the expected output is missing, re-run the command. - Do not repeat completed work. Before repeating an action with external effects (a push, a message, a deployment), check whether it already took effect. If you cannot tell, report the blocker. - The prompt does not say that re-running is always safe. Process teardown (#19727) has not merged. - `doc/execution-semantics.md`: one additive sentence. ## Verification - `native-session-runtime.test.ts`: the three restart continuation tests (Codex, ACPX, OpenCode) now check the new guidance. They were red before the prompt change and are green after it. - Guard: "uses the admitted session cwd in the task envelope" now checks that fresh turns and ordinary resumed turns do not contain the restart guidance. - `npx vitest run src/native-session-runtime.test.ts src/drivers/codex/ src/live/runnerd-restart-continuation.test.ts` in `packages/paperclip-runner`: 603 passed. - `tsc -p tsconfig.json` and `tsc -p tsconfig.test.json` for `packages/paperclip-runner` are clean. - Not run: a live restart-chaos replay with a real model. ## Risks - Low. This changes only the prompt text of the restart continuation. Ordinary turns do not change. - An agent may now re-run a command whose earlier run had partial side effects. The prompt asks it to check external effects first, and to re-run only when the expected output is missing. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
ef83d566ca |
fix(native): ask for the result once after a restart continuation ends without one (#19771)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs continue a turn that a control-plane restart lost, in the same conversation (#19501, #19699) > - In restart storms 5, 6b and 8, ACPX Claude runs crossed restarts, ran the continuation, and then failed with `native_session_retry_exhausted` > - The continuation completed, but Claude ended it with "I'll wait for the notification when it completes" for a background command, and did not call `paperclip_finish` > - The next same-run attempt then sent no turn at all and failed with "session returned no semantic result" > - This pull request makes the continuation ask for the result, and asks once more in the same attempt if the continuation still ends without one > - The benefit is that a restart-interrupted task settles without a Retry, and the restarts that the run survived cannot block the request ## Linked Issues or Issue Description Refs #19501, #19699, #19720 **What happened?** Storm 8, run 84c532f7 and its Retry 0faead66 (ACPX Claude, task CHA-5). The steps are the same in both runs: 1. `resume_dead_runner`, then `session.resumed`, then `turn.failed` with `provider_turn_lost_on_restore` (recoverable). 2. The restart continuation runs. Claude starts the bench in the background and ends the turn with "I'll wait for the notification when it completes". The turn completes with no `paperclip_finish`. 3. Lifecycle: "provider turn completed without a semantic result; same-run disposition recovery persisted" (attempt 2). 4. 30 s later, attempt 3 restores the session. It sends no turn. After about 2 s it fails with `native_finalization_missing: session returned no semantic result`, and the attempt count makes it `native_session_retry_exhausted`. **Expected behavior** The continuation tells the agent to finish and report its result. If the continuation still completes without a result, Paperclip asks for the disposition one time. Restarts that the run already survived do not block that request. An agent that never reports a result still fails closed. **Root cause** - `packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts` `startTurn` (lines 224 and 248): the restart continuation uses the driver's one-shot disposition marker (`dispositionOnlyRecoveryAvailable`). #19699 uses the marker on purpose, so that a crash can adopt the accepted continuation turn. The marker then stays consumed with the continuation turn id. - `packages/paperclip-runner/src/drivers/codex/codex-session-terminal.ts` (line 226): only a *lost* continuation gives the marker back. A *completed* continuation with no result does not. - `packages/paperclip-runner/src/native-session-runtime.ts` (`dispositionRecoveryWasSubmitted`, about line 2761): on the next attempt, the consumed marker looks like a disposition request that already ran. Recovery replays the continuation's terminal and starts no turn. So the disposition request never happens. - `server/src/services/native-runtime/native-session-executor.ts` (lines 9755-9761 and `nativeSessionFailureDisposition` at 7768): the `semantic_result_missing` same-run retry charges a provider attempt on the same counter as restart resumes. In this run, attempt 3 was the last allowed attempt. - The continuation prompt (`restartContinuation`) did not tell the agent to report a result, or not to stop and wait for a background notification. ## What Changed - Codex-protocol driver (used by Codex and by ACPX through runnerd): when a restart continuation completes without a semantic result, the driver gives back the one-shot disposition marker. The disposition request is not a continuation, so it consumes the marker for good. - Native session runtime: after that, the same attempt sends one disposition request (`Paperclip semantic-result recovery after a restart continuation ... invoke exactly one paperclip_finish or paperclip_block`). It does this only when the driver shows that the continuation borrowed the marker and gave it back. Other drivers keep the current behavior. - Restart continuation prompt: it now says to finish in this turn, not to end the turn to wait for a background command or a notification, and to report one `paperclip_finish` or `paperclip_block`. - `doc/execution-semantics.md`: one additive paragraph. - No server change. The disposition request runs in the same attempt, so it uses no provider attempt. This avoids `native-restart-recovery.ts` (#19747). ## Verification Red before the fix, green after: - `codex-app-server-driver.recovery.test.ts` "keeps one disposition request after a restart continuation completes without a result". Red: marker stays consumed on `turn-2`. Green: the marker is given back, one disposition turn starts, and a resultless disposition turn does not re-arm. - `native-session-runtime.test.ts` "settles a restart continuation that ended without a result through one disposition turn (Codex driver)". Real Codex-protocol driver with a fake transport. Red: only one `turn/start`, then `native_finalization_missing`. Green: the disposition turn `turn-3` reports the result and the run completes. - `native-session-runtime.test.ts` "asks once for the disposition when a restart continuation ends without a result" and "fails closed when the driver keeps the continuation's disposition marker". Also run: - `npx vitest run src/native-session-runtime.test.ts src/drivers/codex/` in `packages/paperclip-runner`: 593 passed. - `src/live/runnerd-restart-continuation.test.ts` (real runnerd and fake Codex, killed runner): 5 passed. - `tsc --noEmit` for `packages/paperclip-runner`. - `pnpm verify` exits 2 because the doc change is a global CI input. Required CI supplies the broad coverage. Not run: a live replay of storm 8 t05 with Claude. The chaos harness needs provider keys from the operator's shell. ## Risks - Low to medium. A run whose continuation ends without a result now gets one more provider turn in the same attempt. The bound is one disposition request per continuation. Each continuation still needs an admitted restart, and the provider-attempt budget bounds those. - If the controller crashes after the continuation completes but before its checkpoint, the checkpoint keeps the stricter consumed marker. That case fails closed as before. - The prompt text changes for restart continuations. It does not change a protocol schema or a profile digest. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
fff70fe49e |
fix(runner): relaunch a provider whose runner died before admitting it (#19784)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through `paperclip-runnerd`. When the control plane restarts, recovery starts a new runnerd process, and that process restores the provider (for example the Codex app server) from durable state. > - Each provider launch writes a durable startup fence before it spawns the process. Admission clears the fence. The fence stops a second runner from launching a provider while an unadmitted one may still run. > - A restart chaos wave sent SIGTERM and then SIGKILL to the control plane in quick succession. The second kill landed after the restored Codex provider was spawned and before it was admitted. > - The fence stayed in state with phase `spawned`. Every later runner refused to restore. The recovery probe's `runner.drain` failed with "provider startup ownership remains unadmitted", and the task needed a manual Retry. > - This pull request lets a new runner retire that fence when the recorded provider process and its process group have provably exited. It keeps every other fence closed. > - The benefit is that a double restart during provider relaunch continues automatically. ## Linked Issues or Issue Description **What happened?** A Codex run was active. A graceful restart (K2) suspended it, and recovery started a new runnerd that began restoring the provider. That runnerd spawned the Codex app server (pid recorded, phase `spawned`). A hard restart (K1) 9 s later killed it before admission. At the next boot, recovery chose `resume_dead_runner` and queued its `runner.drain` probe. The new runnerd failed the probe with `command_execution_failed: provider startup ownership remains unadmitted`. The run ended `native_session_recovery_failed`, and the task needed a Retry. The kept provider state had `lifecycle: "turn_active"` and a `startupAttempt` with `phase: "spawned"`, `trigger: "restore"`, `processId` = `processGroupId`, and no recorded exit. That process no longer existed. **Steps to reproduce** 1. Start a Codex task on a local runner. 2. Restart the control plane gracefully during the turn. 3. Kill the control plane again while recovery relaunches the provider, before the provider is admitted. **Expected behavior** Recovery relaunches the provider and continues the run. **Actual behavior** Every later runner refuses to restore, and the run fails. ## What Changed - `provider_backend.rs`: after a runner loads durable provider state, `retire_orphaned_startup_attempt` clears the startup fence only when all of these hold: - the attempt is in phase `spawned` (neither admitted nor failed); - the session is still restorable (`session_open`, `turn_active` or `provider_exited`); - signal 0 returns `ESRCH` for both the recorded process and its process group. It records a `provider_startup_retired` diagnostic event with the attempt, persists the state, and lets the normal restore path relaunch the provider. - Intent-phase fences (OpenCode). Three OpenCode runs after a K6 crash loop, and other runs on master, stopped at phase `intent` with no recorded process. Under load, the runner spends seconds verifying launch artifacts after it records intent. An intent alone could not show whether a process was started. - A new intent now carries `beforeSpawn: true`. - A new `SpawnStarting` observation clears the mark immediately before every provider spawn site in `start_with_tools_observed`. That clear is fsynced, but it emits no extra diagnostic event. - An intent still marked `beforeSpawn` (in a restorable session) proves that no process was started, so the fence is retired. - An intent that reached the spawn, or one written before this mark existed, stays fenced. - A runner that predates the field rejects the state as unknown (`deny_unknown_fields`), which fails closed. - `validate()` only allows the mark on an intent or on a spawn-stage failure. - The fence stays closed for: - an intent that reached the spawn, or one without the before-spawn mark; - a failed startup (initialization or spawn); - a session that recovery closed; - a process or group that may still run; - a permission error or an invalid id. - The existing test `failed_authenticated_identity_commit_keeps_the_durable_startup_fence` now uses a live process group instead of the fixed pid 123. That pid might not exist on a CI host, so the test would no longer prove a running provider keeps the fence. ## Verification - Red before the fix: the new integration test `restart_relaunches_a_provider_whose_runner_died_before_admitting_it` (fake Codex app server; turn active; a `spawned` restore fence with an exited process group; then the recovery `runner.drain` probe) failed with `provider startup ownership remains unadmitted`, the incident error. After the fix it passes. It shows one more `thread/resume`, the retired diagnostic, and a cleared fence. - Unit tests (unix-only module): retirement for each restorable lifecycle, and for an intent still before the spawn. A `SpawnStarting` outside its intent is rejected. The fence stays closed for: - an intent that reached the spawn; - an intent without the before-spawn mark; - a real child that survives its killed leader (process-group probe, no stubs); - a failed spawn; - a failed admission; - a closed session; - a live provider; - a surviving process group. - `cargo test -p paperclip-runner-core --test codex_provider`: 119 passed. The startup event counts are unchanged, because the spawn-start mark emits no event. The existing fence tests (`failed_provider_startup_is_persistently_fenced_before_another_process_can_resume`, `durable_recovery_closes_a_provider_that_reopens_a_settled_turn`) still pass. - `cargo test -p paperclip-runner-core --lib`: 570 passed. `cargo check --workspace --tests` passes. - `cargo fmt --check` passes. ## Risks - Process ids can be reused. A reused id only makes a gone process look alive, so the fence stays closed. Retirement needs `ESRCH` for both the pid and the group. - A provider process that moved to a new process group is outside this check. Before admission the runner has sent it no turn, so it cannot repeat provider work. - Intent fences written by older runners, including the kept incident states, have no before-spawn mark. They stay fenced and still need the Retry. New launches carry the mark. - A runner-side change takes effect on local runners after the runner build. Daytona needs a new image. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, test execution). Default context window. Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
35d41f11bb |
fix(runner): keep Codex thread usage reported before a run's first turn as history (#19783)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A new Codex run can attach to a Codex thread that earlier runs used > - In restart storm 9, a comment-wake run attached to that thread after a restart. Before its first turn, it received the thread's usage from the earlier run > - The runnerd transport gave that usage an empty turn owner. The driver rejected it as an unowned authoritative notification and failed the run > - The task then stopped with `native_event_replay_conflict` and needed the next run to finish > - This pull request keeps usage that arrives before a run's first turn as history > - The benefit is that a run is not lost to earlier work's usage, and it is not billed for that usage ## Linked Issues or Issue Description Refs #19718, #19722 **What happened?** Restart storm 9 (seed 7111), task CHA-5 (Codex). Run A succeeded. A K2 restart fired. The comment-wake run B then started on a fresh runnerd and attached to the same durable Codex session. Before run B started a turn, runnerd committed a `usage.reported` event for the thread's earlier usage. The transport rehydrated it as `thread/tokenUsage/updated` with `turnId: ""`, because no turn was bound yet. `classifyCodexNotification` classifies an empty id as `invalid_authority`. The driver failed the run with "Provider authoritative notification thread/tokenUsage/updated did not name its execution owner." Recovery then stopped with `native_event_replay_conflict`. **Expected behavior** Usage that arrives before a run's first turn does not fail the run. If the session has no usage baseline yet, that usage is history, so the run is not billed for it. **Steps to reproduce** 1. Use fake Codex with `--usage-before-first-turn`. It reports thread usage right after `thread/start`. 2. Open a driver session over the real runnerd transport. Wait until runnerd commits the usage, then start a turn. 3. Before this change, the run fails with "did not name its execution owner". ## What Changed - `runnerd-codex-transport.ts` (`rehydrateRunnerdUsageNotification`): the transport omits `turnId` when no turn is bound. It no longer sends an empty owner. - `codex-session-notifications.ts`: usage reported before the session's first turn, with no baseline yet, sets the baseline (history). Usage during or after a turn keeps its existing accounting. - Fake Codex: new `--usage-before-first-turn` flag. - New test `src/live/runnerd-codex-pre-turn-usage.test.ts` with real runnerd, the real Codex driver and fake Codex. ## Verification - Red, run twice before the fix: the test failed with `NativeProviderTerminalFailure: Provider authoritative notification thread/tokenUsage/updated did not name its execution owner.` This is the live message. - Green, run three times after the fix: the turn completes. The run's usage `runDelta` is 7/2 (thread total 12/3 minus the 5/1 reported before the turn). - `npx vitest run src/drivers/codex src/live/runnerd-codex-transport src/native-session-runtime`: 991 of 992 tests pass. One settlement test timed out under parallel load. It passes alone (19/19), and this change does not touch it. - `pnpm run typecheck` in `packages/paperclip-runner` passes. `cargo fmt` is clean. - Not run: the live restart-chaos replay. It needs provider credentials, which this session does not load. ## Risks - Low risk. The accounting change applies only when no turn is active, no turn is pending, the session has no settled turns and there is no baseline. In that state, the usage cannot belong to this run's work. - The test reproduces the failing shape (usage committed before the first turn) with a fresh session. It does not reproduce the exact restart and run-attach sequence from the storm. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing). Default context window and reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
|
af0fe8fe81 |
Keep personal MCP connections functional while idle tenants sleep (#19427)
## Thinking Path > - Paperclip manages work for AI agents. > - Personal MCP connections should remain enabled while an idle tenant sleeps. > - Unfinished OAuth, transfer and delivery work must retain its owner and deadline. > - Cloud can receive bounded metadata and incoming work while the tenant remains the company-scoped authorizer. > - Current identity, revision and delivery acknowledgments establish when ownership has transferred. > - This change preserves useful wake, retries and revocation while making settled ownership inspectable without writes. ## Linked Issues or Issue Description Refs: #19377 and #19750. Hosted companion: [paperclip-cloud#1217](https://github.com/paperclipai/paperclip-cloud/pull/1217). **What happened?** Enabled personal MCP grants and subscriptions could keep an otherwise idle tenant awake. Ignoring those records would lose discovery, authorization, delivery and revocation obligations. **Expected behavior** Unconsumed authorization codes, device requests, file handoffs and delivery work retain their original deadlines. Settled connections remain enabled and become quiet after the current external owner acknowledges responsibility. Incoming useful work resumes normal authorization and delivery; revocation still takes effect for the current company. **Steps to reproduce** Complete normal PKCE/device consent and company selection, synchronize client/device and revocation state, retain the connection while idle, then exercise useful tools/events and revocation. Sleep eligibility requires current incoming/deadline ownership throughout. ## What Changed - Add bounded signed client/device metadata and revocation synchronization with exact revision, identity and ACK checks, startup recovery and finite legacy-device cutover. - Preserve company selection, authorization and operation identity. Register grant work before its INSERT in the same transaction; retain rollback, expiry and retry behavior. - Require current managed identity/resource before hosted publication. Receipt-cleared re-enable must obtain a fresh source ACK before device creation. - Reuse read-only settings and inspection budgets; connect MCP owners to normal startup, drain and shutdown. - Bound metadata pages and the actual signed request body while preserving the default limits of other operations. - Generate migration 0440 after the merged Legacy migration. Preserve all 438 existing executable migrations and the journal prefix; retain the normal five newest snapshots. - Preserve the merged transaction-owner forwarding and regression coverage from #19750 through normal ancestry integration. ## Verification - Current own Shared/SDK, database, eval-kernel and Runner builds passed, together with all seven normal application TypeScript programs. Missing generated package exports/workspace links were restored through normal setup; the original setup failures were retained. No source, compiler option, test assertion or timeout was weakened. - Normal migration generation changed only the five intended MCP tables; its SQL is byte-identical to the reviewed MCP delta. Fresh reference databases and both retained master/Muse histories upgraded through 0440, preserving historical hashes, row fields, schema and idempotence. Direct assertions cover the Legacy null default and MCP null/zero defaults. Two selected history cases passed; the other 21 database cases were unselected. - All six current MCP lifecycle cases passed with zero skips: normal PKCE approval intent, ACK recovery after outage, mutation during publication, disable/re-enable with original expiry, company-deletion rollback/commit and key rotation. - All 81 checks in five current whole files passed with zero skips: OpenAPI 34, read-only settings 2, diagnostic catalog 25, PostgreSQL diagnostics 7 and dependency scheduling 13. PostgreSQL fixtures ran normal migrations before test imports. - Earlier canonical proof at App `cd1c8c6` / Cloud `9f67531` includes real 60-second rate and three 15-second quiet windows, useful work and three revokes. Later paired proof at App `43af514` / Cloud `914eb0f` plus its reviewed candidate includes normal DCR, PostgreSQL metadata, a 144,017-byte signed publication, actual ACK, cap/default-413 controls and a real 15-second quiet window. The relevant producer/signing/wire inputs remain byte-identical in this integration; shared changes are qualified above. These remain source-scoped observations, not a fresh full canonical run on the integrated tree. - All known test owners, process groups and OS sockets closed; an independent final PostgreSQL audit found no remaining test sessions. One dependency-test trace recorded 46 opens and 45 close events. Its cause is unknown; this is not claimed as proof of graceful closure. - Fresh exact-head remote checks passed: 59 successful and 4 skipped contexts, with a complete current 5/5 Greptile review and all seven review threads resolved. Targeted local checks do not constitute a new whole-repository test run. ## Risks App and Cloud must agree on identity, revision, namespace, deadlines and revocation. Partial rollout, unavailable authority or missing ACK must retain awake debt. Shared startup/shutdown and settings changed since the earlier canonical profile, so historical observations carry only within their stated source scope. No deployed Railway/Neon suspension, Cloud alarm, savings or complete hosted Dot/Muse acceptance is claimed. ## Model Used OpenAI `gpt-6.1-sol` produced code and verification, with a Codex coordinator reviewing and integrating the changes. The coordinator's exact runtime model ID and context-window sizes were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]> Co-authored-by: Codex <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
ee1e1a3994 |
fix(runner): adopt the turn a dying controller's turn.start already started (#19772)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote Runner hosts the provider (here Claude through ACPX) and
keeps working while the control plane restarts
> - A controller can die after the Runner accepts its `turn.start` but
before the controller checkpoints the accepted turn or emits
`turn.accepted`
> - The provider keeps running that turn. On reattach, recovery saw no
checkpointed active turn, and the Runner's replayed `turn/started`
failed the session with "Provider started an unexpected turn"
> - The run then failed with `native_event_replay_conflict`, replay was
blocked, and a finished issue moved from done to blocked
> - This pull request lets recovery adopt that turn when the Runner's
authenticated command result proves it is the controller's own turn
> - The benefit is that a controller crash loop no longer fails a run
that the provider is still doing correctly, and a foreign turn still
fails closed
## Linked Issues or Issue Description
**What happened?**
In a restart chaos run (Claude through ACPX on a remote Runner, crash
loop fault), a follow-up comment woke a new run. The controller sent
`turn.start` and the crash loop began about one second later. After the
restarts, the controller reattached to the same Runner and failed with
"Provider started an unexpected turn". The run failed with
`native_event_replay_conflict` and recovery required workspace repair.
**What did you expect?**
The new controller generation adopts the turn that the old generation
started, waits for its terminal, and finishes the run.
**Evidence**
The durable controller journal for the run shows command 4, `turn.start`
with `turnId: turn_lab_18ac…`, completed with `{ providerTurnId:
turn_lab_18ac…, status: accepted }`. The Runner emitted `turn.started`
for that turn 1.4 s after the command, after the old controller died.
The replacement's `session.snapshot` returned `activeProviderTurnId:
turn_lab_18ac…`. The controller's own checkpoint had no active turn and
the run had no `turn.accepted` event.
Related: #19711, #19691, #19718
## What Changed
- `codex-app-server-driver-impl.ts` `recoverSession`: when the
checkpoint has no active turn, no semantic result, and no autonomous
goal or disposition turn applies, adopt the single in-progress provider
turn if the transport proves that this run's latest completed
`turn.start` accepted that exact turn id. Any other active turn is not
adopted and still fails closed.
- `codex-session-state.ts`: a recovered session with an adopted turn
emits the missing `turn.accepted` before any of that turn's
notifications.
- `app-server-transport.ts`: new optional transport method
`controllerAcceptedTurnId()`.
- `runnerd-codex-transport.ts`: implement it from the durable command
journal (`controllerAcceptedTurnStartId`). The latest `turn.start` must
be completed and accepted, its result must name the requested turn id,
and no run attachment may come after it.
- `DURABLE_TRANSPORT.md`: document the adoption rule.
## Verification
- Red/green with the real `paperclip-runnerd` and the scripted ACPX
Claude sidecar: `npx vitest run
src/control-plane/acpx-controller-recovery.test.ts -t RC-59`. The old
controller sends `turn.start` and stops before the provider accepts it.
The provider accepts and starts the turn while no controller is
attached. A replacement controller adopts the live Runner through the
production transport and `CodexAppServerDriver.recoverSession`. Before
the fix: `activeTurnId` is null (the session then fails on
`turn/started`). After the fix: events are `turn.accepted`,
`turn.started`, `turn.completed` for the same turn, with one
`turn.start` to the provider.
- Driver tests: `npx vitest run
src/drivers/codex/codex-app-server-driver.recovery.test.ts -t
"controller that died"`. Adoption passes. With no accepted result, or a
result for a different turn, `startTurn` still fails with "Provider
started an unexpected turn."
- Journal unit tests: `npx vitest run
src/live/runnerd-codex-transport.test.ts -t
controllerAcceptedTurnStartId` (pending, failed, rejected, mismatched
turn, later attachment, later pending start all return null).
- Regression: `src/drivers/codex` (all files),
`src/live/runnerd-restart-continuation.test.ts`,
`src/live/runnerd-codex-transport-adoption.test.ts`,
`src/live/runnerd-codex-transport-release-resume.test.ts`,
`src/control-plane/acpx-controller-recovery.test.ts` pass. `pnpm run
typecheck:typescript` passes in `packages/paperclip-runner`.
- `pnpm verify` asks for broader coverage (exit 2). Required CI supplies
it.
## Risks
- Low to medium. The change only acts when the checkpoint has no active
turn and the Runner reports exactly one active turn. Adoption needs the
Runner's authenticated, run-scoped `turn.start` result for that exact
id. Without that proof, behavior does not change.
- If a `turn.start` result is still pending when the replacement reads
the thread, recovery does not adopt and fails closed as before.
- Not proven on Daytona. I did not run the paid replay.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
|
||
|
Devin FoleyandPaperclip
|
2d4f80504e |
fix(runner): retain exact ACPX checkpoints after rejected turn starts (#19762)
Preserve the exact idle ACPX conversation when a matching turn.start response rejects work with retryable SESSION_RESUME_REQUIRED. Require verified identity, no pending work, confirmed process cleanup and the provider lifetime fence before normal drain and suspension can publish a reusable checkpoint. Unknown acceptance and arbitrary closed states remain closed. The real Runner regression completes the next turn on attempts 2 and 3, requires successful terminal status for the exact submitted turn, and checks expectedIdentity on both resume requests. Controlled wire faults reject failed, cancelled and missing-identity recovery. Validation: 193 affected Rust tests on unchanged Rust source/fixtures; 134 recovery integration tests, 14 environment-gated skips; 21 Claude effort tests; all server typecheck layers. Exact-head CI: 59 successful checks and statuses, 6 intentional skips. Greptile Apex 5/5, no unresolved findings. Hosted deployment and task acceptance remain separate. Fixes: #19751 Refs: #19366 Refs: #19745 Co-Authored-By: Paperclip <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
7475577cdb |
fix(daytona): keep a runner sandbox alive through the controller reconnect grace (#19754)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote agent run executes in a Daytona sandbox and keeps working while the Paperclip server restarts > - The server reattaches that runner within a 7-minute reconnect grace > - Daytona stops an idle sandbox after its `autoStopInterval`, and it counts only SDK and lifecycle calls as activity > - With the server down, nothing touches the sandbox, so a 5-minute auto-stop kills the runner before the server returns > - This pull request makes the host tell the provider its reconnect grace, so Daytona never auto-stops a runner sandbox inside that grace > - The benefit is that a controller outage of up to 7 minutes no longer turns a healthy remote run into a blocked task ## Linked Issues or Issue Description Refs #19199 (the same floor for the warm runner idle window). Refs #19741 (follow-up delivery through restarts; no overlap). **What happened?** A restart-chaos wave on Daytona killed the server for 4m51s while a per-turn remote run was active. The environment used `autoStopInterval: 5`. The last sandbox activity was at 06:09:16. The server came back at 06:14:19 and then crashed again. At 06:15:21 recovery called the sandbox. The plugin found it not started and started it again. The runner identity marker was still on disk, but its process was gone: `native.remote_runner_recovery_failed {stage: process_verification, reason: process_not_running}`, then `runner_remote_process_identity_unavailable`. The task went from done to blocked. An earlier wave showed the same failure after a 4m45s outage. **Expected behavior** A sandbox that holds a live remote runner survives any controller outage up to the reconnect grace. The server reattaches the runner and the run continues. **Steps to reproduce** 1. Create a Daytona environment with `autoStopInterval: 5` and a `paperclip_runner` agent. 2. Start a remote run and kill the server for more than 5 minutes (less than 7). 3. Restart the server. Recovery reports `process_not_running` and the task becomes blocked. ## What Changed - `server`: add `resolveRunnerReconnectGraceMsForRun`. It returns `REMOTE_RUNNER_RECONNECT_GRACE_MS` for `paperclip_runner` on a sandbox environment. - `server`: forward `runnerReconnectGraceMs` to the provider worker on lease acquisition and on the recovery reattach resume. It uses the same provider-only path as `warmRunnerIdleTimeoutMs`. It is not lease identity, lease metadata, or saved environment config. - Daytona plugin: the effective auto-stop is at least the largest host floor (warm idle window or reconnect grace) plus 5 minutes. With the current grace this is 12 minutes. The plugin applies it at creation and raises an older sandbox's shorter interval on resume (bounded, best effort). - Daytona plugin: keep the forwarded grace out of the normalized config. Update the `autoStopInterval` manifest description. - `doc/execution-semantics.md`: document the reconnect-grace floor next to the warm floor. ## Verification - Red/green Daytona plugin tests: `cd packages/plugins/sandbox-providers/daytona && npx vitest run --config vitest.config.ts`. The new cases failed before the fix (for example `expected 120000 to be >= 720000`, and no `setAutostopInterval` call on resume). All 605 tests pass after the fix. - Server tests: `cd server && npx vitest run src/__tests__/environment-run-orchestrator.test.ts src/__tests__/environment-runtime.test.ts`. 247 pass. They prove acquisition forwards the grace only for `paperclip_runner` on a sandbox, that the reattach resume forwards it, and that it does not reach lease metadata or saved config. - `pnpm typecheck` for the plugin and `tsc --noEmit` for the server are clean. - `pnpm verify` asks for broad CI coverage because a doc changed. Required CI supplies it. - Evidence from the kept wave database: lease metadata shows `autoStopInterval: 5`, `reuseLease: false`, `runnerLifecycleMode: per_turn`. The run had no events between 06:09:16 and 06:14:19. ## Risks - Cost: a runner sandbox that leaks (no cleanup) now auto-stops after 12 minutes instead of a shorter configured value. Cleanup still stops or destroys released sandboxes. Explicit `0` and longer values are unchanged. Auto-archive and auto-delete are unchanged. - I did not inspect the sandbox state in Daytona directly. Cleanup destroyed it before this investigation. The proof is the timing, the lease config, and the `process_not_running` result after a restart that kept the disk. - No live Daytona replay ran. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edit, code execution) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Nicky Leach
|
8be6d70538 | fix(runner): select Claude effort again after a cold session resume (#19708) | ||
|
Nicky Leach
|
167e39c4ef | fix(runner): keep warm run.attach events on ordinary accounting (#19707) | ||
|
|
96bc021edb |
fix(server): certify settled legacy processes for future retries (#19377)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Bounded retries must preserve current authority, budgets and deadlines. > - The tenant can wait for supported future work after the source process and its cleanup stop. > - Retained process identity alone cannot prove that cleanup finished. > - This change records host-owned settlement and checks its exact identity before a retry can cover its wait. > - Uncertain process state, changed authority, live claims and cleanup debt keep the tenant awake. ## Linked Issues or Issue Description Refs #19001. Related dependencies: #19019 and #18988. Related #19340 covers explicit Native task Retry; this change covers future Legacy process retries. Current integrated source: `1645970a9322d1213ba25fc7b52366c285682977`, normally merged with actual master `131b566ca4fa866c4348739c75cfe48c805b3398`. The prerequisites above are merged. This PR targets `master`. **What happened?** A future issue, company or review retry can retain its failed source process identity. The deadline reader could not prove that host cleanup finished. Supported waits stayed blocked. Ordinary issue retries also needed an exact source-settlement check, and represented coalesced requests needed exact input coverage. **Expected behavior** A supported future retry covers only its existing wait, under current company and actor authority, after exact host settlement. Missing, malformed, stale, cancelled, Native or unfinished source state stays blocked. No retry gains new execution authority. **Steps to reproduce** Fail an ordinary process or fail environment preparation before adapter entry. Let normal cleanup schedule the existing retry. Inspect its settlement and deadline, stop and reconstruct the owner, and admit one useful successor at the saved deadline. Hold cleanup or change authority to verify that the reader refuses coverage. ## What Changed - Add nullable internal `legacyProcessSettlement` state through normally generated migration `0439_cultured_impossible_man`. No backfill or adapter result creates the record. - Write exact host settlement after stopped-process and cleanup checks. Compare the current run and reject live claims or cleanup debt. Optional persistence failure preserves normal cleanup and keeps sleep coverage blocked. - Add a separate preparation receipt for a claimed Legacy run that failed before any adapter or Native dispatch. Require the same boot, run times, preparing stage, released claim and completed cleanup. A null PID alone proves nothing. - Clear settlement on a new claim, process metadata or Native selection. Omit the internal record from public serialized data. - Check exact source lineage and current authority for ordinary issue, company and taskless retries. Cover completed coalesced requests only when the admitted retry represents their exact inputs. - Add the closed `current_review_retry` wait with current reviewer, stage and existing issue authority checks. Require a nonempty current stage ID. - Add actual producer, cold reconstruction, SQL negative, redaction and optional-persistence controls. - Repair CI fixture expectations for the additive column, startup mock, budget owner and popup return URL. Bind the controlled Native comment server explicitly to loopback. - Replace temporary Slack clock logging with an exact persisted-deadline assertion. Wait until that real deadline before the existing single default poll. Keep both forced polls, business assertions and timing limits. ## Verification - On final integrated source `1645970a`, normal Shared/SDK prerequisites, Runner TypeScript and the entire normal App layered typecheck pass. Tracked source and lockfile remain unchanged. - All 26 whole-file settlement, recovery-reader and bootstrap controls pass. Seven separate real PostgreSQL producer/negative cases pass at `cf16c6a`; each has one selected passing case and 18 filtered registrations. Filtered registrations are not acceptance. - Master then changed the preparation source-lock scope. Its three intersecting cases were rerun on final `1645970a`: an actual failed company process, an actual failed ordinary issue process, and held preparation cleanup. Each preserves its source identity and original deadline and admits one useful successor. Issue and preparation cases exercise normal automatic retry; the company case uses the normal explicit bounded-retry producer. - The unchanged 26 controls and four other PostgreSQL cases retain their `cf16c6a` proof after a scoped source comparison. Optional settlement/preparation persistence failures keep certification blocked while normal cleanup completes. An entered adapter without process identity stays uncertified. This is scoped carry, not a claim that every earlier case ran again on the final head. - All recorded test owners and controlled groups closed. Final read-only SQL audits returned no test sessions. One earlier postcase resource barrier observed a transient unattributed database backend; that database and original evidence remain preserved. Its later disappearance is recovery, not an explained cause. No assertion, deadline or test body was relaxed. - Migration0439 was generated normally after master history. All executable SQL and journal history are preserved; the five retained snapshots are0435–0439. Every Legacy-owned patch and unrelated actual-master blob/mode is preserved. - Published exact head `1645970a9322d1213ba25fc7b52366c285682977` passes all current required checks: 68 successful contexts and eight expected skips after exhaustive pagination and superseded-run accounting. The fresh complete Greptile review scores this exact head 5/5. All four inline review threads are resolved. The latest Ponytail turn stopped without a verdict; one normal retry was requested, with no active turn or new finding observed. No old-head score is used. The original old-base policy failure, its dependent aggregate failure, and the corrected description-gate failure remain recorded. No full local repository suite is claimed; remote typecheck, test and build gates pass. To repeat the real database coverage, set both `PAPERCLIP_RECOVERY_DEADLINES_TEST_DATABASE_URL` and `PAPERCLIP_RECOVERY_DEADLINES_TEST_ADMIN_DATABASE_URL` to the same fresh dedicated test database and run the selected `recovery-deadline-acceptance.test.ts` cases with one worker. An unset URL, skipped case or filtered registration is not acceptance. The fixtures use controlled local processes and real PostgreSQL. Reconstruction creates new service owners inside the same test process. They do not prove deployed Cloud alarms, physical host/database suspension, new-host recovery or Native capacity settlement. The original30-second retry remains below the Cloud270-second minimum wake lead. The merge projection also includes actual master `c3180398faca815fed7a62674bd4b0c4e88884ae`, whose common completion path now sends deferred/recovery notifications. Its unpublished normal merge tree is `16b028d57150f8caa576e913f38343a365b542dd`. Normal Shared/SDK, Runner TypeScript and the full App graph pass on that combined source. One unchanged ordinary-issue acceptance case was rerun there: an actual failed process retains the original automatic retry deadline, admits one useful successor, and passes the settlement, authority and no-duplicate assertions. It passed in 45.055 s; 18 filtered registrations are not acceptance. All known owners and groups closed, and the independent final database audit found zero remaining test sessions. The successful database was removed normally. Recorded worker socket peak is unavailable for this run; parent-process zero values are not used as worker measurements. All 29 Legacy-owned paths and every unrelated master blob and mode are preserved. Two outside-diff review observations concern existing waiting-chat compaction and cross-host Native result-lease recovery. Their files are byte-identical to current master. This focused Legacy change does not fix or qualify those separate paths. ## Risks Settlement proves exact local closure. It does not cover live provider work, Native ownership, cancellation, costs, claims, cleanup debt or changed authority. Independent checks remain. Ship normal migration0439 with the reader and writer. Preserve executable history and the newest five snapshots0435–0439. Generate the MCP child's next migration normally after this parent. The prerequisites are merged; the final remote checks must use this integrated exact head. This proof uses controlled local processes and external PostgreSQL. It does not establish deployed Cloud alarm acceptance, physical host or database suspension, cold host recovery, or R5 Native capacity settlement. The30-second retry is below the Cloud minimum wake lead. This focused repair follows the roadmap's bounded recovery and Cloud deployment work. It adds no new provider policy. ## Model Used OpenAI `gpt-6.1-sol` assisted with readers, fixtures, source review and verification through Codex. A coordinating GPT-6-family agent implemented and reviewed shared lifecycle wiring and integration. Tools included Git, native TypeScript checks and controlled tests. Exact served revisions and context limits were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and its available capability details - [x] I have checked ROADMAP.md and related work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked existing public work and described the issue - [x] I have not included internal issue IDs or private links - [x] My branch name describes the change - [x] Required focused checks pass at their explicitly recorded source/carry scopes above - [x] I have added or updated meaningful tests - [x] I have recorded the internal contract and migration requirements - [x] I have documented risks and evidence limits - [x] All required published exact-head Paperclip CI gates are green - [x] Published exact-head Greptile is 5/5 with no open findings in this focused diff; separate unchanged master observations are identified above - [x] I will address reviewer findings before requesting merge --------- Co-authored-by: Paperclip <[email protected]> Co-authored-by: Codex <[email protected]> |
||
|
Nicky LeachandClaude Opus 5.5
|
bf45f66619 |
docs(runner): separate the legacy-adapter paragraph in the Runner README (#19753)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native Runner package has a README that explains its contracts > - One section explains how a resumed run gets a fresh-conversation handoff > - The next text, about legacy adapters, has no blank line before it > - Markdown renders the two texts as one paragraph, so the legacy-adapter text looks like part of the native handoff text > - This pull request adds the missing blank line > - The benefit is a README that separates native and legacy behavior clearly ## Linked Issues or Issue Description **Issue type** Formatting error in package documentation. **Where is the issue?** `packages/paperclip-runner/README.md`, the paragraph that starts "Legacy adapters advertise `supportsToolRefreshOnResume`". **What's wrong?** The paragraph has no blank line before it. Markdown joins it to the previous paragraph about the native fresh-conversation handoff. **Suggested fix** Add a blank line before "Legacy adapters advertise". ## What Changed - Add one blank line in `packages/paperclip-runner/README.md` before the legacy-adapter paragraph. ## Verification - `git diff --stat`: one file, one line added. - I read the changed section. The legacy-adapter text now starts its own paragraph. - No code changed, so no tests are necessary. ## Risks - Low risk. This is a documentation-only change. - The Runner build-input digest includes every file in `packages/paperclip-runner`. This change therefore changes that digest, and the next release builds a new Runner image. The Runner behavior does not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
464e6e16e0 |
fix(native): resume copyback after a restart kills its same-host owner; one board action otherwise (#19717)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs save the agent's workspace files after the provider turn finishes (copyback) > - One controller owns copyback at a time. The run records that owner in `runnerProfileJson.nativeWorkspaceFinalizationOwner` (host, PID, token) > - A control-plane restart can kill the controller during copyback. The dead controller can never confirm its own stop > - Recovery then waited for a board confirmation. No surface offered that confirmation, so the run stayed `running` and the task stayed `in_progress` with no visible action > - This pull request lets the next controller on the same host prove the stop by itself. It also gives the board one clear action when that proof is not possible > - The benefit is that a restart during copyback resumes by itself, and a task never waits silently for a person ## Linked Issues or Issue Description **What happened?** In a Daytona restart-chaos wave, a kill during workspace copyback left the run `running` and its task `in_progress`. Recovery reported `native_workspace_finalization_owner_unverified`. The run was still `running` when the wave timed out. **Steps to reproduce** 1. Start a native remote (sandbox) run. 2. Kill the control-plane process (SIGKILL) after the provider result is accepted, during copyback. 3. Restart the control plane on the same host. **Expected behavior** If the previous controller and its copyback children are proven stopped on this host, recovery resumes copyback without a new agent turn. Otherwise, the task shows one board action to confirm that the previous server stopped. **Actual behavior** The recorded owner was cleared only by `resumeNativeWorkspaceAfterOwnerStop`. Same-host recovery also required a completed-copyback barrier, which an interrupted copyback never has. The task presentation showed `working`, and the UI had no control for the confirmation. Refs #19578 (boot-ID fencing for unstarted runs; same proof style). ## What Changed - Root cause: `server/src/services/native-runtime/native-workspace-finalization-ownership.ts` sent a dead same-host owner to `requireStopVerification` when no `workspace_finalize` operation had succeeded. That case is exactly a restart during copyback. - New `native-workspace-owner-stop-proof.ts`. The owner receipt now records the controller's process group (`processGroupId`). Copyback children (tar, Git) are not detached, so they keep that group. A successor on the same host releases the receipt only when the receipt names another controller boot, the PID is dead or recycled (process start time), and the group is empty or has no member outside the successor's own process tree that started after the old controller. Age alone never proves a stop. - A still-live group member returns `local_stop_pending`. The finalization worker rechecks it on its existing ownership recheck interval. The board action stays visible meanwhile. - A successful local proof writes `issue.workspace_owner_stop_verified` with the proof and marks the recovery `resuming`, like the existing platform proof. - A receipt from another host, a receipt without a process group, or a host without `ps` keeps the fence. The task interruption is now `waiting_for_user` with `userAction: "confirm_owner_stop"` (new shared enum value), not `working`. - UI: the task chat shows one **Confirm the previous server stopped** action. It asks for stop evidence and calls the existing `resume-workspace-finalization` API. It is shown only to roles that hold `runtime:confirm_owner_stop` (company owner or administrator, local board, instance administrator). The inbox names the same action. - The copyback advisory lock, the coordinator finalization lease, and remote save fencing are unchanged. - Docs: `doc/native-workspace-finalization-recovery.md`. ## Verification - Red then green on embedded Postgres (`server/src/services/native-runtime/native-workspace-finalizer.test.ts`): - same-host controller killed with SIGKILL, its process group empty: before `owner_unverified`, after `acquired` with the proof logged and no agent wake - a non-detached orphan child of the dead controller still runs: `local_stop_pending`, no copyback; after the child exits, `acquired` - a receipt without a process group: still `owner_unverified` - a foreign-host owner: the task presentation was `working` (red), now `waiting_for_user` / `confirm_owner_stop`; after the board confirmation it is `working` again - The existing orphan test (a detached child) and all other finalizer tests pass: `npx vitest run src/services/native-runtime/native-workspace-finalizer.test.ts src/__tests__/native-workspace-finalization-work.test.ts src/services/task-interruption-presentation.test.ts` (135 passed). - UI: `npx vitest run src/components/WorkspaceExportRecovery` (34 passed), including the action call body and the non-administrator case. - `pnpm check:token-gates` clean. Server and UI typecheck clean for the changed files. - Live Daytona restart-chaos wave: results are added in a comment below. ## Risks - A process-table check that reports a member wrongly as live only keeps the existing fence. The board action is still available. - A subreaper that adopts orphans does not change their process group, so the proof does not depend on the parent PID. - A child that a future change spawns with `detached: true` would leave the group. Copyback paths do not do that today. The existing detached-orphan test still fails closed, because a receipt without a process group is not released. - Old receipts have no process group. They keep the board action. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
fc7e3a6503 |
fix(recovery): never replace a native session after a deterministic refusal (#19733)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs keep a saved provider session, and same-run
recovery resumes it after a failure
> - When resume fails and the checkpoint allows it, recovery archives
the saved continuity state and opens a fresh replacement session
> - Recovery used that path for every resume failure, including
deterministic refusals that left the saved state unchanged and repeat on
every retry
> - In five staging tasks a refusal (a protected launch argument, or a
predecessor tool call with no result) escalated into a replacement. The
remote archive move failed part way, the next attempt found no runner
state, and each task was blocked for workspace repair
> - This pull request stops before replacement after a closed-class
refusal, keeps the saved state, and hands the run to an operator with a
named reason
> - The benefit is that a deterministic refusal can no longer damage
saved state or open a replacement over an unsettled external action
## Linked Issues or Issue Description
Refs #19659, Refs #19516, Refs #19334
**What happened?**
Five Codex tasks on Daytona sandboxes failed on staging. The first
attempt failed with a deterministic refusal:
- `run.attach cannot change protected launch arguments` (four tasks,
fixed separately in #19659), or
- `native_runner_source_semantic_reconciliation_required` with a
predecessor `paperclip_call_assigned_tool` call still pending (one
task).
The same-run retry then opened a replacement session.
`archiveContinuityState` failed with
`runner_continuity_break_archive_failed` and no detail. The third
attempt failed with `runner_remote_state_unavailable: exit=1`, and the
task was blocked with "Workspace repair required".
**Steps to reproduce**
1. Save a native session whose checkpoint policy is
`allow_replacement_after_resume_failure`.
2. Make the resume fail with a refusal that does not change state (an
attach refusal, or a pending tool call).
3. Same-run recovery opens a replacement session and archives the saved
continuity state.
**Expected behavior**
A deterministic refusal stops the run for an operator. The saved state
stays intact. A run with a pending side-effecting call never reaches
replacement. Transport drops keep the existing retry behavior.
**Root cause**
`executeNativeSession` checked only the checkpoint's replacement policy.
The closed failure class (#19334) only had an effect when replacement
was forbidden. Attach refusals and pending tool calls had no class at
all, and the source-reconciliation refusal was a plain `Error`.
## What Changed
- Runner (`provider_backend.rs`, `durable/mod.rs`): `run.attach`
refusals that would change the durable Codex profile or a protected
launch argument carry the closed classification
`provider_attach_refused`. An attach blocked by
`durable_pending_tool_calls` carries `provider_side_effect_pending`.
These refusals already leave the saved state unchanged.
- Contracts (`harness-driver.ts`): new closed classes
`PROVIDER_ATTACH_REFUSED` and `PROVIDER_SIDE_EFFECT_PENDING`. A typed
`NativeRunnerSourceReconciliationRequiredError` replaces the plain
`Error` at the five `native_runner_source_*_reconciliation_required`
throw sites. `runnerRecoveryFailureClass` reads the authenticated
command-result classification or the typed error, never message text.
- Runtime (`native-session-runtime.ts`): after either class, recovery
never opens a replacement session, whatever the checkpoint policy.
- An attach refusal ends with the existing
`NativeProviderStateIncompatibleError`, which maps to the existing
operator path `native_provider_state_incompatible`.
- A pending side effect ends with the new
`NativeProviderSideEffectPendingError`.
- Server (`native-session-executor.ts`, `native-session-resume.ts`,
shared `EXECUTION_RECONCILIATION_CAUSES`):
`native_provider_side_effect_pending` is a permanent failure and an
execution-reconciliation cause, so the board gets a reconciliation
(record whether the action ran) instead of a generic Retry. The
reconciled continuation does not rebind the saved session, which still
refuses attachment. It starts a fresh provider session in the same
workspace. The saved session and workspace are unchanged.
- Archive (`archiveContinuityState`): a failed remote move now records
the bounded (512 characters), redacted exit status and output in
`runner_continuity_break_archive_failed: exit=… timedOut=… <output>`.
The remote move already ran before the local move on `master`, so a
failed remote move leaves the local state in place.
`restart-interrupted-turn.ts` accepts the detailed message for its
reattachment-loss retry.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe the new
rule.
## Verification
Red before the fix, green after (each test first run against unchanged
source from `origin/master`
|
||
|
|
839d41aabb |
fix(server): retire owned plugin listeners before idle sleep (#19358)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Plugins receive durable agent lifecycle events through a registered host. > - The host uses a PostgreSQL LISTEN connection to wake delivery. > - An idle report could return `none` while that host-owned connection stayed open. > - This change gives that listener an exclusive owner and closes it during the current idle hold. > - Release or expiry restores listening before journal replay. > - Failed close and pending work continue to block sleep. ## Linked Issues or Issue Description Refs #18988, which is now merged. This focused ten-file change now targets master. Current integrated candidate: `0f83005122ef1bdb1f1e08f8f58424c9a840ebb1`, with actual master `644ee66f5d4ae8365fe8b9eca57df2be7405b5ab` preserved through normal merges. No history was rewritten. **What happened?** A normal local App with an installed lifecycle plugin could return an idle `none` report while its lifecycle LISTEN backend and socket stayed open. **Expected behavior** The host must close its own listener before it reports idle safety. It must retain other subscriptions, durable events, company scope, retry deadlines and useful reactivation. **Steps to reproduce** Start the normal server with a lifecycle plugin. Create an agent and wait for its durable acknowledgment. Stop the server, create another agent while it is stopped, and restart with the same files. Request an owned idle hold after ordinary replay. Check the exact listener backend and matched Node socket before shutdown. Let the valid hold expire naturally, then make an ordinary authorized agent update and check its durable plugin acknowledgment. Live executing or persistent preview services remain active. The separate hosted preview dependency [paperclip-cloud#478](https://github.com/paperclipai/paperclip-cloud/pull/478) retains its own App and operator prerequisites. This PR does not claim deployed preview or suspension acceptance. ## What Changed - Add a private, exclusive database listener owner that inherits the normal client policy. - Prepare plugin lifecycle retirement after native retirement under the same current idle hold. - Use one delivery-eligibility helper for dispatch and idle journal inspection. It preserves the same manifest, capability, company-disable query, and skip rules. - Close only the owned listener. Keep failed close ownership and inspection failures visible. - Resume listening on normal release or expiry, before journal replay. - Preserve disabled-feature retry history and the original deadline on re-enable. - Await lifecycle disposal during normal shutdown and consumer teardown. - Add owner regressions and document the lifecycle contract. ## Verification **Current integrated source:** `0f83005122ef1bdb1f1e08f8f58424c9a840ebb1`. Fresh normal Runner TypeScript passed in 0.785 seconds. The complete normal App layered semantic check passed in 28.284 seconds. All tracked source, the lockfile and the 18 separate private witness inputs stayed unchanged. Known compiler owners closed and the isolated temporary home was removed. **Current product qualification:** at measured integration `896cc9b62993e6ed10418d7a7417e1ddbf8dc89c`, all three whole files passed with no failures or skips: lifecycle delivery 40, runtime hooks 20, and runtime assignments 24. The actual physical LISTEN and natural-expiry controls passed. All owned clients and captured processes closed. Independent final session checks were zero before normal deletion of each successful test database. **Normal full-App physical retirement:** the original case passed in 47.574 seconds. It exercised real startup, readiness, SDK callbacks, durable ACK, foreign-company exclusion, physical stop, offline input and cold replay. The initial unknown hold and its ordinary release remain in the evidence. After the original 30.5-second cadence, a new valid hold returned quiescent. The exact current cold-App LISTEN backend and its matched Node socket closed before server stop. **Normal full-App natural expiry:** the separate original case passed in 77.391 seconds under its original 90-second body cap. The final valid 30-second hold expired naturally. A new listener backend and matched current App socket appeared before fresh input. An ordinary authorized agent-name PATCH produced journal event 20, the actual SDK callback and durable ACK. Foreign exclusion and readiness assertions passed. The final measured hold had no manual release, replacement hold or shortened TTL. The original preparatory unknown hold, ordinary release and 30.5-second cadence remain explicit. Both App cases used the normal current server helpers and actual SDK bundling/worker entry. A private compatibility change supplied the same owned HOME at the exact existing child-fork boundary. It did not change the product helpers, business assertions or timing limits. These separate acceptance fixtures remain outside the focused product diff. **Source carry to the current candidate:** all nine other owned files, listener mechanics, actual producer paths, SDK paths and measured hold behavior are unchanged. The sole owned intersection from newer master is an independent environment-delete cleanup callback in `app.ts`. The plugin witnesses do not call that route. Normal merges preserve every unrelated master blob and file mode. The fresh current semantic checks above qualify the integrated source. New master Native crash-recovery and saved-copyback Retry capabilities are not claimed by these plugin cases. **Actual master merge target:** newer master contains retained plugin Runner release/resume and other Native recovery fixes. The projected combined target passed fresh normal Runner TypeScript in 0.799 seconds and the full App semantic check in 29.715 seconds. All ten plugin-owned blobs and modes remain exact to the published head; every unrelated master change is preserved. The measured plugin cases create no Native provider turn or warm Native session. The unchanged native-first/plugin-second ordering still checks the native result before listener retirement. Later sparse-checkpoint and Native response-publication changes do not enter those paths. This is a scoped source carry, not new hosted Runner acceptance. **Failures retained:** an earlier reactivation fixture's real timer returned one millisecond before its original wall-clock expiry. Its private fixture now waits any positive remainder and retains the same expiry assertion and limits. The earlier partial-composition `unknown` result remains distinct from normal-startup proof. Current qualification also exposed missing normal workspace dependencies and a private guard that assumed loose Git refs. Normal owned dependency setup and an exact loose-or-packed reader fixed those setup failures. No business assertion, timeout, permission boundary or product sleep gate was relaxed. Repeat focused product coverage with a fresh external fixture database and the normal server Vitest command, one worker: ```sh pnpm --filter @paperclipai/server exec vitest run src/__tests__/plugin-lifecycle-delivery.test.ts --maxWorkers=1 --no-file-parallelism pnpm --filter @paperclipai/server exec vitest run src/__tests__/agent-runtime-hooks.test.ts --maxWorkers=1 --no-file-parallelism pnpm --filter @paperclipai/server exec vitest run src/__tests__/agent-runtime-assignments.test.ts --maxWorkers=1 --no-file-parallelism ``` Exact current head `0f83005122ef1bdb1f1e08f8f58424c9a840ebb1` has 60 successful checks, three skips and one neutral result, with no pending or failing contexts. Greptile completed its full review at 5/5 with no actionable issue. All root and nested review pages were read; the earlier eligibility finding is resolved. Ponytail's latest turn stopped before a verdict. A normal retry was requested; no current Ponytail verdict or score is claimed. Previous `3ca86ec` CI and scores are historical. A full local repository typecheck/test/build or browser run is not claimed. Controlled local App/SDK/PostgreSQL proof does not establish live provider, deployed Railway/Neon suspension, signed Cloud alarms or global exactly-once delivery. ## Risks - A listener close failure or uncertain inspection must continue to block sleep. - Release, expiry, configuration changes, and journal arrival can race with retirement. Epoch and current-hold checks retain conservative results. - Delivery remains at least once across uncertain ACK boundaries. This change does not add global exactly-once delivery. - The database helper closes only its exclusive listener client. Other owners and subscriptions keep their resources. ## Model Used OpenAI `gpt-6.1-sol`, reasoning and code/tool use. The managing Codex agent implemented shared database/composition seams and reviewed the change; its exact model ID was not exposed. The runtime does not expose a verified context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]> Co-authored-by: Codex <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
c45c7cb854 |
fix(workspaces): ignore killed-transfer scratch in sparse checkpoints (#19730)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run in a remote sandbox copies its workspace back to the host after the turn. A sparse checkpoint copies only changed entries, and the host validates each entry against the sandbox's own manifest > - Every Daytona transfer stages a `.paperclip-upload-<uuid>` scratch file at the sandbox workspace root and removes it only in the same process > - A control-plane kill during any transfer leaves that scratch behind. The sparse checkpoint then recorded it as a workspace file, but providers never export it. Every later copyback and every saved-work recovery attempt failed with "Workspace checkpoint payload mismatch" > - The failed recovery then left an accepted Retry showing "Paperclip is finishing the saved result" with no run > - This pull request stops recording transfer scratch, makes the checkpoint export an exact copy, names the failing entry, and gives a Retry behind a spent recovery an honest reason and a working action > - The benefit is that a restart during a transfer no longer makes a task's saved work permanently unrecoverable ## Linked Issues or Issue Description Refs #19721 (RC-50: the killed copyback owner; this PR covers the separate RC-50b cause) **What happened?** Restart-chaos waves s12/t01 and s14/t05 (Claude on Daytona). After a SIGKILL of the control plane, the native result was accepted, and then workspace copyback failed with "Workspace checkpoint payload mismatch". In s14 all 6 saved-work recovery attempts failed the same way. A Retry accepted during recovery kept `retryWait.reason = finalizing_saved_result` after the recovery budget was spent. No run was created. **Steps to reproduce** 1. Start a native Claude run in a Daytona sandbox. 2. SIGKILL the control plane while any provider transfer is in flight. 3. Let the run finish its turn and copy its workspace back. **Expected behavior** Copyback succeeds. Provider scratch is not workspace content. A Retry behind a spent recovery names the failure and can start a new recovery. **Actual behavior** The s14 sandbox (6f4a0b6a) still held `/home/daytona/paperclip-workspace/.paperclip-upload-d99fe36b-5acc-437b-96ad-85d5ff9e79c4.tar`. Its mtime was 03:29:23, the second of the K4 SIGKILL. The Daytona provider stages transfer scratch at `remoteDir`, which is the lease `remoteCwd` (the workspace root). The runtime workspace wipe deliberately preserves `.paperclip-upload-*`. The checkpoint helper (`workspace-checkpoint.ts`, `checkpointMain`) recorded the stale tar. The ordinary export excludes `.paperclip-upload*`, so the host received nothing for that entry. `validateCheckpointPayload` then threw "payload mismatch" on every attempt. The file-mode hypothesis was disproved: the sandbox umask is 022 and the scratch is mode 644. ## What Changed - `packages/adapter-utils/src/exclude-patterns.ts`: new `isSandboxTransferScratchPath`, which matches a root-level `.paperclip-upload-*` entry. This agrees with the wipe's preserve rule in `buildWorkspaceTarExtractCommand`. - `packages/adapter-utils/src/workspace-checkpoint.ts`: the sandbox checkpoint helper skips that scratch. A mismatch error now names the entry and its recorded, baseline and received forms (kind, octal mode, short hash; never contents). - `packages/adapter-utils/src/sandbox-managed-runtime.ts`: the checkpoint export requests `preserveSourceEntries: true`. The checkpoint output is an exact copy that the host validates entry by entry. An ordinary export applies the host umask and drops `._*` and `.paperclip-upload*` names at any depth. That gives the same permanent mismatch for a legitimate user file such as `._summary`, `notes/.paperclip-upload-plan.md`, or a `0775` script. - `server/src/services/explicit-native-continuation.ts` and `task-interruption-presentation.ts`: when the run's saved-work recovery is exhausted, a waiting Retry reports `saved_work_recovery_failed` with Retry available, instead of `finalizing_saved_result`. The existing task Retry path starts one new recovery generation. - `doc/execution-semantics.md`: a short additive note. ## Verification - `packages/adapter-utils/src/workspace-checkpoint-capture.test.ts` uses the real checkpoint helper and an export that behaves like Daytona's ordinary export: - Before the fix: `Workspace checkpoint payload mismatch at ".paperclip-upload-d99fe36b-….tar": sandbox recorded file mode=644 …; baseline absent; received absent`. That is the wave error. - After the fix: the scratch is not recorded, and validation passes. - A second case shows that reserved-looking user names and `0775` modes need the exact export. - `packages/adapter-utils/src/workspace-checkpoint.test.ts` is an end-to-end `prepareSandboxManagedRuntime` restore through a provider model. In that model, ordinary exports drop reserved names and apply the umask, and preserve exports copy exactly. Two red checks: - Without `preserveSourceEntries`, it fails with the mismatch on `._summary`. - Without the scratch skip, the stale tar leaks into the host workspace. - Green: the agent file, the `._summary` file and the `0775` mode arrive, and no scratch arrives. - `server/src/services/workspace-recovery-preparation-gates.test.ts`, "names a spent saved-work recovery behind an accepted Retry, and the next Retry finishes it" (s14 shape): - Red: `finalizing_saved_result` after 6 failed attempts. - Green: `saved_work_recovery_failed` with `canRetry: true`. The next Retry starts a new generation, recovery completes, and the Retry continues in exactly one new run. - Suites: adapter-utils checkpoint, capture, sandbox-managed-runtime, seed-reuse and managed-seed (168 passed). Server preparation-gates (47), explicit-native-continuation and task-interruption-presentation (455), workspace-recovery admission and retry-copyback, and native-workspace-finalizer. `tsc --noEmit` is clean for adapter-utils and server. - Live evidence: I started the stopped s14 sandbox read-only, listed its workspace, and stopped it again. No Daytona replay was run. The runner content ID of this checkout matches no built image. ## Risks - A user file named `.paperclip-upload-*` directly at the workspace root is no longer captured by a sparse checkpoint. Providers already drop that name from every ordinary export, and the runtime already reserves it at the root. - The checkpoint export now uses the provider's exact-copy mode. A workspace file with setuid, setgid or sticky bits is refused by that mode, as it is for the existing exact-copy transfers. The ordinary export would have stripped those bits and failed validation anyway. - The stale scratch tar stays in the sandbox. It is ignored, not deleted, because a concurrent transfer can own a file with the same prefix. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, test runs, embedded Postgres, a read-only Daytona sandbox listing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
d1ee25d0f7 |
fix(runner): fail a pending Codex turn start at once with the runner's own terminal failure (#19722)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane starts each Codex turn through runnerd and waits for the turn's start event (`#startTurnWork` in `runnerd-codex-transport.ts`) > - In restart storm 5, runnerd failed an accepted continuation turn before it started (`turn_binding_mismatch`, fixed at its source in #19718) > - The transport held that terminal for the pending start and waited the full 30 s bound > - It then reported "runnerd did not report the provider turn identity", which hid the real cause > - This pull request makes a pending start fail at once with the runner's own terminal failure > - The benefit is a faster failure that names its real cause, so recovery and operators act on the correct fact ## Linked Issues or Issue Description Refs #19718, #19699, #19608 **What happened?** Restart storm 5 (seed 7106), task t01 (Codex). Runnerd accepted the restart continuation turn. About 0.5 s later it committed a `harness.diagnostic`, a non-recoverable `turn.failed` (`turn_binding_mismatch`) and a failed `run.terminal`. The transport's event pump holds a turn terminal while a turn start is pending, so the start loop never saw it. The start waited 30 s, the #19608 adoption check found no active turn, and the run failed with "runnerd did not report the provider turn identity". **Expected behavior** When runnerd reports a final failure of the pending turn (a non-recoverable `turn.failed` or a failed `run.terminal`) before the turn starts, the start fails at once. The error names the runner's code and message. **Steps to reproduce** 1. Use fake Codex with `--foreign-usage-before-turn-started --turn-started-delay-ms 20000`. 2. Start a turn through the runnerd Codex transport. 3. Before this change, `turn/start` rejects after the full bound with "runnerd did not report the provider turn identity". ## What Changed - `runnerd-codex-transport.ts`: new `runnerdPendingTurnStartFailure()`. It finds a non-recoverable `turn.failed` or a failed `run.terminal` among the terminals held for the pending start and the committed events after the cursor. - `#startTurnWork` checks it while it waits for `turn/started`, before the adoption snapshot, and in the post-adoption wait. It throws `runnerd failed the provider turn before it started: <code>: <message>`. - The check runs only when the turn has not started. A recoverable terminal does not trigger it. The #19608 late-start adoption path is unchanged. - Fake Codex: new `--foreign-usage-before-turn-started` flag. It sends usage for a turn the runner does not own right after it accepts a turn. This makes runnerd fail the turn with `turn_binding_mismatch` before it starts. - New test in `runnerd-codex-transport-deadlines.test.ts` with real runnerd and fake Codex. ## Verification - Red before the fix: `npx vitest run src/live/runnerd-codex-transport-deadlines.test.ts` → the new test failed after 16.4 s with "runnerd did not report the provider turn identity". - Green after the fix: the start rejects in under 5 s with `turn_binding_mismatch`, and fake Codex saw exactly one `turn/start`. - The #19608 tests ("adopts a durably accepted turn whose start notification outlives the turn-start bound" and "does not adopt an accepted turn the runner no longer reports active") still pass. - `npx vitest run src/live/runnerd-codex-transport`: 9 files, 399 tests pass. - `pnpm run typecheck` in `packages/paperclip-runner` passes. `cargo test --bin fake-codex-app-server` passes. `cargo fmt` is clean. - `pnpm verify` exits 2 (broad input: the fake Codex binary). Required CI supplies the broad coverage. ## Risks - Low risk. The new failure fires only when runnerd itself already reported a final failure and no turn start was observed. In that state the start could not succeed, and before this change it failed after the bound anyway. - If a provider sends a non-recoverable terminal before its own turn start for a turn that later starts, the start now fails earlier. Codex sends `turn/started` before its terminal, and runnerd's terminal for a protocol failure is final, so I do not expect this. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing). Default context window and reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
42f6b3a870 |
fix(runner): keep a restart-lost Codex turn's late usage from failing its continuation (#19718)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex runs continue a turn lost to a control-plane restart once, in the same conversation (#19699) > - In restart storm 5, a Codex run crossed one graceful restart. The continuation started, but the task stopped and needed a Retry > - The runner log shows the cause. The resumed Codex thread sent the usage of the lost turn after it accepted the continuation turn. The runner rejected that usage as `turn_binding_mismatch` and failed the run > - Recovery marks the lost turn as settled in the durable ledger. It does not tell the live Codex provider, so the provider treats the old turn's usage as a foreign event > - This pull request records the lost turn as settled in the live provider too > - The benefit is that a restart-lost Codex turn continues without a Retry ## Linked Issues or Issue Description Refs #19699, #19608, #19501, #15297 **What happened?** Restart storm 5 (seed 7106). Task t01 (Codex, B5) crossed one K2 SIGTERM restart. Recovery resumed the dead runner. Runnerd restored the conversation and recorded `turn.failed` with `provider_turn_lost_on_restore` (recoverable). The control plane started the continuation (#19699). Runnerd accepted it as a new provider turn. About 0.5 s later, Codex sent `thread/tokenUsage/updated` for the lost turn id. The runner's durable state records these events: - `turn.accepted` for the new turn `01a1b3e2-…` - `harness.diagnostic` `turn_binding_mismatch`, method `thread/tokenUsage/updated`, `expectedTurnId` = new turn, `receivedTurnId` = lost turn `01a1b3e0-…` - `turn.failed` (`recoverable: false`) and `run.terminal` failed The control plane did not see `turn/started` and failed after 30 s with "runnerd did not report the provider turn identity". The task needed a manual Retry. **Expected behavior** Usage for the lost turn is history. The runner keeps it as a resume usage snapshot. The continuation runs, and the task continues without a Retry. **Steps to reproduce** 1. Start a Codex turn through the durable runner. 2. Kill the runner while the turn is active. Restore so that the provider reports no active turn (`provider_turn_lost_on_restore`). 3. Start the continuation turn. Let the provider send `thread/tokenUsage/updated` for the lost turn after it accepts the new turn. 4. Before this change, the runner fails the continuation with `turn_binding_mismatch`. ## What Changed - `provider_backend.rs`: when recovery finds that the previous active turn ended while the runner was offline, it now also restores that turn id into the live provider's settled-turn set. The durable ledger already settled it. The existing settled-turn path then turns its late usage into `paperclip/resumeUsageSnapshot` and does not bill its `last` usage to the run. - The code skips this step when the settled-turn epoch needs a rollover, the same as the durable ledger. - Fake Codex: new `--stale-usage-after-turn-start` flag. With durable turn ids, it sends usage for the previous turn right after it accepts the next turn, before `turn/started`. This matches the live Codex order. - New test `restart_continuation_ignores_late_usage_for_the_turn_lost_on_restore` in `tests/codex_provider.rs`. It uses the real durable Codex executor with fake Codex. ## Verification - Red before the fix: `cargo test -p paperclip-runner-core --test codex_provider restart_continuation_ignores_late_usage` failed with the same diagnostic as the live run (`turn_binding_mismatch`, method `thread/tokenUsage/updated`, `receivedTurnId: provider-turn-1`, `expectedTurnId: provider-turn-2`). - Green after the fix: the same test passes. The continuation emits `turn.started` for `provider-turn-2`, there is no `turn.failed` or `run.terminal`, and fake Codex saw exactly two `turn/start` calls (the original and one continuation). - `cargo test -p paperclip-runner-core`: all suites pass. `acpx_park_readiness::hosted_park_acpx_preserves_late_goals_and_rejects_foreign_or_busy_observations` failed once and passed on rerun. This change does not touch ACPX. - `cargo fmt --check` is clean. - Not run: the live restart-chaos replay of `w-20261107T023921-s7106/t01`. The local harness needs provider credentials that were not available to this session. ## Risks - Low risk. The change adds one already-ended turn id to the live provider's settled set. That set is only used to ignore stale events and to reject reuse of a settled id. So a provider that later reports the lost turn id as a new active turn is now rejected, as the durable ledger already does. - The change does not address a second, separate problem: after the runner failed the continuation, the control-plane transport waited the full 30 s turn-start bound and reported "runnerd did not report the provider turn identity" instead of the runner's `turn_binding_mismatch`. That message only applies when such a failure occurs. This PR removes this specific trigger. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing). Default context window and reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
ece09c156a |
fix(runner): keep an accepted result when ACPX reports its turn lost on restore (#19691)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex, OpenCode, and ACPX runs survive control-plane restarts: runnerd restores the conversation, and the control plane continues the task from its checkpoint > - #19658 made recovery keep a durable result when a Codex restore loses the turn that proposed it. It covered the recovery read path only > - ACPX reports the lost turn differently. After the restored conversation reopens, a live `turn.failed` arrives with `provider_turn_lost_on_restore` > - That live terminal still turned an accepted `done` result into a failed run. The run ended `failed` (`adapter_failed`), and the finished work needed a Retry > - This pull request moves the rule into the shared terminal mapping. A restore-lost turn with a held result for that exact turn settles as completed, whichever path reports it > - The benefit is that ACPX and OpenCode tasks that already finished keep their accepted result after a restart, as Codex tasks now do ## Linked Issues or Issue Description Refs #19658, #19525, #19501, #19539 **What happened?** In a restart chaos wave, an ACPX Claude task (workload A1) wrote its brief and proposed a `done` result (`run.result.proposed`, then a completed `paperclip_finish` tool call). Then a K1 and a K6 crash loop hit the control plane, and `resume_dead_runner` ran at controller generations 2 and 3. The restored ACPX conversation emitted `turn.failed` with `provider_turn_lost_on_restore` (`recoverable: true`) for the checkpointed turn. The control plane then emitted `run.result.accepted` with the `done` result, but `run.terminal` came back `runTerminalState: failed` / `turnTerminalState: failed` / `reportedWorkDisposition: done`. The run ended `failed` / `adapter_failed`, "Native session failed". **Steps to reproduce** 1. Start a native ACPX (or OpenCode) task through runnerd. 2. Let the agent call `paperclip_finish`, and let the control plane persist the proposal. 3. Kill the control plane and runner before the turn ends. 4. Let restart recovery resume the run. The restored runner reports the turn lost on restore as a live terminal. **Expected behavior** Recovery finalizes the accepted result, and the run succeeds. **Actual behavior** ACPX and OpenCode use the same Codex harness session as Codex (`codex-native-backend.ts`). Recovery restored the held result for the turn from the durable proposal. But the live `turn/completed` notification with `status: failed` and `provider_turn_lost_on_restore` went through `mapTerminalTurn`, which emitted `turn.failed`. The backend maps `turn.failed` to `runTerminalState: failed`. #19658 handled only the `thread/read` reconciliation path. ## What Changed - `codex-session-terminal.ts`: a new `settleRestoreLostTurnWithHeldResult` helper. When a terminal is `failed` with `provider_turn_lost_on_restore` and the session holds the semantic result that this exact turn proposed, the turn settles as `completed`. `mapTerminalTurn` applies it first, so live notifications and recovery reads follow one rule. - `codex-session-terminal.ts`: the helper also covers `acpx_active_turn_recovery_closed`. Runnerd's ACPX backend (`acpx_provider_backend.rs`, `restore_session_if_needed`) emits that code instead of restoring the conversation when its own state already recorded the accepted result. It is the same restart loss, with the result already accepted. #19525 already keeps accepted results when close cannot prove suspension, and session close still handles `provider_exit_unconfirmed`. - `codex-harness-session.ts`: `reconcile()` keeps only the missing-turn case from #19658. The restore-lost read case and the prior-terminal check now use the shared helper. - `doc/execution-semantics.md`: the restart paragraph now states that the rule covers Codex, OpenCode, and ACPX, and both report paths. ## Verification - Red before the fix, with the real Codex native backend (`createCodexNativeSessionBackend`) and a fake transport through `executeNativeSession`: `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/native-session-runtime.test.ts -t "journaled completion proposal"`. The new `restore-lost live` scenario failed. Green after the fix: the run completes with the journaled result on `turn-1` and a succeeded terminal, and no `turn/start` is sent. The `restore-lost read` scenario passes on both (#19658 already covered it). - Driver red/green: `src/drivers/codex/codex-app-server-driver.recovery.test.ts -t "live restore-lost"`. With a held result, the live lost terminal was `turn.failed` before the fix and is `turn.completed` after it. With no held result, it stays `turn.failed` with the lost cause, so the #19501 continuation is unchanged. - The driver case was extended to `acpx_active_turn_recovery_closed` (red with a held result before the extension, then green). It was found in a live replay where the run ended `failed`/`adapter_failed` with an accepted `done` result. - Suites: `src/drivers/codex/` and `src/native-session-runtime.test.ts` (580 tests), plus `src/backends/` and `src/live/runnerd-restart-continuation.test.ts`, pass. `typecheck:typescript` passes. - Live chaos replays (ACPX Claude): results and a separate new signature are in a comment below. ## Risks - The rule needs a held result whose turn id equals the lost turn. Recovery restores that result only from the durable `run.result.proposed` for the checkpointed turn, so a result for another turn does not settle it. - A restore-lost turn without a held result keeps its failure and the bounded same-conversation continuation. - Real provider failures and other terminal states are unchanged. - No schema, profile, pack contract, or database changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Nicky Leach
|
03498f90ee | Add explicit Runner session release and resume support (#19683) | ||
|
DottaandClaude Opus 5.5
|
f09b4704b6 |
fix(recovery): keep the agent's work when Retry continues after exhausted or blocked recovery (#19666)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Task recovery restarts agent work after a control-plane restart or a failed run > - When automatic recovery runs out or is blocked, the user clicks Retry > - Retry then started a fresh isolated attempt cloned from committed HEAD, so the agent lost its uncommitted files, its task branch and the worktree Git link > - A restart must never lose the agent's work > - This pull request makes Retry keep the task's own workspace when that is safe, and tells the agent where the old files are when a fresh attempt is still required > - The benefit is that a Retry continues the real work instead of starting over ## Linked Issues or Issue Description **What happened?** After `native_restart_recovery_blocked` or an exhausted recovery generation, a Retry ran in `isolated-workspace-attempts/<company>/<issue>/<run>/workspace`. That directory was a detached clone of the base ref. It did not contain the uncommitted edits from the issue's git worktree. It was not on the task branch. The agent was not told where the old files were. A restart-chaos wave graded this as an R2 failure (`pre_fault_files_lost`). A second defect: when a restart hit the bootstrap of a fresh attempt, preparing the same run again failed with `isolated_task_directory_binding_mismatch`, so the Retry could not finish. **Expected behavior** Retry continues in the task's own workspace with its edits and `.git` intact when no other writer can hold it. When a fresh attempt is really needed, the old directory stays as it was and the agent gets its path and its changed files. Preparing the same attempt again after a restart is idempotent. **Paperclip version or commit** `master` at `e482f0f632`, restart-chaos waves on 2026-11-06. **Steps to reproduce** 1. Start a native task in a git worktree. Let the agent write an uncommitted file. 2. Kill the control plane so recovery ends in `native_restart_recovery_blocked` or an exhausted generation. 3. Click Retry. 4. The successor runs in a fresh clone without the uncommitted file or the task branch. Root cause: - `server/src/services/fresh-workspace-continuation.ts` `canSelectFreshWorkspace` returns a fresh intent for every task Retry that has a system recovery record. - `server/src/services/explicit-native-continuation.ts` (fresh selection in the source loop) then always persisted `fresh_isolated_workspace`. - `server/src/services/fresh-workspace-seed.ts` `materializeFreshWorkspaceSeed` clones the project source at the base ref with `--no-local`, checks out a detached commit and removes `origin`. Uncommitted work and the branch are not carried over. - RC-35: `server/src/services/heartbeat/workspace-preparation.ts` rejected any reusable existing workspace while `fresh` was true, including the attempt directory that the same run had already created and bound. ## What Changed - New continuation mode `reuse_issue_workspace`. At Retry admission, after all stop proofs pass, `retainedIssueWorkspaceForRetry` checks that the task's bound workspace is the single local directory the failed run used, owned by the task, not a shared workspace, not under remote storage authority, with no other lease, failed cleanup or running workspace operation, a settled recovery generation, no workspace-export failure, and a readable Git state rooted at that directory. Then the Retry continues there with a fresh session. Otherwise the existing fresh attempt is kept. - The reuse decision consumes the generation's one alternative attempt (`freshAttemptRunId`), sets the Retry strategy to `continue`, and records `issue.retry_workspace_reused`. - Environment selection pins the reused workspace binding and fails with `retry_workspace_binding_mismatch` if the binding changed before dispatch. - Fresh fallback: the run's continuation context gets `priorWorkspace` (path, branch, bounded list of uncommitted files). Git status is read with optional locks disabled. The prompt tells the agent to inspect it and carry over the work. The old directory is not changed. - RC-35: preparing the same fresh attempt again after a restart reuses the attempt directory that this run already bound. - Reuse checks run Git through the shared workspace scan scheduler with short deadlines, because admission can hold the issue lock. - `doc/task-recovery.md` and `doc/execution-semantics.md` describe the new contract. ## Verification - `server/src/services/retry-workspace-continuity.test.ts` (embedded Postgres, real git worktree with uncommitted edits): - exhausted-generation Retry and blocked-recovery Retry continue in the same worktree, `.git` file and branch intact. Both failed before the fix (readiness was `fresh_isolated_workspace`). - Blocked recovery without a generation and duplicate holds for one run also reuse the worktree. A running operation orphaned by a restart of the stopped source does not block reuse. - The fresh attempt is kept when the workspace is unsafe: live lease, running workspace operation, corrupt `.git`, missing directory, remote authority, shared workspace, running export generation. `priorWorkspace` is omitted for the unsafe writer and shared cases. - `workspace-preparation.test.ts`: - A `reuse_issue_workspace` Retry prepares in the original worktree with its uncommitted file and branch. - RC-35 red/green: re-preparing the same fresh attempt reuses its bound directory and keeps files written before the restart. Another run still gets `isolated_task_directory_binding_mismatch`. - The fresh fallback sets `priorWorkspace` with the old path and `unpublished.txt`. - `environment-selection.test.ts`: the new mode pins the binding and rejects a changed binding or cwd. - `packages/adapter-utils/src/server-utils.test.ts`: the prompt names the retained directory and its files. - Regression files run locally: `workspace-preparation`, `environment-selection`, `task-retry`, `quarantined-task-retry`, `explicit-native-continuation`, `workspace-recovery`. - Restart-chaos (`tests/restart-chaos`, local, B1+B5, codex+claude, K1/K4/K6, concurrency 2, rc06-masked): - Final head: wave `w-20261107T014633-s7206` on `c6776283f3`: 8/8 PASS, all 8 restart-crossing. 5 tasks needed a Retry (t03-t07: codex B1 K6, codex B1 K1, codex B5 K6, codex B5 K1, claude B5 K6). All five finished in the original git worktree with `.git` and every pre-fault file present (`workspaceMovedTo` empty, `lostFiles` empty); no "Retry keeps a fresh attempt" log line. - Wave `w-20261107T004829-s7205` on `29373a6b6a`: 8/8 PASS, all 8 restart-crossing. 3 tasks needed a Retry (t02, t06, t07). All three finished in the original git worktree with `.git` and every pre-fault file present; no "Retry keeps a fresh attempt" log line. (`cc178492ef` only makes the operation filter stricter for run-less jobs.) - Wave `w-20261107T000412-s7204` on `d197082107`: 8/8 PASS, 7 restart-crossing. 3 tasks needed a Retry (t02, t04, t06). All three finished in the original git worktree (`workspaceMovedTo` is empty), R2 passed with `.git` and every pre-fault file present, and no "Retry keeps a fresh attempt" log line appeared. - Wave `w-20261106T231416-s7203` on the first commit: 6/6 PASS, but its t01 Retry still moved to `isolated-workspace-attempts`. That instance was not kept, so the exact rejection was not recorded. The likely cause is a `running` workspace operation that a killed server left on the stopped source (the source restarted three times). The second commit treats such restart orphans as not live and logs the rejection reason. A `--replay` of that t01 on `d197082107` finished in the original worktree with its `.git` file and no attempt directory. - Review follow-ups in `d197082107`, `29373a6b6a`, `cc178492ef` and `c6776283f3` (run-less workspace jobs still block reuse; red/green covered by the `running_operation` case): holds are grouped by source run; the readiness fallback (blocked recovery without a generation) also reuses the task workspace; Git probes go through the shared scan scheduler with 10-15 s deadlines; `priorWorkspace` is only set for a task-owned directory with no other writer; the reuse rejection reason is logged; `priorWorkspace` uses the same task-owned, settled, single-writer predicate as reuse; Git probes share one budget per workspace; the RC-35 check is a pure path check; renames are parsed from either porcelain column. ## Risks - A Retry now writes into the old directory instead of a new one. The checks above prevent a second writer. A stale binding fails the run with `retry_workspace_binding_mismatch`; it does not write to another directory. - If the cause of the failure was inside the workspace itself, reuse could repeat it. Workspace-export failures, restore failures and required workspace recovery keep the fresh attempt. - A workspace operation of the stopped source that started before the source settled and before the current server process no longer blocks reuse (a restart orphan). Operations with no run, of another run, or started in the current process still block it. - `priorWorkspace` is an optional new field in the continuation envelope. Older adapters ignore it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
a894b11a2b |
fix(runner): retry a rejected runner suspension and log every cleanup quarantine (#19653)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run's close must prove that its runner durably suspended
before the session's next run may start. If it cannot, the runtime
quarantines the session (`NativeSessionCloseUnrecoverableError`)
> - In a restart chaos storm, an ACPX Claude run succeeded and its
normal close quarantined its own session. The follow-up run on that
session failed 2 s later with `native_session_cleanup_quarantined`, and
no control-plane restart came between the two runs
> - The close sent `runner.suspend` once, the ACPX sidecar rejected it
right after the turn completed, and the barrier gave up at the first
rejected command. No log line showed the quarantine or its cause
> - This pull request retries a rejected suspension within the close
deadline, and logs every cleanup quarantine with its cause when it is
created
> - The benefit is that a successful run no longer blocks its own
follow-up because of a one-time suspension rejection, and any remaining
quarantine can be traced
## Linked Issues or Issue Description
**What happened?**
Run A (ACPX Claude) succeeded. The run log of its close shows:
- `runner did not prove durable suspension before checkpoint`
- `native_session_settlement_incomplete
{"runnerSuspended":false,"providerDrained":true,"suspensionState":{"commandStatus":"failed","runnerLifecycle":"suspended",...},"semanticTools":{"pending":[]},"finalProviderState":{"activeProviderTurnId":null,"providerSettled":true}}`
The runner journal shows `runner.drain` completed, then `runner.suspend`
failed with `failed to suspend ACPX provider: ACPX sidecar command
session.suspend was rejected (retryable=false,
classification=unclassified)`. The follow-up run B on the same session
was created 200 ms after A finished and failed 2 s later at admission
(`retryQuarantinedSessionCleanups`, operator recovery branch). The
server log had no line for the quarantine. One Retry completed the task.
**Steps to reproduce**
1. Run an ACPX task whose sidecar rejects the first `session.suspend`
after the turn completes.
2. Start a follow-up run on the same session.
**Expected behavior**
The close asks again, proves suspension, and the follow-up runs. If a
quarantine is still needed, it is logged with its cause.
**Actual behavior**
The close fails on the first rejection, quarantines the session, and the
follow-up fails with no logged cause.
## What Changed
- `runner/.../durable/runner.rs` (`process_command`): a failed
`runner.suspend`, on first execution or replay, now keeps the runner in
`Continue` instead of `Suspend`. Before, the runner recorded `suspended`
over a provider whose session was still open, then exited. The RC-34
journal shows exactly that: runner lifecycle `suspended`, ACPX provider
`session_open`. A failed `runner.shutdown` still stops the runner.
- `durable-prp-control-plane.ts`: a failed or rejected `runner.suspend`
now lifts the connection's terminal-lifecycle fence, so the next pending
command (the fresh suspend) is delivered on the same connection.
Completed suspensions and every shutdown keep the fence. New test:
"delivers a fresh suspension after a rejected one on the same
connection". Before the fix, the retry was never sent and the test timed
out waiting for the frame.
- `runnerd-codex-transport.ts` (`awaitRunnerSuspensionBarrier`): after a
rejected `runner.suspend`, the barrier sends a fresh suspend command, up
to 3 attempts, 250 ms apart, inside the same close deadline. An exited
runner or a spent budget still fails closed. A rejection changes no
state: the runner returns before it records suspension.
- `native-session-runtime.ts`: new
`onNativeSessionCleanupQuarantined(listener)`. It is called when a
cleanup quarantine is created or escalates to operator recovery, with
company, run, issue, session, `operatorRecoveryRequired`, a bounded
cause (error name, code, message) and bounded settlement facts.
- `server`: `logNativeCleanupQuarantines` logs each quarantine as a
warning. The subscription is process-wide and installed once, so closes
that fail during shutdown, after HTTP closes, are still logged.
## Verification
- Red before the change: `pnpm --filter @paperclipai/paperclip-runner
exec vitest run src/live/runnerd-codex-transport.test.ts -t "rejected
suspension"`.
- "retries a rejected suspension…" got `false`.
- "stops retrying…" saw 1 attempt instead of 3.
- Green after the change: both pass.
- Rust: new test
`rejected_suspend_keeps_the_runner_serving_its_controller` covers a
failed suspend, its replay, and an accepted retry with a fresh command.
`failed_lifecycle_commands_replay_their_terminal_transition` now expects
`Continue` for suspend and `Shutdown` for shutdown. The `pnpm run
check:runner` unit suites pass (559). `acpx_park_readiness` has two
timing tests (`< 500ms`) that fail at this host load on unchanged master
as well.
- New runtime test: "announces every cleanup quarantine with its cause
when it is created" (operator-required quarantine, cause and settlement
reported once). `src/native-session-runtime.test.ts` passes (212).
- The suspension cases in `runnerd-codex-transport.test.ts` pass, except
"proves local suspension after an event backlog… (1024 suffix deltas)".
That case also times out on unchanged master at this host load (27–35);
I confirmed this by stashing the change and rerunning.
- Server `tsc --noEmit` passes after the runner package build.
## Risks
- A provider that keeps rejecting suspension now gets three tries
instead of one, within the existing close deadline. The final outcome
and quarantine are unchanged.
- The cause of the ACPX sidecar's first rejection is not proven. The
logged evidence fits the sidecar's "not at a safe suspension point"
guard (`turnId`, tools or inputs still set) racing the just-completed
turn. This change makes close tolerate it. It does not change the
sidecar.
- Log volume: one warning per quarantine.
- The Rust change ships in the runner binary. Hosted images need a
rebuild to pick it up.
- After a failed suspend, the runner stays alive with its provider. If
every retry fails, close still treats suspension as unproven and kills
the owned runner process, exactly as before.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Refs #19532, #19609.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
|
||
|
DottaandClaude Opus 5.5
|
c37bc749ac |
fix(native): keep restart continuations alive through early results and repeat restarts (#19699)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs continue a turn lost to a control-plane restart once, in the same conversation (#15297, #19501) > - In restart storm 3, two Codex runs each crossed one restart and still needed a Retry > - The continuation was attempted, but the driver failed it with `turn_binding_mismatch`, and the run then reported "runnerd exited before provider turn startup" > - A second restart that loses the continuation turn itself also needed a Retry, because the one-shot marker was already used > - A restart should not cost a click when continuing is safe, and a failure should name its real cause > - This pull request fixes the start race, reports the driver's own failure, and gives each admitted restart its own continuation > - The benefit is that interrupted tasks finish without a Retry, within the existing recovery budget ## Linked Issues or Issue Description Refs #19501, #15297, #19658, #19539 **What happened?** Restart storm 3 (seed 7103, RC06 mask on, load 9–12). Codex B5 Retry runs crossed one K1 restart mid-turn. After the restart, recovery resumed the dead runner. The runner restored the conversation and recorded `turn.failed` with `provider_turn_lost_on_restore` (`recoverable: true`). The control plane started the same-conversation continuation (`resumeInterruptedTurn` → `startTurn`). About 1 s later the transport closed with `protocol_failure:turn_binding_mismatch`. The run failed as `native_runner_process_exited: runnerd exited before provider turn startup`. The task needed another Retry. I reproduced it with the focused cell (`--workloads B5 --faults K1 --triggers retry`) and a temporary diagnostic. The failure is "Provider semantic result did not name the active turn": a `paperclip/runResult` reached the Codex driver while the continuation's `turn/start` was still pending and no turn was bound. The driver waits for a pending start before it handles `turn/completed`, but not before it handles `paperclip/runResult`. **Expected behavior** After a restart that loses a turn, the run continues automatically when that is safe. Each admitted restart attempt can continue once, within the provider-attempt budget. A failure names its real cause. **Steps to reproduce** 1. Run `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex --workloads B5 --faults K1 --triggers retry --tasks 2 --concurrency 2 --seed 4040 --rc06-masked`. 2. When a lost turn's continuation receives a semantic result before its start settles, the run fails with `turn_binding_mismatch` and the task needs a Retry. The race is intermittent: 1 of about 14 lost turns in my runs. **Root cause** - `packages/paperclip-runner/src/drivers/codex/codex-session-notifications.ts`: the `paperclip/runResult` branch checked the active-turn binding without waiting for `state.turnStartSettled`. The `turn/completed` branch already waits. - `codex-harness-session.ts` `startTurn`: the driver's own protocol failure closed the transport. The pending start then reported the runner exit, which hid the cause. - `codex-session-terminal.ts` `mapTerminalTurn`: a consumed one-shot marker blocked every later continuation in the run, even when the newly lost turn was that continuation itself. The one-continuation allowance is per run, not per issue or session lineage: attaching a new run resets the marker. The storm runs did attempt their continuation; the start race made it fail. ## What Changed - Codex driver: a `paperclip/runResult` waits for a pending turn start to settle before it is bound, like a terminal. - Codex driver: when this session's own protocol failure ends a pending turn start, `startTurn` throws `codex_turn_start_protocol_failure: <code>: <message>` instead of the runner exit. CodexRpc errors and protocol-integrity errors keep their own types. The server now classifies this as `native_session_interrupted` instead of `native_runner_process_exited`; both are retryable. - A later restart that loses the continuation turn itself re-arms the one-shot marker: the driver resets it as it was before the first continuation. It does this when it maps the lost terminal, including during recovery from a checkpoint taken while the continuation ran. The checkpoint gate `nativeRestartInterruptedTurnId` stays strict: a consumed marker still means the allowance is used. Each continuation still needs an admitted dead-runner restart. The existing provider-attempt budget bounds the repeats, and an exhausted budget still blocks with Retry. - `doc/execution-semantics.md`: an additive paragraph. ## Verification Red before the fix, green after: - `codex-app-server-driver.events.test.ts` "binds a semantic result that arrives before its pending turn start settles". Red: `session.failed`. Green: the result binds to `turn-1` and the turn completes. - `codex-app-server-driver.events.test.ts` "names its own protocol failure when that failure ends a pending turn start". Red: "runnerd exited before provider turn startup". Green: `codex_turn_start_protocol_failure: turn_binding_mismatch`. - `codex-app-server-driver.recovery.test.ts` "offers another restart continuation when a later restart loses the continuation turn itself". Red: `null`. Green: `turn-2`. A checkpoint with a consumed marker still returns `null`. - `codex-app-server-driver.recovery.test.ts` "re-arms the continuation when recovery finds the checkpointed continuation turn lost". This recovers from a checkpoint taken during the continuation, then starts the next continuation. Red without the re-arm (the marker stays consumed); green with it. - `native-runner-restart-recovery.integration.test.ts`: the same checkpoint (continuation active, consumed marker naming it) is claimed as `resume_dead_runner`. This is regression coverage. Suites run locally on the final code: - Runner: Codex driver, `native-session-runtime`, `runnerd-restart-continuation` and contracts (745 tests). - Server: `native-runner-restart-recovery.integration.test.ts`. - `tsc` for the runner tests and the server. - A few real-process tests timed out at about 6 s while six live agents ran. Each passed when rerun alone. The settlement suite passed both with and without this change at the same load. Live (local, real Codex, macOS arm64), focused restart-chaos cell `--adapters codex --workloads B5 --faults K1 --triggers retry,time --tasks 6 --seed 4042 --rc06-masked`: 6/6 crossed a restart and passed. R1 passed for all 6, so each continued without a Retry. ## Risks - A run can now continue more than once in the same conversation, once per admitted restart. Each continuation asks the agent to reconcile unfinished actions before repeating them. The provider-attempt budget still caps the repeats. - A semantic result that arrives during a pending start now waits for that start. If the start fails, the result is still rejected by the existing binding check. - The start race was intermittent (1 of about 14 lost turns), so the live cell shows absence of the failure, not a forced repro. The driver regression reproduces it deterministically. - No schema, API or migration change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, code execution, and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
9a52a2dfeb |
fix(native): retry ACPX session.open startup timeouts and give cold Claude a 60 s bound (#19661)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs start an ACPX provider through Runner. `session.open` opens the provider session before any prompt. > - On a loaded host, a cold ACPX Claude `session.open` took longer than its 30-second bound. The controller reported `runner_local_connect_failed: PRP command session.open timed out`. > - The server classified this as the generic `native_session_interrupted` with `ambiguous_state`, so the task was blocked at once. A person had to click Retry, sometimes several times. In one restart-chaos storm, one task needed three Retry clicks, and on another task both Retry runs timed out too. > - The server already has a bounded same-run bootstrap retry for failures that come before provider work (#18854). It did not recognize this timeout. The 30-second bound also cut off normal slow startups. > - This pull request gives cold ACPX Claude a 60-second `session.open` bound, like Pi and OpenCode (#19487). It also retries a startup `session.open` timeout automatically on the same run, from clean state. > - The benefit is that a slow cold start on a busy host completes, or retries itself, instead of waiting for a person. ## Linked Issues or Issue Description Refs #18854 (bounded bootstrap retry for definite `session.open` failures), #19487 (60-second cold OpenCode admission), #19665 (the bootstrap proof that this change reuses). **What happened?** ACPX Claude runs failed with `runner_local_connect_failed: PRP command session.open timed out` about 35 to 40 seconds after they started. The coordinator recorded `native_session_interrupted` with `recoveryMode: "ambiguous_state"` and `providerSessionEstablished: false`. The task was blocked with a board-owned action. No provider event and no checkpoint existed. The host 1-minute load was 17 to 44, sometimes with no restart nearby. **Expected behavior** A normal slow cold start completes. A `session.open` that still times out during runner startup, with no provider event, no checkpoint and no turn, retries on the same run with backoff. At most two automatic attempts follow the first. A run where provider work may have started is never retried automatically. **Steps to reproduce** 1. Run ACPX Claude tasks on a host with a 1-minute load of about 25 or more. 2. Observe a run fail after about 35 to 40 seconds with `PRP command session.open timed out`. 3. Observe the task blocked with `ambiguous_state` and a manual Retry. **Paperclip version or commit** `master` at `6c7cc2c5d6`. ## What Changed The bound (evidence below): - `coldAdmissionTimeoutMs` gives cold ACPX Claude 60 seconds (live adoption keeps 30 seconds). The runner's Rust sidecar `session_open_timeout` and the sidecar's provider handshake (`acpxSessionHandshakeTimeoutMs`) give Claude and Pi 60 seconds too. All three bounds must agree, or the shortest one fails first. Claude's warm run attachment uses the same cold budget, like Pi. The automatic retry: - `nativeSessionFailureSourceCode` returns `provider_session_open_timeout` only for exactly `<startup failure code>: PRP command session.open timed out`. The startup failure codes are `runner_local_connect_failed`, `runner_direct_wss_failed` and `runner_ingress_unavailable`. The transport uses them only before startup completes. After startup the code is `provider_transport_failed`, which is unchanged. - `nativeProviderRecoveryEvidence` treats the cause as pre-session. Durable evidence still decides: a bootstrap retry needs no provider event (`session.started`, `turn.started` and the others) and no checkpoint. The existing budget applies: three attempts with backoff of about 30 seconds, then about 60 seconds. When the budget is used, the task is blocked with the existing "Provider startup failed on 3 attempts before any provider work began" action. - A timeout is indeterminate: the runner can open a provider session after the controller stops waiting. Live runs showed both leftovers, so the retry now starts from clean state: - `prepareNativeSessionOpenTimeoutRetry` (server) runs when the failure settles. It quarantines a local runner root only when the restart-recovery proof (`runnerdStateProvesIncompleteBootstrap`, #19665) shows that no turn began and every recorded process is gone. Any other root, and any remote runner, stays `ambiguous_state` as before. - `resetAbandonedBootstrapRecord` (runner ACPX adapter) runs on a fresh open (no runner-recorded identity). If ACPX has a saved record for the session key with no messages, no prompt and no events, it is marked `reset_on_next_ensure`. Claude saves nothing before the first prompt, so reusing such a record fails the first turn with `session_resume_required`. A record that holds any conversation is unchanged. - A chat retry of an exhausted `provider_session_open_timeout` run is accepted like the other pre-provider causes. - `doc/execution-semantics.md` and the Runner README describe the cause and the bound. ## Verification **Measured startup distribution** (ACPX Claude `runner.session.bootstrap` / `runner.session.startup` spans from restart-chaos run events, 1-minute load 8 to 44): | Source | Bound | Successful cold bootstraps (s) | Failures | | --- | --- | --- | --- | | Earlier storm evidence | 30 s | 2.9, 4.7, 7.4, 8.6, 8.7, 26.2, 27.4, 27.6, 29.2 | startups ended at 34.9 to 40.3 s (the 30 s `session.open` bound plus spawn and `run.prepare`) | | This branch, wave s3601 | 60 s | 6.3, 7.8, 8.1, 13.4, 14.4, 22.3, 22.8, 22.9, **37.3** | 0 `session.open` timeouts | | This branch, wave s3602 | 60 s | 13 samples, p50 10.5, p90 17.9, max 25.7 | 0 `session.open` timeouts | | This branch, wave s3610 (final head, load 28 to 42) | 60 s | 10.4, 11.3, 11.7, 12.0, 13.3, 14.2, 29.6, **30.4**, **30.9** | 0 `session.open` timeouts | Under load, successes clustered just under 30 s and failures sat just above it, so the 30 s bound was cutting the distribution off. With 60 s, cold starts of 30.4, 30.9 and 37.3 s succeeded that the old bound would have failed. 60 s is about 1.6 times the largest observed success, and it matches Pi and OpenCode. **Live restart-chaos waves** (local, ACPX Claude only, 4 in flight, `--rc06-masked`, real provider, K1/K2 restarts): - s3601 (8 tasks, load 19 to 32): 7/8 PASS. No `session.open` timeout. The one failure is an R3 Retry signature after K1 (`native_restart_recovery_blocked`), not this cause. - s3602 (8 tasks, load 17 to 39): 8/8 PASS. No `session.open` timeout. Three Retry clicks were for `native_provider_terminal_failed` (`provider_turn_lost_on_restore`), a different restart signature. - Forced timeout, s3609 (4 tasks, B5, K2): a local, uncommitted fault injection gave every other cold startup a 1.5 s `session.open` deadline. Result 4/4 PASS. Run 4fe6537a timed out, retried itself on the same run, and then succeeded through four K2 restarts with no person involved. Run 02799a85 hit the injection three times in a row; the budget stopped it after attempt 3, and one Retry completed it. - Before the two clean-state fixes, the same forced wave (s3607 and s3608) showed the retry reusing leftover state: `turn.start` failed with `session_resume_required`, and then "ACPX session is not open". Those runs are the red evidence for the two clean-state fixes. - Final head, s3610 (8 tasks, K1/K2, no injection, load 28 to 42): **8/8 PASS**, 0 `session.open` timeouts. The one Retry click (t06) was for `provider_turn_lost_on_restore`, a different restart signature. **Tests:** - Red before the fix: `native-startup-retry.integration.test.ts -t timeout` (embedded PostgreSQL, real executor, real resume claim). The 3 retry cases failed with `native_session_interrupted` / `ambiguous_state`. - Green: `native-startup-retry.integration.test.ts` passes 36/36. New cases: - for each startup failure code, a timeout retries on the same run with backoff, keeps the task `in_progress` with an agent-owned `resume_native_run` wake, and blocks after attempt 3 with a board-owned action; - no automatic retry when `session.started` or `turn.started` is durable, when the code is `provider_transport_failed`, or when another command timed out; - a real runner root with an opened session and no turn is quarantined; a root with a turn command stays `ambiguous_state` and intact; a remote runner is never prepared. - `codex-runtime-adapter.test.ts`: 113 pass. New cases: a 45 s delayed handshake is admitted for Claude and rejected for Codex; a never-prompted record is reset only on a fresh open; records with messages, a prompt or events are kept. - `abandoned-bootstrap-record.test.ts` (real `createAcpRuntime` with `createRuntimeStore` and a controlled ACP peer that, like Claude, cannot load a never-prompted session): without the reset, a fresh open reuses the record and the first turn fails with SESSION_RESUME_REQUIRED after `session/load`. With the reset, the provider receives `session/new` and the turn completes. - `vitest run src/drivers/acpx src/cli` in the runner: 107 files, 1,935 tests pass. - `native-session-executor.test.ts -t "session.open|startup|bootstrap"`: 144 pass. `runnerd-codex-transport.test.ts -t "cold process admission"`: 4 pass. `cargo test -p paperclip-runner-core --lib admission_budget` passes. - `npx tsc --noEmit -p server` and the runner `tsc -p tsconfig.test.json` pass. ## Risks - **This changes Runner Rust code. The Daytona runner image must be rebuilt after merge** for remote sandboxes to get the 60-second sidecar bound. Until then, a remote Claude `session.open` still fails at 30 seconds in the sidecar (as a definite failure, which already retries). A remote timeout stays operator-owned, because a remote root cannot be proven clean here. - A cold ACPX Claude start that hangs now fails after about 60 seconds instead of 30 seconds. - A timed-out `session.open` may have created an empty provider session. No prompt was sent. The retry quarantines that runner root and asks ACPX for a new session. - Same-run retries of a fresh-attempt Retry run can still fail with `isolated_task_directory_binding_mismatch`. This is RC-35 (workspace re-preparation), which has its own fix in progress. - No schema or API change. The new error code appears on runs and in run error codes. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
20fc37778f |
fix(runner): keep settled OpenCode sessions across a controller proxy rebuild (#19641)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native Runner keeps an OpenCode conversation in durable state,
so a task can continue after a control-plane restart.
> - Saved OpenCode state binds one digest of the launch profile: the
Node command, the proxy bundle and the OpenCode executable, each by path
and SHA-256.
> - The Node command and the proxy bundle are controller runtime. A
rebuild or deploy between a hard kill and recovery replaces their bytes.
> - The recovery probe then refused a settled conversation with
`provider_state_incompatible`, although the OpenCode release and the
conversation were unchanged.
> - ACPX already re-binds a settled session after a Runner upgrade
(#18879). OpenCode did not.
> - This pull request records the OpenCode release digest in saved state
and re-binds a settled session when that release is unchanged.
> - The benefit is that a task continues automatically after a restart
that also changed the controller build. A real release change still
fails closed.
## Linked Issues or Issue Description
Refs #19558, #19501, #18879.
**What happened?**
A restart-chaos wave killed the control plane with SIGKILL while an
OpenCode run was settled between turns. The first recovery attempt
restored the session. The controller's runner `dist` was then rebuilt in
the same checkout. The next recovery attempt failed in the
`runner.drain` resume probe with `OpenCode runner launch profile changed
across durable recovery` (`native_provider_state_incompatible`). The
saved state matched the original proxy bytes exactly. Only the proxy
bundle bytes had changed. The OpenCode executable, the paths and the
model were the same.
**Expected behavior**
A settled OpenCode session continues after a restart when only
controller runtime bytes changed. A changed OpenCode release, or a
session with work in flight, fails closed. The existing disposition then
offers one Retry that continues in a fresh provider session.
**Steps to reproduce**
1. Start an OpenCode run and let it settle between turns.
2. Kill the runner and the controller.
3. Replace the qualified proxy bundle with valid new bytes (rebuild or
deploy).
4. Recover the run. The resume probe fails with
`provider_state_incompatible`.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
e05bced39f |
fix(runner): adopt an accepted Codex turn whose start outlives the bound (#19608)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs submit each turn through runnerd. The transport
answers `turn/start` only after it sees the provider's `turn/started`
for the turn the durable command result named
> - That wait has a fixed 30 s bound
> - At host load around 50, after two hard restarts, Codex accepted a
turn and started it about 27 s after the command, beyond the bound once
scheduling delay was added. The transport failed the start, but the
provider kept running the turn and made a tool call
> - No controller owned that turn. Its close could not settle, and the
task's Retry was refused at once
> - This pull request adopts the accepted turn when the bound expires,
but only when the runner still reports that turn active
> - The benefit is that a slow but live provider turn stays owned, so
its events, Stop and close keep working
## Linked Issues or Issue Description
**What happened?**
In a restart chaos replay (two SIGKILLs about 20 s apart, local runner,
real Codex, load 50), a resumed run failed with `runnerd did not report
the provider turn identity`. The runner journal shows the durable
`turn.start` result with the provider turn id, then `turn.started` for
that same turn 27 s later, followed by items and a semantic tool call.
The run's close then stayed incomplete (`semanticResultsSettled:
false`), and a Retry of the task failed with
`native_session_cleanup_quarantined: prior session cleanup remains
incomplete`.
**Steps to reproduce**
1. Use a Codex provider that answers `turn/start` and emits
`turn/started` later than the transport's turn-start bound.
2. Start a turn through the runnerd transport.
**Expected behavior**
The accepted turn is owned and observed by the controller.
**Actual behavior**
The start fails after the bound, while the provider still runs the turn.
## What Changed
- `runnerd-codex-transport.ts` (`#startTurnWork`):
- Near the end of the bound, if `turn/started` has not arrived, the
transport reads an authenticated `session.snapshot`. It reserves at most
5 s (or a quarter of the bound) for this read.
- It binds the accepted turn only when the turn is provably live:
- the runner reports that exact provider turn as active;
- the runner attributed no provider launch
(`provider_startup_ownership`) to a command after this `turn.start`;
- for a local provider, the same process (pid and start time) that
accepted the turn is still alive.
- Otherwise it keeps waiting for `turn/started` until the configured
bound and fails exactly as before. A turn lost across a provider
relaunch keeps the same-run resume path.
- `fake-codex-app-server.rs`: new test options `--turn-started-delay-ms`
(a delay between the `turn/start` response and `turn/started`) and
`--exit-before-turn-started`.
## Verification
- Red before the fix: `pnpm --filter @paperclipai/paperclip-runner exec
vitest run src/live/runnerd-codex-transport-deadlines.test.ts -t "adopts
a durably accepted"` (real runnerd and fake Codex, 12 s start delay, 8 s
bound) failed with `runnerd did not report the provider turn identity`.
This is the replay's error.
- Green after the fix: the start resolves with `provider-turn-1`, and
`turn/started` and the turn's terminal follow in order. There is one
provider `turn/start`.
- Negative case (`does not adopt an accepted turn the runner no longer
reports active`): the provider accepts the turn and exits before
starting it. The runner relaunches the provider and still lists the
turn, but the turn is not adopted. The start fails within the bound plus
scheduling slack. A snapshot-only version of this change adopted that
dead turn, and it hung for 20 s with no terminal.
- `src/native-session-runtime.test.ts`,
`src/live/runnerd-restart-continuation.test.ts` and
`src/live/runnerd-codex-transport-deadlines.test.ts` pass. The package
typecheck passes.
- Live chaos replays: results are added in a comment.
## Risks
- A provably live provider that accepts a turn and never starts it now
waits on the turn's own timeout and Stop paths instead of failing after
30 s. A dead or relaunched provider still fails at the bound.
- Remote runners can adopt only on the runner's own evidence (active
turn and no later provider launch), because their provider pid is not a
local process.
- A stale `turn/started` for a previous turn that arrives after the new
command result is still rejected. The transport cannot attribute that
turn's following events, so rejecting it fails only the transport, and
the same-run resume continues the task (seen in the replay). Delivering
it would attach the old turn's items and usage to the new turn.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Refs #19532, #19539.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
|
||
|
DottaandClaude Opus 5.5
|
785ead5c7a |
fix(tasks): hold a Retry while the task's own native session close is retained (#19609)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native provider session close fails, the runner keeps a cleanup owner or quarantine in process memory. Its admission gate refuses new sessions for that task (scoped to the task since #19532) > - The server could not see that state. A task's Retry was accepted (202), its run was dispatched, and the runner refused it at once with `native_session_cleanup_quarantined: prior session cleanup remains incomplete` > - The user saw a Retry that failed instantly, and nothing said what to do > - This pull request lets the server read the runner's per-task cleanup gate and hold the Retry with a clear reason. The Retry starts automatically when the close is released > - The benefit is that a Retry is never dispatched only to fail on a block that is already known ## Linked Issues or Issue Description **What happened?** After two hard restarts, a Codex run's close stayed incomplete. The board Retry for that task was accepted, and its run failed in about 1 s with `native_session_cleanup_quarantined: prior session cleanup remains incomplete`. The next Retry did the same. **Steps to reproduce** 1. Let a native session close fail or stay pending for a task. 2. Press Retry on that task. **Expected behavior** The Retry waits with a clear reason, and starts when the cleanup finishes. If the cleanup needs an operator, Retry is not offered and the one working action is shown. **Actual behavior** The Retry is accepted and its run fails at once. ## What Changed - `native-session-runtime.ts`: - New `nativeSessionCleanupBlock({ companyId, issueId })`. It is a read-only view of the in-process admission gate for one task: `pending` (a retained close or a bounded automatic close recovery still owns the task's provider resources), `operator_required`, or `null`. It grants no authority. - New `onNativeSessionCleanupSettled(listener)`. It is called whenever a cleanup owner or quarantine entry is released. - `explicit-native-continuation.ts` (the continuation gate for a Retry or a new message): - `pending`: the input is held with `local_cleanup` ("Retry is waiting for the previous execution to finish cleanup. Your message is saved."). It starts automatically when the close is released. - `operator_required`: handled exactly like a recorded `native_session_cleanup_quarantined` run. Retry is not offered. A new message runs the existing stopped-process proof (`verifyStoppedNativeSessionForContinuation`), which retires the runner quarantine when the old processes are proven stopped. - `native-cleanup-release-wake.ts` (registered in `server/src/index.ts`): each cleanup release wakes deferred execution for the company in a transaction, so a held Retry re-evaluates. - `server/src/vendor/paperclip-runner/index.ts`: exports the two runtime functions. ## Verification - Red before the gate (server change stashed): `pnpm --filter @paperclipai/server exec vitest run src/services/quarantined-task-retry.test.ts -t "holds a Retry"` dispatched a successor run while the task's close was retained (`expected [ { … } ] to have a length of +0 but got 1`). - Green after the gate: - The Retry is held (`deferred_issue_execution`, `executionWait.reason = local_cleanup`), and no successor is dispatched. - A different task is not blocked. - When the close settles, the release is announced, and the next pass dispatches exactly one successor. - The whole `quarantined-task-retry.test.ts` file passes (28 tests). `src/native-session-runtime.test.ts` passes. Server `tsc --noEmit` passes after the runner package build. - Live chaos replays: results are added in a comment. ## Risks - The gate reads process memory, so it applies only where the server and the runner session share a process (local runner). Other deployments keep today's behavior. - A held Retry re-evaluates on the release wake (sent in its own transaction) or on the existing 30 s deferred-execution pass. When the bounded automatic close budget is spent with nothing else scheduled, the view reports `operator_required`. Retry is then no longer offered, and a new message reaches the stopped-process recovery instead of waiting forever. - No schema or API shape changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #19532. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Nicky LeachandClaude Opus 5.5
|
fb3cf1a072 |
feat(ci): retry a failed Vitest case once in PR CI and report each retried pass (#19637)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors wait for PR CI before merge, so CI wall-clock time sets the pace of work > - PR CI runs Vitest shards with 4 workers on 4-vCPU runners, so timing-sensitive tests sometimes time out under load > - One such timeout fails the whole run, and a re-run costs another 15 minutes or more > - In one 7-hour window on 2026-11-06, about 30 failed jobs were test flakes on commits that passed on a later attempt > - This pull request retries a failed Vitest case once in PR CI only, and reports every pass that needed the retry > - The benefit is fewer whole-run re-runs, while each flaky test stays visible so that we can fix it ## Linked Issues or Issue Description **What existing behavior does this improve?** PR CI fails the whole run when one Vitest case fails once because of runner load. **Subsystem affected** CI: the PR Vitest lanes (general, serialized, workspace and Runner package lanes). **Current behavior** Vitest uses `retry: 0` everywhere. One timeout under load fails the run. A person must notice it and re-run the failed jobs. About 20% of recently merged PRs needed at least one re-run. **Proposed behavior** PR CI retries a failed Vitest case one time. A test that passes only on the retry gets a GitHub warning annotation and a row in the job summary. Local runs, release, cloud readiness and master keep `retry: 0`. **Reason and benefit** A re-run of one test case costs seconds. A re-run of the CI run costs 15 minutes or more. The report keeps flaky tests visible. Separate pull requests fix the known flaky tests. **Breaking changes** None. ## What Changed - Add `scripts/ci/vitest-retry.mjs`. It sets 1 retry when `GITHUB_ACTIONS=true` and `GITHUB_WORKFLOW=PR`, and 0 in other contexts. `PAPERCLIP_VITEST_RETRY` overrides the value. - Add `scripts/ci/retry-reporter.mjs`. It writes a `::warning` annotation and a step-summary table row for each test that passed only on a retry. - `scripts/run-vitest-stable.mjs` passes `--retry` to `vitest run`. The `vitest list` collection step does not retry. - The root `vitest.config.ts` and `packages/paperclip-runner/vitest.config.ts` add the retry reporter beside the existing reporters. The Runner config sets `retry` from the same gate. - `.github/workflows/pr-trusted.yml` reads the `CI_VITEST_RETRY` repository variable. Set it to `0` to turn retries off without a code change. - Add `scripts/ci/retry-reporter.test.mjs` to `test:release-registry`, and document the behavior in `doc/DEVELOPING.md`. ## Verification - `node --test scripts/ci/retry-reporter.test.mjs`: 11/11 pass. The tests cover the gate matrix, the reporter output, a real Vitest run with a test that fails once, the wrapper arguments, and the Runner config. - `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`: 39/39 pass. - `node --test .github/scripts/tests/*.test.mjs`: 744/744 pass. - Manual check: a temporary test that fails once ran through `run-vitest-stable.mjs` and through the Runner package config. With the PR environment it passed and wrote the annotation and the summary row. With `GITHUB_WORKFLOW=Release` it failed. The temporary test is removed. ## Risks - A retry can hide a flaky test. The annotation and the summary row reduce this risk. - A test that fails for a real reason now runs twice in PR CI, so a failed run takes slightly longer. - `pr.yml` loads `pr-trusted.yml` from master, so the `CI_VITEST_RETRY` switch works only after merge. The retry itself is in the scripts and starts with this pull request. - Low risk otherwise: release and master verification are not changed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. A Claude sub-agent wrote the change, and the parent session reviewed it. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
DottaandPaperclip
|
a0ac81e114 |
fix(ui): show task image results in compact thumbnail grids (#19626)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task feeds show agent messages, documents, and files in time order. > - Each image currently uses a full feed card. > - A turn with seven screenshots makes the conversation hard to scan. > - This pull request groups adjacent images from the same run in a compact grid. > - Users can scan the results and open each image in the existing gallery. ## Linked Issues or Issue Description **What existing behavior does this improve?** Image results in task history and the live task feed. **Current behavior** Each image uses a separate full-width card. Native image upload receipt messages can also separate related screenshots. **Proposed behavior** Adjacent images from one run share a wrapping thumbnail grid. New image results join the grid. Documents, video, and authored messages keep their positions. The grid opens the existing gallery. **Reason and benefit** Users can see a batch of screenshots without scrolling past one large card for each file. **Breaking changes** The feed layout changes. The stored comments, files, APIs, and permissions do not change. Related public work: Refs #17228. Refs #8785. Refs #7667. These changes supplied the artifact cards and media gallery that this pull request reuses. ## What Changed - Add one image grouping function for settled history, turn history, and live results. Group only images with the same known run. Keep the first row key and each image key when results arrive. - Add a token-based grid that reuses the media artifact cards and image gallery. - Add an optional typed attachment-preparation marker to file registration and chat-file reuse. Preserve the marker through comment normalization. Use it with matching image, author, and run links to hide generated receipts. Keep the exact historical sentence as a fallback for old receipts. - Add nine production component stories with seven sample-account screenshots. Cover mobile, Classic, mixed results, single images, missing images, streaming, gallery navigation, and the complete feed projection. - Add grouping and production-feed regressions. Add 18 browser cases across light and dark themes. - Improve contrast for the shared fallback label and gallery counter after the accessibility checks found low contrast. - Match alternate original-image paths when opening the gallery. Keep a separate Open original link beside Download in the gallery. - Record the feed layout in DESIGN.md. ## Verification - Final qualification at `09b12381ce85bee6593050bd8a2fa18cbe2c45a6`: [required CI passed](https://github.com/paperclipai/paperclip/actions/runs/42317080212/job/128237094513), including typecheck, build, all test lanes, E2E, and verify. The snapshot has 58 successful check runs, three conditional skips, one neutral advisory job, and a successful legacy Snyk status. No pending or failed checks remain. Greptile and Paperclip Review both score this exact head 5/5; all review threads are resolved. - Current-head local checks: 544 focused UI tests passed across 11 files, with one existing skip. UI/shared typechecks and token gates pass. All 18 focused Storybook browser cases passed after the CSS merge, with retries disabled. - Two database integration cases verify file registration and chat-file reuse receipt markers. All 52 issue-validator tests and six upstream default-model tests passed. - Regressions failed before the fixes for unknown-run grouping, receipt wording, and authored progress text with an image. The repair uses a typed receipt marker and requires matching author, attachment, image, and run links before hiding a generated receipt. All five review findings are resolved. - Browser checks cover loaded images, wrapping columns, overflow, missing-image fallback, streaming stability, keyboard gallery navigation, original-file links, and downloads. Fixtures use production renderers. These checks do not prove deployed persistence or provider behavior and do not compare visual baselines. - Local limits: the preflight helper could not finish because Cargo was absent. The earlier low-memory server typecheck and dependency-free selector did not replace final-head CI. The now-passing final CI supplies broad typecheck, build, server, Runner, and E2E coverage. - Prior independent CI failures remain recorded: one Slack timer case passed a single diagnosed rerun; its exact timer cause was not proved. A later chat shard lost its self-hosted runner without a test assertion failure. The current revision passes that shard and all applicable gates. Master conflict resolution retained both image-grid and patch CSS rules. ## Risks - Images without a run ID remain separate. Messages and other resource types end each batch. - Receipt suppression requires a matching preparation marker or the exact legacy sentence, plus matching attachment, image work product, run, and author. Tests preserve human messages and native progress text with attachments. The optional marker uses existing JSON metadata and needs no database migration. - Shared gallery and fallback text now have stronger contrast. ## Model Used OpenAI GPT-6, through Codex. The agent used reasoning, repository tools, code execution, and browser tests. The runtime did not expose a more specific model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
7c673e695f |
fix(chat): name tasks by identifier, not UUID, in tool rows and agent text (#19635)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - New users meet Paperclip through the onboarding chat with their
first agent, and through task briefs that agents write
> - A browser walk of onboarding found internal handles and
control-plane jargon in those user-facing surfaces
> - A tool row showed `Read taskId: e3ecf9bc-…`, an agent-written brief
cited `task … (id 2b330993-…)`, and a Codex agent told a new user about
"receipts", "execution policy", and a "rejected policy patch"
> - Raw UUIDs and mechanism narration make the product look broken and
give the user nothing to act on
> - This pull request makes tool rows name a task by its identifier, and
tells agents to reference identifiers and to describe outcomes, not
mechanics
> - The benefit is that a new user sees "Read <task identifier>" and
plain progress updates
## Linked Issues or Issue Description
**What happened?**
In the task chat, a collapsed tool row rendered the raw call argument
`taskId: <uuid>`. An agent-written task description included a raw task
UUID next to the task identifier. A Codex chief of staff narrated
internal mechanics (receipts, execution policies, a rejected policy
patch) to a new user in chat.
**Expected behavior**
Tool rows name the task by its identifier. Agent-written text links
tasks and documents by identifier. Agents tell the user what happened,
what it means, and the next step.
**Steps to reproduce**
1. Run onboarding with the Codex API runner and accept a single-task
proposal.
2. Watch the handoff in the chief-of-staff chat.
3. See the task lookup tool row and the narration. With the Claude API
runner and the plan path, open a child task created from the plan and
read its description.
## What Changed
- `ui/src/components/task-chat/transcript-adapter.ts`: a tool row's
target never contains a UUID argument. A `taskId`/`issueId` UUID is kept
as `taskRefId` on the tool item.
- New `ui/src/components/task-chat/task-reference.ts`:
`useTaskReferenceLabel` reads the task from the shared issue detail
query cache and returns its identifier. It returns "task" until the task
resolves or when it cannot.
- `TaskChatRunnerActivityGroup` and `TaskChatToolCard` show that
identifier as the row subject. They load the task only for rows that
have a task reference.
- `skills/paperclip/SKILL.md` (Comment Style): two new required rules.
"No raw IDs in prose" and "Describe outcomes, not mechanics".
- First-task onboarding skill: a "Talk in outcomes" rule. Non-inline
handoff inputs use an identifier link, not a raw UUID.
- Chief-of-staff persona (`AGENTS.md` for the first agent): the same
plain-language rule, so it applies in every chat.
- `skills/paperclip-converting-plans-to-tasks/SKILL.md`: task
descriptions reference other tasks and documents by identifier link,
never by raw UUID.
- No change to `skills-releases/` snapshots. They are frozen releases.
## Verification
Red on master (fix files reverted, new tests kept):
```
ui tool-row-task-reference.test.tsx
FAIL ... > falls back to a plain word while the task cannot be resolved
AssertionError: expected 'ReadtaskId: e3ecf9bc-099a-4b10-8169-9…' not to contain 'e3ecf9bc-099a-4b10-8169-9ce490a91e3d'
Tests 3 failed (3)
server user-facing-wording-guidance.test.ts
Tests 6 failed (6)
```
Green with the fix:
```
ui tool-row-task-reference.test.tsx Tests 3 passed (3)
server user-facing-wording-guidance.test.ts Tests 6 passed (6)
```
Neighbouring suites:
- `cd ui && npx vitest run src/components/task-chat
src/components/TaskChatThread src/components/transcript`: 70 files, 1448
passed, 1 skipped.
- From the repo root: `onboarding-first-task-assets`,
`paperclip-skill-utils`, `onboarding-first-task-admission`,
`hiring-operational-examples`, `cli-invocation-safety`, and the new
guidance test all pass.
- `pnpm check:token-gates`: all gates clean. The change adds no styles.
- `ui` typecheck: no errors in the changed files.
- Not run locally: `agent-skills-routes.test.ts` hit a 30 s `beforeAll`
import timeout on a cold cache. CI covers it.
## Risks
- Low risk. A row that names a task loads that task once through the
shared issue detail cache. The thread usually has it cached already. A
failed load shows "task" and does not retry.
- Rows now show other non-UUID arguments after the identifier (for
example `<identifier> · status: done`). Rows without a task reference do
not change.
- The skill text changes the guidance that agents read. They do not
change permissions, tools, or API behavior.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run in Claude Code
with tool use (shell, file edits, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
|
||
|
DottaandPaperclip
|
19c2bebc9c |
fix(runner): settle durable provider exits on open streams (#19657)
## Thinking Path > - Paperclip manages AI agents and their work. > - Runner keeps the native session and its event stream until a turn ends. > - A provider can exit with a terminal error while its event stream stays open. > - Waiting for that stream to close leaves the run and its resource owner active. > - The existing durable failure receipt already identifies the failed current turn, including an authoritative replay. > - This pull request uses that event to enter the existing failure and cleanup path. > - Recovery and later idle retirement can then proceed from a real terminal result. ## Linked Issues or Issue Description Refs #18988 for native resource retirement. This is a separate terminal-event defect found during failed-turn recovery qualification. A search of open and closed PRs found no duplicate `provider_exited` fix. The related merged #12596 changes Rust test polling; it does not change this TypeScript session owner. **What happened?** `executeNativeSession` records a nonrecoverable `provider_exited` event, but can continue to wait on an open event iterator. A proposed result does not authorize success, and stream EOF may never arrive. The run cannot reach its existing failure and resource cleanup path. **Expected behavior** Settle the committed or authoritative duplicate terminal failure for this exact run, session and provider. Close through the existing owner. Preserve recoverable goals, completed-turn and identity guards. A duplicate must not add another committed event count. **Steps to reproduce** 1. Supply a native session whose iterator emits a current, nonrecoverable, unexpected `provider_exited` failure. 2. Keep that iterator open until the session owner closes it. 3. Observe that the old implementation does not settle within the existing test bound. 4. Apply the guard below. The same execution fails with the typed terminal error and calls the existing close owner once. **Paperclip version or commit** The original regression was reproduced against master source `05a19768ae4d337a4ae88ecdbc259f251406487c`. Current candidate `f4bb8f1b68e2793062dd640ee56b87a9208b4fd9` normally merges fetched master `07a548073ab48f13cb93b2fdc31553a5da88a070`. All unrelated master paths are preserved exactly. The two-file PR diff contains only the guarded settlement and its tests. ## What Changed - Add a guard after the existing durable event receipt. It requires a committed or duplicate receipt, a nonrecoverable unexpected exit, no completed current or previous turn, and exact provider/run/session identity. - Throw the existing typed provider failure so the existing failure and close owners run. Add no scheduler, polling loop, provider parser or cleanup owner. Event counting remains committed-only. - Cover open streams with and without a result proposal, authoritative duplicate replay, all six safety predicates, and a recoverable goal that later completes. Preserve accepted-result and close-once assertions. ## Verification - The original open-stream regression fails before the production change. Review then found that the duplicate test released EOF before asserting settlement. The corrected duplicate control fails on the prior production code at its original timely-settlement assertion; it passes after accepting the authoritative duplicate. All 21 selected exit, safety, goal and compaction controls pass. - The normal durable transport validates authenticated current run/session/turn/item identity and canonical replay equality before delivery. The App port validates current authority and binding; its durable event transaction returns duplicate only for matching source identity and canonical payload. No absent or conflicting event authorizes settlement. The six negative consumer tests use a deliberately permissive mock port; they prove this consumer guard, not that production accepts foreign events. - The first remote Typecheck job found five errors in new mocks that accessed PRP fields without narrowing a native-or-PRP event. Both mocks now explicitly reject the wrong event shape. Complete original CI evidence is retained. No cast, suppression or narrowed test compiler program is used. - On current integrated candidate `f4bb8f1`, normal Runner emission and the full normal `tsconfig.test.json` source/test program pass. All 26 selected cases pass: the same 21 controls plus all five structured provider-failure/model-retry rows affected by the incoming master change. The other 197 cases were filtered, not counted as passing. - Before master integration, all 411 declaration paths and bytes were identical across the settlement change. Incoming master independently adds one OpenCode declaration and changes three OpenCode declarations; those deltas are recorded separately. This fix changes no runtime declaration. No Rust source or binary is changed by this PR. - Actual guarded workers recorded zero PG sockets, active listeners or blocked attempts. All known worker processes/groups closed. The working tree is clean after qualification. No install, Rust build, App run, provider connection or PostgreSQL suite was part of the final freshness loop. - Fresh exact-head remote CI and complete bot reviews remain required. A local whole-repository test/build, successful task retry or live-provider acceptance is not claimed. Run the selected controls from a normal dependency installation: ```sh pnpm --filter @paperclipai/paperclip-runner exec vitest run src/native-session-runtime.test.ts --testNamePattern='settles committed or duplicate provider exits while their stream stays open|retains EOF settlement when provider exit fails the|preserves a recoverable provider-exit goal until its later authoritative completion|preserves a durable session failure when its stream closes|preserves session-goal failure priority|context compaction|never continues task work when a restart loses a compaction turn|preserves structured provider failure and model retry classification' --maxWorkers=1 --no-file-parallelism ``` ## Risks - A false terminal classification could stop useful work. The guard requires the existing durable receipt and explicit current-turn failure fields; recoverable and completed-turn controls remain covered. An exact durable duplicate may settle the same failure without repeating accounting. - Close can still fail. This change reuses the existing close owner and does not manufacture resource retirement or clear retained failure debt. - This proves Runner failure settlement under a controlled session boundary. It does not establish successful task retry, whole-tenant quiet, live provider closure, or deployed suspension. ## Model Used OpenAI `gpt-6.1-sol` produced the delegated fix and qualification. OpenAI Codex, based on GPT-6, performed managing technical review and integration. Both used repository tools and code execution. The coordinator's exact backend snapshot and context-window size were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the selected tests locally and they pass - [x] I have added tests for the changed behavior - [x] I have reviewed documentation; no command or public contract changes require a documentation edit - [x] I have considered and documented the risks above - [ ] All Paperclip CI gates are green — fresh exact-head checks pending - [ ] Greptile is 5/5 with no open findings — fresh exact-head review pending - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
e33aa8a46a |
fix(runner): let saved Codex threads follow the current feature policy (#19659)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip runner keeps a saved Codex thread so the next run of a
task continues the same conversation
> - On each new run, the controller sends `run.attach` with the current
Codex launch arguments, and the runner refuses any change to "protected"
arguments
> - #19516 removed `--disable image_generation` from the Codex launch
arguments, but every thread saved before that deploy still records the
flag
> - The runner treated the feature toggle as protected, so every
pre-deploy Codex thread failed with `run.attach cannot change protected
launch arguments`
> - Same-run recovery then tried to replace the thread, the replacement
failed, and the tasks became blocked for an operator
> - This pull request treats Codex feature toggles as controller launch
policy, so a saved thread adopts the current toggles and restarts on the
same thread
> - The benefit is that a release that turns a Codex feature on or off
no longer strands every saved Codex thread
## Linked Issues or Issue Description
Refs #19516
**What happened?**
After a staging deploy that included #19516, Codex agent runs on tasks
with a saved thread failed at the start. The run log showed
`native_session_recovery_failed: PRP command run.attach failed:
run.attach cannot change protected launch arguments` (error code
`native_session_interrupted`). Same-run retries then failed with
`runner_continuity_break_archive_failed` and
`runner_remote_state_unavailable`, and the tasks were blocked with
"Workspace repair required". Four tasks on one Codex agent showed this
first failure. Each one last ran successfully before the deploy.
**Steps to reproduce**
1. Run a Codex task so that its thread is saved with the launch
arguments from before #19516 (`--disable image_generation`).
2. Deploy a revision that includes #19516.
3. Wake the same task again. `run.attach` sends the current arguments
without the toggle, and the runner rejects them.
**Expected behavior**
The saved thread continues under the current launch policy.
**Root cause**
`CodexCommandExecutor::stable_launch_args` drops only the per-run
filesystem and shell-environment entries before it compares saved and
current arguments. All other arguments, including Codex
`--enable`/`--disable` feature toggles, must be identical. The
controller computes the toggles from Paperclip policy for every launch,
so any change to that policy made every saved thread impossible to
attach.
## What Changed
- `provider_backend.rs`: `stable_launch_args` also skips `--enable
<feature>` and `--disable <feature>` pairs and the
`--enable=<feature>`/`--disable=<feature>` form. The function contract
comment now lists the rebindable arguments. `run.attach` then adopts the
controller's current toggles. Because the arguments changed, the
existing path stops the provider and restarts it on the same thread.
- New test
`saved_thread_adopts_the_controllers_current_codex_feature_policy`:
- A thread saved with `--disable image_generation` attaches with the
current arguments, keeps its thread ID, and records the new arguments.
- A new `--disable` toggle (a tighter policy) is also adopted.
- A new `--enable` toggle is adopted in both flag forms, and the thread
ID stays the same.
- A sandbox bypass flag, a different `default_permissions` profile, and
a different subcommand are still rejected with "protected launch
arguments", and the saved state does not change.
## Verification
- Red on `master` (
|
||
|
DottaandClaude Opus 5.5
|
f4b19e4231 |
fix(runner): finalize a durable proposed result when restore loses its Codex turn (#19658)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex runs survive control-plane restarts: runnerd restores the provider conversation, and the control plane continues the task from its checkpoint > - A hard kill can land after the agent proposed its result (`run.result.proposed`) but before the provider turn ended > - The restored runner then proves that the turn ended with the accepted result. It records `session.reconciled` and `run.terminal`, but no provider turn terminal > - The Codex recovery read did not see that turn as ended, so recovery failed with "persisted active turn … is missing from thread/read". Retries ran out, and the Retry re-ran finished work > - This pull request lets recovery settle that turn with its durable result > - The benefit is that a task that already finished its work continues to finalization after a restart, instead of failing and redoing the work ## Linked Issues or Issue Description Refs #19525, #19591, #19539, #19501 **What happened?** In a restart storm, a native Codex task (B1) proposed a complete `paperclip.run_result.v1` with passing tests. The control plane accepted the completion tool call. About 9 s later, a K1 (SIGKILL and restart of the control plane) killed the controller, runnerd, and the Codex app-server. After startup, `resume_dead_runner` resumed the same run two times. Each resume spent about 25 s in `runner.session.startup` and failed with `native_session_recovery_failed: HarnessReconciliationError: persisted active turn <id> is missing from thread/read`. Then `native_session_retry_exhausted`, replay was blocked (`preserve_without_replay_v1`), and the run ended `native_session_interrupted`. The task needed a Retry, which re-ran the finished work. **Steps to reproduce** 1. Start a native Codex task through runnerd. 2. Let the agent call `paperclip_finish`, and let Paperclip accept the result. 3. Kill the controller, runnerd, and Codex before the provider turn ends. 4. Let restart recovery resume the run. Durable evidence from the kept instance: - The run events have `run.result.proposed` for the turn, then a `completion.tool_accepted` span, then the kill. - The saved checkpoint has `activeTurnId` set to that turn and `semanticResult: null`. - The Codex rollout on disk does contain the turn. The provider did persist it. - The runnerd journal for the first resume has `session.reconciled {previousProviderTurnId: <turn>, activeProviderTurnId: null}` and then `run.terminal {turnTerminalState: completed, runTerminalState: succeeded, reportedWorkDisposition: done}`. It has no `turn.*` event for that turn. The runnerd Codex provider state shows `completedTurnAuthoritative: true` and `completedProviderTurnId: <turn>`. **Expected behavior** Recovery finalizes the accepted result. It does not re-run the task. **Actual behavior** The runnerd Codex transport builds the recovery `thread/read` reply from runnerd's journal. It looked only for `turn.completed|failed|interrupted|cancelled` events that name the turn. Runnerd's "turn ended with an accepted result" branch (`provider_backend.rs`, `recovered_turn_ended_with_result`) emits only the reconciliation and the run terminal. So the reply omitted the turn, and `CodexHarnessSession.reconcile()` threw at `codex-harness-session.ts:796`. The second resume read the same journal and failed the same way. ## What Changed - `runnerd-codex-transport.ts`: a new `runnerdRecoveredTurnTerminal` helper selects the terminal for a restored turn. A provider `turn.*` terminal that names the turn wins, as before, with its cause. A `run.terminal` that follows a `session.reconciled` that names the turn as the ended previous turn (no active turn, no later `turn.started`) now also settles that turn, with the run terminal's `turnTerminalState`. The `thread/read` wait loop uses this helper. - `codex-harness-session.ts`: `reconcile()` now settles a checkpointed active turn that provider history no longer contains when the session holds the semantic result that this exact turn proposed (`resultTurnId` equals the turn). Recovery restores that result from the durable `run.result.proposed` event. Without such a result, a missing turn still fails closed. - `codex-harness-session.ts`: the same rule applies when the restored runner reports that turn `failed` with `provider_turn_lost_on_restore`. This happens when the runner cannot prove the accepted result itself. Later recoveries keep the settled outcome. Without a held result, a restore-lost turn keeps its existing failure and same-conversation continuation. - `fake-codex-app-server`: a test-only `--hold-after-terminal-result` flag keeps the turn open after an accepted completion report. - `doc/execution-semantics.md`: one additive paragraph on this recovery rule. ## Verification - Red before the fix, real `paperclip-runnerd` and fake Codex: `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-restart-continuation.test.ts -t "durable proposed result"` failed with `native_session_recovery_failed: HarnessReconciliationError: persisted active turn provider-turn-1 is missing from thread/read` at `native-session-runtime.ts:2267`. This is the same stack as the incident. - Green after the fix: the run finalizes on `provider-turn-1` with the proposed result and `runTerminalState: succeeded`. It has no `session.failed` or `turn.failed`, and Codex received only one `turn/start` (no re-run). With the transport fix alone, the test passes in about 5 s. With only the session fix, it also passes (about 30 s, through the 5 s read wait). - Driver red/green: `src/drivers/codex/codex-app-server-driver.recovery.test.ts -t "missing persisted turn"`. The settle case failed before the fix (`recovered: false`) and passes after it. The guard case (the held result belongs to another turn) still fails closed. The existing "fails recovery explicitly when the persisted active turn is missing or replaced" test still passes. - Driver red/green for the restore-lost case: `-t "restore-lost turn"`. The held=true case failed before the fix and passes after it. The held=false case still maps `turn.failed` with the lost cause. - Transport unit test: `runnerd-codex-transport.test.ts -t "reconciled result terminal"` covers the run-terminal rule and its negative cases. - Suites: `src/drivers/codex/`, `src/native-session-runtime.test.ts`, `src/backends/`, `src/live/runnerd-restart-continuation.test.ts`, and `src/live/runnerd-codex-transport.test.ts` pass. Some load-sensitive tests time out at host load 25–30 and pass when rerun alone. The two `proves local suspension after an event backlog … (1024 suffix deltas)` cases also time out on unmodified master. - `pnpm --filter @paperclipai/paperclip-runner typecheck:typescript` passes. - Live chaos replay with real Codex: K1 0–3 s after `run.result.proposed`, 3/3 PASS (R1). Each case resumed the same run, which ended `succeeded`. No Retry was needed and no work was re-run. Details are in the comment below. ## Risks - The transport rule accepts a run terminal only directly after runnerd's own reconciliation that names the ended turn with no active turn. Runnerd emits that pair only after it restored the conversation and proved the turn ended. A provider turn terminal still wins. Other run terminals do not settle a turn. - The session rules use only a result that Paperclip already holds for the exact turn. The control plane accepted that result before the kill. A later provider terminal for that turn cannot change the outcome. The finalizer can already interrupt a provider after an accepted result. - A missing turn without a durable result still fails closed. Its outcome is unknown, and the existing Retry stays available. - No profile, schema, pack contract, or database changes. The Rust change touches only the fake Codex test binary. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Nicky LeachandClaude Opus 5.5
|
97953e9b59 |
fix(ci): repair the weekly test-weight refresh and extend it to e2e and Runner lanes (#19650)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors wait for PR CI before merge, so CI wall-clock time sets the pace of work > - CI test shards are balanced with per-file durations that a weekly workflow refreshes > - The weekly refresh has failed since 2026-11-02, and the e2e weights have not been measured since 2026-09-15. Today e2e shards 3 and 6 take 7.0–7.7 minutes of tests, and shard 1 takes 4.3 minutes > - A single file can also grow until it sets the time of its shard, and nothing tells us about it > - This pull request repairs the refresh, extends it to the e2e and Runner package lanes, and warns about test files over 180 seconds > - The benefit is balanced shards that stay balanced, and early notice of oversized test files ## Linked Issues or Issue Description **What existing behavior does this improve?** The weekly refresh of the test-duration weights that balance CI shards. **Subsystem affected** CI: `scripts/ci/refresh-test-durations.mjs`, `.github/workflows/ci-test-durations.yml`, and the Vitest and Playwright CI reporters. **Current behavior** - The 2026-11-02 refresh (run 41408033686) failed. `gh api …/logs` now refuses output that contains terminal escape sequences. - Most successful `cloud-readiness.yml` master runs are admission-only and have no server jobs. The script took the newest 2 runs, so it would fail with "No server timing jobs found" on most weeks. - The e2e weights (`scripts/e2e-shard-durations.json`) have no refresh and are from 2026-09-15. - No signal tells us when one test file becomes too slow. **Proposed behavior** The refresh runs again, selects master runs that ran the server jobs, refreshes the e2e and Runner package-lane weights from merged PRs, and lists oversized files. CI writes a warning for each test file over 180 seconds. **Reason and benefit** Unbalanced shards add minutes to every PR. The current e2e weights put 472 seconds on the slowest shard. The new weights put about 300 seconds on each shard. **Breaking changes** None. Weights change only shard placement, never which tests run. ## What Changed - Add `scripts/ci/gh-api.mjs`. It always passes `--allow-escape-sequences`, retries without the flag when an older `gh` does not know it, and strips ANSI codes from logs. - `scripts/ci/refresh-test-durations.mjs`: - scans up to 50 master runs and uses the 2 newest that ran the server jobs; - adds the `--dry-run`, `--pr-body=<path>` and `--lanes=server,e2e,runner` options; - validates every lane before it writes a file. If a merged-PR lane has fewer than 2 valid runs, it keeps the current weights and writes a warning. - Add `scripts/ci/pr-run-durations.mjs`. It uses the final passing `pr.yml` run of each merged PR, and it validates: - the PR is merged into `master`; - the merge commit is in the checkout; - the run's head SHA is the PR's final head SHA; - every shard of the run succeeded. It parses Playwright per-spec times and Runner per-file times. - Add `scripts/ci/oversized-test-files.mjs` (Vitest) and `scripts/ci/oversized-test-files-playwright.mjs` (Playwright). In GitHub Actions they write a `::warning` for each test file over `PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS` (default 180). They never fail a job. The root and Runner Vitest configs and the e2e Playwright config load them. - `scripts/ci/test-timing-reporter.mjs`: move the per-file duration calculation into an exported `moduleDurationMs()` helper. The behavior does not change. - `.github/workflows/ci-test-durations.yml` and `.github/ci-duration-pr.md`: - the bot PR can also update the e2e and Runner manifests; - the PR body lists oversized files; - the verify step also runs the new tests and the e2e shard tests. - Refresh `scripts/e2e-shard-durations.json`: 56 of 56 specs measured (before: 30), from merged PRs #19613, #19191 and #19176. The file now uses the shared schema with `sources` and `coverage`. ## Verification - `node --test scripts/ci/test-selection-pr-timings.test.mjs scripts/ci/test-selection-timings.test.mjs scripts/__tests__/e2e-shard.test.mjs`: 32/32 pass. These include the e2e shard balance check with the new weights. - `node scripts/ci/refresh-test-durations.mjs --dry-run` against real GitHub data, all lanes: | Manifest | Measured | Sources | | --- | --- | --- | | `general-server-shard-durations.json` | 1488/1534 files | master runs 42296943853, 42294279748 | | `serialized-shard-durations.json` | 167/167 files | the same two runs | | `e2e-shard-durations.json` | 56/56 specs | PR runs 42296374365, 42297700476, 42296994179 | | Runner `vitest-lane-durations.json` | 310/311 files | the same three PR runs | - e2e placement over 8 shards, with the new medians as the real costs: - with the old weights, the slowest shard has 472 seconds; - with the new weights, every shard has about 300 seconds. - The dry run lists 12 files over 180 seconds. The largest is `server/src/services/workspace-recovery.test.ts`, at 636 seconds. - In this PR, only the e2e weight file changes. The weekly run refreshes the server weights. The Runner manifest is added by #19646. ## Risks - The `--allow-escape-sequences` path ran only in unit tests. The local `gh` did not support the flag, so the real dry run used the fallback. The next weekly run is the first real test of the flag. - The changes to the bot-PR shell steps did not run in Actions yet. - The e2e weights come from 3 PR runs. Real shard times vary more than the estimate. - This PR and #19637 both edit the `reporters` lists in `vitest.config.ts` and `packages/paperclip-runner/vitest.config.ts`. The second PR to merge needs a small merge resolution. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. A Claude sub-agent wrote the change, and the parent session reviewed it. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
DottaandPaperclip
|
374a918d67 |
fix(runner): retain qualified OpenCode artifacts for cold recovery (#19622)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native recovery must keep admitted provider state and useful workspace files. > - OpenCode binds its saved session to exact executable paths and hashes. > - A replacement image can remove those files even when session data is saved. > - Retaining a manifest alone cannot restore the missing launch files. > - This pull request saves the qualified files before provider work and restores the same owned paths. > - The benefit is cold continuity without relaxing the Runner guard or replaying a prior turn. ## Linked Issues or Issue Description Refs #19558. That fix retains the selected pack for ordinary continuation. This change adds durable file retention for a cold allocation. Related #19487 adjusts the admission budget; it does not retain launch files. A search of open and closed OpenCode cold/durable artifact PRs found no duplicate. **What happened?** A saved OpenCode session could outlive the image that supplied its Node executable, proxy, and native executable. A new image supplied different bytes. The Runner then correctly refused the changed launch profile. The saved harness did not contain the old files. **Expected behavior** Core must save qualified launch files before the first turn. It must restore those exact files under the same owned account and absolute paths. Unknown ownership, missing files, and changed hashes must still stop recovery. **Steps to reproduce** Start an admitted remote OpenCode session. Save its workspace, encrypted checkpoint, and harness data. Destroy the original allocation and controller files. Restore into a new allocation with an image whose qualified proxy has different bytes. Resume the same session. The new opt-in local qualification command automates this path with actual OpenCode and a local synthetic model. ## What Changed - Store company-bound, encrypted launch objects and canonical run/backup references. Generate migration 0438 through the normal database workflow. - Stage qualified Node, proxy, and native OpenCode files under a private content-addressed OS-account home. Record the account UID and exact launch paths. Preserve the Rust profile guard. - Gate first provider work on durable capture under the current native claim. Bound Stop/deadline waits while the actual late transfer keeps its worker and scratch ownership until settlement. Restore cold caches with verified staging and atomic rename. Keep live reattachment and historical image-rooted sessions on their existing paths. - Reuse bounded canonical expiration owners. Atomically fence old checkpoint authority with a strict company/run-scoped retirement marker before external deletion. Keep its indexed cleanup anchor across failed deletes and restart. Preserve earlier resume handles and reject later header or source-consumer writes. Add no idle scan or network call inside a transaction. - Add storage, race, executor, filesystem, and actual Rust resume regressions. Add the opt-in local Linux qualification command and operating limits in `doc/opencode-cold-recovery.md`. ## Verification - RED before repair: real PostgreSQL/AES cold restore lacked the launch files; normal executor preparation chose the new image pack. Separate regressions exposed lost upload receipts, missing cache repair, last-owner retirement, an account-UID mismatch, and a resume-handle race during external deletion. Each expected failure was captured before its repair. - Local qualification before the final master integration: 1,169 executor tests, 233 storage/archive/admission/expiration tests, all seven server typecheck layers, migration safety, and the actual Rust cold-resume test passed. - The maintained Linux command passed with two cached immutable Paperclip images whose proxy hashes differ. It removed the original account/controller trees, restored the production published cwd under a new lease, resumed the same actual OpenCode session, kept a useful file and prior model context, and made one new synthetic model request. No external model call or network was available in either container. - The object-store fixture uses real encrypted StorageService streams with an independent owned local directory. It is not hosted S3 acceptance or a complete task HTTP/actor workflow. Historical tasks still require normal effects reconciliation and recovery after deployment. - Review regressions reproduced partial-cache publication, resume-versus-deletion, fresh-writer and source-consumer resurrection, and unbounded Stop/deadline waits before each repair. They now pass through real filesystem, encrypted storage, PostgreSQL, and SDK boundaries. File validation is split from database ownership (384/831 source lines). - Clean runtime head `aadab7acc69ff959678030db46ffe2dcfbefe28f`: the maintained actual OpenCode cold qualification passed with `dirty:false`. It preserved the provider session and recorded launch paths, restored useful workspace content and prior model context, and made exactly one resumed synthetic model request. The 314,560,000-byte launch capture took 23.118 seconds; the later harness was 833,536 bytes. Subsequent repairs change backup-reference retirement and governed continuity-cache classification; capture, restore, launch, and published-workspace inputs are unchanged, so that runtime receipt remains scoped to those paths. The continuity-break consumer is verified separately. - Retention repair `c14d386cecf7123257918ba6a8d47307df2da397`: 68/68 affected real PostgreSQL/AES storage tests and 15/15 focused executor authority/admission cases pass. All seven server typecheck layers pass (source layers and unchanged test shards, with the final 590-file test layer repeated after fixture edits). All review threads have evidence-backed resolutions. Its subsequent CI caught the separate lifecycle-classification gap described below. The refreshed Paperclip review reported paused work, not a completed new review. - Final head `4d1cd3e3309e328b11abc85adc6d78c6d0c7f55d`: all 19 focused root-entry/executor cases and all three source declaration layers pass. [Greptile reviewed this exact head at 5/5](https://github.com/paperclipai/paperclip/pull/19622#issuecomment-6695984874). Final-head [CI verification](https://github.com/paperclipai/paperclip/actions/runs/42308908167/job/128214100339) passes: 60 successful checks, five conditional skips, and the explicitly neutral paused Paperclip review. No check is pending or failing. - CI also caught the launch cache missing from the session-root lifecycle contract. Both the writer scan and a real governed-replacement consumer failed locally before repair: old launch bytes remained in the active root. The cache now archives with the prior session state; the consumer verifies exact bytes are preserved in the archive and absent from the active root. - The latest review found that backup retirement could delete an independent run pin while a selected session retained the checkpoint. A real PostgreSQL/AES regression failed before the repair. Backup-only ownership now uses a NULL-safe exact backup predicate; the run pin belongs to fenced checkpoint expiration. The regression verifies checkpoint decryption and launch-file restoration after pending-backup cleanup, followed by deletion only after removing the final resume handle. - An executor first-session case checks new-session durable capture through the real capture helper and receipt propagation into the actual harness manifest at the backup boundary. Real storage cases verify the run-profile receipt and canonical backup/run pins. - Sequential closure reuse and simultaneous first admission are different scopes. A deterministic two-uploader test measures both independent uploads, one verified ready winner, loser-only deletion, and usable restoration for both runs. Each capture is bounded to 512 MiB and five minutes; up to three pending receipts per run are retained. There is no claim of global single-flight capture or a company-wide egress bound. - Earlier optional warm-ownership verification encountered missing local binaries. After exact-source setup, all nine ownership assertions passed; two other suites encountered local PostgreSQL setup/teardown contention, so those interrupted attempts are not passes. Required current-head CI supplies the broad final coverage. No deployment or hosted historical recovery is claimed. ## Risks - This adds two tables and a forward migration. The file object is bounded to 512 MiB. Capture adds one initial transfer; later harness archives exclude those large files. Company-scoped ready objects are shared and retained until the last canonical owner retires. Concurrent initial admissions can duplicate bounded transfers; each keeps independent ownership and cancellation rather than waiting on another controller. A continuity break retains the controller cache with its prior state until normal session-root cleanup, adding up to one bounded launch closure per retired state. - Cold continuity requires the same qualified Linux platform, supported runtime dependencies, owned account UID/home, and published absolute cwd. A mismatch fails closed. No alias or Rust guard change is included. - Existing image-rooted profiles have no retained file object. This change cannot backfill missing historical binaries. A fresh governed session must reconcile prior effects first. - Live reattachment does not read artifact objects. Storage or provenance failures before a new turn hold admission. They do not grant new execution, model, tool, or replay authority. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution, and local browser tools. The exact served model ID and context window are not exposed in this session. No claim is made for unavailable model metadata. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
fd58378ee6 |
fix(runner): never dispatch an OpenCode tool call its MCP client abandoned (#19546)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Paperclip runner runs native OpenCode agents. OpenCode calls Paperclip tools through an MCP bridge inside the OpenCode proxy. The proxy sends each call to runnerd as an `item/tool/call` frame on stdout. > - runnerd reads the proxy stdout only after the controller acknowledges earlier events. A backlog of event frames can build in the proxy. A tool call waits behind that backlog. > - The OpenCode MCP client can time out while the call waits. The agent then gets `MCP error -32001: Request timed out`. But the queued frame still reaches runnerd later, and runnerd still runs the call. > - This is a safety bug. A mutating call (for example, create a gateway token) can succeed after the agent was told that it failed. The agent can then repeat it. > - This pull request makes sure that a call whose MCP client abandoned it is never sent to runnerd while it is still queued in the proxy. > - The benefit: "timed out" now means "did not run" for every call that had not left the proxy. ## Linked Issues or Issue Description **What happened?** In chat-feature evals with native OpenCode, the agent got `MCP error -32001: Request timed out` for a POST tool call. The server still ran the POST and returned 201 about one second later. Other Paperclip tools that took 0.03–0.5 s on the server reached OpenCode 10–22 s later. The delay grew over the run, and then the calls failed with -32001. Sometimes they failed at once. **Expected behavior** If the MCP client abandons a tool call before Paperclip dispatches it, Paperclip must not run it. **Steps to reproduce** 1. Run a native OpenCode agent (`paperclip_runner` adapter, runnerd path) for a long chat turn with many tool calls. 2. Make the controller acknowledge runnerd events slowly, so a backlog builds in the proxy. 3. Let the OpenCode MCP client time out on a mutating call that waits in the backlog. 4. See that the agent gets -32001, but the call still runs on the server. **Where the per-call lag comes from (finding)** - runnerd (`provider_backend.rs`, `poll_current_provider_with_control`) does not read proxy stdout while any event is not acknowledged. It reads at most 128 events per poll. - Each poll that admits events, and each acknowledgement, calls `persist_state`. That function writes the full provider state JSON again (up to 16 MiB, and it grows during the run) and does two fsyncs. So each round trip gets slower during the run. - Pipe writes from Node are asynchronous on POSIX. OpenCode event frames and `item/tool/call` requests therefore wait in one FIFO buffer in the proxy. A tool call waits behind all earlier event frames, and the results come back through the same gated path. - OpenCode starts its MCP timeout when it calls the tool, not when runnerd reads the frame. The -32001 event frame can itself wait in the backlog, so it can look "instant" when it arrives. - Evidence: the delay from runner emit to server receipt stays below 0.5 s. But runnerd's `item.completed` for a call comes 10–22 s before OpenCode's `tool.execution.completed`, and the gap grows during the run. - The fix for the lag is not small. It needs a change to runnerd persistence: do not gate stdout reads on a full acknowledgement, or use an append-only event log instead of a full state rewrite with two fsyncs for each batch. This pull request does not change runnerd. ## What Changed - `mcp-bridge.ts`: abort the call signal when the MCP client drops its HTTP connection before a response. The `notifications/cancelled` path already aborted it. - `opencode-server-driver.ts`: send the bridge abort signal to `dynamicToolHandler` (new optional `signal` field). - `opencode-proxy-output.ts`: add `createBackpressuredFrameWriter`. It gives the stream only what its buffer accepts and keeps the backlog in its own queue. Frames keep their order. If a queued frame's signal aborts, the writer removes the frame and calls `onAbandoned`. - `opencode-app-server-proxy.ts`: write stdout through the new writer. Tool-call requests carry the call signal. An abandoned request fails with "abandoned by the MCP client before it was dispatched; it did not run". Shutdown drains all live frames. - Tests: a new regression test uses the real bridge and the writer with a stalled stdout. It covers a cancel after timeout, a dropped connection, and a live call. There are also unit tests for the writer. ## Verification - `npx vitest run src/drivers/opencode src/cli/opencode` in `packages/paperclip-runner`: 23 files, 226 tests pass. - Red check: I removed the disconnect abort and the queue gating. Then 2 of the 3 tests in `opencode-proxy-abandoned-tool-call.test.ts` failed: the abandoned `item/tool/call` reached stdout. After I restored the code, they pass. - `pnpm verify` exits 2 because the changed proxy files are outside its resolved test graph. The required CI jobs supply broad coverage. - `tsc --noEmit` for the runner package shows no errors in the changed files. One error remains in `src/eval/workflow-harness.ts`. It comes from a workspace package that is not built and is not related to this change. ## Risks - A call that already left the proxy (it is in the kernel pipe buffer or in runnerd's reader) still runs after a client timeout. The window is about 64 KiB of pipe buffer plus Node's 16 KiB high-water mark. To close it fully, runnerd must get a deadline or a cancellation for each call. That is a follow-up. - Frames now wait in a proxy queue instead of the Node stream buffer. Delivery order does not change. The new code drains this queue on shutdown before the last stdout flush. - When a client disconnects, a durable call is now cancelled. Before this change it stayed pending. The client cannot get the result in either case. - No schema, contract, or ACPX profile changes. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Nicky LeachandClaude Opus 5.5
|
a2aa69bc62 |
perf(ci): split the Runner package Vitest lane into three duration-balanced lanes (#19646)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors wait for PR CI before merge, so CI wall-clock time sets the pace of work > - The Runner package Vitest lane is the PR CI critical path. It finishes a median 15.3 minutes after the push > - That lane runs all 311 package test files in one job with 2 workers: 1,381 seconds of serial work and 727 seconds of wall time > - Ten files account for 61% of the time. Vitest's `--shard` splits by file count, not by duration, so it cannot balance this work > - This pull request splits the package files into three lanes balanced by duration, and runs the native server group in its own lane > - The benefit is about 232 seconds of tests per package lane instead of 727 seconds ## Linked Issues or Issue Description **What existing behavior does this improve?** The time that PR CI needs to run the Runner package Vitest tests. **Subsystem affected** CI: the `Verify Paperclip Runner` Vitest lanes in `pr-trusted.yml`, and `packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs`. **Current behavior** The `vitest package` lane takes a median of 13.6 minutes. It is the last job to finish in most PR runs. The lane keeps a 2-worker limit so that Runner and sidecar processes can start within their deadlines. Average CPU use is 29%. **Proposed behavior** Three package lanes each run about one third of the files, balanced by measured duration. Each lane keeps the 2-worker limit. The native server group runs in its own lane, as before. **Reason and benefit** The Runner lane stops being the critical path. More workers per lane would add CPU contention and timeouts, so this change adds lanes instead. The cost is about 3 runner-minutes per run for the extra setup. **Breaking changes** None. The `master` ruleset requires only `ci / verify` and `ci / e2e`, so the new job names do not change required checks. ## What Changed - Add `packages/paperclip-runner/scripts/vitest-package-lanes.mjs`. - It gets the file inventory from `vitest list --filesOnly --json`, so it follows `vitest.config.ts`. - It partitions the files with the `partitionGeneralServerSuites` function from `scripts/general-server-shard.mjs`. - Add `packages/paperclip-runner/scripts/vitest-lane-durations.json`. - It uses the same schema as `scripts/general-server-shard-durations.json`. - It is seeded from CI run 42290920882. It measures 310 of the 311 files. - `run-pr-vitest-lane.mjs` accepts two new arguments: - `--package-lane=K/N` runs one partition. - `--native-server` runs only the native server group. - The `--shard=1/2` and `--shard=2/2` arguments keep their exact behavior for the matrix that master still sends. - `.github/ci-task-matrix.json` and the fallback matrix in `pr-trusted.yml` have three `vitest package (K/3)` lanes and the `vitest native server` lane. The labels keep the `vitest` prefix because `scripts/ci-profile/trace.mjs` matches `Verify Paperclip Runner (vitest`. - Tests: - the lanes cover the inventory exactly once, checked with real `vitest list`; - the old `--shard` commands plan the same commands as before; - the manifest schema is correct; - bad lane arguments are refused. - Update the Runner Vitest lane paragraph in `doc/DEVELOPING.md`. ## Verification - `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`: 44/44 pass. - `node --test scripts/__tests__/runner-release-crate-cache.test.mjs .github/scripts/tests/ci-plan.test.mjs`, together with the file above: 71/71 pass. - Related workflow, cache, routing and profile tests: 446/446, 153/153 and 14/14 pass. - Local end-to-end run of `GITHUB_WORKFLOW=PR node scripts/run-pr-vitest-lane.mjs --package-lane=3/3` on macOS: - the prep steps ran, and the lane ran 104 of 311 files in 123 seconds; - 2 tests failed: a macOS `kill EPERM` and a 5-second timeout under full local load without the CI worker limit; - both files pass when run alone (35/35 tests). - This PR's own CI runs the matrix from master, so it uses the old `--shard` commands. The new lanes run in CI only after merge. Reviewers can compare the job durations of the first PR runs after the merge. | Lane | Files | Serial estimate | Wall estimate with 2 workers | | --- | --- | --- | --- | | 1/3 | 103 | 464.9 s | ~232 s | | 2/3 | 104 | 465.0 s | ~232 s | | 3/3 | 104 | 465.0 s | ~232 s | ## Risks - The weights come from one CI log and do not include collection or setup time. The balance can be uneven until a refresh measures real lane runs. - Each lane passes its file paths to Vitest as filters, and Vitest matches a filter as a substring of the path. A path that is a substring of another path would run a file in two lanes. No current path does this. If one does, the script prints a warning, and no file is dropped. - Each package lane runs `ensure:eval-build-deps` and `build:rust` again, which adds about 3 runner-minutes per run. - A comment in `scripts/run-vitest-stable.mjs` (about line 110) still says that the last Runner shard runs the native group. A follow-up will correct it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. A Claude sub-agent wrote the change, and the parent session reviewed it and ran the local lane check. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
DottaandPaperclip
|
2e340d6235 |
fix(ui): show unread message counts in agent chat pickers (#19627)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - People switch between agent chats to read replies. > - The agent picker does not show how many messages are unread. > - People need that count before they select a chat. > - The chat list already has the viewer's read state. > - This pull request adds numeric badges from that shared list. > - People can find unread replies without extra requests for each agent. ## Linked Issues or Issue Description **What existing behavior does this improve?** The chat header agent picker and the secondary sidebar's Add chat picker. **Subsystem affected** Agent Chat UI, shared task types, and the chat read-state service. **Current behavior** The pickers do not show the unread message count for each agent. **Proposed behavior** Show the unread count beside each agent. For example, show 3 for three unread agent messages. Hide the badge when the count is zero. Keep switching available while chat data loads. **Reason and benefit** People can see which chats need attention before they switch. The count uses the existing shared request and cache. **Breaking changes** The chat response gains an optional `unreadMessageCount` field. Existing fields and read receipts keep their behavior. There is no database migration. **Additional context** Related: #19425 changes chat structure. This change adds counts to the current agent pickers. The duplicate search found no PR for these count badges. ## What Changed - Count unread agent comments in the existing grouped read-state query. Keep company and viewer scope. - Exclude human messages, deleted comments, system notices, and fast-response receipts. - Pass memoized count maps to both agent pickers. Use a direct lookup for each row. - Show numeric badges and accessible singular or plural labels. - Test count changes, account isolation, cached opens, search, and large agent lists. - Update the chat stories, fixture data, and development guide. - Make the Slack owner retry-deadline test deterministic after reproducing its CI failure. ## Verification - On current head `05f5fa6201`, the [main CI workflow](https://github.com/paperclipai/paperclip/actions/runs/42305845930) passed, including build, typecheck, all applicable test shards, browser shards, and final verification. The prior server shard 2, 6, and 7 failures do not recur. - [Greptile's current-head review](https://github.com/paperclipai/paperclip/pull/19627#issuecomment-6696161631) is 5/5, its check succeeded, and both previous review threads are resolved. Current head check inventory is complete: 55 successful checks, three intentional skips, and no pending or failed checks. The Paperclip application review summary reports paused; no new completed Paperclip review is claimed. Its existing findings are resolved, and the independent current-head Greptile review is 5/5. - Current head `05f5fa6201` merges master `8621bcf53e` to resolve an import conflict. Master now includes the same model-default test repair. This branch keeps its upstream import, so that file no longer differs from master. - After this merge: all 21 server tests in the Slack owner and model-default suites pass, all 22 tests in the three picker/sidebar suites pass, and token gates pass. The standard preflight still cannot build Rust because `cargo` is absent. Fresh CI must supply the broad checks. - Head `7d125fb8c7` replaces the Slack owner test's real timer wait with the worker's existing clock input. It checks that the intent stays pending one millisecond before the retry deadline and clears at the deadline. All 15 Slack owner integration tests pass locally. Runtime retry behavior is unchanged. - On `4cfb1e9d55`, both review checks gave 5/5. CI [server shard 2](https://github.com/paperclipai/paperclip/actions/runs/42302458830/job/128183671349) failed the Slack credential-lease retry test. The same assertion failed in an unchanged local run. A diagnostic run with added logging passed; the exact timer deviation was not captured. The new test removes the real-time wait assumption and checks the clock boundary directly. The earlier model-default and cleanup failures did not recur in that CI run. - On `a3378dfc40`, both Greptile and Paperclip Review gave 5/5 with no open findings. Typecheck, build, all eight browser shards, and most test shards passed. - That CI attempt failed two server shards. The model-default test still expected `gpt-5.6-sol`, while the current default is `gpt-6.1-sol`. I reproduced both failures locally. Commit `4cfb1e9d55` uses the exported default-model constant; all six tests in that suite now pass. Production model selection does not change. - The retained-cleanup backlog test exceeded its 15-second limit in [server shard 6](https://github.com/paperclipai/paperclip/actions/runs/42299858777/job/128175193320). The unchanged test passes locally with the same limit. Its CI timeout cause is not established. No assertion or timeout was changed. The fresh CI run must recheck it. - Review repair: commit `0db631b215` keeps the existing-chat and unread-agent arrays stable when chat data and the active agent do not change. A new 500-agent test fails before the repair and passes after it. The test also checks count refreshes. - The first CI typecheck failed on duplicate `IMAGE_ONLY` and `TESTOPS_REQUESTED` keys in `tests/runner-e2e/workflow-target.test.ts`. This was present in upstream code. Master commit `e482f0f632` removes the duplicate keys. This branch now includes that commit. All nine workflow-target tests pass locally. - After the latest master merge: all 22 tests in the three affected picker/sidebar suites pass. Before that merge, all 38 focused UI tests passed. - The current head is `05f5fa620134ad199a3d85b66fb2eb93d3bb6077`. All current status checks are complete, the aggregate verify gate succeeds, mergeability is CLEAN, and no unresolved review threads remain. The maintainer has instructed normal merge after checks pass. - After merging current master: 37 focused UI tests pass across AgentChatPicker, CoordinatorAgentSwitcher, AgentConversationsSidebar, useAgentChatRead, and SidebarAgentChats. - After merging current master: all 38 proactive-chat integration tests pass with a real disposable PostgreSQL database. They check counts of 2, 1, and 0 at exact read cutoffs. - The prior revision passed UI typecheck and build, shared build, isolated low-memory server typecheck, token gates, and 30 map/catalog tests. - The 500-agent test bounds index work and count lookups. Search does not rebuild the existing-chat index. - Browser checks used production components with simulated API data. Dark, light, mobile, and Add chat cases passed. Counts of 3 and 12 stayed visible through search. Read agents had no badge. Cached opens made no chat API requests. - Twenty local cached opens took 34.5–57.6 ms, with a 38.4 ms median. This is an unthrottled fixture measurement. - Earlier full local checks could not complete because Rust cargo is absent and the workspace has limited memory. The standard worktree preflight also stops at missing cargo. Required CI must supply broad checks for this PR head. - The first server typecheck ran out of memory while Storybook was active. Its isolated low-memory retry passed. No visual baseline comparison or live provider delivery is claimed. ## Risks - Low risk. The count replaces a boolean aggregate in the same grouped query. Very long chat histories still need that existing scan. - A count can be stale until the existing chat-list cache refreshes. Read receipts still control the unread cutoff. - The change adds no polling, per-agent history request, database migration, or permission change. ## Model Used OpenAI GPT-6 through Codex, with code editing, terminal tools, and browser verification. The runtime did not expose a more specific served model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
7941f6b051 |
fix(runner): show OpenRouter's invalid model ID as a model rejection (#19548)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs use the Paperclip runner with drivers such as OpenCode
> - When a provider rejects the selected model, the product shows "Model
unavailable" and offers a model choice
> - OpenCode reports provider HTTP errors as `APIError` with the message
under `data`, not at the top level
> - The runner then reported "Provider turn failed", so an invalid model
ended as a generic adapter failure
> - This pull request classifies OpenRouter's invalid model ID with a
closed provider-failure category, and makes the server keep that
classification
> - The benefit is that a person sees "Model unavailable" and the real
provider message, and can choose another model
## Linked Issues or Issue Description
**What happened?**
A chat evaluation selected a model ID that OpenRouter does not know.
OpenCode reported `turn.failed` with `error: { name: "APIError", data: {
message: "<model> is not a valid model ID", statusCode: 400,
isRetryable: false } }`. The run was saved with `errorCode:
adapter_failed`. The task chat showed "The OpenCode run failed before it
completed." and no "Model unavailable" marker. The provider notice said
only "OpenCode session failed.".
**Expected behavior**
A non-retryable rejection of the selected model is saved as
`native_provider_model_rejected`, so the product shows "Model
unavailable" and the model choice. The notice shows the provider's
message.
**Steps to reproduce**
1. Configure an OpenCode agent with an OpenRouter model ID that does not
exist.
2. Send the agent a message.
3. Before this change, the run fails as `adapter_failed` and the notice
says "OpenCode session failed.".
## What Changed
- `drivers/opencode/provider-failure.ts`: a non-retryable OpenCode
provider error whose message says the model is not valid (`not a valid
model`, `model_not_found`, model not found or not supported) gets the
closed category `model_unavailable`. A provider that allows a retry
keeps the existing category.
- `native-session-runtime.ts`: a non-recoverable failure with
`model_unavailable` is a model rejection. The runtime reads the closed
category, not provider text.
- `native-session-executor.ts`: a non-recoverable
`NativeProviderTerminalFailure` that the runner marked as a model
rejection is saved as `native_provider_model_rejected`. Before, every
such terminal was saved as `native_provider_terminal_failed` before the
marker was read.
- `drivers/opencode/opencode-server-driver.ts`: the provider notice
summary shows the provider message from `data` when there is no
top-level message. Provider text is not used as the turn failure
message, so it does not reach `run.error` or Sentry.
- Tests: provider-failure mapping (retryable and non-retryable), runtime
classification by category, driver notice summary with no failure
message, and an executor test for the saved code and terminal
disposition.
## Verification
- `npx vitest run src/native-session-runtime.test.ts
src/drivers/provider-failure.test.ts src/drivers/opencode` in
`packages/paperclip-runner`: 16 files, 412 tests passed.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts` in
`server`: 1,152 tests passed.
- Negative controls: reverting the provider-failure mapping, the runtime
check, or the executor branch each makes its new test fail.
## Risks
- Low risk. Classification uses a closed category and a runner-set
marker, not provider text.
- The model check on the provider message is text-based inside the
OpenCode mapper. A provider that uses "not a valid model" for another
reason with a non-retryable error will show "Model unavailable".
- Provider text appears in the notice summary, as provider notices
already did for top-level messages.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
|
||
|
DottaandClaude Opus 5.5
|
02a2299173 |
feat(codex): default Codex to gpt-6.1-sol, half the gpt-5.6-sol price (#19623)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Codex agents run through Paperclip Runner or the legacy
`codex_local` harness, and the runner E2E suites run paid Codex cells
> - The shared Codex default and many selections preferred `gpt-5.6-sol`
> - The reviewed OpenAI rate card in
`server/src/services/codex-pricing.ts` (`openai-standard-2026-09-30`)
prices `gpt-6.1-sol` at $2 input / $0.10 cached / $10 output per 1M
tokens, and `gpt-5.6-sol` at $4 / $0.40 / $20
> - So every default run and every paid Codex eval cell cost twice as
much as necessary
> - This pull request moves every place that prefers or selects
`gpt-5.6-sol` to `gpt-6.1-sol`, and keeps historical facts, rate cards
and compatibility fallbacks on `gpt-5.6-sol`
> - The benefit is half the token price for new default Codex agents and
for the Codex E2E and TestOps cells
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The default Codex model for new agents and the Codex model that the
runner E2E suites and TestOps plans select.
**Subsystem affected**
Adapters (`adapter-utils`, `codex-local`), server native runtime
fallback, UI AI connection pools, Paperclip Runner devtools and live
evals, runner E2E catalog and TestOps mappings.
**Current behavior**
New Codex agents without an explicit model store `gpt-5.6-sol`. The
runner E2E `runner-codex` profile, the Reliability `queue-steer` plan
cell, the Product regression TestOps mapping and the ACPX Codex
qualification selection all use `gpt-5.6-sol`.
**Proposed behavior**
They use `gpt-6.1-sol`. Existing agents that store a model keep it.
`gpt-5.6-sol` stays selectable.
**Reason and benefit**
`gpt-6.1-sol` costs half as much per token on the in-repo reviewed rate
card. The current OpenAI pricing page (retrieved 2026-11-06) shows the
same short-context rates and also lists long-context rates of $4 / $0.20
/ $15 for `gpt-6.1-sol` against $8 / $0.80 / $30 for `gpt-5.6-sol`.
**Breaking changes**
None for stored configurations. See Risks.
## What Changed
Changed defaults and selections (all now `gpt-6.1-sol`):
- `PAPERCLIP_RUNNER_DEFAULT_MODELS.codex` in
`packages/adapter-utils/src/paperclip-runner-permissions.ts`. This also
changes `DEFAULT_CODEX_LOCAL_MODEL`,
`resolvePaperclipRunnerModel("codex", …)`, runner config normalization,
`codex_local` → Paperclip Runner conversion, the UI new-agent and
agent-config defaults, the composer default model and the public MCP
effort lookup.
- `codex-local` `models`: the 5.6 entry is now the literal
`gpt-5.6-sol`, because it used to be `DEFAULT_CODEX_LOCAL_MODEL`. The
list order and contents do not change.
- AI connection pool: a new OpenAI member profile uses
`PAPERCLIP_RUNNER_DEFAULT_MODELS.codex` instead of a hard-coded
`gpt-5.6-sol`.
- Paperclip Runner devtools ACPX Codex default and preset, the
`runner-codex-sol` local provider smoke, and the `acpx-codex-sol` live
workflow candidate.
- `tests/acpx-qualification-models.ts` Codex selection (runner E2E and
runner acceptance ACPX Codex profiles).
- Runner E2E: the `runner-codex` profile follows the default, so all
native Codex cells move. This includes the `runner-codex` cells among
the 52 standard `everyday-workflows` cells and the seven Codex
reliability cells. `runner-codex-mini` stays on `gpt-5.4-mini`. The
Reliability plan `queue-steer` `steeringProfile` is now
`['runner-codex', 'gpt-6.1-sol']`. Its executionId
(`everyday-workflows.runner-codex.daytona.queue-steer`), its fullName
and its TestOps case identity do not change. No separate profile is
necessary.
- `product-regression-testops.mjs`: the declared model for every
`runner-codex` case is now `gpt-6.1-sol` (it was already that for
`configured-repository`).
- `tests/runner-e2e/installed-cli.ts`: the credential-free installed
startup proof opens a session on the default model.
- `tests/runner-e2e/scheduled-prices.json`: add a `gpt-6.1-sol` price
(`openai-standard-2026-11-06`, including 30m/12h cache writes and
long-context rates from the OpenAI pricing page). The weekly scheduled
Codex cells need an exact price for their model, or admission fails
closed.
- Storybook: the composer model picker fixtures, the connection pool
stories and host, and the agent settings preview show `gpt-6.1-sol` as
the default.
- Docs: `doc/DEVELOPING.md` (the Codex conversion default) and
`tests/runner-e2e/EVERYDAY-WORKFLOWS.md` (local matrix, reliability
extras, `queue-steer` row).
- Tests that assert the default now expect `gpt-6.1-sol`. New
assertions: the default has the `0.159.0` CLI floor, the compatibility
fallback is floor-free, the `queue-steer` plan cell runs `runner-codex`
on `gpt-6.1-sol`, and all native Codex everyday cells (standard and
reliability) use `gpt-6.1-sol`.
Deliberately kept on `gpt-5.6-sol`:
- Rate cards: `server/src/services/codex-pricing.ts` and
`packages/paperclip-runner/src/evals/model-pricing.ts`. Old receipts
need them. (The runner eval rate card has no `gpt-6.1-sol` entry. An
unpriced candidate is reported as unpriced, not as zero. I did not add a
rate there because that card has its own version and source.)
- The remote Codex CLI compatibility fallback. A new
`CODEX_LOCAL_COMPATIBILITY_FALLBACK_MODEL = "gpt-5.6-sol"` replaces the
default in `codexModelFallbackCandidates`. `gpt-6.1-sol` needs Codex
0.159.0 or newer, so it cannot be the floor-free last resort. Fallback
behavior is unchanged.
- `tests/runner-e2e/provider-usage.ts`: the implicit request
cache-policy attribution stays qualified only for `gpt-5.6-sol`. I did
not verify that policy for `gpt-6.1-sol`. A `gpt-6.1-sol` response that
omits the write-TTL split keeps its cache-write cost unknown, which is
the existing behavior for unqualified models.
- `tests/runner-e2e/scheduled-measurements.json`: historical
`gpt-5.6-sol` measurements. They no longer match the current suite
definitions, so the forecast reports them as historical only. The
forecast tests now build their synthetic cohorts on the current
definition hash.
- Model catalogs that list `gpt-5.6-sol` as available (`codex-local`,
`cursor-local`, `opencode-local`, Speko voice diagnostics, Pi picker
fixture).
- The `gpt-5.6` → `gpt-5.6-sol` alias for old stored configs.
- Dated or evidence docs: `doc/plans/*`, `doc/adapter-model-audit-*`,
`QUALIFICATION-*`, `doc/connections/MEMORY.md`,
`doc/connections/TASK-REVIEWS-VERIFICATION.md`,
`tests/runner-e2e/CHAT-FEATURES.md`, the Slack eval evidence, and the
S49 soak signature text (the soak failure happened on `gpt-5.6-sol`).
- Test fixtures that parse, price, migrate or round-trip an existing
`gpt-5.6-sol` value (Rust runner tests, driver tests, pricing tests,
stored-config tests, stories that show an existing agent).
## Verification
- `npx vitest run` on every server, UI, adapter-utils and codex-local
test file that mentions `gpt-5.6-sol` or the default, plus all of
`packages/adapter-utils` and `packages/adapters/codex-local`: 5,483
passed. Three timeouts under load (`sandbox-managed-runtime`,
`acp.quota`) passed when rerun.
- `packages/paperclip-runner`: `vitest run` for `src/eval`, `src/evals`,
`src/drivers/acpx`, `src/drivers/codex`, `src/cli`, `src/contracts`,
`src/backends`: 2,641 passed.
- `node --test tests/runner-e2e/reliability-testops.test.mjs
tests/runner-e2e/product-regression-testops.test.mjs`: 28 passed.
- `node tests/runner-e2e/run-unit.mjs`: all model, catalog, reliability,
scheduled and TestOps tests pass. Failures remain only in
`copilot-protection-flow`, `denied-target-isolation-flow`,
`pi-controls-flow`, `workflow-target` and `unit-launcher`. These five
also fail on unmodified master in this local environment (filesystem
watch and git environment), and they do not touch models.
- `vitest run --config tests/runner-acceptance/vitest.config.ts`: 19
passed.
- `pnpm test:e2e:runner -- --list --suite everyday-workflows --group
reliability`: the seven `everyday-workflows.runner-codex.daytona.<case>`
cells show `gpt-6.1-sol` with `OPENAI_API_KEY,DAYTONA_API_KEY`.
`--matrix-json` gives 16 reliability cells with the same credentials as
before. `--list --all` shows no `gpt-5.6-sol` cell.
- Typecheck: `adapter-utils`, `adapter-codex-local`, `server`, `ui`,
`paperclip-runner` (TypeScript), and `pnpm test:e2e:runner:typecheck`.
- `pnpm check:token-gates`: clean.
- Not run: paid cells, Rust tests (no Rust change), the browser suites,
and the full `pnpm -r typecheck` / `pnpm build` (CI runs them).
## Risks
- Agents without an explicit model change model on their next save or
normalization, and Paperclip Runner rows with a blank model use
`gpt-6.1-sol` at execution. Stored models, including `gpt-5.6-sol`, are
never rewritten. No migration is necessary.
- `gpt-6.1-sol` needs Codex CLI 0.159.0 or newer with ChatGPT sign-in.
The runner pins `@openai/[email protected].0`. On a remote target with an
older image CLI, the existing compatibility check falls back to
`gpt-6-sol` and then `gpt-5.6-sol`, and that probe now runs for
default-model agents too. A legacy `codex_local` agent created with the
new default on a host with an older Codex CLI gets the existing
actionable "upgrade Codex" Test result.
- Qualification: the native Codex profile uses `adapter_constant` model
qualification, so it follows the default. No qualification gate is
dropped. The ACPX Codex qualification selection moves to `gpt-6.1-sol`.
Its next qualification run uses the new model. The provider-usage
cache-policy attribution stays limited to `gpt-5.6-sol`.
- Runner E2E history: the Codex cells change their suite definition
hashes, so earlier `gpt-5.6-sol` results are a separate cohort and do
not count as current.
- The `gpt-6.1-sol` scheduled price uses the file's existing `expiresAt`
(2026-11-19). It needs the same review as the other entries before then.
- No `.github/` change. The Reliability queue-steer cell keeps the
`runner-codex` profile id, so the existing workflow credential and setup
steps still match.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, running in
Claude Code with tool use (shell, file edits, web fetch of the OpenAI
pricing page).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Related: #19604 (commits the Reliability TestOps identities; this PR
changes only `steeringProfile` in the same file).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <[email protected]>
|
||
|
Nicky LeachandClaude Opus 5.5
|
05a19768ae |
fix(runner): require normalizedSessionId when validating PRP events (#19615)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip Runner reports native session activity as PRP events > - `validatePrpEvent` checks an event against the published event schemas and then returns it as a `PrpEvent` > - The `PrpEvent` type requires `normalizedSessionId`, but the schemas leave it optional, so the validator accepts events that the type forbids > - Every producer sets the field, and the consumers compare it to the expected session > - This pull request makes `validatePrpEvent` reject an event without `normalizedSessionId` > - The benefit is that a validated event now always matches its type ## Linked Issues or Issue Description No issue exists. #18839 listed this under its product questions and did not change it. #19495 fixes the other Runner contract types from that list. **What existing behavior does this improve?** `validatePrpEvent` in `packages/paperclip-runner/src/protocol/replay-contract.ts` returns `{ ok: true, event: PrpEvent }` for an event without `normalizedSessionId`. **Current behavior** - The TypeScript `PrpEvent` type requires the field. The Rust event structs also require it (`normalized_session_id: String`). - The published schemas (`protocol/schemas/event*.schema.json`) leave it optional. - Validated events go into event arrays, reducers and the control-plane callback, and all of them expect the field. A plain type change to optional shows 12 such places in the Runner. **Proposed behavior** After the schema check passes, `validatePrpEvent` rejects an event that has no `normalizedSessionId`. The issue has the code `schema_validation` and the path `/normalizedSessionId`. The schema still checks the field's type and length when the field is present. **Reason and benefit** The validator, the type and both producers then agree. The published schemas, their manifest hashes and the generated standalone validators do not change. **Breaking changes** An event without `normalizedSessionId` now fails validation. No producer in this repository creates such an event. The replay fixtures and golden snapshots include the field. ## What Changed - `packages/paperclip-runner/src/protocol/replay-contract.ts`: `validatePrpEvent` requires `normalizedSessionId` after the schema check. - `packages/paperclip-runner/src/protocol/replay-contract.test.ts`: new test. A fixture event passes, and the same event without `normalizedSessionId` fails with the new issue. ## Verification - Runner, in `packages/paperclip-runner` after `pnpm run test:typescript:prep`: - `pnpm exec vitest run` (the full Runner TypeScript suite): 299 of 309 files passed, and 16 tests in 9 files failed. - I ran those 9 files again by themselves: 231 of 233 tests passed. The 2 remaining failures were 20-second timeouts in `src/drivers/cursor-cloud/artifact-delivery.test.ts` and `src/drivers/cursor-cloud/cursor-cloud-driver.test.ts`. - I ran those 2 files again by themselves: all 117 tests passed. They also pass with the master validator. So all 16 failures were load timeouts, not results of this change. - `pnpm run check:replay-goldens`: "Replay golden snapshots and parity summaries are current." - `tsc --noEmit` passes for the Runner `tsconfig.json`, `tsconfig.surfaces.json` and `tsconfig.test.json`. - Server: - `cd server && pnpm typecheck`: passes. - I ran the 29 server test files that import `acknowledged-native-stop` or `native-session-executor`. 27 files passed. - `dot-runner.test.ts` failed at first because `cargo` was not on PATH. With `cargo` on PATH, it passes. - `heartbeat-process-recovery.test.ts` fails on my machine with timeouts. These failures are not caused by this change: two of the failing tests fail in the same way when I use the master `replay-contract.ts`. No failure output mentions PRP events or `schema_validation`. CI runs this file. - `pnpm verify` selects a broad run for this change (exit 2). Required CI supplies that coverage. ## Risks - Medium-low risk. This is a runtime change to a protocol validator. The server validates stored events again (`native-session-executor.ts`, `acknowledged-native-stop.ts`). A stored event without the field would now fail validation. The producers that wrote those events always set the field, so I expect no such rows. - The Rust event structs already reject an event without the field. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Nicky LeachandClaude Opus 5.5
|
a7d3d4b78c |
fix(runner): rotate local and external authority through one resumable path (#19547)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner moves a native session from one run to the next
with an authority rotation, for Codex and for every ACPX provider
> - A rotation archives the prior run's controller state under
`authority-epochs/` and archives the Runner state, locally or on an
external Runner host
> - If the run fails after the rotation but before the new Runner
starts, retry preparation restores the prior run's controller checkpoint
into the session root
> - The retry then finds an active `control-plane` next to the archive
marker and fails with `native_runner_authority_archive_conflict`. Every
later run for the session fails the same way
> - The local and external rotations were two copies of the same steps,
with different retry rules, and the sandbox archive script had a third
set of rules
> - This pull request gives every provider and Runner placement one
resumable rotation path, with the same retry rules for controller state
and Runner state
> - The benefit is that one failed startup no longer makes a native
session unusable
## Linked Issues or Issue Description
No public issue exists. Bug description:
**What happened?**
A warm Codex session on an external (plugin) Runner failed its first
attempt after the authority rotation finished. The two automatic retries
both failed with `native_session_recovery_failed: Error:
native_runner_authority_archive_conflict`, and the run stopped. On disk,
the session root had
`authority-epochs/epoch-<key>/control-plane/control-plane-state.json`
and an active `control-plane/control-plane-state.json` with the same
identity (in this case, also the same sha256). The remote Launcher had
already archived the Runner state under the same key.
**Expected behavior**
A retry resumes the interrupted rotation. The rotation already supports
this when only the archive marker exists.
**Steps to reproduce**
1. Run a native Codex or ACPX session on an external Runner.
2. Start a new run for that session. Make the new Runner fail to start
after the rotation finishes.
3. Let the run retry. Retry preparation (`restoreWarmRunnerdController`
/ `restorePerTurnRunnerdController` →
`restoreNativeControllerProjections`) restores the prior run's
controller checkpoint into the session root.
4. The retry fails with `native_runner_authority_archive_conflict`. All
later runs for the session also fail.
The new `authority rotation after the prior controller state is
restored` tests reproduce steps 2–4 directly.
**Paperclip version or commit**
`master` at
|
||
|
github-actions[bot]andlockfile-bot
|
e28158b353 |
chore(lockfile): refresh pnpm lockfiles (#19617)
## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description **What happened?** Dependency manifests on master changed after the last committed lock refresh. **Expected behavior** Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. **Steps to reproduce** Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <[email protected]> |
||
|
DottaandPaperclip
|
b55d8a9201 |
fix(runner): retain admitted OpenCode pack on ordinary continuation (#19558)
## Thinking Path > - Paperclip manages AI agents and preserves their work across runs. > - Native OpenCode turns can restore a saved provider session after the prior daemon stops. > - The Rust provider binds that saved state to exact qualified launch paths and artifact bytes. > - Ordinary continuation selected the current provider pack, although live reattachment retained the admitted pack. > - A new proxy or a diagnostics setting could therefore replace the launch profile behind saved state. > - This pull request retains the recorded pack for ordinary OpenCode continuation with a verified durable binding. > - The benefit is compatible continuation without weakening artifact integrity or granting replay authority. ## Linked Issues or Issue Description Refs: #17512, #18237. These added saved authority and compatible pack reuse. This change covers ordinary OpenCode continuation. **What happened?** An ordinary remote OpenCode continuation could select a newer pack while restoring its existing provider session. The Rust recovery guard then rejected the changed launch profile. The same error can occur after a controller release changes the bundled proxy. **Expected behavior** A continuation with a valid saved binding and pack receipt uses the same admitted artifacts. If those artifacts are unavailable, it holds the saved state instead of selecting another pack or creating a provider session. **Steps to reproduce** 1. Admit a remote OpenCode session with a qualified protected provider pack. 2. Persist its pack receipt and suspended durable Runner state. 3. Change the controller bundle to a valid pack with a different proxy digest. 4. Start an ordinary continuation with diagnostics enabled. The new executor regression fails before the repair because selection rejects the available original pack in favor of the new bundle. It passes after the repair. ## What Changed - Retain the saved provider pack for ordinary OpenCode continuation with a verified durable session binding. - Keep provider-pack retention independent of surviving-daemon identity: a new daemon records its current binary, and an authorized continuity break restores normal compatible pack selection. - Test retained pack selection after a proxy upgrade and fail closed when only a newer pack is available. - Test the real Rust executor with changed, valid proxy file bytes. It rejects drift without changing saved state and accepts the original artifacts again. - Document the selection rule and its limits. ## Verification - Initial prevention focused Vitest: 220 passed across native session execution, pack authority, pack verification and pack preparation. The executor regression was observed failing before the production change. - Rust native provider tests: all five OpenCode tests passed. These include the existing drift guard, actual changed proxy bytes, original artifact restoration and restart-lost turn behavior. - Advisory review follow-up: reproduced stale daemon identity in the actual executor and incompatible pack selection after a governed continuity archive. Both fail before their corresponding repairs and pass after them. Added an executor regression for a missing authority receipt; it remains at the current release integrity gate without creating a new receipt. - Updated head `639c080856` integrates master `17901d74ba` by normal merge. Affected Vitest: 202 passed across four files, including all new review regressions; all seven server typecheck layers passed. Exact-head CI and fresh Greptile/Paperclip review are pending. The prior head had 57 successful checks plus successful Snyk and Greptile 5/5; those results do not qualify this revision. - Environment: macOS arm64, Node 26.4.0, locked pnpm 9.15.4 dependencies, controlled remote command boundary and real Rust provider executor. No paid provider calls or hosted retries. - This is a bounded prevention change. It does not deliver historical provider artifacts to a replacement sandbox or replacement controller. Missing saved receipts or unavailable old artifacts remain a separate cold-recovery gap. No full hosted recovery is claimed. ## Risks - A continuation now requires its exact recorded pack when it has a durable binding. A newer installed pack cannot substitute for missing historical bytes. - Fresh launches, other providers and explicit continuity replacement keep their existing paths. Current ownership, company, model, permission and receipt binding checks remain in effect. - The repair does not relax the Rust launch-profile guard, replay prior tool effects, or authorize a new provider conversation. Existing published review receipts remain history. ## Model Used OpenAI Codex, GPT-6 family. The host does not expose the exact model ID or context window. Capabilities used: reasoning, code editing, shell execution, tests and authenticated read-only browser diagnostics. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
24508a8ba5 |
perf(dev): add agent preflight with a shared content-keyed build cache (#19555)
perf(dev): add agent:preflight with a shared content-keyed build cache Agent worktrees each built the Runner from nothing: a cold cargo debug and release build, the Runner TypeScript build and its checks, about 8 minutes per fresh worktree on a loaded machine. `pnpm agent:preflight` replaces the manual install + Runner build sequence. It skips steps whose inputs are unchanged and reuses other worktrees' results from ~/.cache/paperclip/agent-build: - Runner and eval-kernel dist, keyed by the Runner build-input digest and the toolchain - Runner debug binaries, keyed by every file the Rust build reads - a cargo seed with third-party crates only; the Runner crate is scrubbed because cargo records its out-of-package includes by absolute path Entries are written once with an atomic rename and restored as APFS clones. Also exclude provider-assets from the Runner build inputs. The build writes that directory itself, so the first build in every fresh checkout saw its inputs change and never recorded a build stamp. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
DottaandClaude Opus 5.5
|
86e9c9354d |
test(opencode): pin Stop during a pending MCP tool call (#19550)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runner runs OpenCode agents and settles their cost after each run > - An operator can stop a run while the agent waits inside a Paperclip MCP tool call > - Before #18610, OpenCode then kept calling the model after the Stop. The turn never closed, so the run cost never settled > - #18610 fixed the cause, but no test covers a Stop during a pending MCP tool call > - This pull request adds that regression test at the driver boundary > - The benefit is that a change to empty-answer recovery or interruption cannot bring back runaway model work after a Stop ## Linked Issues or Issue Description Refs #18610 (the fix that this test pins) Refs #19497 (accounting that waits on a terminal run receipt) **What happened?** A board operator stopped a native OpenCode run. The agent was blocked inside a gated `paperclip_call_assigned_tool` call. The runner acknowledged the interrupt, but OpenCode made 8 more model requests in the next 60 seconds. No `turn.cancelled` event arrived. The run stayed at `costAccountingPending: true` and `accountingReceiptReady: false`, so cost accounting never settled. **Expected behavior** A Stop aborts the provider. The turn closes with `turn.cancelled`. The provider makes no new model requests. Cost accounting then settles from the final receipt. **Steps to reproduce** 1. Start an OpenCode turn with a dynamic Paperclip tool whose handler does not return. 2. Let the provider call that tool through the MCP bridge. 3. Fail the pending call (runnerd does this first on Stop), then interrupt the turn. 4. Let OpenCode report the abort as a bare `session.idle`. 5. Before #18610, the driver sent a corrective "give a final answer" prompt. The provider then continued work. **Root cause** The driver sent `/session/:id/abort` correctly. OpenCode 1.18.34 aborts correctly, and sometimes reports the abort only as `session.idle`. Empty-answer recovery treated that idle as an incomplete answer and sent a new prompt. #18610 added `#interruptedTurnId`, so recovery now stops for an interrupted turn. The failing eval ran on `3b66c3c144`, which is earlier than #18610. ## What Changed - Add `packages/paperclip-runner/src/drivers/opencode/interrupt-pending-tool-call.test.ts`. It calls the real MCP bridge over HTTP, holds the dynamic tool call open, and interrupts the turn. It checks three abort reports: - `idle-after-settle`: runnerd fails the call, then the interrupt, then a bare idle. - `idle-before-settle`: an idle while the call is still pending, then the late failure, then an idle. - `aborted-error`: `MessageAbortedError`. - Each case asserts one prompt only, exactly one terminal event (`turn.cancelled`), and no empty-response correction diagnostic. - No product code changes. Master already has the fix. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/opencode/interrupt-pending-tool-call.test.ts`: 3 passed on this branch. - The same test at `3b66c3c144` (before #18610): `idle-after-settle` and `idle-before-settle` fail with 2 prompts (the corrective prompt). `aborted-error` passes. This matches the eval symptom. - `vitest run src/drivers/opencode/`: 14 files, 185 tests passed. - `tsc -p tsconfig.test.json --noEmit` for the runner: no new errors. A pre-existing error in `src/eval/workflow-harness.ts` comes from an unbuilt local eval-kernel package. - `server` `native-stop-accounting.integration.test.ts`: 5 passed. This confirms that settlement completes after a Stop when the terminal event and final receipt arrive. - Manual, not committed: the same flow with the real OpenCode 1.18.34 binary, a mock OpenAI-compatible model, and a gated tool. I tested it through the driver alone and through runnerd, the proxy, and the driver. The interrupt produced `turn.cancelled`, and the model received no more requests. I also tried different gate-release and model-latency timings. - The new test waits for the terminal event with a bounded `expect.poll`, not a fixed delay (review feedback). - `pnpm verify` exited 2 (broader verification required) because the new test file is outside the resolved test graph. CI supplies the broad coverage. ## Risks Low risk. This change adds one test file only. The test uses the existing fake OpenCode fixture and an injected `fetch`. It uses real timers with short waits. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edits, code execution) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]> |