paperclip

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-07 06:16:06 -10:00

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	7475577cdb	fix(daytona): keep a runner sandbox alive through the controller reconnect grace (#19754 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote agent run executes in a Daytona sandbox and keeps working while the Paperclip server restarts > - The server reattaches that runner within a 7-minute reconnect grace > - Daytona stops an idle sandbox after its `autoStopInterval`, and it counts only SDK and lifecycle calls as activity > - With the server down, nothing touches the sandbox, so a 5-minute auto-stop kills the runner before the server returns > - This pull request makes the host tell the provider its reconnect grace, so Daytona never auto-stops a runner sandbox inside that grace > - The benefit is that a controller outage of up to 7 minutes no longer turns a healthy remote run into a blocked task ## Linked Issues or Issue Description Refs #19199 (the same floor for the warm runner idle window). Refs #19741 (follow-up delivery through restarts; no overlap). What happened? A restart-chaos wave on Daytona killed the server for 4m51s while a per-turn remote run was active. The environment used `autoStopInterval: 5`. The last sandbox activity was at 06:09:16. The server came back at 06:14:19 and then crashed again. At 06:15:21 recovery called the sandbox. The plugin found it not started and started it again. The runner identity marker was still on disk, but its process was gone: `native.remote_runner_recovery_failed {stage: process_verification, reason: process_not_running}`, then `runner_remote_process_identity_unavailable`. The task went from done to blocked. An earlier wave showed the same failure after a 4m45s outage. Expected behavior A sandbox that holds a live remote runner survives any controller outage up to the reconnect grace. The server reattaches the runner and the run continues. Steps to reproduce 1. Create a Daytona environment with `autoStopInterval: 5` and a `paperclip_runner` agent. 2. Start a remote run and kill the server for more than 5 minutes (less than 7). 3. Restart the server. Recovery reports `process_not_running` and the task becomes blocked. ## What Changed - `server`: add `resolveRunnerReconnectGraceMsForRun`. It returns `REMOTE_RUNNER_RECONNECT_GRACE_MS` for `paperclip_runner` on a sandbox environment. - `server`: forward `runnerReconnectGraceMs` to the provider worker on lease acquisition and on the recovery reattach resume. It uses the same provider-only path as `warmRunnerIdleTimeoutMs`. It is not lease identity, lease metadata, or saved environment config. - Daytona plugin: the effective auto-stop is at least the largest host floor (warm idle window or reconnect grace) plus 5 minutes. With the current grace this is 12 minutes. The plugin applies it at creation and raises an older sandbox's shorter interval on resume (bounded, best effort). - Daytona plugin: keep the forwarded grace out of the normalized config. Update the `autoStopInterval` manifest description. - `doc/execution-semantics.md`: document the reconnect-grace floor next to the warm floor. ## Verification - Red/green Daytona plugin tests: `cd packages/plugins/sandbox-providers/daytona && npx vitest run --config vitest.config.ts`. The new cases failed before the fix (for example `expected 120000 to be >= 720000`, and no `setAutostopInterval` call on resume). All 605 tests pass after the fix. - Server tests: `cd server && npx vitest run src/__tests__/environment-run-orchestrator.test.ts src/__tests__/environment-runtime.test.ts`. 247 pass. They prove acquisition forwards the grace only for `paperclip_runner` on a sandbox, that the reattach resume forwards it, and that it does not reach lease metadata or saved config. - `pnpm typecheck` for the plugin and `tsc --noEmit` for the server are clean. - `pnpm verify` asks for broad CI coverage because a doc changed. Required CI supplies it. - Evidence from the kept wave database: lease metadata shows `autoStopInterval: 5`, `reuseLease: false`, `runnerLifecycleMode: per_turn`. The run had no events between 06:09:16 and 06:14:19. ## Risks - Cost: a runner sandbox that leaks (no cleanup) now auto-stops after 12 minutes instead of a shorter configured value. Cleanup still stops or destroys released sandboxes. Explicit `0` and longer values are unchanged. Auto-archive and auto-delete are unchanged. - I did not inspect the sandbox state in Daytona directly. Cleanup destroyed it before this investigation. The proof is the timing, the lease config, and the `process_not_running` result after a restart that kept the disk. - No live Daytona replay ran. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edit, code execution) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>	2026-11-07 01:36:18 -06:00
github-actions[bot]andlockfile-bot	e28158b353	chore(lockfile): refresh pnpm lockfiles (#19617 ) ## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description What happened? Dependency manifests on master changed after the last committed lock refresh. Expected behavior Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. Steps to reproduce Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <[email protected]>	2026-11-06 21:08:43 +00:00
Nicky LeachandClaude Opus 5.5	7dfdc97b9f	fix(deps): let Dependabot update the sandbox providers (#19489 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Sandbox providers (`packages/plugins/sandbox-providers/`) are standalone published packages. `pnpm-workspace.yaml` keeps them out of the root workspace on purpose. > - `.github/dependabot.yml` watches only `/`, so Dependabot never saw the providers. > - Because of this, the Kubernetes provider stayed on zod 3 (GHSA-mrhx-c7xc-m3w5) after the rest of the repo moved to zod `^4.6.6`. #18839 fixed it by hand. > - We do not want to keep these versions current by hand in many places. > - This pull request lets Dependabot update the providers, and keeps each provider standalone. > - The benefit is that a provider dependency that falls behind gets a pull request that can pass CI. ## Linked Issues or Issue Description Refs #18839 (adds `scripts/typecheck-sandbox-providers` and fixes Kubernetes zod by hand). Refs #18998 (a Dependabot pull request for the Pi distribution that fails CI by design). What existing behavior does this improve?* Dependabot version updates. They skip every package that the root pnpm workspace excludes. Subsystem affected CI and dependency automation: `.github/dependabot.yml`, `scripts/typecheck-sandbox-providers`. Current behavior Dependabot does not update the sandbox providers. A provider can stay on an old or unsafe version, and nobody gets a pull request. Proposed behavior Dependabot updates the providers each week. Each pull request changes the provider manifests and the lockfiles that CI installs frozen, so the pull request can pass CI. Reason and benefit We stop having to find and fix old provider dependencies by hand. ## What Changed - `.github/dependabot.yml`: add an npm entry for `/scripts/typecheck-sandbox-providers`. Dependabot treats this workspace as one unit. One pull request per dependency changes every provider that uses the dependency and the shared `pnpm-lock.yaml` together. So the entry needs no `groups`. One entry per provider would open a separate pull request for each provider, and each one would fail the frozen install in CI. - `.github/dependabot.yml`: add an npm entry for `/packages/plugins/sandbox-providers/daytona`. Daytona is the only provider with its own lockfile, and the release build and the Daytona end-to-end job install it frozen. Each lockfile has exactly one Dependabot entry, so every pull request changes a manifest and its lockfile together. - Take Daytona out of the shared typecheck workspace. This pull request does not commit `scripts/typecheck-sandbox-providers/pnpm-lock.yaml`: since #19522, only the lockfile refresh workflow writes it. After merge, the bot removes Daytona's packages from it (no other version changes; checked with pnpm 9.15.4). Until then CI resolves it in place. `scripts/typecheck-test-programs.mjs` now installs Daytona from its own lockfile (`LOCKED_PROVIDERS`, with `--ignore-workspace`). A stale lockfile is resolved in place for the run, for both installs. - `.github/dependabot.yml`: add a comment that tells why the Pi and Kimi runtime distributions have no entry (see Risks). - `scripts/typecheck-sandbox-providers/pnpm-workspace.yaml`: list each provider by its full path. Dependabot does not expand a workspace glob that starts with `../`, so with the glob it saw no providers. - `scripts/typecheck-test-programs.test.mjs`: add tests. They check that the workspace list has no glob and matches `SANDBOX_PROVIDERS` without `LOCKED_PROVIDERS`, that each locked provider has its lockfile, that a locked provider installs frozen with `--ignore-workspace`, and that a stale locked-provider lockfile is resolved in place. - Add `.npmrc` (`auto-install-peers=true`) in `scripts/typecheck-sandbox-providers` and in the Daytona provider. Without it, Dependabot reads the root `.npmrc` (`false`). That rewrites the lockfile settings, and the frozen install then fails. npm and pnpm never publish `.npmrc`. - `scripts/typecheck-sandbox-providers/package.json`: add `"packageManager": "[email protected].4"`, the version CI uses. Without it, Dependabot guesses pnpm 10. - `doc/DEVELOPING.md`: say that contributors do not refresh or commit Daytona's lockfile either, and that a new provider goes in both lists. No provider moves into the root workspace. There is no pnpm catalog. The root `pnpm-lock.yaml` does not change. ## Verification I ran the real Dependabot updater on this branch with the Dependabot CLI (`dependabot update ... --local`, updater image `dependabot-updater-npm@sha256:c5a9451c…`): - Before the change, the workspace entry loaded only `cloudflare/bridge-template` (the one path without a glob). It skipped all the other providers. - After the change, `--dep typescript` gave one pull request. It changed the 9 non-Daytona provider `package.json` files and the shared `pnpm-lock.yaml`, and did not touch Daytona. - `--dep wrangler` changed only the wrangler closure in the shared lockfile. Before the `.npmrc` fix, the same run changed `autoInstallPeers` to `false` and 757 lockfile lines. - The Daytona entry (`--dep @daytonaio/sdk`) changed Daytona's `package.json` and its own lockfile in one pull request, and kept `autoInstallPeers: true`. It keeps the exact pin exact: `0.203.0` becomes `0.227.3`. - Both frozen installs (shared workspace, and Daytona with `--ignore-workspace`) pass with pnpm 9.15.4. - Dependabot installed `[email protected].4` from the new `packageManager` field. Other checks: - `.github/dependabot.yml` passes the SchemaStore `dependabot-2.0` schema (Ajv). A wrong key fails the same check. - With pnpm 9.15.4, `install --resolution-only --ignore-scripts --no-frozen-lockfile` in `scripts/typecheck-sandbox-providers` (the lockfile bot's command) gives exactly the lockfile that passed the frozen installs earlier in this pull request. - The lockfile rule from #19522 allows this pull request's diff: it changes no lock content. - `node --test scripts/typecheck-test-programs.test.mjs`: the new test and the provider tests pass. Two other tests need a root `node_modules` (`tsc`), which my worktree did not have. CI runs them. These checks cannot run before merge. GitHub reads `dependabot.yml` only from the default branch. After merge, open Insights → Dependency graph → Dependabot and make sure that the two new entries show no errors and run on Monday. ## Risks - A dependency in both a non-Daytona provider and Daytona (for example `typescript`) gets two pull requests: one from each entry. Each one passes alone, and they can merge in any order. - The typecheck job now typechecks Daytona's tests against Daytona's own lockfile instead of the shared one. Its versions are a little older (for example vitest 4.1.11), so a type difference can show up there. - Majors of shared dev tools (for example TypeScript 7, Vitest 5) come as one pull request across all providers. It can need code changes before it passes. That is the same as the root entry. - Pi and Kimi get no version updates. Every direct dependency there is a pinned provider release, and the pins feed the provider's ACPX `commandDigest`. The Runner checks compare the Pi lock with `PI_DISTRIBUTION_PINS`, and the Kimi build compares the lock SHA-256 with a pin. So any Dependabot change there fails CI (see #18998), and a real upgrade starts saved sessions fresh. GitHub security alerts still cover both lockfiles. - The Pi zod `3.25.76` pin is still affected by GHSA-mrhx-c7xc-m3w5 (open alert). It is deliberate: `[email protected]` has `zod ^3.25.0` as a direct dependency, not a peer. A fix needs a `pi-acp` release that allows zod 4. This pull request does not change it. - `packages/plugins/examples/plugin-orchestration-smoke-example` is also outside the root workspace. It uses `workspace:*` dependencies, so it cannot install alone, and this pull request does not cover it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker runs of the Dependabot CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>	2026-11-06 11:08:39 -08:00
github-actions[bot]andlockfile-bot	17b4f972bf	chore(lockfile): refresh pnpm lockfiles (#19534 ) ## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description What happened? Dependency manifests on master changed after the last committed lock refresh. Expected behavior Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. Steps to reproduce Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <[email protected]>	2026-11-06 18:09:43 +00:00
DottaandPaperclip	c6573962f8	fix(workspaces): recover full sandboxes through read-only export (#19465 ) ## Thinking Path > - Paperclip preserves agent work across failed runs and server restarts. > - A remote workspace must have a durable save before its source can be released. > - Portable capture normally writes a Git mirror and a full archive inside the sandbox. > - A full sandbox can prevent capture before the provider's streaming download starts. > - Retrying that write cannot create the required free space. > - This change transfers a checked standalone source and seals its archive on the controller. > - The source remains retained unless the existing durable save completes. ## Linked Issues or Issue Description Related: #18644 reduced temporary capture space. #18172 added the provider's disk-free directory transfer. This repair connects portable capture to that transfer when the archive itself cannot be created. What happened? Workspace capture returns `remote_workspace_sandbox_disk_full` after provider work has finished. File recovery repeats the same sandbox write. Expected behavior Recover a stable standalone workspace without requiring another full archive on the source disk. Preserve staged changes, untracked work, confined links and the original source. Steps to reproduce 1. Create a standalone Git repository with a commit, staged changes and an untracked result. 2. Use the existing real-command transport fixture. Inject ENOSPC into the sandbox helper write or archive write. 3. Run portable capture. Before this change it rejects the helper failure. 4. With the change, restore the saved archive after removing the fixture source. Check the commit, staged content, untracked result and Git connectivity. ## What Changed - Attempt one controller-side capture after an exact sandbox disk-full failure. - Read a source inventory without installing a helper. Use the existing provider bulk transfer with confined links and runtime exclusions. - Compare the transferred inventory with source fingerprints before and after sealing. - Reject copied Git pointers, alternates and administrative symlinks. Parse config as data without includes; allow only portable keys before controller Git normalization. - Preserve socket exclusions. Retain the source and remove incomplete controller staging after failure. - Update execution semantics and focused regression coverage. No database migration or new execution authority. ## Verification - TDD red on base `2f35eb1080`: the new full-sandbox test failed with `remote_workspace_sandbox_disk_full` before production edits. - Green: 202 transport and archive tests passed before the additional Git configuration guards. At head `8541086cb5`, all 116 transport tests and the PostgreSQL durability acceptance pass. The final `e24e173ec9` exclusion guard passes those 116 cases plus its new Git-history regression across the full run and focused rerun. Environment: macOS arm64, Node 26.4.0; real Node/Git commands and filesystem transfer. - Review regressions were also red before their fixes: an internal Git config symlink bypassed the check, and `core.attributesFile` could select a controller path. Git administrative symlinks and nonportable config now fail before normalization. The focused safety selection passes 11 cases. - Cases cover helper and archive ENOSPC, commit/staged/untracked continuity after source removal, confined links, sockets, changed source, corrupt or extra copied files, unsafe Git metadata, retained diagnostic text and failed download. - Real PostgreSQL acceptance passes: normal workspace save recovers one sandbox ENOSPC, commits the encrypted snapshot, clears its writer, loses the original allocation, then independently reloads the saved result twice. The first acceptance attempt used an incorrect expected status (`available`); the documented state is `committed`. No production change was needed for that correction. - Review regressions for untracked Python/Node environments and excessive Git roots failed before their fixes. Dependency exclusions now match normal capture. Controller parsing starts only after whole-tree/repository budgets and layout checks; refspec/settings budgets and a shared two-minute verification/normalization deadline bound further work. The excessive-root test verifies zero Git subprocesses. - Real provider-boundary red/green: Daytona direct tar and capacity-fallback PTY transfers first dropped `._result.txt`. The new optional mapping contract preserves all user names and excludes only current transfer scratch. Kubernetes implements the same contract. Explicit caller exclusions and archive validation remain in force. - Standalone Linux qualification on Node 24.22.2/Debian bookworm: fresh shared/SDK build; Daytona and Kubernetes typecheck and build; Daytona file-sync/capacity/diagnostics 68/68 pass; Kubernetes file-sync 31/31 pass, including the actual `/proc` case. Qualification was refreshed again on the final `be6f7feacc` source tree: current shared/SDK rebuilt; all 815 staged source/config/manifest hashes match the delivered tree before and after testing. An initial container run as root bypassed a read-only file test; the same source passes as uid 1000. Installing zstd enabled eight initially skipped cases. - A further review regression proved `.git/pyvenv.cfg` could exclude Git history as a dependency. The inventory now handles `.git` before dependency filtering, matching normal capture. The full capture/restore test verifies the original commit and saved work. The first green attempt incorrectly compared command-result object identity; correcting the test to compare stdout passed with no production change. - Further red/green regressions cover read-only private-copy cleanup, a regular file replaced by a FIFO before open, case aliases of Git administrative metadata, and permission preservation under restrictive umask. Nonblocking no-follow file handles verify identity before and after reading. Cleanup changes only private controller directories. Provider extraction preserves ordinary permissions, refuses special mode bits before writing, and does not adopt source ownership, ACLs, attributes or flags. Final macOS Daytona tests pass 65/65; the final Linux qualification above was repeated on these exact provider sources. - Focused strict TypeScript compilation of the inventory and archive modules passes. - The first sandboxed archive run failed three Unix socket fixtures with EPERM. The permitted run outside that filesystem sandbox passed all tests. - Initial full local server typecheck could not resolve the unbuilt Runner package and produced cascading errors. Exact-head `e24e173ec9` CI typecheck, build, Runner and browser checks pass. That CI attempt failed two server shards: the unchanged retained-cleanup backlog test timed out at 15 seconds and passes locally unchanged; ten cache-reclaim fixture oracles failed because directory fallback consumed their sealed-archive counters/faults. Final commit `caed4e4f83` isolates the intended unavailable-directory boundary while preserving archive counts, recovery status, integrity and no-second-reclamation assertions. Those ten failures reproduced locally before the fixture edit. The full affected file passes 394/395; its sole other HTTP socket-hang-up case passes unchanged in an isolated rerun. This is combined coverage of all 395 cases, not a single complete green run. The final branch integrates master `53645a0104` at `98ea07225592930cbb0d18bcf250bc64f6bb13b4`, preserving upstream workspace phase timings. After integration: 117 transport cases, the PostgreSQL durability acceptance, the existing object-storage/timing regression, and 12 affected cache cases pass. The expanded recovery file now has 421 cases. Final-head `be6f7feacc45c1e32a26a2417e973f1e3752fd8d` CI passes all 55 applicable checks (three skipped, one neutral review), including the expanded recovery suite, typecheck, builds and Runner coverage. [CI result](https://github.com/paperclipai/paperclip/actions/runs/42251345554). - Provider exclusion regressions were red through four real Linux transfer paths: Daytona direct archive, direct safe-link repair, capacity PTY repair, and Kubernetes streaming. A root socket and nested regular file share a basename; literal glob-character files/subtrees also survive. Preserve mode now anchors tar exclusions as literal root-relative paths and escapes `find -path` pruning. Default callers keep glob semantics. Receipt-count regressions were also red before correcting preserved-output counting. All 31 Kubernetes and 68 Daytona tests, builds and typechecks pass on the final source with 815 matching inputs. - The final semantics documentation explicitly records special mode bits, the stricter config allowlist (including innocuous unlisted keys), and provider qualification limited to Daytona/Kubernetes; these address the optional current-head review observations without broadening controller Git authority. - No Daytona credential is available in this session. Provider transfer is a local filesystem boundary in these tests. Hosted task recovery and rollout remain unverified. ## Risks - Recovery reads the source repeatedly and transfers the standalone tree. It can take longer than normal capture. - Linked worktrees, initialized submodules, Git administrative symlinks, nonportable Git configuration, special mode bits and alternate object stores remain retained if the normal capture cannot finish. This fallback does not follow their pointers onto the controller. - A provider without usable directory export, a changed source, or insufficient controller space cannot produce a receipt. The original source stays retained. - The server, SDK and affected provider plugins must be built from this revision together. An older provider can fail inventory verification on names it omits; it cannot publish an incomplete save. No merge, publication or deployment is included. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, code editing, shell tools and automated tests. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-11-06 11:00:32 -06:00
Nicky LeachandClaude Opus 5.5	e1332b997f	fix(plugins): align tool run context and interaction input types (#19493 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Plugins add tools and issue-thread interactions through `@paperclipai/plugin-sdk` > - The SDK types are the public contract between the host and plugin authors > - Two parts of that contract do not match what the host does at runtime > - First, `ToolRunContext.projectId` says it is always a project UUID. The tool gateway sends an empty string when a run has no project. The direct tool route rejects such runs with 400 > - Second, three interaction helpers ask for the parsed output type, so plugin authors must pass fields that the host fills in with defaults > - This pull request makes the types and both tool paths agree with the runtime > - The benefit is that agents with no project can run plugin tools, and plugin authors can omit the fields that have defaults ## Linked Issues or Issue Description Fixes #3273 Related: #3274 removes the `projectId` requirement from the route. That change alone is not enough today: the route now also looks up the project to check its company, and that lookup needs a value. This pull request also skips that lookup when the run has no project. It gives plugins one value for "no project" on both tool paths. The new test typecheck in #18739 and #18839 also listed both problems as possible product problems. ## What Changed - `server/src/routes/plugins.ts`: `POST /api/plugins/tools/execute` accepts a `runContext.projectId` that is missing, `null`, or an empty string. It passes an empty string to the plugin, the same value that the tool gateway sends. It checks the project's company only when a project ID is present. A `projectId` that is not a string returns 400. - `server/src/routes/openapi.ts`: the request schema marks `runContext.projectId` as optional and nullable. - `packages/plugins/sdk/src/types.ts`: - The `ToolRunContext.projectId` comment says the value is an empty string when the run has no project. - `suggestTasks`, `requestConfirmation` and `requestCheckboxConfirmation` take `CreateIssueThreadInteractionInput`, as `askUserQuestions` already does. The host parses this input with the schema, so the defaults apply. - `packages/plugins/examples/plugin-kitchen-sink-example/src/worker.ts`: the example passes `undefined` instead of an empty project ID to `issues.create`. - `doc/plugins/PLUGIN_SPEC.md` (section 13.10): documents the empty project ID. - Tests: - `server/src/__tests__/plugin-routes-authz.test.ts`: tests a run with an empty, missing, or `null` project ID, which skips the project lookup. Also tests that a project ID that is not a string returns 400. - `packages/plugins/sdk/tests/testing-actions.test.ts`: removes the cast that the output type required. ## Verification - `pnpm exec vitest run server/src/__tests__/plugin-routes-authz.test.ts server/src/__tests__/openapi-routes.test.ts packages/plugins/sdk/tests/testing-actions.test.ts`: 3 files, 81 tests pass. - `cd server && pnpm typecheck`: passes. This includes the source layers and the three test-tree shards. - `tsc --noEmit` passes for `packages/plugins/sdk` (`tsconfig.json` and `tsconfig.test.json`) and for the kitchen-sink example. - Not run locally: the full `pnpm test:run` and the root `pnpm build`. Required CI covers them. ## Risks - Low risk. The direct tool route now accepts calls that it rejected before. It still checks that the agent and the run belong to the company. With no project ID, there is no project to check. - A plugin that reads `runCtx.projectId` can get an empty string on both tool paths. The tool gateway already sent this value before this change. Only the documentation is new. - The SDK parameter types are wider. Existing plugin code still compiles. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>	2026-11-06 16:25:43 +00:00
Nicky LeachandClaude Opus 5.5	83c6832ccb	fix(build): typecheck package, Playwright, and e2e test files in CI (#18839 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Runner, plugin, and Playwright tests verify how those agents run, connect, and appear in the board > - Most package tsconfigs exclude tests, and Vitest strips types, so a type error in those tests fails nothing > - The tests drifted to 803 real type errors, and some of those errors hid assertions that checked nothing > - paperclipai/paperclip#18739 and paperclipai/paperclip#18738 check the server test tree and four test projects, but not these files > - This pull request adds a required check for these test files, fixes every error, and fails CI if a test file is in no program > - The benefit is that a type error in any test file outside `server/src` now fails CI ## Linked Issues or Issue Description Refs #18739 Refs #18738 What happened? 517 test files were in no program that required CI checks. A type error in those files did not fail CI. Expected behavior CI fails on a type error in any test file. Steps to reproduce Add `export const value: number = "text";` to a test file under `packages/paperclip-runner/src` on master. CI stays green, because `tsconfig.json` excludes test files and Vitest strips types. Measured gap Each row is a program with its tests included, on one thread. File and error counts are for head `7d4b5e907a`. Memory peaks were measured before the two latest merges of master. \| Group \| Test files \| Real errors \| Peak \| \| --- \| ---: \| ---: \| ---: \| \| `paperclip-runner` Node source \| 290 \| 644 \| 1.09 GiB \| \| `paperclip-runner` surfaces \| 9 \| 12 \| 0.42 GiB \| \| `paperclip-runner` devtools and examples \| 14 \| 0 \| 0.68 GiB \| \| `sandbox-providers/kubernetes` \| 21 \| 65 \| 0.44 GiB \| \| 8 other sandbox providers and `bridge-template` \| 32 \| 4 \| 0.34 GiB \| \| `plugins/sdk` \| 22 \| 15 \| 0.41 GiB \| \| `plugin-workspace-diff` \| 5 \| 0 \| 0.37 GiB \| \| `plugin-orchestration-smoke-example` \| 1 \| 1 \| 0.15 GiB \| \| `kv-demo-mcp-server` \| 3 \| 0 \| 0.13 GiB \| \| `packages/db` scripts \| 1 \| 0 \| 1.01 GiB \| \| 8 Playwright suites under `tests/` \| 52 \| 25 \| 1.72 GiB \| \| `tests/runner-acceptance` \| 3 \| 0 \| 0.58 GiB \| \| `tests/e2e`, through layer declarations \| 63 \| 37 \| at most 1.59 GiB \| \| `ui` `AttentionInteractionResolver.test.tsx` \| 1 \| 0 \| n/a \| \| Total \| 517 \| 803 \| \| 27 of the 803 errors came from master while this pull request was open. Each merge of master brought new test code that no check had read. Errors that were config errors, not real errors: - One TS2688 for each provider, because the root install does not install provider dependencies. - 9 in the Runner Node program (TS7016 and TS6059). - 2 in `tsconfig.browser.json`. TypeScript 7 removed `baseUrl`, so that config was invalid. - 213 in the Playwright suites (`@/` aliases and Vite types). - About 60 in `tests/e2e` (import extensions and aliases). `packages/plugins/create-paperclip-plugin` has no gap. Its tsconfig excludes its test, but `cli/tsconfig.json` reaches that test through an import. `server/test-baselines` also has no gap, because `tests/lifecycle-baseline` reads it. ## What Changed - Added `scripts/typecheck-test-programs.mjs` and its tests. The `Typecheck + Release Registry` job runs it. It checks each program in `PROGRAMS` on one thread. - The script fails if a tracked `.test.ts(x)` or `.spec.ts(x)` file outside `server/src` is in no program. The compiler file list decides this, because `include` drops `foo.test.tsx` when `foo.test.ts` exists. - Added a `tsconfig.test.json` to each uncovered package, and `tests/tsconfig.playwright.json`. - Added `tests/e2e` to `PROJECTS` in `scripts/typecheck-test-projects.mjs`, with a `tsconfig.json` and a `tsconfig.layers.json`. - Added `scripts/typecheck-sandbox-providers`. It is a small workspace that installs provider SDKs from one lockfile. I did not add a lockfile inside each provider, because `scripts/build-standalone-public-packages.mjs` uses a frozen install when one exists, and that changes what a release resolves. - Corrected `packages/paperclip-runner/tsconfig.browser.json` for TypeScript 7. - Set `rootDir` to `..` in `packages/paperclip-runner/tsconfig.test.json`, because one Runner test imports from `packages/adapter-utils/src`. The program does not emit files, so this changes no build output. - Moved the Kubernetes provider from `vitest ^3.2.4` to `^4.1.8`, the version the other providers use. Vitest 3 requires `tinypool ^1.1.1`, which has two critical advisories (GHSA-85c8-ppgw-ccpr and GHSA-5gmw-xhrv-c9v3). The first patched version is 2.1.2. Its tests have the same result on both versions: 189 passed, and the same 12 `file-sync` tests failed. - Moved the Kubernetes provider from `zod ^3.24.2` to `^4.6.6`, and gave its two `z.record` calls a key schema. This is the only production change, and it stays in this pull request. See "Production change for review". - Renamed `AttentionInteractionResolver.test.ts` to `AttentionInteractionResolver.replace.test.ts`, so TypeScript reads the `.test.tsx` beside it. - Fixed all 803 errors, with changes to test files and test support only. The fix for `contracts/native-execution.test.ts` is the same as the fix in paperclipai/paperclip#18739, so those two changes merge cleanly. - Updated `doc/DEVELOPING.md`. This pull request does not edit `server/scripts/typecheck-layers.mjs`. paperclipai/paperclip#18739 and paperclipai/paperclip#18738 have both merged, and this branch includes them. ### Production change for review The new lockfile made dependency review flag `[email protected]` as a high-severity advisory (GHSA-mrhx-c7xc-m3w5). The first patched version is 4.6.6, so 3.x has no fix. The published Kubernetes plugin uses zod to validate operator config. The zod 3 source does not compile against zod 4, so a typecheck-only override cannot fix this. The existing tests do not prove this change, because they also passed with the invalid one-argument `z.record`. So I parsed 45 inputs with the same schema on zod 3.25.76 and zod 4.6.7: \| Result \| Inputs \| Detail \| \| --- \| ---: \| --- \| \| Identical \| 23 \| Includes defaults, valid config, and a `__proto__` key \| \| Same outcome, new error wording \| 21 \| For example, `Invalid url` is now `Invalid URL`. `validateConfig` shows this text to operators. \| \| Different outcome \| 1 \| `jobTtlSecondsAfterFinished: 2 * 60` was valid and is now rejected. The new limit is `Number.MAX_SAFE_INTEGER`. \| I did not test this against a real cluster. The decision is to keep this change in this pull request. Since then, paperclipai/paperclip#17364 has moved the rest of the repository to zod 4.6.x, so the Kubernetes plugin now matches it. ### Behavior change for review `reducedMotion` under `use` is not a Playwright option. It had no effect in 11 configs and 2 `test.use` calls, so those suites ran with `no-preference`. The options are now under `contextOptions`, so reduced motion now takes effect. `app-shell-design.spec.ts` asserts a real transition, so it now asks for `no-preference` itself. The baseline manifest has `snapshotCount: 0`, so this invalidates no published baseline. A frozen trial reference PNG can differ. ### Findings These are product questions. I did not change them, and I did not hide them with casts: 1. `PrpEvent` and its schema disagree. The TypeScript type in `src/protocol/replay-contract.ts` requires `normalizedSessionId`. The `required` list in `protocol/schemas/event.schema.json` does not include it. `validatePrpEvent` accepts events that the type forbids. 2. `requestCheckboxConfirmation` is stricter than its runtime. It uses the output type, so plugin authors must pass `continuationPolicy` and `allowDeclineReason`, although the schema gives them defaults. `z.input` would fix this. 3. `RetainedRunnerdMaintenanceInput.backend.kind` is a free string. The quarantine domain comes from `descriptor.kind`, but the release uses `proof.backend.kind`. If they differ, `completeRetainedNativeSessionCleanup` returns 0 and the quarantine stays. The production caller passes `"runner"` today, so this is not a live bug. 4. `CodexHarnessSession.close(input?)` is optional, but the contract requires `{ reason }`. These are tests that verified nothing. I corrected them, and they pass: - `backends/native-instruction-measurement.test.ts` filtered on `placement === "core"`, which is not a valid placement. It measured an empty tool list. It now measures 14 tools. - `drivers/acpx/codex-acpx-driver.test.ts` checked for `usage.updated`, which is not an event type. - The `hostFixture()` helper in `drivers/acpx/runtime-host.test.ts` ignored two overrides, so `expect(prepareSandbox).not.toHaveBeenCalled()` could not fail. - `live/live-session.test.ts` set `agent: "pi"` before a spread that overwrote it. - `native-distribution-integrity.test.ts` passed an AbortSignal where `openCommand` expects `{ signal }`. These are gaps that this pull request does not close: - The root lockfile still has some old zod versions. paperclipai/paperclip#17364 moved the direct zod dependencies to 4.6.x. `pnpm-lock.yaml` still resolves `[email protected]`, `4.1.11`, and `4.4.3`, which are below the patched 4.6.6. `packages/paperclip-runner/scripts/pi-distribution/package.json` pins `zod` to `3.25.76`, and some agent SDKs resolve `4.4.3`. Dependency review does not flag them, because this pull request does not add them. - Dead GitHub coverage. `chat-adapters-ui-providers.spec.ts` returned early for GitHub, so four later `provider === "github"` branches never ran. I removed them. They included the private key file and paste match, and the reconnect flow. Nothing asserts those behaviors in that spec now. - Provider tests do not run in CI. The root `vitest.config.ts` lists only Daytona. The other 8 providers, `bridge-template` included, are typechecked now, but CI does not run their tests. - One `packages/db` test never runs. `scripts/clean-poisoned-claude-sessions.test.ts` is outside the `include` of `packages/db/vitest.config.ts`. - `plugin-orchestration-smoke-example` does not collect. `@paperclipai/shared` is not linked in its `node_modules`. It passes 3 of 3 with an alias. ## Verification Run these commands: ```sh node scripts/typecheck-test-programs.mjs --low-memory pnpm --filter @paperclipai/server typecheck node scripts/typecheck-test-projects.mjs --low-memory --reuse-declarations node --test scripts/typecheck-test-programs.test.mjs ``` Results on macOS (arm64), Node 26.12. CI uses Node 24, and its `Typecheck + Release Registry` job also passes. At head `7d4b5e907a`, the test programs, `tests/e2e`, and the script tests pass again with 0 errors. The times and peaks below come from the run before the two latest merges of master: \| Check \| Result \| \| --- \| --- \| \| Test programs, `--low-memory` \| Exit 0, 0 errors, 42 seconds, peak 1.78 GiB \| \| Server layers, low memory \| Exit 0, 0 errors, 38 seconds, peak 3.41 GiB \| \| Test projects, including `tests/e2e` \| Exit 0, 0 errors, 13 seconds, peak 1.59 GiB \| \| Script tests \| 12 passed \| \| Injected type error in each of 23 places \| Each one failed its program with TS2322 \| \| Coverage check \| 0 unchecked files, 10 seconds \| I did not run this in a 4 GiB container. The peaks above are `maximum resident set size` for the largest single process. Runtime check of the fixes. Types do not prove that a test still works, so I ran the suites before the two latest merges of master: \| Suite \| Result \| \| --- \| --- \| \| Runner, full suite \| 4,885 passed, 3 failed, 13 skipped \| \| Sandbox providers, 10 packages \| 939 passed, 12 failed, 36 skipped \| \| Plugin SDK \| 200 passed \| \| `kv-demo-mcp-server` \| 12 passed \| \| Renamed UI test and its sibling \| 4 passed \| All 15 failures exist without these fixes or come from the local machine: - 3 Cursor Cloud timeouts. `artifact-delivery.test.ts` has 1 and `cursor-cloud-driver.test.ts` has 2. They time out in a full local run, and all 133 tests in those files with `codex-security-config.test.ts` pass when run alone. They fail in the same way with those two files at their unedited versions. - 12 Kubernetes `file-sync.test.ts` failures. They fail in the same way before and after the fixes. The first failure is that `/proc/self/fd` does not exist on macOS. After each of the two latest merges of master, I ran only the Runner test files that I changed: \| Merge \| Result \| \| --- \| --- \| \| `885c249cb9`, 8 files \| 534 passed \| \| `7d4b5e907a`, 7 files \| 672 passed, 16 failed \| The 16 failures are in the two `source-reconciliation` test files. They fail in the same way with master's unedited versions of those files. 15 of them fail with `ACPX sidecar exited at initialize`, so I think my local build of the Rust and sidecar binaries is out of date. I did not rebuild them to check. CI builds them from scratch, and all Runner jobs pass in CI at `7d4b5e907a`. Not run: - `tests/e2e` and the 8 Playwright suites need a server, so most of them have only compile checks. `playwright test --list` loads 308 tests from the main e2e config. - `packages/plugins/sandbox-providers/kubernetes/test/integration` needs a cluster, so it has only compile checks. - I did not run the full `pnpm test:run` or root `pnpm build`. Required CI runs them. Reduced motion. I ran 6 motion-sensitive Storybook specs in system Chrome with a temporary config, once with `reduce` and once with `no-preference`. Both runs had 45 passed and the same 5 failed. So reduced motion did not change those results. The 5 failures do not depend on motion. At least one comes from the browser, because system Chrome prints `oklch(55.6% 0 0)` where the test expects `oklch(0.556 0 0)`. I did not run the other Storybook specs. The `storybook-visual` label runs that workflow. ## Risks - A new lockfile to maintain. A change to a provider `package.json` now fails the typecheck job until someone refreshes `scripts/typecheck-sandbox-providers/pnpm-lock.yaml`. The failure message gives the command. - The job has a new install. The step downloads provider SDKs, so it depends on the npm registry. It uses `--frozen-lockfile` and `--ignore-scripts`. - The local install replaces provider `node_modules`. A contributor with a standalone provider install must reinstall that provider after running the script. - Reduced motion can change Storybook results. See "Behavior change for review". That workflow is not part of required CI. - Kubernetes plugin validation changed. A release of this plugin ships zod 4. Error wording changes, and very large integers are now rejected. See "Production change for review". - Master moves fast. Each merge of master brought new test code with type errors, 27 so far. After this merges, the check reports those errors in the pull request that adds them. ## Model Used - Provider: Anthropic Claude, through Claude Code - Model: Claude Opus 5.5 (`claude-opus-5-5`) - Capabilities: tool use, code execution, and subagents for the type fixes in separate files ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details. The app generated the branch name. - [ ] I have run tests locally and they pass. See the failures explained in Verification. - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>	2026-11-06 07:33:59 -08:00
Nicky LeachandPaperclip	26071353d2	feat(plugins): define settled task release and resume (#19468 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Environment plugins provide Runner tasks through the public SDK. > - A settled turn can leave resources that a later run can reuse. > - Core needs a shared contract to end active use and request the same logical session later. > - The host must retain control of capacity and report when reuse is unavailable. > - This pull request adds an optional release and resume contract with strict validation and documentation. > - The benefit is a common boundary for later host and Core integration across Codex and ACPX. ## Linked Issues or Issue Description Subsystem affected packages/plugins — plugin SDK and its environment task documentation. Problem or motivation The SDK has stop and completion operations. It has no explicit contract to release active use after a settled turn and later request the same logical session without promising retention or the same process. Proposed solution Add a versioned optional capability, release for the identified session, and resume with fresh transient authentication. Define two explicit resume outcomes: resumed or unavailable. Keep authorization, safe retries and ordinary attachment requirements explicit. Alternatives considered A provider-specific command would couple Core to one host. Reusing stop, completion or archive would change their existing meaning. A new checkpoint format or Runner mode is unnecessary. Roadmap alignment This extends the existing plugin and sandbox boundaries in ROADMAP.md. It adds no Core caller, provider implementation or runtime activation. Additional context The duplicate search found #19371, which fixes Core failure cleanup for a session resume owner. Its scope is separate. Related recovery work in #19157 and admission work in #19196 remain unchanged. ## What Changed - Add strict release and resume operation/result schemas and optional releaseResumeVersion: 1. - Reuse the existing task identity, inferred SDK exports and worker RPC types. - Define settled release, host retention policy, atomic reuse reservation, fresh authentication, retries that cannot release a newer owner and ordinary recovery. - Add schema and serialized worker RPC tests, including legacy replies and failure cases. ## Verification - SDK typecheck: `node_modules/.bin/tsc -p packages/plugins/sdk/tsconfig.json --noEmit` passed. - From `packages/plugins/sdk`, `../../../node_modules/.bin/vitest run --config vitest.config.ts tests/environment-task-release-resume.test.ts tests/environment-tasks.test.ts tests/environment-task-error.test.ts tests/environment-task-error-code.test.ts tests/worker-rpc-host.test.ts` passed: 94 tests across five suites. - `git diff --check` passed. - [Full GitHub CI](https://github.com/paperclipai/paperclip/actions/runs/42238743059) passed on `a55376801c91af29b68a13304d085706e4d7ed6f`. Standard Greptile scored that head 5/5 with no comments or unresolved review threads. - Full regressions run in GitHub CI, per the requested verification scope. No VM, deployment or live provider test was run. These tests prove the SDK boundary, not host retention or live attachment. ## Risks - This is an optional contract. Runtime behavior stays unchanged until later host and Core work uses it. - Older strict Core schemas reject the new description field. Providers must coordinate capability advertisement with a compatible Core. - Host implementations must enforce settlement, exact authority, exclusive claims and safe retries. Unknown outcomes remain errors. An unavailable result never proves physical termination. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, with repository inspection, code editing and local test tools. Context window size was not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <[email protected]>	2026-11-06 07:23:28 -08:00
DottaandClaude Opus 5.5	a4bb059f59	fix(heartbeat): retry transient sandbox provider lease refusals and explain the block (#19412 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task runs in an execution environment. A managed sandbox environment gets a lease from a sandbox provider (for example Daytona) before the agent starts. > - When the provider answered sandbox create with a temporary error (HTTP 502), the run failed as the generic `setup_failed`. > - `setup_failed` is the fail-closed code for unknown setup errors. It gets no automatic retry, so the task went to `blocked` at once. The task showed only "Couldn't finish this attempt". > - A user had to find and press Retry. The Retry then worked, which shows that the failure was temporary. > - This pull request classifies temporary provider refusals during lease acquisition from structured fields. It retries them within the existing bounded bootstrap budget, and it explains the block when the retries are used up. > - The benefit is that a short provider outage no longer stops a task. When the outage continues, the task shows the cause and a Retry that works. ## Linked Issues or Issue Description What happened? A task was assigned to an agent that uses a managed Daytona sandbox environment. Sandbox creation returned `HTTP 502`, and the cleanup of the provisional sandbox could not be confirmed. The run failed with `setup_failed` and the error `Failed to acquire lease for environment "Paperclip Sandbox" (sandbox): Daytona sandbox creation failed (HTTP 502); cleanup could not be confirmed for paperclip-create-…`. Three seconds later the task was `blocked`. No automatic retry was scheduled. The task showed no specific reason. A manual Retry started a new run that completed normally. Expected behavior A temporary provider error (HTTP 5xx, 429, timeout) before the agent starts must retry automatically with bounded backoff. The task must show why it waits. If the retries are used up, the task must show the cause, a next action, and a Retry that works on the first click. Steps to reproduce 1. Configure an agent with a sandbox environment. 2. Make the provider return HTTP 502 on sandbox create. 3. Assign a task to the agent. 4. See that the run fails with `setup_failed` and the task goes to `blocked` with no retry. Paperclip version or commit `master` at `612da1e8e0`. ## What Changed - Plugin SDK: an `environmentAcquireLease` error now carries a closed diagnostic envelope (HTTP status or a closed errno code) across the worker RPC. It takes the status of the create call only, not of the compensating cleanup lookup. The existing creation-cleanup ownership envelope is unchanged. New module: `environment-acquire-error.ts`. The resume diagnostic helpers are shared. - Server: new `EnvironmentProviderUnavailableError` and code `environment_provider_unavailable`. The environment run orchestrator sets it as the cause of `lease_acquire_failed` only when structured evidence shows a transient status (408, 425, 429, 500, 502, 503, 504, or a closed connection errno). It never parses message text. Saved-sandbox resume and stopped-worker refusals keep their own codes. - Setup failure (`failRunSetup`): a fresh, pre-dispatch run records `environment_provider_unavailable` with a `providerFailure` (phase `startup`, retryability `transient`). The run uses the bounded bootstrap retry (30 s, then 60 s). An accepted interaction continuation keeps its own continuation retry, as for `plugin_worker_unavailable`. - Recovery: the new code is registered with the other bootstrap codes. It is not sent to the generic continuation retry. It does not create an execution-reconciliation hold, because no provider work started. On exhaustion the stranded notice has the title "Sandbox provider was temporarily unavailable" and a specific next action. - UI: `ProviderFailureNotice` labels the `sandbox` provider as "Sandbox provider". The task notice shows the provider failure, "Paperclip will retry automatically" while a retry is scheduled, and Retry after exhaustion. - Docs: `doc/execution-semantics.md` describes the policy and the sandbox ownership rule. ## Verification Red before the fix, green after the fix: - `server/src/__tests__/heartbeat-process-recovery.test.ts` › "transient sandbox provider refusal during lease acquisition": - "retries automatically and runs exactly one successful successor without user action": the real heartbeat. The lease runtime returns a Daytona 502 once, then delegates. Before the fix: one `failed` run and no successor. After the fix: one `scheduled_retry` successor with no hold and a `retry_scheduled` presentation. After promotion, exactly one successor succeeds. - "explains the block with a working Retry once automatic retries are used up": three refusals. The task is `blocked` with a "Sandbox provider was temporarily unavailable" notice. The presentation is `{ state: "failed", canRetry: true, providerFailure: { httpStatus: 502 } }`, and no execution-reconciliation hold exists. One `requestTaskRetry` is accepted, and its run succeeds. - `server/src/services/heartbeat/run-completion.test.ts` › "routes a … sandbox provider lease refusal through bounded bootstrap recovery": the real orchestrator wrapping. Only the fresh, structured 502 is retried. A message-only 502, a 400, a run after provider dispatch, and a retained native run stay `setup_failed`. - `packages/plugins/sdk/tests/environment-acquire-error.test.ts`: the real worker RPC. The test checks the create status (not the 404 cleanup lookup), the errno, no provider text, the unchanged cleanup envelope, and that forged envelopes are rejected. Commands: ```sh pnpm --filter @paperclipai/plugin-sdk exec vitest run tests/environment-acquire-error.test.ts tests/environment-creation-cleanup.test.ts tests/environment-resume-error.test.ts cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts src/services/recovery/ src/services/heartbeat/run-completion.test.ts src/services/heartbeat/retries.test.ts src/services/legacy-execution-recovery.test.ts src/services/task-interruption-presentation.test.ts src/services/task-retry.test.ts src/__tests__/environment-runtime.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts cd packages/plugins/sandbox-providers/daytona && npx vitest run pnpm --filter @paperclipai/plugin-sdk typecheck && pnpm --filter @paperclipai/server typecheck ``` I did not run a live Daytona 502 on staging. Re-check on staging after deploy (see Risks). ## Risks - Sandbox leaks or double creation: each retry is a new run with a new durable creation record and a new provider name (`paperclip-create-<operationId>`). The failed attempt stays owned by its own record, and the existing creation reconciler settles it. A retry never reuses or abandons it. While a provisional sandbox waits for reconciliation, up to two more sandboxes can be created for the same task. This is the same as a manual Retry today. - Plugins built with an older SDK do not send the envelope. Their refusals stay `setup_failed`, which is the current behavior. - An in-process error with `status`/`statusCode` 500–504 or a connection errno during lease acquisition is now retried twice. The budget is bounded and shared with the other bootstrap codes. - An RPC timeout on the host side is not classified by this change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code as an agent with tool use (shell, file edits, test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>	2026-11-06 00:08:44 -06:00
dependabot[bot]Nicky Leach Claude Opus 5.5	c9c4949c86	build(deps): bump zod from 4.4.3 to 4.6.6 (#17364 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The shared validators use zod to check API input, and the Runner contracts check some of the same values again > - zod below 4.6.6 has a high-severity memory issue (GHSA-mrhx-c7xc-m3w5), so this bump is necessary > - zod 4.6 also changes string `min`, `max`, and `length`: they now count Unicode code points, not UTF-16 units > - The Runner question parser and the goal readers still use JavaScript string length, so emoji text can pass zod and then fail later > - This pull request bumps zod and keeps the string-length bound on those schemas > - The benefit is that we get the security fix, and these inputs still fail at validation ## Linked Issues or Issue Description Refs GHSA-mrhx-c7xc-m3w5. What happened? With zod 4.6, three CI tests failed on this bump. Two showed an emoji body that is over the agent commentary ceiling and that zod now accepts. The third is the scheduler timeout described under Verification. Expected behavior A dependency bump does not widen an API limit. Text over the limit gets a validation error. Impact Emoji text between the two bounds passed validation and failed later: - A question set is created, and then each answer gets a 422. The question stays pending and nobody can answer it. A schema and parser run confirmed the mismatch. This was not run over HTTP. - The Muse route returns a 500 because `parsePaperclipQuestionSet` throws an error that is not an `HttpError`. This comes from reading the code. It was not run. - A Runner goal normalizes to null and replaces the stored goal, or it is marked blocked. This comes from reading the code. It was not run. ## What Changed - Bump zod to `^4.6.6`. The lockfile resolves 4.6.7. (Dependabot) - Add `maxStringLength` to `packages/shared/src/validators/text.ts`. It checks zod `max` and JavaScript string length. - Use `maxStringLength` for the question-set text fields, the Runner goal objective, and the agent commentary body. - Add regression tests for the question-set bound and the goal objective bound. - Dependabot recreated the branch on current `master` (`@dependabot recreate`) to resolve `pnpm-lock.yaml` conflicts. This happened twice. The lockfile is Dependabot's. Each time, the two fix commits were applied again without changes. ## Verification - Compared zod 4.4.3, 4.6.5, 4.6.6, and 4.6.7 directly. The change to `max` is in 4.6.5 or earlier, and 4.4.3 does not have it. - Passed locally: `packages/shared` tests `agent-commentary`, `validators/issue`, and `runner-goal`, plus `tsc --noEmit` in `packages/shared`. - Passed locally on the first fix commit: server `agent-commentary.integration`, `agent-commentary-helper`, and `openapi-routes`. The later commit changes these schemas again. CI passed these tests at `b5e9dc7352`. - Local testing was focused. GitHub CI is the full regression check. CI passed at `b5e9dc7352`, including `ci / verify` and `ci / e2e`. - Greptile reviewed `b5e9dc7352`. The score is 5/5, with no inline comments. - After the recreate, the fix commits applied cleanly as `bad62c3ba5` and `3294552e65` on Dependabot's `46789d1487`. Their diff is the same as the diff Greptile reviewed. The lockfile still resolves zod 4.6.7. The focused `packages/shared` tests (`validators/issue`, `runner-goal`) and `tsc --noEmit` in `packages/shared` passed again locally. A second `pnpm-lock.yaml` conflict came before CI finished at `3294552e65`. - Second recreate: Dependabot ignored the first `@dependabot recreate` (no reaction and no update job after 20 minutes). It acted on the second command and pushed `83c328c15c`. The fix commits applied cleanly as `dd3e87d93e` and `95b450048f`, with the same diff. The lockfile still resolves zod 4.6.7. The same focused tests and `tsc --noEmit` passed again locally. Greptile first scored `95b450048f` 4/5 with one P2: the description did not explain the source changes. Greptile read Dependabot's replacement description, which was live during that review. The description was restored and the thread resolved. The new review scored `95b450048f` 5/5 with no findings. CI failed at `95b450048f` in two shards. Neither failure comes from the zod change: - `server (9/12)`: `server/scripts/reporting-lock-policy.test.mjs` fails with "Reporting lock changed; review the dependency correction before install". #19283 added this test today. It requires the committed `pnpm-lock.yaml` hash to match `tests/runner-e2e/report-lock.json`. Every pull request that changes the lockfile fails it until the hashes are renewed, for example #17172 and #17911. - `workspaces-a (1/2)`: `cli/src/__tests__/company-import-export-e2e.test.ts` timed out waiting for its child server. The same test failed the same way on two unrelated branches in the same hour (runs 42115721271 and 42109846042). A rerun of this shard at `95b450048f` passed. The rerun of `server (9/12)` failed again on the reporting-lock test. - #19356 fixed the reporting-lock gate for Dependabot lockfiles and merged. A re-run reuses the old merge commit, so the empty commit `482d76686e` starts a run against current `master`. `pnpm-lock.yaml` and the reporting-lock proof on `master` did not change since this branch's base. - CI at `482d76686e`: `ci / policy` passed, and `server (9/12)` skipped the reporting-lock test as #19356 intends. `e2e shard (6/8)` failed once in `cloud-auth.spec.ts` ("apiResponse.json: Response has been disposed", a Playwright race in a route handler). The same test passed at `95b450048f` with the same code. A rerun passed. All checks pass, including `ci / verify` and `ci / e2e`. - Earlier failure: `issue-monitor-scheduler.test.ts` ("lets the board check a scheduled monitor on an in_progress task") timed out in CI at `d60b5f56d8`. It passed 5 of 5 local runs with the first fix. The cause is unknown. The test waits 3 seconds for heartbeat runs to settle, so CI load is possible, but this is not proven. The test passed in the CI run at `b5e9dc7352`. This shows recovery. It does not explain the earlier timeout. ## Risks - All other zod string bounds now count code points. Emoji text can be up to 2N UTF-16 units and 4N UTF-8 bytes (it was N units and at most 3N bytes). This pull request does not change those bounds. - An audit of those bounds found no other concrete failure. It checked index limits, byte limits, `.length` guards, provider limits, and security `min` checks. It used code search and sampling, so it is not exhaustive. - Not verified: Slack `topic` and `purpose` use `.max(250)`, and we did not confirm how Slack counts that limit. - `.min(N)` is now stricter for emoji text. A single emoji no longer passes `.min(2)`. - `@dependabot rebase` or `@dependabot recreate` on this branch will remove the manual commits. This happened twice to resolve lockfile conflicts, and the commits were applied again each time. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused tests only; see Verification) - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documentation change needed) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (at `482d76686e`) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (5/5 at `95b450048f`) - [x] I will address all Greptile and reviewer comments before requesting merge --- Bumps [zod](https://github.com/colinhacks/zod) from 4.4.3 to 4.6.6. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/colinhacks/zod/releases">zod's releases</a>.</em></p> <blockquote> <h2>v4.6.6</h2> <blockquote> <p>Six malicious commits pushed to <code>main</code> on September 21 with a compromised git credential were reverted the same day in <a href="https://redirect.github.com/colinhacks/zod/issues/6620">#6620</a> and have since been removed from the history of <code>main</code>. None were ever published to npm. The npm provenance for 4.6.6 names 284243fb, the pre-rewrite copy of 9c219833 with identical contents.</p> </blockquote> <h2>Commits:</h2> <ul> <li>0cd3af5e9f50ad11bbcd4407a46a21b8848ba574 docs: load the en locale directly in Zod Mini</li> <li>9d5e60f162997b764b40d45aa02481888db50dea docs: import locales from zod/locales, not z.locales</li> <li>34a626f307871ecbb0123947559328dc52ed5e3e docs: tighten the z.locales tree-shaking note</li> <li>74304ce5b5796b5edb9a88ceb8e170b6ea3e82df release: stage npm versions instead of publishing them (<a href="https://redirect.github.com/colinhacks/zod/issues/6622">#6622</a>)</li> <li>b2ef983e0711ff6517d6020b021071e2e5653efa docs: add Code for Japan to bronze sponsors</li> <li>8fb2faf0584828a6a027d8279db36f6f52784462 docs: crop the Code for Japan logo and list it first in bronze</li> <li>bacad518c41610d654658172d325874bf0ee6290 docs: square the Code for Japan logo to match the other bronze tiles</li> <li>c42596bd1c87e941d7241d10e012c5c196adcc10 release: hold the run on the release-approval environment until the staged versions are approved (<a href="https://redirect.github.com/colinhacks/zod/issues/6638">#6638</a>)</li> <li>a2ba4a67986a3e6b2b131be30084a2f460d9ea67 fix(types): declare record output and input as members, not base type arguments (<a href="https://redirect.github.com/colinhacks/zod/issues/6645">#6645</a>)</li> <li>a0ebf970f2d752e5f58527cd2942ff48be4f46bd fix(v4): check array, set and map size bounds before parsing elements (<a href="https://redirect.github.com/colinhacks/zod/issues/6652">#6652</a>)</li> <li>529f78dd9431b07f49647fee3aa0819b3f8b66b7 4.6.6</li> <li>9c219833c3a3764fa1f1ea9c5ecec8254a63c0b1 ci: stage at verbose so a failed OIDC token exchange says why</li> </ul> <h2>v4.6.5</h2> <h2>Commits:</h2> <ul> <li>d2b135cfb7a3582b9eb515756b9166bcb9521f4a docs: add the 4.6.x patch highlights to the 4.6 post</li> <li>f1448f7cee00df9fe1e9ad84a000aa1828cc8bc1 docs: fold the 4.6.x patch highlights into the 4.6 post's own sections</li> <li>de65a5cb39ed22a507fac935788f718fa88d104f docs: lead the properties section with the check and add a Zod Mini tab (<a href="https://redirect.github.com/colinhacks/zod/issues/6598">#6598</a>)</li> <li>56222cd1532c07bcb91b67df529cab4c0a215330 feat(instanceof): key the .properties() shape off the instance type (<a href="https://redirect.github.com/colinhacks/zod/issues/6600">#6600</a>)</li> <li>ca0229a404818290e6cdcfefcd7eb2d04bcbb543 Revert "feat: add z.currencyCode() over a vendored ISO 4217 list, refreshed weekly by CI (<a href="https://redirect.github.com/colinhacks/zod/issues/6595">#6595</a>)"</li> <li>cc4cd4ee9c52fcaa10964e48cc144541e41a5ed9 Revert "Revert "feat: add z.currencyCode() over a vendored ISO 4217 list, refreshed weekly by CI (<a href="https://redirect.github.com/colinhacks/zod/issues/6595">#6595</a>)""</li> <li>0f3f5ee3ca56c7574bf849e54f79e9a6e02562ee 4.6.5</li> <li>59bbc03e10c636b9eb3c393dfeb552819774ec21 chore: re-pin the integration peers to the workspace zod after the 4.6.5 bump</li> </ul> <h2>v4.6.4</h2> <p>A patch on top of <a href="https://github.com/colinhacks/zod/releases/tag/v4.6.3">4.6.3</a>.</p> <ul> <li><a href="https://github.com/colinhacks/zod/commit/d6bc1e30"><code>d6bc1e30</code></a> feat: add <code>z.currencyCode()</code> over a vendored ISO 4217 list, refreshed weekly by CI (<a href="https://redirect.github.com/colinhacks/zod/pull/6595">#6595</a>)</li> <li><a href="https://github.com/colinhacks/zod/commit/ad32d751"><code>ad32d751</code></a> perf: <code>z.url()</code> rejects an invalid URL with <code>URL.canParse()</code> instead of a throwing constructor, about 50x faster; fewer allocations on the validation path (<a href="https://redirect.github.com/colinhacks/zod/pull/6588">#6588</a>)</li> <li><a href="https://github.com/colinhacks/zod/commit/2bb08717"><code>2bb08717</code></a> chore: re-pin the integration peers to the workspace zod after the 4.6.4 bump</li> <li><a href="https://github.com/colinhacks/zod/commit/f6e1701a"><code>f6e1701a</code></a> chore(deps): bump next to 15.5.25 and vite to 7.3.6 (<a href="https://redirect.github.com/colinhacks/zod/pull/6153">#6153</a>)</li> </ul> <h2>v4.6.3</h2> <p>A patch on top of <a href="https://github.com/colinhacks/zod/releases/tag/v4.6.2">4.6.2</a>.</p> <ul> <li><a href="https://github.com/colinhacks/zod/commit/413cce9a"><code>413cce9a</code></a> fix(v4): make z.properties() a check again (<a href="https://redirect.github.com/colinhacks/zod/pull/6594">#6594</a>) — removes the standalone <code>z.properties()</code> schema from 4.6.0; <code>z.instanceof().properties()</code> and <code>.check(...z.properties())</code> are unchanged</li> <li><a href="https://github.com/colinhacks/zod/commit/75d63ee1"><code>75d63ee1</code></a> docs: show only the <code>.properties()</code> method form in the 4.6 post</li> <li><a href="https://github.com/colinhacks/zod/commit/46da9572"><code>46da9572</code></a> docs: match the error-message examples to what the parsers emit</li> </ul> <h2>v4.6.2</h2> <p>A patch on top of <a href="https://github.com/colinhacks/zod/releases/tag/v4.6.1">4.6.1</a>.</p> <ul> <li><a href="https://github.com/colinhacks/zod/commit/9446b5cc"><code>9446b5cc</code></a> fix: preserve undefined prefault outputs and object keys (<a href="https://redirect.github.com/colinhacks/zod/pull/6587">#6587</a>) — closes <a href="https://redirect.github.com/colinhacks/zod/issues/6585">#6585</a></li> <li><a href="https://github.com/colinhacks/zod/commit/0c483c58"><code>0c483c58</code></a> docs: the <a href="https://zod.dev/blog/zod-4-6">Zod 4.6 announcement post</a> (<a href="https://redirect.github.com/colinhacks/zod/pull/6546">#6546</a>)</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/colinhacks/zod/commit/9c219833c3a3764fa1f1ea9c5ecec8254a63c0b1"><code>9c21983</code></a> ci: stage at verbose so a failed OIDC token exchange says why</li> <li><a href="https://github.com/colinhacks/zod/commit/529f78dd9431b07f49647fee3aa0819b3f8b66b7"><code>529f78d</code></a> 4.6.6</li> <li><a href="https://github.com/colinhacks/zod/commit/a0ebf970f2d752e5f58527cd2942ff48be4f46bd"><code>a0ebf97</code></a> fix(v4): check array, set and map size bounds before parsing elements (<a href="https://redirect.github.com/colinhacks/zod/issues/6652">#6652</a>)</li> <li><a href="https://github.com/colinhacks/zod/commit/a2ba4a67986a3e6b2b131be30084a2f460d9ea67"><code>a2ba4a6</code></a> fix(types): declare record output and input as members, not base type argumen...</li> <li><a href="https://github.com/colinhacks/zod/commit/c42596bd1c87e941d7241d10e012c5c196adcc10"><code>c42596b</code></a> release: hold the run on the release-approval environment until the staged ve...</li> <li><a href="https://github.com/colinhacks/zod/commit/bacad518c41610d654658172d325874bf0ee6290"><code>bacad51</code></a> docs: square the Code for Japan logo to match the other bronze tiles</li> <li><a href="https://github.com/colinhacks/zod/commit/8fb2faf0584828a6a027d8279db36f6f52784462"><code>8fb2faf</code></a> docs: crop the Code for Japan logo and list it first in bronze</li> <li><a href="https://github.com/colinhacks/zod/commit/b2ef983e0711ff6517d6020b021071e2e5653efa"><code>b2ef983</code></a> docs: add Code for Japan to bronze sponsors</li> <li><a href="https://github.com/colinhacks/zod/commit/74304ce5b5796b5edb9a88ceb8e170b6ea3e82df"><code>74304ce</code></a> release: stage npm versions instead of publishing them (<a href="https://redirect.github.com/colinhacks/zod/issues/6622">#6622</a>)</li> <li><a href="https://github.com/colinhacks/zod/commit/34a626f307871ecbb0123947559328dc52ed5e3e"><code>34a626f</code></a> docs: tighten the z.locales tree-shaking note</li> <li>Additional commits viewable in <a href="https://github.com/colinhacks/zod/compare/v4.4.3...v4.6.6">compare view</a></li> </ul> </details> <br /> 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Nicky Leach <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>	2026-11-05 17:41:43 -08:00
Nicky LeachandClaude Opus 5.5	e6c74982fe	fix(workspaces): report why a workspace provider operation failed (#19362 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A project can store its task workspaces with a workspace storage provider plugin. Core calls the plugin's `workspaceOperation` method to prepare, inspect, finalize and remove a workspace. > - `callWorkspaceProvider` catches every rejection of that call and throws the same conflict: "Workspace storage operation is incomplete; retry the same workspace". > - A provider error, an RPC timeout, a crashed worker and a worker that never got the call all look the same in the run, the log and the API. The message tells the user to retry, also when the failure is a fixed configuration problem that a retry cannot fix. > - In one diagnosis, three different configuration failures in a row showed this same message. Finding each cause took trace span counts and database reads. > - This pull request lets a provider send a reason from a closed set, and makes Core tell transport failures apart from provider failures. > - The benefit is that a failed run says what failed and whether a retry can help. Provider text and values still never cross into Core. ## Linked Issues or Issue Description What happened? A workspace provider's `workspaceOperation` failed. The run failed with "Workspace storage operation is incomplete; retry the same workspace", `errorCode: setup_failed`, and no reason in the run record or the `heartbeat execution setup failed` log. Retrying did not help, because the cause was a provider configuration problem. Expected behavior Core tells these cases apart: the worker did not receive the call, the call timed out, the worker stopped during the call, or the provider reported a failure. For a provider failure, Core shows a bounded reason. Core tells the user to retry only when a retry can help. Steps to reproduce 1. Install a workspace storage provider plugin whose `onWorkspaceOperation` throws during `prepare` (for example, because of missing provider settings). 2. Run an agent task in a project that uses that provider. 3. See the failed run: the error is the generic retry message, and the run record and logs do not show the cause. Paperclip version / commit `master` at `5ac05871b5`. Deployment mode Any deployment with a workspace storage provider plugin. ## What Changed - SDK reason contract. New `packages/plugins/sdk/src/workspace-operation-error.ts`: `WORKSPACE_OPERATION_FAILURE_REASONS` (11 reasons), `PluginWorkspaceOperationError`, `workspaceOperationErrorData` and `readWorkspaceOperationFailureReason`. It uses the same envelope design as `environment-task-error.ts`. The worker host attaches the envelope only to `workspaceOperation` errors. It copies only the reason, never the message. - Duck-typed reason. The worker host also forwards an own `workspaceFailureReason` property from any error, when the value is in the closed set. A provider that builds against an older SDK can set the reason before it can import the class. An inherited or unknown value is ignored. The value can only select one of Core's fixed messages. - Core classification. New `server/src/services/workspace-provider-failure.ts`. `callWorkspaceProvider` maps each rejection to a `WorkspaceProviderFailureError` (409) with `kind`, `reason` and `retryable`: - `worker_unavailable`: no request was sent. The `PluginWorkerUnavailableError` stays as the cause, so run setup retries it as `plugin_worker_unavailable` (the existing bounded bootstrap retry). - `timeout` and `worker_exited`: the outcome is unknown, so the message still says to retry the same workspace. - `transport`: any other rejection. The message is the old message. - `provider` with a reason: a fixed message per reason. Only transient reasons say to retry. - `provider` without a known reason: "Workspace storage provider reported a failure; retry the same workspace". - Run settlement. `failRunSetup` records `resultJson.workspaceProviderFailure` (`kind`, `reason`, `retryable`). The `heartbeat execution setup failed` log carries the same fields and the run's `errorCode`. The run keeps `errorCode: setup_failed` (except for the worker-not-running case above), so the existing setup-failure retry, recovery and chat paths do not change. - Span allowlist. The host accepts a `plugin.workspace.failure` span and a `plugin.failure_reason` attribute. The value must be a reason from the closed set or `unknown`. - Docs. `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` describes the failure kinds and the reasons. `doc/observability.md` lists the new span and attribute. ## Verification - `cd packages/plugins/sdk && npx vitest run`: 21 files, 261 tests pass. The new `tests/workspace-operation-error.test.ts` sends errors through the real worker RPC host. - `cd server && npx vitest run src/__tests__/workspace-provider-failure.test.ts src/__tests__/plugin-host-services-span.test.ts src/services/heartbeat/run-completion.test.ts`: 3 files, 118 tests pass. These tests cover: - every transport kind, and that a worker refusal is still found as the cause; - a real worker RPC rejection with a reason, and without a reason; - a fixed message and retry decision for every reason; - span clamping of `plugin.failure_reason`; - run setup for a provider reason (no retry, reason in `resultJson` and in the log) and for a stopped worker (bounded retry). - `cd server && npx vitest run` for the 4 plugin workspace integration suites (`plugin-workspaces`, `plugin-fresh-workspaces`, `project-workspace-provider`, `workspace-delivery-cleanup`): 91 tests pass. Their lost-reply fixtures still get the old "incomplete" message. - `npx tsc --noEmit` in `packages/plugins/sdk`: no errors. `npx tsc --noEmit -p server/tsconfig.json` shows no errors in the changed files. The remaining errors come from workspace packages that I did not build locally. - Not run locally: the full `pnpm -r typecheck`, `pnpm test:run` and `pnpm build`. CI runs them. ## Risks - Behavior change for a stopped worker. A `workspaceOperation` call that the worker did not receive now gets `errorCode: plugin_worker_unavailable` and the bounded bootstrap retry, as other plugin calls already do. Before, it was a `setup_failed` run with no retry. No request reached the provider, so the retry is safe. - Message changes. Provider-reported failures and timeouts get new fixed messages. Code that matches the exact old message only sees it for `transport` and `remote_incomplete` failures. I did not find a match on this message outside tests. - No new error codes. Provider failures keep `setup_failed`. Retry, recovery and chat-retry routing does not change. - Duck typing. Any error with an own `workspaceFailureReason` from the closed set is forwarded. A provider can only select one of Core's fixed messages. It cannot send text or values. - Observability. The new span attribute is a closed set and stays inside the host allowlist. No Telemetry change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, test runs). Extended thinking was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: [paperclipai/paperclip#19326](https://github.com/paperclipai/paperclip/pull/19326) added the same envelope design for Runner revision mismatches. I found no duplicate PR for this change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>	2026-11-05 16:28:49 -08:00
Devin FoleyandPaperclip	8080620fc9	fix(workspaces): size archive deadlines and preserve resume budgets (#19364 ) Give large workspace archives a bounded per-transfer deadline without changing provider settings. Preserve configured post-upload command limits and the saved provider budget when resuming an exact retained sandbox. Verification: 341 focused Core/SDK/adapter tests and 588 Daytona tests passed. Workspace and provider typecheck/build passed. Hosted CI: 54 successful checks, two intended Storybook skips. Greptile 5/5 with no unresolved review threads. The bounded full local run passed all six chat shards and stopped at twenty minutes during general server suites; that full local run remains incomplete. Co-Authored-By: Paperclip <[email protected]>	2026-11-05 16:09:32 -08:00
Nicky LeachandClaude Opus 5.5	a6ab6a301c	fix(server): retry a run when its remote Runner is out of date (#19326 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent can run on a remote Runner that an environment plugin provides > - A release can update the app before the remote Runner, so for a short time the app sees an older Runner > - In that window the plugin can refuse the task, but Core cannot tell this refusal from other errors > - Today the run fails as `setup_failed` with no retry, and the task goes to board recovery. Before #19246, it was also reported as a project authentication conflict > - Agent verification also counts the refusal as a failed harness test, and it parks the agent after three tries > - This pull request gives the plugin a stable error code for a Runner mismatch, and Core waits and retries when it gets that code > - The benefit is that a release no longer blocks tasks or parks agents, and a real authentication conflict keeps its own clear notice ## Linked Issues or Issue Description Refs #19246 (it stopped reporting every precheck error as a project authentication conflict) Refs #19249 (the same bounded bootstrap retry for a stopped plugin worker) Refs #19302 (shares `environment-task-error.ts`; its diagnostic envelope is unchanged) Refs #19172 (related open PR: it reports an undescribable plugin Runner as a retryable check in the same precheck) Refs #18667 (a real `ai_connection_incompatible` conflict, which now gets its own notice) What happened? A run started while its remote Runner was older than the app. The environment plugin refused the task because the installed Runner did not match the revision it expects. Core recorded a generic setup failure, scheduled no retry, and blocked the task. Before #19246, Core reported this as a project authentication conflict. An agent in `verifying` counted the same refusal as a failed harness test. After three tries it stopped verification until a person chose Retry. Expected behavior Core treats a Runner mismatch as temporary. A run retries without blocking its task. A `verifying` agent checks again until the Runner matches. Only a real `ai_connection_incompatible` result blocks the task, with a notice that names the conflict. Steps to reproduce 1. Use an agent on a plugin environment whose installed remote Runner revision differs from the revision the plugin expects. 2. Wake the agent on a task, or put a new agent through verification. 3. The run fails with `setup_failed` and the task is blocked. Verification parks after three tries. Paperclip version or commit master at `377e98f1a2`. Deployment mode Authenticated deployment with a plugin environment that provides a remote Runner. ## What Changed - Plugin SDK: add `PluginEnvironmentTaskError` and `ENVIRONMENT_TASK_ERROR_CODES` with one code, `runner_revision_mismatch`. The worker RPC host sends it as structured `environmentTask` error data with the schema `paperclip/environment-task-error-code/v1` (`environmentTaskErrorCodeData`, read with `readEnvironmentTaskErrorCode`). This uses the same envelope method as the other environment RPC errors. Core reads only the code, never the message. The source refusal diagnostic from #19302 keeps its own schema and API unchanged, and it takes precedence on a source `submit`. - `executeEnvironmentTask` still replaces each worker error with a generic error. After the #19302 diagnostic check, it now keeps only that closed code. - Run setup: a Runner mismatch before provider dispatch gets the error code `runner_revision_mismatch`. Setup schedules a bounded retry with its own reason, `runner_revision_wait`, and budget (10 attempts, 60 s apart). The reason is in the same resource-wait lane as workspace and AI connection waits, so a wait does not spend the failure retry budget. The guards are the same as for `plugin_worker_unavailable`. Recovery and legacy reconciliation treat the code like the other bootstrap-only codes. `scheduleBoundedRetryForRun` uses the Runner wait reason and budget for a mismatch run when a caller gives no reason, and the recovery sweeper reports exhaustion from that budget (`failedRunRetryBudgetSpent`). An answered interaction that hits a mismatch also waits for the Runner, and its successor keeps the resolved interaction continuation identity. - Move the managed project auth classification from `heartbeat.ts` to `classifyManagedProjectAuthFailure` in `run-preparation.ts`. Only a found `ai_connection_incompatible` conflict becomes a configuration failure. All other errors keep their own code. - Recovery notices: `ai_connection_incompatible` now has its own notice instead of the "secret/env bindings" text. An exhausted Runner wait gets the notice "Remote Runner is out of date". - Agent verification: the Runner probe and the managed project auth precheck report a mismatch as a retryable check. `assertHarnessTestPassed` maps it to the new lifecycle code `runner_revision_mismatch`. That code never counts toward the three-failure park. The agent checks again every 60 s and becomes ready when the Runner matches. - Document the error code in `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md`. ## Verification - `cd packages/plugins/sdk && npx vitest run tests/environment-task-error-code.test.ts tests/environment-task-error.test.ts tests/worker-rpc-host.test.ts`: the code crosses the real `environmentTask` RPC. A plain error with the same message gets no code. Other methods get no envelope. The #19302 diagnostic tests still pass. - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts`: a fresh or wrapped mismatch gets `runner_revision_mismatch` and the runner retry budget, with no configuration failure. A started, untrusted (message only) or native run keeps `setup_failed` with no retry. A real `ai_connection_incompatible` stays `configuration_incomplete` with no retry. Other errors, such as an unreachable execution host, are not reported as an authentication conflict. - `npx vitest run src/services/execution-recovery-attempt.test.ts`: Runner waits do not change `failureRetries`, and earlier failures do not shorten the wait. - `npx vitest run src/services/heartbeat/retries.test.ts src/modules/run-dispatch` (embedded Postgres): a recovery caller without options gets the Runner wait budget and its exhaustion. An answered interaction waits for the Runner, keeps its continuation identity and does not spend failure retries. - `npx vitest run src/__tests__/environment-task-runtime.test.ts`: sanitization keeps only the code. No worker text or cause survives. - `npx vitest run src/modules/agent-lifecycle/__tests__/verification-retry.test.ts` (embedded Postgres): a `verifying` agent keeps checking after five mismatches. It never gets `retryRequired`. It becomes ready when the next check passes. - `npx vitest run src/services/native-runtime/plugin-runner-probe-runner-mismatch.test.ts src/__tests__/agent-harness-verification-remote.test.ts src/services/recovery/stranded-notice.test.ts src/services/recovery/service.pause-durability.test.ts src/services/legacy-execution-recovery.test.ts`. - `npx tsc --noEmit` passes for `packages/plugins/sdk` and `server`. - One test in `src/__tests__/heartbeat-process-recovery.test.ts` ("redacts opaque environment-bound credentials from Sentry diagnostics") fails on my machine. It fails the same way on the base commit without this change. - I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm build`. CI covers them. ## Risks - Low to medium. The change adds one new setup failure code and one lifecycle code. Other failures keep their current behavior. - A Runner mismatch must come from a plugin that throws `PluginEnvironmentTaskError`. Until an environment plugin does that, behavior is the same as today. - The run retry budget is 10 minutes. If a Runner update takes longer, the task goes to board recovery with the Runner notice. - A remote Runner run without a managed AI connection reaches the Runner check only after native dispatch. That configuration is not supported, so this change does not handle it. Such a run keeps the native session retry (3 attempts, 30 s apart). - #19172 edits the same precheck call in `agent-environment-test.ts`. Whichever PR merges second needs a small rebase. Keep the mismatch check before its generic retryable check. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (code search, shell and test runs). Reasoning mode: standard agentic coding. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>	2026-11-05 14:34:23 -08:00
Nicky LeachandPaperclip	9a79384958	Relay bounded Runner source refusal diagnostics (#19302 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An environment plugin can submit a Runner continuation from a retained source. > - The host can refuse that continuation when its source is unverified or has an unresolved operation. > - The worker and Core currently discard that bounded reason. > - Raw provider errors can contain credentials, text or foreign identities. > - This pull request relays only a closed diagnostic and keeps the existing admission and redaction rules. ## Linked Issues or Issue Description Refs #19134 and #19157. This independent fix is based on master. A search found no duplicate relay. What happened? A source submit can receive HTTP 409 with `SESSION_STATE_UNAVAILABLE`. The worker drops its structured data. Core then replaces the error with a generic message. The caller cannot distinguish the two known diagnostic reasons safely. Expected behavior Carry only the fixed status, code and reason through SDK worker RPC and Core. Keep all provider text and identities out of the relayed error. A diagnostic must not authorize a retry or change state. Steps to reproduce Submit an authorized retained source through a task provider. Have the provider throw the recognized state-unavailable error. Read the error after worker RPC and Core dispatch. ## What Changed - Export `PluginEnvironmentTaskErrorDiagnostic`, `environmentTaskErrorData` and `readEnvironmentTaskErrorDiagnostic`. - Accept only `httpStatus: 409`, `errorCode: "SESSION_STATE_UNAVAILABLE"` and reason `"unresolved_operation"` or `"source_state_unverified"`. - Attach the versioned envelope only to an environment task `submit` with `resumeFrom`. Replace its provider message with a fixed message. - Revalidate and reconstruct the closed error in Core. Keep other failures under the existing generic redaction. - Document older SDK behavior and the diagnostic-only boundary. No grant, retry, discard, lifecycle or admission rule changes. ## Verification - Rebased head `7d02bb3e4c` includes master’s UI fixture fix from #19342. The sole conflict was in the worker integration test imports. It now imports both the physical-task result schema from #19300 and the diagnostic reader and RPC error type from this PR. The 39 SDK/worker cases and 57 Core/runtime cases passed, including PostgreSQL worker integration. The SDK build and Runner/server TypeScript checks passed. [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/42111889388) passed without retries. All 56 check-rollup entries were green, including Snyk and Socket; [standard Greptile review 2](https://github.com/paperclipai/paperclip/pull/19302#issuecomment-6665823815) scored this exact head 5/5, with zero unresolved threads. The PR was marked ready on November 5 at 21:37 UTC. - Both new worker relay cases fail against the baseline worker and pass with the fix. - 46 SDK/worker cases passed, including both reasons, malformed/foreign envelopes, extra identities, unsafe getters and non-source operations. - 51 Core/runtime cases passed. The integration uses real SDK serialization and dispatch with PostgreSQL through the existing authorized retained-source submit. It verifies both reasons, malformed RPC rejection, extra-field removal, preserved source outcome and existing dispatch bookkeeping. A valid envelope on an ordinary submit or status stays generic. - Runner and SDK builds and server TypeScript check passed. An initial Runner build lacked its eval-kernel build prerequisite; it passed after that prerequisite was built. - The first expanded DB fixture used a shared environment name and failed its unique constraint. It now uses its unique fixture ID; both complete focused files pass. - Diff and added-line secret scans passed. [GitHub workflow 42083410387](https://github.com/paperclipai/paperclip/actions/runs/42083410387) passed on its second attempt. All CI checks completed, and Snyk passed. Standard Greptile scored exact head `38559ec4df8c` 5/5 with no unresolved threads. That result applies to the previous head. - The first GitHub attempt failed in the unchanged concurrent-budget case in `decision-models.test.ts` (one provider call observed, two expected). That exact case passed locally without code or assertion changes. The workspace shard also passed 4,960 tests before the unchanged CLI company import/export fixture exceeded its 30-second child-server startup wait. No port-collision cause is claimed; that fixture deliberately reserves the requested port. Both initial logs are retained. One failed-job-only retry passed without source changes. ## Risks Older SDKs keep the generic failure. The envelope is diagnostic evidence only. Provider operation and call identities are never accepted. Specific user-visible operation reporting must use Core's retained event or receipt under its existing authority. Master does not contain the new claimed predecessor orchestration. This PR tests the existing retained-source submission boundary. It does not claim the #19157 source recovery or old-journal end-to-end qualification. Runtime pins and deployments remain separate. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning. Used repository tools, TypeScript, SDK RPC and real PostgreSQL tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR using the bug template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change - [x] I have run the focused tests locally and they pass - [x] I have added tests for the new behavior and rejection paths - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no unresolved findings - [x] I will address reviewer comments before requesting merge Co-authored-by: Paperclip <[email protected]>	2026-11-05 13:38:55 -08:00
Devin FoleyandPaperclip	c3550cfb38	fix(daytona): send archive chunks in bounded windows (#19339 ) ## Thinking Path > - Paperclip manages agents and saves their work when execution stops. > - The Daytona plugin saves an archive through a terminal when disk capacity prevents the normal archive or download copy. > - That fallback sent one 40 KiB chunk and waited for its host write acknowledgement. > - Network delay then applied to every small chunk. A large archive could exceed the recovery deadline even while bytes were moving safely. > - This pull request sends a bounded window of chunks and still acknowledges every completed host write. > - The benefit is faster recovery over delayed connections, with bounded memory and the existing size, space and completion checks. ## Linked Issues or Issue Description Refs #19034, which added the capacity fallback. What happened? The archive fallback waits for one network round trip after each 40 KiB chunk. A saved workspace archive can be much larger than that chunk. Repeating a timed-out transfer does not remove this throughput limit. Expected behavior Allow a small amount of data to travel before waiting for acknowledgements. Keep memory bounded, write bytes in order, and stop after any write or protocol failure. Steps to reproduce 1. Run the real Bash archive sender against a host that delays each acknowledgement by 400 ms. 2. Transfer 1 MiB plus a partial final chunk with a four-second deadline. 3. The original implementation times out. The bounded-window implementation completes and preserves the exact source bytes. Paperclip version or commit The failing regression uses the original implementation from `6a66c09f7f7`. ## What Changed - Send at most sixteen 40 KiB chunks before collecting their acknowledgements. Drain the final partial window before the completion receipt. - Serialize host writes and reject more than sixteen pending frames. Stop queued writes on the first failure. - Keep per-frame size checks, total archive limits, the 1 GiB host space reserve, the transfer deadline and bounded teardown. - Add a real Bash regression with delayed acknowledgements, plus overflow and failed-write coverage. Update the plugin documentation. - Update the agent-creator test fixture for the required directory membership role introduced by #15183, which otherwise breaks merged CI typechecking. ## Verification - Failing first: the final real-Bash regression fails against the original source with `Daytona archive stream timed out`. It passes with the change. The empty-stream case passes too. - Standalone Daytona package `pnpm test`: 578 passed, 8 skipped. The skips are the existing opt-in live tests. - Standalone package `pnpm typecheck` and `pnpm build`: passed. - `pnpm test src/file-sync-capacity.live.test.ts` with a staging Daytona key: 2 passed on isolated test sandboxes. This covers actual terminal transport, stop/start, archive and file-copy capacity failures, exact saved hashes, safe links, cleanup and a second export. The tests removed their own sandboxes. - The production dependency assumption is bounded: at most 640 KiB of decoded payloads can wait for host writes. The parser also has a bounded encoded buffer. No background work or idle timer is added. - Initial hosted CI failed with TS2741 in `ui/src/lib/agent-creator.test.ts` after #15183 made `membershipRole` required. Rebased onto that change and added the nullable field to the fixture. UI `pnpm typecheck`, all 4 existing creator-label tests, and `pnpm check:token-gates` now pass locally. Hosted CI subsequently passed on the updated head. - `git diff --check` and a local secret and PII scan: passed. - Full repository checks were not repeated locally for this standalone plugin change. Hosted verification at `bf6b3cb6a622a9523c72b86fdfe616555c8f3ef8`: all 53 check runs passed, with the 2 opt-in Storybook checks skipped; both required gates (`ci / verify`, `ci / e2e`) passed. Greptile is 5/5 at this exact head, with no review threads or merge conflicts. The separate non-required `security/snyk (cryppadotta)` status remains pending; it is not claimed as passed. ## Risks - A burst can queue up to sixteen chunks instead of one. The receiver enforces this bound and acknowledges a chunk only after its ordered host write. - The timeout budget is unchanged. A connection that cannot finish within that budget can still fail. This change does not retry a provider turn or change file-recovery authority. - Source confinement, archive validation, successful producer completion and publication rules are unchanged. No schema migration. ## Model Used - OpenAI `gpt-6.1-sol` with xhigh reasoning, in Codex through T3 Code. Tool use included source edits, real Bash and filesystem tests, and isolated Daytona tests. The runtime does not expose its context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-11-05 13:12:11 -08:00
Nicky LeachandPaperclip	2506bcdaf4	Admit exact stopped Runner sources under the current run lease (#19134 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution keeps provider state and a durable controller journal. > - A completed process can stop before the next run attaches. > - The current run must own recovery of that exact source. > - An old controller must lose permission to change the consumed source journal. > - This pull request adds explicit source admission and protected task dispatch. > - The benefit is a narrow recovery boundary that preserves the original successful result. ## Linked Issues or Issue Description Native prerequisites #19132 and #19153 are merged into master. This PR targets master and updates the TypeScript Runner driver contract alongside Core and SDK admission. This change adds the SDK and Core admission boundary. Executor selection and fresh source preparation are a separate change. Related #17916 supplies the workspace export holds preserved here. ## What Changed - Plugins advertise `codexSourceRecoveryVersion: 1` for Codex App Server and `acpxSourceRecoveryVersion: 1` for ACPX. Hosts can qualify either, both, or neither. Core selects the matching field, because a stopped-source check for one provider cannot establish the other provider’s recovery guarantees. ACPX orchestration is tracked separately in #19196. Use the existing exact source identity and `resumeFrom` under the authenticated current run. No public recovery mode is added. Require the claimed source, current account and prepared launch at protected dispatch. - Keep the first source binding pinned on the consumer lease, including after uncertain dispatch. A newly acquired coordinator can refresh only the same source. - Claim one stopped successful source under the current run's acquired native lease. Store the token-free consumer reference in existing lease metadata. Preserve the source status and result. - Check the exact six source identity fields, encrypted journal header, company, task, agent, environment, driver, workspace, account selection, and current coordinator generation. Keep workspace export and legacy settlement holds. - Serialize claims and checkpoint writers in the same coordinator lock order. Reject stale source writes after consumption. Check the current consumer before recovered source writes. - Check source admission before and after protected worker dispatch. Keep the request and lease owned by the current run while the native identity remains the source. - Allow the same owner to prepare B after a fresh source submission, using the exact consumed-source proof instead of a generic warm-reuse flag. Recheck that proof after dispatch. - Preserve ordinary cached adoption and repeated preparation after launch. The current authenticated owner can differ from the adopted PRP identity without requesting a new source launch. Keep ordinary suspended recovery and live task attachment checks. Extract the shared source resolver to avoid a module cycle, and make ordinary selection honor the same durable source consumer. ## Verification - Current head `cac9cf188a` includes merged #19300 and the symmetric capabilities introduced in `b7e25a64b4`. The task-owner and warm-rebind fixtures now identify their physical task. All 29 SDK schema cases, 95 Core ownership and source-admission cases, 25 native Codex and ACPX recovery cases, and the binary metadata check passed on this head. The SDK and Runner builds and server typecheck passed. Both native providers reject the other provider’s capability. GitHub CI is green and [standard Greptile scored this commit 5/5](https://github.com/paperclipai/paperclip/pull/19134#issuecomment-6645574114) with no open threads. - Previous head `bda49271a4ae362a15baf3fe6d6d297e25fd2000` is directly based on master after #19132 and #19153 landed. [GitHub workflow 42072192023](https://github.com/paperclipai/paperclip/actions/runs/42072192023) passed on its first attempt. All check runs completed successfully or as intentional workflow skips; Snyk passed. Standard Greptile scored this exact head 5/5. That review had no unresolved findings. - Runner and SDK TypeScript builds and server typecheck passed. The focused SDK, task-owner, task-runtime, PostgreSQL source and authority suites passed all 161 cases. The production transport-factory getter regression and 14 native-lane selection checks also passed. - The Linux native lane executed the new ordinary warm A→B process case. It kept one Runner/provider process and completed two turns on the same thread. The eager-getter mutation fails with `native_runner_source_attachment_pending`; preserving the live getter passes. The production factory test changes the current owner after transport creation and checks that same options object. - PostgreSQL cases cover both concurrent source-writer/claim orders, exact source submission under the current caller, source-switch denial after uncertain dispatch, same-source coordinator refresh, stale same-boot generation, corrupted journal headers, competing consumers, account changes and export holds. - The protected dispatch sequence submits A, prepares B and retries B preparation. It rejects undispatched sources, changed accounts and foreign target bindings. Source status and accepted result remain unchanged. - Ordinary adopted A preparation under current B, repeated ordinary preparation after launch and ordinary suspended continuation retain their existing path. Absent or false claims preserve ordinary identity denial; malformed claims fail closed. - Earlier development attempts exposed ordinary preparation regressions and the live-getter omission; the final regressions cover those fixes. The first embedded PostgreSQL attempt needed missing library aliases repaired in ignored dependencies. Those attempts are not passing evidence. - Historical, superseded CI heads had unchanged UI fake-timer and Playwright health-response-lifetime failures. Their isolated cases and scoped retries passed. The response-lifetime cause remains unproven. No UI/test timeout was changed. The final rebased-head workflow above passed without a retry. - Whitespace and added-line secret/PII scans passed. Full workspace checks ran on GitHub. No live vendor probe or deployment was performed. ## Risks - Any driver that used the earlier Codex TypeScript option must use `codexSourceRecoveryVersion`. The new plugin description rejects the generic spelling. The Launcher’s existing Codex-specific HTTP header can map to this property when the plugin adopts the new SDK. The change requires no journal migration or ROM change. - This is an inactive admission foundation. Executor orchestration is separate in #19157. Native restore integrity and provider stopped-source/account validation must be present before executor activation. The native prerequisite uses ordinary recovery followed by attachment. The composed executor proofs are in #19157. - The source claim is durable. A later controller must recover the same consumer or keep the hold. It cannot select another source after an ambiguous launch. - Admission adds bounded database checks around protected startup calls. Busy issue locks fail closed; other lock waits have a one-second bound. - No migration, snapshot format, proactive receipt, recurring worker, or deployment is included. Full workspace checks and platform coverage run in CI. These tests do not prove vendor history durability or a hosted rollout. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, through Codex with repository tools and code execution. The runtime does not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on the current head - [x] Standard Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-11-05 12:55:06 -08:00
Devin FoleyandClaude Opus 5.5	ee72132a44	fix(daytona): bound one-shot exec requests by their command deadline (#19298 ) Bound the Daytona one-shot executeCommand request at its command deadline plus 5 s and return an attributed timed-out result, instead of waiting for the host RPC deadline. Bound the stdin cleanup the same way. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-05 11:39:00 -08:00
Devin FoleyandClaude Opus 5.5	7f5d0793bd	fix(daytona): record a bounded errno when a sandbox command fails (#19297 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Daytona sandbox provider copies the agent's workspace back to the host after each turn (syncOut) > - When the sandbox `tar` fails in the `archive_create` step, the restore diagnostic records only `exitCode: 2` and `errorCode: "unknown"` > - The command output is not sent across the plugin RPC boundary, so an operator cannot tell a full disk from a permission error or a path that is too long > - This pull request maps the system error text at the end of a failed sandbox command's output line to the existing allowlisted `errorCode` > - The benefit is that the next failure records the real errno, and the output text stays private ## Linked Issues or Issue Description Refs #18172 Refs #18212 What happened? A hosted Daytona run failed its workspace restore. The `archive_create` step exited 2 on all three attempts. The run log recorded `{"step":"workspace_transfer","transferStep":"archive_create","transferFailureKind":"command_failed","exitCode":2,"errorCode":"unknown"}`. Two causes are possible. The sandbox disk was full, or a path was longer than PATH_MAX, for example from deeply nested GitHub helper config directories before #18212. The diagnostic cannot tell these causes apart. Expected behavior A failed sandbox command records a bounded errno such as `ENOSPC`, `EACCES` or `ENAMETOOLONG` when its output shows one. The output text itself must not be recorded. Steps to reproduce 1. Run `performSyncOut` against a sandbox whose `tar -c` exits 2 with `tar: ./x: Cannot open: Permission denied`. 2. Read the environment sync diagnostic from the thrown error. 3. See that `errorCode` is `unknown`. Paperclip version or commit `master` at `3fca8a1`. ## What Changed - `packages/plugins/sandbox-providers/daytona/src/file-sync.ts`: `assertSandboxCommandOkWithOutput` maps the strerror suffix of a failed command's output to an allowlisted errno. It reads only the last 16 KiB in memory. A line must end with `: <message>`. Fatal causes are checked first (ENOSPC, EIO, ENAMETOOLONG, EACCES, EPERM, ENOTDIR, EISDIR, ENOENT), so a warning about a vanished file does not hide the error that stopped `tar`. The error message, capacity fallback (#18172), restore and retain-on-failure behavior do not change. - `packages/plugins/sdk/src/environment-sync-error.ts` and `packages/adapter-utils/src/workspace-restore-diagnostics.ts`: add `ENAMETOOLONG` to both allowlists. Both sides still validate independently. - Tests: 7 new Daytona cases, plus one allowlist case each in the SDK and adapter-utils. ## Verification - `vitest run --config packages/plugins/sandbox-providers/daytona/vitest.config.ts packages/plugins/sandbox-providers/daytona/src/file-sync-diagnostics.test.ts`: 18 passed. Without the `file-sync.ts` change, the 7 new cases fail with `errorCode: "unknown"`. - `packages/plugins/sdk`: `vitest run tests/environment-sync-error.test.ts`: 20 passed. - `packages/adapter-utils`: `vitest run src/workspace-restore-diagnostics.test.ts`: 50 passed. - `tsc --noEmit` passes for the SDK and adapter-utils. `git diff --check` is clean. - I did not run the Daytona package typecheck or its full suite locally because the sandbox disk was full. CI covers them. ## Risks - Low risk. The change adds a diagnostic field only. Control flow, error messages and the capacity fallback regex do not change. - Localized `tar` output does not match, so it stays `unknown`. That is the current behavior. - `ENAMETOOLONG` is a new allowlisted code on both sides. It is a fixed string and carries no path or output text. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in an agent harness. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>	2026-11-05 11:34:41 -08:00
DottaandClaude Opus 5.5	037c58bb22	fix(daytona): keep the provider auto-stop strictly after the warm runner idle window (#19199 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm remote agent keeps its sandbox running after a turn. The host keeps a warm runner for an idle window (`runnerIdleTimeoutMs`, or the agent's `idleTimeoutMs`) > - The Daytona provider also stops an idle sandbox after `autoStopInterval` minutes. The default is 15 minutes > - On staging, the warm idle window was 900,000 ms (15 minutes). The two windows were equal > - A follow-up turn at the end of the warm window resumed a sandbox that Daytona was stopping. Daytona refused the resume in about 4 seconds, and the run failed at setup > - #19195 added a bounded retry for that refusal. It did not remove the cause > - This pull request makes the two windows non-overlapping by construction. Daytona derives its auto-stop for a warm reusable sandbox from the same warm idle value that the host uses > - The benefit is that a follow-up inside the warm window always finds a running sandbox, and idle sandboxes still stop ## Linked Issues or Issue Description Refs #19195 (bounded retry for a refused resume; this change removes the common cause). What happened? On a staging soak, three follow-up runs failed at setup with a refused sandbox resume. Each resume began 15 minutes 8 to 25 seconds after the previous turn released the sandbox. The warm idle timeout was 900,000 ms. Daytona's default `autoStopInterval` is 15 minutes. Nothing in the code kept these two values apart. Expected behavior A reusable sandbox that keeps a warm runner must not auto-stop inside the warm idle window or at its edge. The provider backstop must fire after the host's own idle handling, with a margin. Idle sandboxes must still stop. Steps to reproduce 1. Configure a Daytona sandbox environment with `reuseLease` and a warm runner with `runnerIdleTimeoutMs: 900000` (or an agent with `lifecycleMode: warm`, `idleTimeoutMs: 900000`). 2. Run a turn. Do not set `autoStopInterval`. 3. Before this change: the sandbox is created with `autoStopInterval: 15`, equal to the warm window. A follow-up near minute 15 races the provider stop. Paperclip version or commit `c656b95b6b` (current `master`). ## What Changed - Daytona plugin (`plugin.ts`): add `effectiveAutoStopIntervalMinutes`. For a reusable lease that keeps a warm runner, the effective auto-stop is at least `ceil(warm idle / 1 min) + 5` minutes. The plugin applies the floor only when the host forwards `warmRunnerIdleTimeoutMs`, or when the environment itself selects `runnerLifecycleMode: warm`. An explicit `0` (disabled), a longer configured value, `inherit` with a per-turn agent, per-turn, and non-reusable sandboxes keep the configured value. - Daytona plugin: use the effective value at sandbox creation. On resume, raise a shorter auto-stop on an existing sandbox. This update is best effort and bounded to 5 seconds; the #19195 retry still covers a refusal. The stale-handle refresh threshold keeps the configured interval, so a cached handle is not trusted for longer before the provider confirms the new interval. The warm window is not written to the saved normalized config. - Host (`runner-environment-lifecycle.ts`): add `resolveWarmRunnerIdleTimeoutMsForRun`. It returns the run's effective warm idle window (environment `runnerIdleTimeoutMs`, else the agent `idleTimeoutMs`) for a warm `paperclip_runner` sandbox run. It shares the lifecycle-mode resolution with `resolveRunnerEnvironmentForRun`. - Host (`environment-run-orchestrator.ts`, `environment-runtime.ts`): pass that value as `warmRunnerIdleTimeoutMs` to the sandbox-provider driver on acquisition and on remote recovery resume. The driver adds it to the provider worker config only. Lease identity, reuse matching, and lease metadata are unchanged. - Daytona manifest: describe the warm floor on `autoStopInterval`. - `doc/execution-semantics.md`: document the contract. ## Verification - Red (before the fix, same tests, with the floor disabled): 5 failures in `plugin.test.ts` "warm idle window versus Daytona auto-stop". The staging case failed with `expected 900000 to be greater than or equal to 1200000`. The resume case failed because `setAutostopInterval` was not called. - Red (host): the new `environment-runtime.test.ts` case first failed because the provider worker config had no `runnerIdleTimeoutMs`. The service wrapper dropped the field. The fix forwards it. - Green: `npx vitest run --config vitest.config.ts` in `packages/plugins/sandbox-providers/daytona`: 563 passed. This includes a setter that never settles (resume completes after the 5-second bound) and `inherit` with no forwarded window (explicit `autoStopInterval: 2` stays 2). - Green: `npx vitest run src/services/runner-environment-lifecycle.test.ts src/__tests__/environment-runtime.test.ts src/__tests__/environment-run-orchestrator.test.ts` in `server/`: 242 passed. This includes recovery resume forwarding the admitted warm window. - The regression tests assert the relationship `autoStop >= warm idle + margin` for several warm windows (125 s, 300 s default, 900 s, 1,800 s) and for an explicit `autoStopInterval: 15`. They are not constant checks. - `node scripts/typecheck-low-memory.mjs` passes. `pnpm typecheck` in the Daytona package passes. ## Risks - A warm reusable Daytona sandbox now stays running up to the warm window plus 5 minutes after its last activity, instead of 15 minutes. With a 15-minute window this is 20 minutes. Cost is bounded and idle sandboxes still stop. - The resume-time `setAutostopInterval` call is best effort. If it fails, the old interval stays and the #19195 retry still applies. - Other providers are unchanged. They receive one extra host-owned config key (`warmRunnerIdleTimeoutMs`) only for warm `paperclip_runner` runs, and they ignore unknown keys. - No lease identity change, so existing retained sandboxes are still reused. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>	2026-11-05 01:10:19 -06:00
DottaandClaude Opus 5.5	c656b95b6b	fix(environments): retry a refused sandbox resume with the bounded bootstrap budget (#19195 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm remote agent keeps its sandbox after a turn, so a follow-up turn resumes the same sandbox and its files > - The sandbox provider auto-stops an idle sandbox (Daytona: 15 minutes by default). The warm native idle window is also 15 minutes > - A follow-up that reaches the sandbox during that stop gets a provider refusal. The host keeps the sandbox ("the lease was preserved and no replacement was created"), which is correct > - But setup then records a generic `setup_failed`. No retry is scheduled, no cause is recorded, and the reply that woke the agent is effectively dropped > - This pull request gives that refusal its own failure code and routes it through the existing bounded bootstrap retry. The retry resumes the same recorded sandbox > - The benefit is that a follow-up continues a few seconds later on the same saved sandbox, instead of stopping the task with no next action ## Linked Issues or Issue Description Refs #18852 (the bounded bootstrap retry that this change reuses). Related: #19152 and #19187 (copy-back causes), #19178 (Daytona errored-sandbox recovery before stop-only cleanup), #19182 and #19180 (recovery holds). This change does not overlap with them: they cover cleanup, copy-back, and recovery after a run. This change covers a resume refusal before provider work starts. What happened? On a staging soak, two follow-up turns failed at setup after a successful first turn: - A board reply woke the agent. The run failed after 4.9 seconds with `setup_failed`: `Reusable sandbox lease "…" could not be resumed; the lease was preserved and no replacement was created.` The issue stayed in review with no recovery action. The reply was not delivered. - An earlier run on another task failed with the same message. Both resume attempts started 15 minutes 8–25 seconds after the previous turn released the sandbox. The first turns used `keep_running` with a warm idle timeout of 900,000 ms. The Daytona default `autoStopInterval` is also 15 minutes. So the resume reached the sandbox while the provider was stopping it. The provider refused within seconds; this was not a long resume timeout. The provider cause was not recorded, so this timing is the strongest evidence. The exact provider state remains an inference. Expected behavior A refused resume before provider dispatch is a temporary infrastructure condition. The host retries it with a small, bounded budget. Each retry resumes the same recorded sandbox. A refusal does not prove that the sandbox is lost, so the host never destroys it or creates a replacement. Only an explicit provider not-found or a workspace identity mismatch with a verified backup permits replacement (unchanged). The run records why the provider refused, without provider text. Steps to reproduce 1. Run a warm remote turn that keeps its sandbox. 2. Make the provider's `environmentResumeLease` throw a non-transient error (for example, a 400/409 while the sandbox stops). 3. Wake the agent with a reply. 4. Before this change: the run fails with `setup_failed`, no retry is scheduled, and the error has no cause. Paperclip version or commit Seen on staging at `5b99bdae`. The touched code is unchanged on current `master` (`f7a4c5678f`). ## What Changed - Move `ReusableSandboxResumeError` to `server/src/services/reusable-sandbox-resume-error.ts` (re-exported from `environment-runtime.ts`). Give it the code `reusable_sandbox_resume_failed`. - Add the provider refusal to the error message by closed labels only: `(cause=<label> status=<http> code=<errno>)`. The labels come from the existing transport classifier, now exported as `sandboxProviderFailureLabel`. Provider text, paths, and IDs are never copied. - Carry the provider status across the plugin worker RPC. A JSON-RPC error has only a message and a numeric RPC code. The worker host now attaches a validated `paperclip/environment-resume-error/v1` envelope (HTTP status and an allowlisted errno only) to `environmentResumeLease` failures. The host re-validates it and adds it to the recorded cause. - In setup-failure settlement, find this error through the orchestrator wrapper. Record `reusable_sandbox_resume_failed` only when provider dispatch did not start and the run has no native session or runner identity (the same guard as `remote_workspace_prepare_timeout`). - Add the code to `isTransientWorkspacePreparationCode`. The existing scheduler then gives two retries (30 s, 60 s) with its issue gate, Stop, pause, and dependency checks. - Exclude the code from generic continuation recovery and from legacy reconciliation (no unknown provider actions; bootstrap evidence proves none started), like the template. - Add a specific stranded-recovery notice: "Saved sandbox could not be resumed". - Document the contract in `doc/execution-semantics.md`. ## Verification - Red (before the fix, same tests): 6 failures. The `fresh` case recorded `setup_failed` and did not schedule a retry. The error had no cause. Legacy reconciliation held the run. Generic recovery granted another budget. - Green: `npx vitest run src/services/heartbeat/run-completion.test.ts src/services/legacy-execution-recovery.test.ts src/services/recovery/ src/__tests__/heartbeat-retry-scheduling.test.ts src/__tests__/environment-runtime.test.ts src/services/remote-workspace-transport.test.ts src/services/heartbeat/retries.test.ts` (in `server/`): 619 tests pass. Also `src/services/recovery/stranded-notice` and `src/services/remote-workspace*`: 411 pass. - Safety regressions: - `run-completion.test.ts`: only a pre-dispatch legacy failure with no native identity gets the code and a retry. `started`, `retained` (native), and an untyped copy of the message stay `setup_failed` with no retry. The recorded error contains the closed cause and not the provider path. - `heartbeat-retry-scheduling.test.ts` (real PostgreSQL): two concurrent schedulers across a restart create exactly one successor per attempt. The budget ends after two attempts. Stop, pause, and a new dependency cancel the scheduled retry. Generic stranded recovery creates no extra run after exhaustion. - `packages/plugins/sdk/tests/environment-resume-error.test.ts`: through the real worker RPC, the `JsonRpcCallError` has no status, and the envelope gives 409 and the errno. Provider text does not cross. Other methods get no envelope. Malformed envelopes are rejected. - `environment-runtime.test.ts`: a throwing resume still preserves the exact lease. The runtime does not call destroy or acquire. The resume throws a `JsonRpcCallError` whose message has no status. The thrown message names `cause=provider_error status=409` from the envelope, and not the provider text. - `node scripts/typecheck-low-memory.mjs` passes. `git diff --check` passes. - Not verified: a live staging run at this revision. The original tasks were not retried. This PR is not deployed. ## Risks - Low. The change affects only runs that fail with this typed error before provider dispatch. Each retry resumes the same lease and inherits the source wake context, so a reply is delivered once and no second executor starts. - If the provider cannot start the sandbox for longer than about 90 seconds, the two retries end and the task gets the specific recovery notice. The sandbox and its saved files are kept. - The error message now has a parenthetical cause before the semicolon. No code parses this message. - An issue in review does not get the immediate stranded notice (unchanged existing behavior). The retry, not the notice, delivers the reply in that case. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, browser reads of staging diagnostics). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>	2026-11-05 00:20:21 -06:00
Devin FoleyandPaperclip	895cee28b3	fix(daytona): recover an errored sandbox before stop-only cleanup (#19178 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote agent runs use sandbox provider plugins. Daytona is one of them. > - After a failed run, Paperclip stops the sandbox and keeps its files ("stop-only cleanup"). A task can continue only when every remote lease of the failed run has a provider stop receipt. > - Daytona rejects Stop for a sandbox in its `error` state. A sandbox that filled its disk went to that state. The plugin called Stop on it, so every cleanup sweep failed with the same generic error. > - The lease stayed in pending cleanup without a stop receipt. Each Retry or comment waited forever with reason `remote_cleanup`. A manual Daytona Recover was the only fix. > - This pull request makes the Daytona stop-only hook recover a recoverable sandbox once, with a bound, and then stop it. When it cannot stop the sandbox, it returns a typed failure. The server escalates a terminal failure at once instead of retrying it like a transient error. > - The benefit is that a full-disk sandbox no longer blocks its task, and a sandbox that cannot recover is visible to operators on the first attempt. ## Linked Issues or Issue Description No public issue exists. Bug description: What happened? A task sandbox filled its 10 GB disk. Daytona then reported the sandbox as `state: error`, `desiredState: started` and `recoverable: true`. Its `errorReason` was a Docker `no space left on device` error. The Daytona `stopLease` hook calls Stop on any sandbox that is not `stopped` or `archived`. Daytona rejects Stop in the `error` state. The pending cleanup sweep ran about every 30 minutes, and it failed on every pass. The lease stayed `pending_cleanup` with `cleanup_status=failed` and no stop receipt. Continuation admission (`server/src/services/explicit-native-continuation.ts`) requires a stop receipt on every remote lease of the failed run. So each Retry and comment was deferred with "Waiting for the previous environment to stop. Your message will start automatically." It never started. A manual Daytona `POST /api/sandbox/{id}/recover` fixed it. The sandboxes reached `started` in about one minute. The next sweep stopped them, recorded the receipt, and the tasks continued. Expected behavior Stop-only cleanup recovers a recoverable errored sandbox and then stops it. When the sandbox cannot be recovered, cleanup returns a clear, typed failure that the server can tell apart from a transient error. Cleanup never deletes the retained sandbox, and it never loops on Recover. Steps to reproduce 1. Run a task in a Daytona sandbox until the sandbox fills its disk and Daytona shows `state: error`, `recoverable: true`. 2. Let the run fail, so the server records a `sandboxStopAndRetain` intent and calls `environmentStopLease`. 3. Before this change, Stop fails with `Sandbox failed to stop with status: error` on every sweep, and the task cannot continue. 4. The new tests in `plugin.test.ts` reproduce this with a mock sandbox whose Stop fails from `error`. Paperclip version or commit Observed on a hosted deployment. Reproduced in unit tests on `master` at `6452e964a36`. Deployment mode Managed Daytona sandbox execution with plugin-backed stop-only cleanup. Related: #18172 (Daytona exports under disk pressure) and #19050 (session reset on every Daytona restart path). This PR does not change the resume path in `ensureSandboxStarted`, which already recovers a recoverable sandbox. ## What Changed - Daytona plugin (`packages/plugins/sandbox-providers/daytona/src/plugin.ts`): after `stopLease` confirms the retention policy, a sandbox in `error` goes through `recoverSandboxForStop`. - Recoverable: the hook calls `sandbox.recover()` once. The SDK waits for `started`. The plugin bounds the wait to 20 s, plus the existing liveness margin for a hung connection. Then the hook stops the sandbox normally and returns the usual `stopped` receipt. - Not recoverable: the hook makes no Recover or Stop call. It throws `PluginEnvironmentStopError("resource_unrecoverable")`. - Recover fails: the hook reads the provider state once. A sandbox back in `error` gives `recovery_failed`, or `resource_unrecoverable` if Daytona now marks it so. A sandbox that is still starting, or a failed read, gives `recovery_incomplete`. - The recovered sandbox fails again before Stop finishes (for example, the disk fills again): a fresh read that shows `error` gives `recovery_failed`. The hook never calls Recover a second time in the same call. - The hook never deletes or archives the sandbox. The non-error stop path is unchanged. - Plugin SDK (`packages/plugins/sdk/src/environment-stop-error.ts`, new): `PluginEnvironmentStopError`, `environmentStopErrorData` and `readEnvironmentStopFailure`. This follows the existing `PluginEnvironmentRunnerIngressError` pattern. The envelope holds only one closed reason. The reader validates the reason and derives `terminal` from it. `resource_unrecoverable` and `recovery_failed` are terminal. `recovery_incomplete` is not. - Worker RPC host (`packages/plugins/sdk/src/worker-rpc-host.ts`): forwards that envelope as JSON-RPC error `data` for `environmentStopLease` only. - Pending cleanup sweep (`server/src/services/heartbeat/recovery.ts`): the catch reads the validated envelope. - On a terminal failure, it escalates on the first attempt. It writes the one-time "needs operator attention" warning and uses the existing 30 minute retry interval. Before, this happened only at the fifth attempt. - The failure log uses `errorKind: "provider_stop_terminal"` and includes the closed `providerStopFailure` reason. The sweep still never logs exception fields. - The lease stays `pending_cleanup`. Automatic cleanup continues, and an explicit Retry still bypasses the cooldown. Untyped and transient failures keep their current behavior. - Docs: the SDK README ("Stop failures"), the Daytona README, `doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md` and `doc/native-workspace-finalization-recovery.md` describe the new behavior. Shared contract note (module boundaries): the SDK envelope is the smallest addition that lets the host tell a terminal provider stop failure from a transient one. The worker host drops all other error data across the RPC boundary, so the existing contracts cannot carry the signal. The only consumer is the pending cleanup sweep. When a provider does not throw the new error, or an older host ignores it, behavior is unchanged. This PR does not touch `server/src/services/workspace-recovery.ts` or `sandbox-stop-and-retain.ts`. ## Verification All commands ran in a clean worktree from `master`. Vitest ran from the real path. - Red on `master` (only the new tests added): - Daytona, with `plugin.ts` from `master`: `npx vitest run --config vitest.config.ts src/plugin.test.ts -t "provider error state"` exits 1. All 7 new tests fail. The main case fails with `Sandbox failed to stop with status: error`, which is the production error. - SDK, with `worker-rpc-host.ts` from `master`: `npx vitest run tests/worker-rpc-host.test.ts -t "typed stop failure"` exits 1 (the typed case fails). - Server, with `recovery.ts` from `master`: `npx vitest run src/__tests__/heartbeat-pending-cleanup-sweep.test.ts -t "terminal provider stop failure"` exits 1. All 3 cases fail (`errorKind: "destroy_failed"`, no escalation). - Green on this branch: - `packages/plugins/sandbox-providers/daytona`: `npx vitest run --config vitest.config.ts` exits 0 (534 passed, 26 skipped live tests). `pnpm typecheck` exits 0. `pnpm build` exits 0. - `packages/plugins/sdk`: `npx vitest run` exits 0 (222 passed). `pnpm --filter @paperclipai/plugin-sdk typecheck` exits 0. - `server`: 15 related suites exit 0 (1498 passed): pending cleanup sweep, `heartbeat/recovery`, environment runtime, environment cleanup work, native startup cleanup, native remote failure retention, observed workspace preservation, retained native session cleanup, idle sleep safety, explicit native continuation, task retry, workspace recovery, native failed and cancelled workspace preservation, and Daytona close in the plugin worker manager. - `pnpm --filter @paperclipai/server typecheck` exits 0. `pnpm check:module-boundaries` exits 0. - New tests: - Daytona: a recoverable errored sandbox is recovered once (`recover(20)`) after the retention check and before Stop, then stopped. An unrecoverable one returns `resource_unrecoverable` with no Recover, Stop, start, delete or archive call. A failed recovery makes exactly one Recover call for each outcome (`recovery_failed`, `resource_unrecoverable`, `recovery_incomplete`). A Recover call that never settles is cut off by the bound. A recovered sandbox that fails again before Stop gives `recovery_failed` with no second Recover. - SDK: the envelope holds only the reason. A forged or non-SDK error gives no data. Unknown reasons, accessor fields and a sender-supplied `terminal` flag are not trusted. The worker host forwards the typed failure and drops a plain error's detail. - Server: terminal reasons escalate on attempt 1 with the long retry and one operator warning. `recovery_incomplete` keeps the short retry. The lease stays `pending_cleanup`, and the provider message never reaches the logs. - Not run: the full `pnpm -r typecheck`, `pnpm test:run`, `pnpm build` and `pnpm run typecheck:build-gaps` repo checks. CI runs them. No live Daytona probe ran. The mocks model the Daytona SDK `recover()` contract: it waits for `started` and rejects when the sandbox returns to `error`. ## Risks - Time budget: the host gives `environmentStopLease` at most 60 s. A recovered stop uses up to 20 s for Recover and up to 30 s for Stop, plus short reads. A slower recovery returns `recovery_incomplete`. The SDK call can continue at Daytona after that, so the next sweep finds the sandbox started and stops it normally. If the host deadline fires while the worker still stops the sandbox, the next sweep sees `stopped` and records the receipt. - Classification: after a failed Recover, the plugin cannot tell a rejected request from a sandbox that started and failed again. Both give `recovery_failed` (terminal) when Daytona still shows `error`. A hung Recover call with a sandbox still in `error` is also terminal. Terminal failures still retry every 30 minutes, and each retry makes at most one Recover call. An explicit Retry also makes one attempt. There is no tight loop. A sandbox that fails Recover every time gets about two Recover calls an hour until an operator acts. - Recover restarts the sandbox container before Stop. The plugin runs no command in it, and the retention policy is confirmed before Recover. - The continuation message for a terminally failed lease still says the message "will start automatically". This PR does not change admission. A follow-up can use the typed reason there. - Compatibility: a host without this change ignores the new error data and keeps the old retry behavior. A plugin without this change sends no typed data. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, code execution for tests and typechecks). The runtime does not expose the context window size to the agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (affected suites; full repo runs are left to CI, as noted above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <[email protected]>	2026-11-04 21:33:48 -08:00
DottaandPaperclip	9e5039f6c8	fix(daytona): retain duplex close ownership after lease cleanup (#19150 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote agents use provider plugins to manage sandbox leases and duplex channels. > - The host verifies the exact route and session when a bound channel closes. > - Daytona lease teardown removed that binding before a later host close. > - The incomplete reply made the host retire the shared worker. A later command then failed during setup. > - This pull request retains bounded close receipts and joins the original cleanup. > - Later work can use the same worker while invalid or unconfirmed closes still fail closed. ## Linked Issues or Issue Description What happened? A normal lease teardown can close a duplex channel before the host closes its bound route. Daytona then replies with only the route ID. The host requires the original session ID too. It logs `duplex channel close not acknowledged; retiring worker`, stops the worker, and rejects a later `environmentExecute` call as stopped. Expected behavior A later close must identify the original route and session and join its existing cleanup. It must not revive the channel or restart the worker. Unknown ownership and an unconfirmed close must retain the existing failure behavior. Steps to reproduce 1. Acquire two leases through one Daytona provider worker. 2. Open a duplex channel on the first lease. 3. Release that lease, then close the bound channel through the host. 4. Execute a command on the second lease. Paperclip version or commit The real host/worker regression fails on `69dea878c0b920b335594debb0286443ba9f6c35`. This branch includes master `3a72efebe5b050e7594bb309e71a0d5f3072db00`. Deployment mode Managed Daytona sandbox execution in Cloud. The regression uses a local real host and SDK worker. Only the external Daytona SDK and PTY are simulated. Related work: Refs #19004, Refs #10429, Refs #18065, Refs #19050. These cover task Retry, managed worker startup, drain release, and stale provider sessions. ## What Changed - Retain at most 4096 retired route/session bindings with their original close promises. - Remove live channel authority before cleanup starts. Concurrent and later closes join the same promise and return the stored binding. - Reject reuse of a retained route ID. Keep unknown and evicted routes unable to provide a session acknowledgement. - Add a real child-process host/SDK-worker/Daytona regression with only the external Daytona SDK and PTY simulated. - Cover release, cancellation, stop, destroy, resume, a stalled close, concurrent closes, failed cleanup, forged IDs and bounded eviction. - Document close receipt scope and its separate provider-termination checks. ## Verification - RED before the repair: the actual host and SDK worker reproduced the retirement log, SIGKILL, stopped state, and later `environmentExecute` rejection. The direct provider cases also failed because the session ID was missing. - Combined-branch provider suite: 364 passed. - Combined-branch host manager, duplex, managed readiness and composed suites: 142 passed. The final test-only manifest-loading change passed all 6 composed cases again. - Full repository typecheck (`node scripts/typecheck-low-memory.mjs`), focused server types and the standalone Daytona build passed. The provider build uses pinned SDK `0.203.0` and TypeScript `5.9.3`. - The composed cases check the same worker/PID, one later command, preserved saved content, revoked channel writes/stops, and continued rejection of commands on closed leases. A stalled close still retires the worker and dispatches no later command. - Reproduce with `pnpm --dir server exec vitest run --config vitest.config.ts src/__tests__/plugin-worker-manager-daytona-close.test.ts` from a checkout with the SDK build dependencies prepared. Run the provider suite with its own Vitest config. - An initial local typecheck lacked the excluded providers' SDK links after an install with scripts disabled. The standard root postinstall linker restored those local links, and full typechecking then passed. The fixture loads its actual built manifest. Failed setup output is retained; no type configuration was relaxed. - The first two composed attempts stopped during fixture setup. They are not the behavioral RED above. Their failed outputs are retained. - The first broad local typecheck stopped at an IPC permission error during preflight. It passed after rerunning with allowed local IPC and the standard SDK links. - [Full repository CI](https://github.com/paperclipai/paperclip/actions/runs/41945022881) passed all 48 jobs on this head. Selection fell back to the complete task plan, so the run was unfiltered. The complete check inventory has 54 successes and two expected Storybook skips; the separate Snyk status passed. Fresh Greptile review on `daebbeb5005b9d9bd59d32ea30094279048b9213` is [5/5 with no actionable comments](https://github.com/paperclipai/paperclip/pull/19150#issuecomment-6646621914). Independent production and fixture review found no actionable findings. Hosted acceptance of this repair has not run. ## Risks - A receipt removed at the 4096-entry bound cannot confirm an old bound close. The existing host guard remains effective in that case. - Close acknowledgements identify channels. They do not certify remote process termination or clear unknown external work. Provider termination receipts and existing recovery gates remain in force. - Hosted evidence confirms the unacknowledged-close retirement class. The historical log does not distinguish a missing session ID from a timeout or another close error. The regression proves and repairs the missing-binding path; it also preserves timeout retirement. - The provider is outside the main pnpm workspace. A new Cloud image must compile its changed source. The unchanged plugin version alone does not prove activation. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, source editing and tool execution. The exact model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-11-04 19:15:50 -06:00
DottaandClaude Opus 5.5	367aa16800	fix(daytona): drop the stale session id on every sandbox restart (#19050 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs can execute inside a Daytona sandbox through the Daytona sandbox provider plugin > - The plugin runs every command for a lease in one persistent Daytona process session and keeps that session id in memory > - Daytona auto-stops an idle sandbox, and a stopped sandbox loses all of its process sessions > - Only the execute and resume hooks dropped the stored session id before a restart. Workspace realize, sync-in, sync-out, runner ingress, and interactive setup also restart a stopped sandbox, but they kept the dead id > - When host-side workspace preparation stays idle for longer than the auto-stop interval, sync-in restarts the sandbox, and the next command sends the dead id. The native run then fails with `native_session_interrupted: session not found` > - This pull request moves the session-id clear into the one shared start helper, so every restart path drops the stale id > - The benefit is that a run that waits through a long preparation opens a fresh session instead of failing ## Linked Issues or Issue Description What happened? Native runs on Daytona failed with `native_session_interrupted` and the error `session not found`. The failure came about 4 seconds after the provider session started. In every observed run, the workspace pack phase took longer than 15 minutes (917 s to 1442 s). This is longer than the default Daytona `autoStopInterval` of 15 minutes. Expected behavior After Daytona restarts an auto-stopped sandbox, the next command opens a new process session and the run continues. Steps to reproduce 1. Run one command on a Daytona lease, so the plugin opens and stores a session. 2. Let the sandbox stop (Daytona auto-stop after an idle gap). 3. Call workspace realize or sync-in on the lease. The hook restarts the sandbox. 4. Run the next command. It uses the stored session id, and Daytona answers `session not found`. No related open PR exists. The undeployed native fixes on master (#18992, #19025, #19026, #19004) do not touch the Daytona session store. ## What Changed - `ensureSandboxStarted` in the Daytona plugin now takes the lease scope. When the sandbox is not running, it clears the stored session id before it starts or recovers the sandbox. - All seven callers (execute, resume, realize workspace, interactive setup, runner ingress, sync-in, sync-out) pass their lease scope. The compiler enforces the scope on any new caller. - The execute hook no longer has its own copy of the clear. The resume hook keeps its clear because it also closes duplex channels before the restart. - Added three regression tests: a fresh session after realize restarts an auto-stopped sandbox, a fresh session after sync-in restarts it, and no new session when realize finds the sandbox already running. ## Verification - Red before the fix: `pnpm exec vitest run src/plugin.test.ts -t "auto-stopped sandbox\|already running"` in `packages/plugins/sandbox-providers/daytona`. The two restart tests failed with `expected "vi.fn()" to be called 2 times, but got 1 times`. The running-sandbox safety test passed. - Green after the fix: the full Daytona plugin suite (`pnpm exec vitest run`) passes, 542 passed and 8 skipped. - `pnpm typecheck` in the Daytona plugin package passes. - Not done: a live Daytona run that idles past the auto-stop interval. The fix needs deployment before the staging failure can be checked again. ## Risks - Low risk. The clear runs only when the sandbox is not in the `started` state. A stopped sandbox has no live session, so no live shell can leak. A running sandbox keeps its stored session, and a test covers this case. - The recover path for a recoverable `error` state now also drops the id. A recovered sandbox restarts, so its old session is also gone. - Duplex channels on a restarted sandbox are still closed only by the resume hook. This PR does not change that behavior. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging run records). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>	2026-11-04 18:24:41 -06:00
Nicky LeachandClaude Opus 5.5	be20d4ad97	fix(projects): keep projects on the storage chosen at creation (#19036 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A project stores its work in Core/local storage or in a storage plugin > - Today, Core finds the installed default each time it reads a project > - So installing a default storage plugin moves existing projects, without a saved change > - Agents that cannot reach the new storage then lose access to new workspaces > - This pull request makes Core apply the default once, when it creates a project > - The benefit is that a project stays on its storage until a person moves it ## Linked Issues or Issue Description No issue exists. This describes the problem. I searched open pull requests for related work and found no duplicate. What existing behavior does this improve? How Core chooses the storage for a project when a plugin claims `defaultForProjects`. Current behavior Installing such a plugin changes the storage of every existing project with no explicit choice. Disabling the plugin changes it back. Nothing is saved, so the project has no fixed owner. Execution environments and storage are separate contracts, so an agent can lose access to new workspaces on a project that it used before. Proposed behavior Core saves the storage choice on the project when it creates the project. A project with no saved choice uses Core/local storage. A plugin-backed project stays attached to its plugin. A Core/local project can move to a plugin. Reason and benefit A project stays on its storage until a person moves it, so installing or disabling a plugin does not change where work goes. Alternatives considered - Keep the default at read time and add a process that saves a choice on existing projects when a plugin arrives. This adds a background step and a race with new projects. - Add a new database column. The existing policy field already holds the choice, so a migration is not necessary. ## What Changed - `resolveProjectWorkspaceProvider` reads only the saved choice. It no longer applies an installed default. - `withDefaultProjectStorage` applies the default when Core creates a project, including a personal private project. With more than one default, a project with no explicit choice is refused. This is the existing conflict rule, now applied at creation. - `projectService.update` refuses a change of storage on a plugin-backed project. An update that omits the field keeps the attachment. - Plugin lifecycle delivery sends project events to a storage provider only for projects attached to it, and for deleted projects. Core acknowledges the other events for that provider. Other plugins continue to receive all project events. - The project settings control has no "Inherit" option. It is locked on a plugin-backed project. - `ProjectWorkspaceProviderConfiguration.source` no longer has the `"default"` value. - A project with no repository or folder ignores its saved storage choice when a task runs. A storage plugin cannot manage this type of project, so its tasks use a plain task directory. Without this change, the second run of a task reset the agent session, and remote agents could not use an empty task directory. - The plugin SDK README and the plugin authoring guide describe the new rules. ## Verification - `npx vitest run server/src/__tests__/project-workspace-provider.integration.test.ts server/src/__tests__/plugin-lifecycle-delivery.test.ts server/src/__tests__/plugin-workspaces.integration.test.ts packages/plugins/sdk/tests/workspace-provider.test.ts`: 93 tests pass. - `npx vitest run src/components/ProjectProperties.concurrency.test.tsx` in `ui/`: 9 tests pass. - `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter @paperclipai/ui typecheck` pass. - `npx vitest run` on `heartbeat-session-preparation`, `workspace-preparation`, `remote-workspace-source`, `project-workspace-provider.integration`, and `plugin-workspaces.integration` in `server/`: 231 tests pass. - I did not run the full test suite locally. CI runs it. - I did not test this change on a running instance. ## Risks - Behavior change at upgrade: an existing project that uses a plugin only through the installed default, with no saved choice, moves to Core/local storage after upgrade. Its plugin files stay in place, but new work does not use them until an operator sets `storageProviderPluginId` on the project. This pull request has no automatic backfill. - Missed lifecycle events: a provider does not receive events for a Core/local project. If the project moves to that provider later, the update event starts its reconcile. - One-way move: a person cannot move a plugin-backed project back through the API or the UI. This is intended. If a plugin is disabled, the project fails closed, as it does today for an explicit choice. - This pull request does not stop a person from assigning an agent to a project with storage that the agent cannot reach. That is follow-up work. - This changes core plugin behavior, and I did not discuss it in `#dev` first. Maintainers can tell me if they want a discussion before review. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>	2026-11-04 13:07:29 -08:00
Devin FoleyandPaperclip	cf0ab4f7f4	chore: enable boxd plugin release publishing (#19020 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Sandbox provider plugins let those agents run in remote environments. > - The boxd provider exists, but its release entry excludes it from CI publishing. > - npm now has the `@paperclipai/[email protected]` bootstrap placeholder. > - This pull request enables CI publishing and documents npm installation. > - A pre-existing OpenCode fixture blocked CI because it qualified a group-writable host Node binary. > - This also gives that test its own Linux Node copy with strict permissions. > - Operators can install a real release after CI publishes it. ## Linked Issues or Issue Description Related implementation: https://github.com/paperclipai/paperclip/pull/17284. No matching public issue or duplicate release-enrollment PR was found. What existing behavior does this improve? Publishing and installing the existing boxd sandbox provider. Subsystem affected `packages/plugins` and the npm release manifest. Current behavior `publishFromCi` is false. npm has only the `0.0.0` placeholder. The README says the package is not published and requires a local-path install. Proposed behavior Include boxd in CI releases. Document stable installation, explicit published versions, and the placeholder limit. Reason and benefit Operators can install the provider through the existing npm plugin installer after a real release ships. Breaking changes None. This changes release enrollment and documentation only. Additional context This follows the existing plugin and sandbox roadmap items. It adds no new runtime capability. ## What Changed - Set `publishFromCi: true` for `@paperclipai/plugin-boxd` in the release manifest. - Replace the README's unpublished-package notice with npm installation guidance. - Explain that the `0.0.0` placeholder has no plugin functionality and that `latest` can point to it until the first stable release. - Keep the local development instructions available. - Fix the existing ingress-retry test to use an owned Linux Node copy and explicit executable permissions. This follows the existing OpenCode transport fixture. It does not change production launch checks. ## Verification - Passed: `node --test scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/bootstrap-npm-package.test.mjs` (31 tests). - Passed: `node scripts/release-package-map.mjs check`. - Passed: `PAPERCLIP_RELEASE_BOOTSTRAP_BASE_SHA=origin/master node scripts/check-release-package-bootstrap.mjs scripts/release-package-manifest.json packages/plugins/sandbox-providers/boxd/README.md`. - Passed: `git diff --check`. - Confirmed that npm resolves `@paperclipai/plugin-boxd` to version `0.0.0`. - Full workspace typecheck, test suite, and build were not run locally. CI checks remain required before merge. - Initial CI failure: [native Runner job](https://github.com/paperclipai/paperclip/actions/runs/41889969053/job/126785636239), OpenCode ingress retry, `qualified OpenCode proxy command must not be group- or world-writable`. The failing fixture is byte-identical to the base commit `3a2249dd576`. It passed `process.execPath` to qualification, while other OpenCode tests already use a private Linux Node copy. - Local focused run: `pnpm exec vitest run server/src/services/native-runtime/native-ingress-bootstrap-retry.integration.test.ts` built the real Rust Runner and passed the Codex case. The OpenCode case stopped at `runner_remote_provider_artifact_incompatible: OpenCode proxy path belongs to the controller host` on macOS (`/Users/...`). This does not exercise the new Linux-only copy. - An unchanged-base OpenCode comparison was attempted with `-t "reacquires opencode ingress"`, but PostgreSQL setup failed with `ENOSPC` before the case ran. The local baseline comparison is therefore inconclusive. Task-owned Rust build outputs were cleaned to recover disk space. - At this stage Linux verification was pending. The successful final run is recorded below. - No new tests were needed. The existing release-tool tests cover manifest enrollment, dependency ordering, and bootstrap checks. - Current-head CI interruption: [workspaces-a shard 1/2](https://github.com/paperclipai/paperclip/actions/runs/41893808561/job/126799012853) was cancelled after the runner received a shutdown signal. The log shows no assertion failure. The job rerun request was rejected with HTTP 403 because the workflow was still running. The subsequent fixture correction starts a new full CI run and supersedes that pending rerun. - CI at `7fffa315`: [native Runner job](https://github.com/paperclipai/paperclip/actions/runs/41893808561/job/126799011711) passed executable permission checks, then failed because Node tried to resolve `/memfd:paperclip-verified-launch (deleted)`. The private executable was named `opencode-node`, which bypassed Runner's Node interpreter recognition. The fixture now names its copy `node`, matching the existing transport fixture and selecting the CommonJS descriptor loader. - Passed a local smoke test of the production CommonJS descriptor loader using an open, unlinked script file. This is macOS evidence only; the successful final Linux run below provides the complete ingress retry evidence. - Final verification on `bcacac542aa0a1b641b26c099c1e9fc4a1653ec4`: all 54 reported checks passed; three unrelated checks were skipped. [Native Runner integration job](https://github.com/paperclipai/paperclip/actions/runs/41895787712/job/126805844773) passed, including the OpenCode ingress retry regression. The fresh full run also passed the workspace shard that was interrupted on the previous attempt. Greptile reports 5/5 for this exact head with no unresolved review threads. Earlier failures and local verification limits remain recorded above. ## Risks - The maintainer confirmed that npm trusted publishing is configured for this package. The first real CI publish will verify OIDC authentication. - Merging enables publication in the next successful canary run. An installation that uses `latest` can still resolve to the placeholder until the first stable release includes this change. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`, with xhigh reasoning effort, repository tool use, and shell execution. The runtime does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-11-04 11:36:04 -08:00
DottaandPaperclip	3a2249dd57	fix(runner): retry proven ingress bootstrap failures (#18981 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution must retain its admitted model and execution authority during recovery. > - A sandbox provider can reject Runner endpoint acquisition before the Runner gets a credential. > - The old error path loses the structured HTTP status and leaves that startup failure ambiguous. > - A zero counter in an old checkpoint cannot prove that no credential was issued. > - This pull request adds a versioned producer proof and carries closed provider status through the existing RPC. > - A proved transient startup failure uses the existing retry budget. A terminal proved-unused run can settle its accounting. ## Linked Issues or Issue Description Refs #18018 and #18934. Those changes restore existing checkpoints and ready retained environments. This change covers endpoint acquisition before Runner dispatch. A search of related public PRs found no duplicate implementation. What happened? A provider HTTP 502 during Runner endpoint acquisition becomes an untyped worker error. Native recovery treats it as ambiguous and stops without retrying. The run's native usage receipt also stays pending after startup ends without model work. Expected behavior Retry a structured transient endpoint failure only when the complete current controller checkpoint proves that this exact run never issued a Runner credential. Preserve authentication, configuration, Stop, ownership and provider-progress gates. Settle zero usage only under the existing terminal accounting owner. Steps to reproduce 1. Use the controlled endpoint in the Daytona ingress integration test to return HTTP 502. 2. Register the actual native Runner driver and persist its bootstrap journal through the encrypted checkpoint writer. 3. Observe the old terminal ambiguous disposition instead of bounded startup recovery. Paperclip version or commit Reproduced on the pre-change master source. The original regression used the production native driver, a real HTTP fault, a real durable control plane and local Postgres. The new contract additionally tests the pinned Daytona SDK and real worker RPC. ## What Changed - The controller owns a v2 persisted state with retained origin and credential history. Current readers still accept v1. Older writers reject v2. Restore and checkpoint decoding do not add missing history. - The proof validates the complete bootstrap journal, all six identity fields and every delivery, credential and progress control. The server rechecks its fingerprint under the current controller claim before retry or accounting changes. - Daytona and the SDK carry a closed acquisition phase and HTTP status through the existing ingress RPC. Both public error wrappers omit response bodies and causes, including Pino cause serialization. - Only structured HTTP 429 or 5xx ingress failures can use the existing three-attempt startup budget. HTTP 401/403, configuration and unknown errors do not gain retry authority. - A terminal proved-unused run wakes existing accounting and records zero provider work. It does not create a semantic result, prove physical termination or release an unknown process/workspace hold. - The normal automatic second attempt rechecks its exact claim, frozen execution and matching complete local/encrypted journal before admitting absent remote Runner state. It keeps the original pending bootstrap command IDs. A proved unused Codex home can rematerialize sealed skills. Full raw local state is checked before streaming-event or policy projection. - Normal saved-state readers and the existing E2E journal evidence matcher accept the closed v1/v2 set. The separate legacy no-launch witness is unchanged. Technical docs describe the contracts. ## Verification Current head: `cd38f70409cdb8664e14bfbf4efa208f468a0419` (final delta adds only fixture artifact preparation). Production files are unchanged from the independently reviewed `93c8a33520` source. Full repository typecheck and build passed at `96417acb58`; the affected server typecheck also passes after the final two-line fixture delta. Focused protocol, raw-proof, real retry, SDK and RPC acceptance pass on this source. Earlier broad suite counts are listed with their limits below: - Durable Runner protocol: 187 tests. - Native executor, driver, startup/accounting and redaction: 1,211 tests, including real Postgres and encrypted checkpoint persistence. - Normal saved-state restore/projection compatibility: 21 focused cases. - SDK: 209 tests. - Standalone Daytona: 544 tests; 8 opt-in live tests skipped. - Explicit standalone `pnpm -C packages/plugins/sandbox-providers/daytona exec vitest run --config vitest.sdk-ingress.config.ts`: 6 cases with real pinned SDK 0.203.0 and worker RPC ingress acceptance. These include lookup 502/401/403/400, private-preview 502 after successful lookup/refresh, missing credentials and private-response exclusion. - E2E journal evidence: 4 cases across both supported formats, with wrong ownership, missing stimulus and unknown-schema controls. - The exact historical controller source rejects a new v2 checkpoint at decode and restore. A v1 control is accepted. This verifies the downgrade fence with the actual older parser. - Startup controls cover issued and pruned credentials, old/truncated journals, provider progress, exhausted retry accounting, and checkpoint/Stop/controller-claim races. Race cases retain the current execution and pending usage. - New exact admission controls: 19 cases through encrypted persistence and cold restore. They cover current/stale/expired claims, Stop, pause, model/trust/account changes, prior progress, issued/pruned credentials, adopted/old/truncated state, local/canonical fingerprint mismatch, hidden delivery, unsupported work and company scope. Two additional red/green cases prove that raw local stream progress and missing event policy are rejected before lossy checkpoint projection. No case mutates run, claim, result or accounting state. - Actual native executor, scheduler and Rust Runner retry: 2 cases. Codex uses its fake app server. OpenCode uses the real proxy and fake OpenCode HTTP server with the frozen DeepSeek model and allow permission configuration. Each path launches no Runner on attempt 1, starts one Runner and provider on attempt 2, reuses both original command IDs, commits one accepted result, validates the persisted local workspace, records nonzero synthetic native usage and settles accounting once. No zero-work accounting flag survives. The controlled SDK and protocol tests use local source, isolated PostgreSQL, a private HTTPS preview and fake provider servers on macOS arm64 with Node 24.22.2. They create no hosted sandbox and make no model call. The root workspace intentionally omits standalone provider dependencies; the explicit SDK target fails if its pinned dependency is absent and has no conditional skip. They do not qualify hosted rollout or repair an old unmarked checkpoint. The first full local typecheck found missing checkout SDK links from an install with scripts disabled; the repository link helper repaired that setup. Current-head Linux CI typecheck and build pass. Independent final review checked 12 source scopes and six raw-proof controls with no remaining finding. Fresh Greptile review is 5/5 with no unresolved thread. The full Runner CI suite found two older control-peer test doubles missing the new proof method. Both now return no proof, with all 27 permission, ordering and cancellation assertions retained and passing. The new Cargo-dependent native retry suite uses the existing Rust-cached CI lane. All 39 lane coverage controls pass and prove complete, duplicate-free suite coverage. The first full test run stopped at an admitted chat Retry polling failure. CI also found a stale retained-allocation fixture, now repaired on master by #18984, and a canceled runner job. A real SDK suite initially relied on a dependency absent from the root workspace; the explicit standalone target now owns that acceptance. The separate Product E2E support run had 12 failures in four untouched files. All 12 reproduce on the original clean base with independent dependencies, compiled base source and verified workspace resolution. These concern workflow expectations, cleanup timeout arguments, a profile snapshot and operating receipts. They are not passing acceptance. The later full local repository test command stopped at a 15-second timeout in the unchanged GitHub pre-allocation retry `controller` case. That exact case passed in isolation with unchanged assertions. The broad Runner run initially omitted `test:typescript:prep` and lacked debug fake-provider binaries. Required preparation then passed 370 + 44 package contract cases (11 skipped); the prepared full package run reported 4,932 passed, 9 failed and 13 skipped. All failed families then passed with unchanged assertions in isolation: 8 ACPX controller tests and 2 credential controls. The paired original-base ACPX control also passed. The full-run failures remain recorded; they are not passing full-suite evidence. The first Rust-cached native CI run found a missing generated OpenCode proxy in a fresh checkout. The fixture now runs the existing verified provider-entrypoint builder. Removing the generated proxy reproduces the exact ENOENT before the fix; both real retry cases pass after the fix from the same absent-artifact state. All current-head CI gates pass, including the aggregate verification check. Linux native CI passes all 10 suites: 177 tests pass and 14 opt-in cases skip. Both real provider retry cases pass. The full Runner package passes 297 suites and 4,950 tests, with 1 suite and 10 tests skipped. The CLI workspace groups pass 4,762 and 286 tests. The first CI attempt had Runner shutdowns in the Runner package, Canary and three general server jobs. One unchanged CLI shard also hit child-server startup and dense ZIP deadlines. Both CLI files passed unchanged in isolation: 18 tests. One bounded rerun of unsuccessful jobs and their dependents passed. The original failures remain recorded. ## Risks - Older controllers reject newly written v2 state on downgrade. Current controllers retain v1 restore support, but old or unmarked state cannot gain this new proof. - An older plugin that omits structured ingress evidence remains held. The server and provider must use the new producer contract for this recovery path. The Docker `cloud` target builds the SDK, server/Runner and standalone Daytona producer from one source tree. The default `production` target does not build the standalone producer. Hosted rollout must confirm the new image and active image-owned plugin worker; a registry version alone does not prove its source. An npm-installed producer needs an explicit matching artifact upgrade. - This prevents future proved startup failures. It does not release historical accounting based on an old checkpoint, change retry budgets, replace a model or weaken sandbox isolation. - Zero model usage is separate from process termination, saved files and resource release. Those owners keep their existing gates. ## Model Used OpenAI Codex with tool use and code execution. This runtime does not expose the exact backend model ID or context window to the agent. Read-only independent agents reviewed checkpoint compatibility and SDK/RPC error boundaries. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (affected suites; full local run limits are recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-11-04 12:16:51 -06:00
DottaandPaperclip	86814c6550	fix(recovery): ready retained sandboxes before preparation (#18934 ) ## Thinking Path > - Paperclip manages work performed by AI agents. > - Restart recovery must preserve each run's workspace and execution owner. > - A retained remote lease can refer to an archived sandbox. > - The reattachment path returned that lease without making it ready. > - A short project-authentication command then had to start the sandbox. > - This PR resumes and verifies the exact retained allocation before preparation. > - Recovery keeps the recorded workspace and preserves saved work if resume fails. ## Linked Issues or Issue Description What happened? A native run resumed after a controller restart. Its saved sandbox had been archived. Reattachment returned its lease without a provider lifecycle resume. The next project-authentication probe tried to start that sandbox with its 15-second command budget. The start failed before the probe could inspect project settings. The heartbeat reported a configuration conflict and stopped work. What did you expect to happen? Make the exact retained sandbox ready through the existing bounded lifecycle operation. Verify its allocation before running short preparation commands. Keep the validated lease and recorded working directory. Never acquire a replacement in this path. Steps to reproduce Hold the provider's exact-lease resume pending and call remote reattachment for an active ephemeral lease. Before this change, acquisition returns immediately. The added tests hold acquisition until resume completes and reject missing, changed or expired allocation evidence. Provider compatibility cases retain the recorded workspace when resume omits directory metadata or returns the allocation root. Deployment mode Native Runner in a sandbox with a retained workspace and managed AI connection. The observed provider was Daytona. Refs #18910, which restores selection of retries using current remote workspace storage. This change repairs the next preparation boundary. Related #17421 and #18559 change project-file inspection; they do not make retained sandbox leases ready. ## What Changed - Resume the exact validated sandbox lease before returning it to preparation. - Require the receipt to confirm the original provider allocation without reporting expiration. - Keep the original database lease and recorded directory. Resume metadata can describe the allocation root. - Pass the existing typed resume-directory input through the runtime facade. - Preserve retained E2B sandboxes when resume timeout setup or workspace inspection fails. Keep new-allocation cleanup and explicit release/destroy behavior. - Preserve plugin-task reattachment and lease identity checks. - Update the Stop-accounting fixture to confirm exact-allocation resume before cleanup, with its budget, ownership and no-paid-work assertions intact. - Document the ordering and add readiness, receipt, failure and ownership regressions. ## Verification - Red: seven initial retained-lease cases failed before the readiness change. Two receipt-compatibility cases and four E2B preservation cases failed before their fixes. - Green: all four runtime, workspace-preparation, heartbeat-recovery and orchestrator suites pass: 442 tests. - The Stop-accounting CI failure reproduced locally. All five integration cases pass with the exact-allocation resume fixture. - All 22 E2B tests, the standalone plugin typecheck and its build pass. - 13 Daytona safety cases pass, including recorded directories, allocation sentinel rejection, saved-file reads and no replacement creation. The SDK boundary is simulated; the saved-file case uses real files and a POSIX shell. - These focused checks cover 482 passing cases. - Full recursive typecheck, final server typecheck and full workspace build pass on macOS arm64 with Node 24.22.2 and PostgreSQL 18.1. - Greptile is 5/5 on head `32f915a475275680bcf2a2aa43ae0c72f4e4b1db`. Both original review threads are resolved. - All current-head CI gates pass. The check rollup has 54 successful checks, three skipped checks and one neutral check. - The separate Paperclip Review check remains queued. Its public response says work is paused and the request is saved. The review task is waiting for saved-file recovery, so no Paperclip assessment of this head is claimed. The branch-required gate has passed. - A read-only deployed trace showed an archived sandbox start fail at the short probe's 15-second deadline. Actual provider reattachment with this revision has not been exercised. Live recovery remains unverified until rollout. ## Risks Recovery now makes one bounded lifecycle call before short preparation commands. A missing resume capability or a missing, changed or expired allocation receipt holds recovery. The existing runtime and provider retain allocation and sentinel verification. The path does not acquire, release, replace or reseed the lease. Warm providers can return their existing allocation. E2B retains the existing sandbox if resume setup fails; explicit lifecycle cleanup still owns its removal. No background polling, credential-policy change, migration, workflow change or lockfile change is added. ## Model Used OpenAI GPT-6 through Codex, with reasoning, source inspection, tool use and code execution. The exact backend snapshot and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-11-04 09:04:13 -06:00
DottaandPaperclip	84aa989bf9	Preserve native sessions through legacy workspace enrollment (#18943 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native session retains its workspace and provider session across turns. > - An ephemeral web server must restore those files without its predecessor's disk. > - Legacy enrollment changes the workspace authority and can change both workspace addresses. > - The retained provider backup checks those addresses as part of its identity. > - This pull request preserves the exact addresses in a verified migration receipt. > - The benefit is that the next turn can use the original session after enrollment. ## Linked Issues or Issue Description Refs #18932, merged as `612d9931a6`. This pull request builds on that all-Runner durability change. Related: #18918 fixes concurrent workspace preservation locks. This change covers identity preservation during offline enrollment. ## What Changed - The native workspace migration owner records the selected session bindings in the successful workspace migration operation. - Runtime selection retains the original logical address only for the exact agent, task and session. - Remote workspace preparation restores the original provider address. It refuses conflicting targets, forged receipts and occupied unowned roots. - The environment owner includes the migration operation in the reusable lease fingerprint. Warm policy stays warm and receives a fresh allocation through normal lease cleanup. - Exact migration evidence permits the existing verified provider backup to restore into that fresh warm allocation. - Preserve verified provider allocation files through a normalized, lease-bound digest receipt. Unknown files and altered markers remain refused; interrupted publication is retryable. - Retain legacy managed-instruction session fingerprints using the exact imported content hash and recorded paths, preserving all other configuration and credential checks. - Migrate existing local skill archive/manifest objects through upload claims, readback verification and receipt compare-and-swap; retain original objects and IDs. - Add explicit two-turn browser qualification for every released native Daytona profile, with warm and per-turn variants. The production maintenance entry runs offline. The old home is deleted only after a verified export and successor readiness. The oracle checks provider identity, native session, addresses, reference bytes, question and final document. ## Verification - Full repository typecheck passed at `b12ecd823e`. The production session fix passed 198 session/credential tests. Full repository build passed at `94bb917947`. The final Rust-only provider fix passed 57 ACPX provider tests and the exact Linux release build. - Workspace and transport checks passed: 117 transport/workspace tests, six final real PostgreSQL migration boundary cases, 178 environment tests, 539 Daytona tests (eight skips), and the native warm-backup admission control. - Local skill migration passed eight PostgreSQL/S3 tests (one existing opt-in skip). Failed readback retains the source. Successful readback still works after deletion of the old home. - Latest Product E2E typecheck and 38 focused harness tests pass. Earlier catalog/admission/report checks passed. Twelve unrelated support-test failures reproduce on unchanged base `1a39dea9aa`; these are not reported as passing. - Live Codex per-turn at `2d5bb9a6d1` retained the native and provider sessions, both workspace paths and exact reference bytes after deletion of the old home. It completed the task and delivered its saved document and original reference. Its original failure was an output-count grading defect. The corrected grader passes all 21 checks against retained, hashed evidence; the original result remains unchanged. - Live OpenCode per-turn and Codex warm at harness `94bb917947` pass all 21 product checks. Their launcher cleanup failed because a read-only diagnostic observer kept its disposable database open. The observer now closes before ordinary database cleanup; a no-provider reproduction confirms the fix. All owned sandboxes were independently verified absent, child keys revoked and buckets removed. Original failed campaign dispositions remain unchanged. - Claude at `94bb917947` kept its sessions and reference bytes but omitted the final clickable document link. That user-delivery failure remains recorded. The fixture now explicitly asks for the link and records migration evidence before checking navigation. The fresh Claude attempt at `b12ecd823e` passed both strict migration checks: exact provider/native session, both paths, original file bytes and independently downloaded attachment. Its full browser workflow failed again: it requested unnecessary confirmation, used the answer sentence as the reference, and omitted the clickable document link. This is retained as a failed user-workflow result, not a full pass. - Earlier live failures exposed and reproduced the skill-object, fresh-allocation marker and managed-instruction fingerprint bugs. The fixes retain original identity checks. No provider session reset or accounting bypass was used. - The real Grok continuation at `b12ecd823e` failed before provider execution because its immutable-instruction attach policy rejected relocated registered file copies. A no-provider regression reproduced the exact rejection. `b05ba3728a` selects the existing strict authenticated asset-path policy for Grok; content digests, unknown context fields, session/workspace identity and same-run mutations remain enforced. The regression and all 57 ACPX provider controls pass. The Grok browser migration now passes all 21 checks with the b05 Linux Runner, including provider identity, exact reference bytes and final user output; the launcher also reports successful cleanup. - `24fabf1d81` applies the same policy to released ACPX Codex after a second no-provider reproduction. All 58 ACPX provider controls and the final Linux release build pass. Its live browser migration passes all 21 checks on the final binary, including exact original session/file preservation and completed user output. - The full native-session executor suite passes all 1,121 tests after updating its existing lease database fixture for the migration-receipt join. The prior CI failure was this unsupported test-double method; the production guard still rejects ordinary unproven recovery. CI and Greptile 5/5 passed at `ca1e0d5fe6`. The first CI attempt lost several workers and had one rotated-recovery transport failure; its seven descriptor cases passed locally, and the single unchanged retry passed. The original failure remains recorded. Staging enrollment has not started. Initial offline enrollment and subsequent rolling availability are separate checks. - The final review follow-up adds real runtime-selection PostgreSQL cases for exact, different-session and absent receipts, plus executor-level warm restoration that copies actual backup files and verifies provider identity and the new lease stamp. Both complete affected suites pass (1,172 tests), and full repository typecheck passes. Removing either production migration connection independently makes its new positive case fail; both source files were restored byte-for-byte after these negative controls. - Final commit `0971d373a3` passes all 63 checks (57 successes, five expected skips, one neutral); Greptile is 5/5 and every review thread is resolved. ## Risks - An identity conflict blocks enrollment and keeps the original files. It does not silently reset the session. - A new migration generation retires a reusable lease through its existing owner. The original sandbox must finish normal cleanup before reuse proceeds. - Initial enrollment has an offline step. The separate rolling suites prove uninterrupted deploy behavior after enrollment. - This adds no schema migration and changes no provider credentials or accounting gate. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model build and context window are not exposed in this session. The agent used reasoning, terminal tools, code edits, PostgreSQL/filesystem tests and isolated live browser qualification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-11-04 04:43:56 -06:00
DottaandPaperclip	89e3c23062	fix(daytona): stream files when export copies cannot fit (#18712 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native task must save its files before finalization can release its writer. > - Remote workspace capture can produce an archive that fits on the sandbox disk. > - Daytona then makes another copy before it downloads that archive. > - The second copy can run out of space and leave completed work behind a save hold. > - This pull request uses the existing bounded stream when that copy cannot fit. > - The benefit is a saved archive without a second archive-sized sandbox allocation. ## Linked Issues or Issue Description Related: #18644 reduced capture scratch space. #18172 added the disk-free directory export. #18705 preserves provider checkpoints on a full disk and reports capture failures. This change handles the later single-file download copy. It does not depend on #18705 and does not replace its checkpoint fix. What happened? A native provider turn returned a successful result and captured its workspace archive. The Daytona outbound file guard then failed with exit 43 and `No space left on device` while it copied the archive to a reserved download file. Finalization and file-preservation retries could not pass this boundary. The workspace kept its writer hold. Expected behavior Download an existing archive through a confined sandbox process when its second copy cannot fit. Keep the source and all finalization holds until the normal save commits. Steps to reproduce 1. Capture a workspace archive in a Daytona sandbox. 2. Leave less free space than the archive needs for another copy. 3. Finalize the task. The outbound single-file guard fails before download. Paperclip version or commit Reproduced against `6d72243d8a`. The incident used the native sandbox runtime. This branch is based on current master. Deployment mode Cloud with native Paperclip Runner execution and remote workspace storage. ## What Changed - Remove partial outbound file copies after a failed snapshot command. Only copy capacity failures select the stream path. Permission, confinement, and transport errors still fail. - Use the existing acknowledged terminal stream to read each file as the sandbox user. Recheck canonical confinement and regular-file status. Use a no-follow open. Keep the sandbox disk limit, host free-space reserve, shared transfer deadline, and bounded teardown. - Keep host files in private temporary files until the complete batch has transfer receipts. Never pass a mutable live source to the privileged download API. - Add real-file and real-terminal regressions, a disposable Daytona acceptance case, and workspace verification notes. ## Verification - Red before repair: a real 5 MiB copy under a controlled file-size limit failed with exit 43. The same test passes after repair and compares every source and destination byte. - The ten capacity cases pass. They cover producer failure after output, source replacement, stream size limits, permission errors, timeout teardown, cleanup, and a successful retry. A two-file case completes the first stream and fails the second. Both old host files stay intact, both source files survive, and retry saves both files. An isolated early-publication mutation makes that test fail. - The initial complete Daytona package suite passes: 537 tests passed, 8 skipped. After the test-only review follow-up, all 60 focused transfer, stream, and diagnostic tests pass. The package typecheck also passes. - Real Daytona acceptance passes: an owned Ubuntu 24.04 sandbox stops and resumes, a copy to `/dev/full` reports ENOSPC, production export saves the exact archive hash, and a second save after reconnect passes. The test deletes only its disposable sandbox. This is provider-boundary acceptance, not a UI journey. - Assembled finalization acceptance passes with real Postgres and a live Linux x86_64 Daytona sandbox. The production control-plane port accepts one fixture result. The production workspace finalizer encounters the failed copy, saves the files, and commits the task as `done`. A new database client reads the committed result and exact file hash. The workspace has no writer or running save, and there is still one run and one result. No model turn runs. The target resolver is bound to the live provider fixture. Object storage uses the production encryption and local-disk service with an S3 admission label; this does not prove an S3 network transfer or a UI journey. - The live tests used the fixed transfer source with SHA-256 `857b3d8c9ad75170707d95277c17c051ce4582fa79a9c38331d51d2800b36af5`. That source is identical in this PR head. - `pnpm -r typecheck` passes locally. The duplicate full local test run passed all six chat shards before it was stopped during the server lane in favor of completed current-head CI; a duplicate local build was skipped. Full current-head CI passes, including typecheck, build, general and serialized tests, Rust/Runner checks, all eight E2E shards, and the `ci / e2e` and `ci / verify` aggregate gates. - Greptile is 5/5 on the current commit with no unresolved review threads. - `node scripts/check-maps.mjs` and `git diff --check` pass. Map checks prove structure only. ## Risks - The fallback requires the Bash and GNU `dd`/`base64` tools already required by the existing stream path. - Terminal streaming is slower than the normal bulk download. It runs only after a copy capacity failure. - The workspace capture itself still needs sandbox space. This change does not fix a full disk before an archive exists. - The test injects a failed copy sink. It does not exhaust a shared host disk. No original task was retried, and no deployment or merge is part of this PR. - No schema, API permission, writer ownership, or retry-budget change. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, code editing, shell tools, browser diagnostics, and automated tests. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-11-03 06:11:41 -06:00
Nicky LeachandPaperclip	6d72243d8a	fix(runtime): recover task creation after a lost acquire reply (#18692 ) ## Thinking Path > - Paperclip manages agent execution and the resources that execution owns. > - A retained environment task can need a replacement before the next turn. > - Core records the creation attempt before it asks the provider to acquire it. > - A lost reply can leave that journal without the resource ID needed for cleanup. > - This change lets an opted-in provider recover the identity from that recorded attempt. > - Core still requires the existing exact-task termination proof before it releases ownership. ## Linked Issues or Issue Description Refs #18405 and #18658. This is the separate lost-reply follow-up from their recovery audit. Draft status: Both prior review findings are fixed in `a66520c8d4e06e0eec6ccb5446190ace0ca3003f`. Fresh standard Greptile is 5/5 with no unresolved threads. Current-head CI is green (53 successful checks/statuses including Snyk, three expected skips). Apex completed 5/5 on this exact head at 2026-11-03T00:54:59Z (request 6592407130), with no unresolved threads. The review cycle is complete; the PR remains draft for human merge review. What happened? If a replacement task's acquire reply is never received, the creation journal can reach manual reconciliation without a resource ID. Startup cleanup cannot complete, and ordinary Stop cannot release the run owner. Expected behavior When the selected provider supports creation inspection, Core should retain enough binding information to inspect the recorded attempt and continue verified cleanup. An inspected identity must not count as termination proof. Steps to reproduce 1. Use a retained task whose provider supports creation inspection. 2. Lose the replacement acquire reply before Core records its resource ID. 3. Restart cleanup and inspect the existing creation journal. 4. Confirm that the exact task is stopped and its termination receipt is saved before cancellation releases the owner. Paperclip version or commit Current head `a66520c8d4e06e0eec6ccb5446190ace0ca3003f`, based on master `3a7bb4dac62dbd7a34a55deb665c55df310bdf64`. Tested with isolated PostgreSQL fixtures. Deployment mode Local source tests with controlled provider responses. No deployment or live model request. ## What Changed - Save the selected inspection capability and prior opaque binding in the existing creation journal. Acquire through that same worker so a worker replacement cannot change the recorded contract. - Make failed opted-in creations due through the existing cleanup notification and saved retry deadline. - Add optional SDK observation field `creationLease`. Use it only to recover the requested creation's identity and binding, then follow the existing Stop, status, completion and receipt checks. - Keep old entries and providers without the recorded opt-in under their existing reconciliation policy. - Add actual PostgreSQL recovery, cancellation and worker-replacement regressions. Document the provider contract. ## Verification - Current correction: 53 distinct targeted cases pass (20 real-worker IPC, 12 task-recovery PostgreSQL and 21 generic-sandbox cases), with server typechecking. The IPC selection was rerun after the final late-observation ordering refinement; the other tested sources were unchanged. - An assembled live-child IPC and PostgreSQL case proves cleanup and cancellation release the owner despite the original worker never replying and remaining alive. Accepted host work, unrelated acquisitions and late observations remain accounted for. Removing the recovery handoff and restoring the old drain snapshot produce distinct assertion failures. - The unsupported-host control skips all 12 PostgreSQL cases without setup; removing the guard fails the controlled setup. These tests do not constitute live VM/provider qualification. - Initial head `5db6882` passed an exact reference-tree comparison against master plus the reviewed follow-up patch, 97 targeted cases and server typechecking. Those checks predate the final inspection correction. - Previous head `59f87fa` passed 32 targeted cases (11 task-recovery and 21 generic-sandbox cases) and server typechecking. Both `pending` and `needs_reconciliation` observations without `creationLease` preserve ownership and get a future retry deadline. Restoring the previous guard makes both new cases fail. - The reproduced failure was a wrong generic cleanup shape that could resolve its journal while leaving the owner held—not a demonstrated false ownership release. The correction reuses the existing missing-identity guard before the generic decoder and leaves generic sandbox behavior unchanged. - Earlier affected selection: 72 cases passed on the original follow-up base. This overlaps the final selection and is not an additional count of unique final-head tests. - Two isolated negative controls restore the missing task inspection and redirected-worker behaviors. Both fail at the expected recovery and dispatch assertions without changing production source files. - The PostgreSQL cases verify one cleanup lease, both late-result orders, recovery after the old journal deadline, ordinary legacy behavior, retained ownership on missing or mismatched identity, and cancellation acknowledgement after exact termination proof. - The tests use actual database and run-control services with controlled worker callbacks. They do not prove live IPC or production recovery timing. - A compatible provider is needed for the new recovery path. No normal successful-startup database or network call is added. No ordinary TTFT improvement is claimed; the repair removes an indefinite ownership hold for supported lost replies. No full local CI suite was run; GitHub runs the full regression gates. ## Risks The provider must recover the identity for the recorded creation operation and preserve its selected binding. `creationLease` does not authorize a new turn or prove that a process stopped. Older journal entries do not infer a new identity after an upgrade. Unavailable original machine bindings or credentials retain the existing uncertainty behavior; this change does not repair machine or credential loss. No schema migration, deployment or ROM qualification is included. ## Model Used OpenAI Codex, GPT-6 family. Exact model ID and context window are not exposed in this session. Tool-assisted implementation, targeted tests and independent subagent review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-11-02 17:45:07 -08:00
DottaandPaperclip	b9e94b85a6	feat(observability): trace Daytona lease acquisition and resume (#18626 ) ## Thinking Path > - Paperclip manages work performed by AI agents. > - The server prepares a remote execution environment before the agent can respond. > - OpenTelemetry traces show the time spent in each preparation phase. > - A retained Daytona sandbox can take minutes to resume from an archive. > - The provider worker previously exposed no child timings for this operation. > - This pull request adds bounded lease phase spans under the existing worker trace. > - Operators can separate provider startup from lookup, teardown waits, and workspace verification. ## Linked Issues or Issue Description What existing behavior does this improve? Daytona environment acquisition and resume tracing. A long worker invocation currently hides the operation that consumed its time. Related tracing foundation: #10999. No duplicate open PR was found. Subsystem affected The Daytona sandbox provider and the server's existing plugin span validation boundary. Proposed behavior Show lookup, start or recovery, teardown waits, workspace verification, and fresh acquisition setup as timed child spans. Group verification steps beneath one parent. Record the bounded provider state before start, including archived. Reason and benefit An operator can identify which step delayed the first agent response. SDK startup includes its internal readiness wait; this change does not claim to measure provider queue time separately. Breaking changes The opt-in trace schema gains fixed span names and one closed state attribute. Execution, authorization, resource ownership, deadlines, and retry policies keep their existing contracts. ## What Changed - Add lease phase spans through the existing invocation tracer. - Add the fixed span names and state values to host validation. Define the state key in the shared startup attribute contract. - Preserve the exact work result or error if tracing fails. Never repeat work through the tracing wrapper. - Test worker receipts for archived and warm resume, recovery, concurrency, failure, disabled tracing, and cleanup. - Document the waterfall and its attribution limits. ## Verification - Daytona plugin suite: 525 passed, 7 skipped. The expanded tracing suite separately passes all 12 cases. - Host span validation and instrumentation tests: 46 passed, 3 skipped. - Standalone Daytona TypeScript check passes. - Module boundary and whitespace checks pass. - Full recursive typecheck and full build pass. - The full local Vitest attempt at `b007a8e64f` stopped in chat shard 1: 337 passed, 2 failed, 1478 skipped. One retry-notice case exceeded 15 seconds. The next case then reported that fake timers were no longer installed. A focused rerun of all four retry-notice cases passed. This is not a full local suite pass. - Final head `8e1f81523b` passes CI and Greptile 5/5, with no unresolved review threads. The CI chat shards pass despite the preserved local timeout attempt. - The expanded lease tracing suite passes 15 tests; host/shared tracing suites pass 66 tests. Disabling the wait flag makes the new teardown test fail, and restoring it passes. Standalone plugin typecheck passes. - The tests drive the real plugin worker RPC and invocation tracer with a controlled provider fixture. They validate receipts and host parent IDs. They do not establish deployment or Honeycomb ingestion. Inspect a new deployed run to verify the exported waterfall. ## Risks - A new provider phase must be added to the closed host allowlist or it appears as other. - SDK-internal start and readiness work remains one measured operation. - Older workers without withSpan run normally without these new detail spans. - Receipt budgets and clock validation can omit detail from very large or invalid receipts. ## Model Used OpenAI Codex, GPT-6, with repository tools and code execution. The exact runtime model ID and context window were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-11-02 11:32:03 -06:00
Nicky LeachandPaperclip	f8a888a072	feat(runner): define bounded startup lifetime contracts (#18490 ) ## Thinking Path > - Paperclip manages AI agents and their work. > - Native sessions and plugin workers have separate startup lifetimes. > - A caller timeout does not prove that a worker or an accepted host write has stopped. > - Shared startup code needs an exact deadline and an actual settlement contract. > - This PR adds those contracts and tests them before any production caller uses them. > - The dependent integration in #18405 will use the contracts with verified cleanup. > - This split lets reviewers assess the lifetime rules separately from runtime activation. ## Linked Issues or Issue Description Refs #18405. This PR extracts the foundation from that change. The integration will depend on this PR. Problem or motivation A bounded startup must distinguish the caller's wait from actual worker completion. A late worker reply or an accepted host request can outlive the original RPC promise. A shorter timeout must not discard that responsibility or authorize a replacement run. Proposed solution Add one Core execution context, optional Runner startup deadlines, and worker settlement and retirement APIs. Test the contracts with real child-process IPC. Keep production context creation, transport projection, capability negotiation, durable cleanup, and Stop/recovery integration in #18405. Alternatives considered One combined PR mixed the lifetime contract with its activation and cleanup policy. Activating a shorter deadline before the complete transport and cleanup path is unsafe. Splitting only the SDK declarations would not provide a useful behavioral review boundary. Roadmap alignment This supports the existing plugin, sandbox execution, and bounded recovery capabilities. It adds no new product workflow, endpoint, migration, provider or runtime dependency. ## What Changed - Add `NativeExecutionContext`. It retains the selected input, binds one transport target, rejects work after cancellation or expiry, and tracks actual worker settlement. Expiry alone does not complete pending work. - Add worker startup groups. A caller timeout retains the group's responsibility. Retirement revokes obsolete invocation scopes, waits for accepted host handlers, and retires only the verified group's bookkeeping. Each cleanup wait is bounded; an unfinished handler remains pending for a later retry. - Add an optional Runner startup signal and absolute deadline for session open and retained attachment. Clean up abort listeners after synchronous failures. Existing callers keep their default timeout path. - Expose the remaining confirmed controller-lease budget without a new query or renewal timer. - Add optional SDK context and preparation-retirement fields. No production bridge sends or advertises them in this slice. - Keep the required lease mock update with the new `remainingMs()` contract so this foundation typechecks independently. Production activation files, including the native executor and environment task bridge, match the current master base. No production caller constructs `NativeExecutionContext` or passes the new Runner startup option. This PR adds no durable cleanup producer and claims no standalone TTFT improvement. The dependent integration must enable the context, transport and verified cleanup together. ## Verification Initial split verification on `8ebc8a2f9c6c02e90f920356138b3742a4067b28`, based on `9a3f395737e7a92faf99971459979061d423531a`: - From `server`: `../node_modules/.bin/vitest run src/services/native-runtime/native-execution-context.test.ts src/services/native-runtime/native-execution-context-worker.test.ts src/services/native-runtime/native-execution-lease.test.ts` — 17 passed. Includes real child IPC, caller timeout with late reply, worker death, accepted host-handler draining, obsolete cleanup invocation rejection, unrelated work preservation, context scope, expiry, and confirmed lease budget. - From `packages/paperclip-runner`: `../../node_modules/.bin/vitest run src/native-session-runtime.test.ts` — 188 passed. Includes existing callers without the new option and new synchronous-failure/expired-deadline cases for fresh and retained sessions. - Direct TypeScript checks for `packages/plugins/sdk/tsconfig.json` and `packages/paperclip-runner/tsconfig.json` — passed. Built only their shared/eval TypeScript prerequisites. - From `server`: `../node_modules/.bin/tsc --noEmit` — passed. - Checked the partition against the reviewed combined candidate. The twelve selected files match exactly. The thirteenth file carries only the required test-mock update. The production executor, bridge and target wiring remain equal to the base. - `git diff --check`, added-line secret/private-data scan, and added-code-comment scan — passed. No generated files or run artifacts are committed. - The first old-base split passed its focused tests but failed server typechecking because the unchanged executor test mock lacked `remainingMs()`. The original failure is retained. Moving that one-line mock update into this foundation resolved the split dependency; the final current-master checks above pass. - Full local CI suites, Rust builds, deployments and provider calls were not run, as requested. GitHub CI will provide full regression coverage. Current-head GitHub CI, standard review and Apex are complete; exact acceptance is recorded below. Review follow-up `d7635f4d280dab757a3666703737bcda955c9361` fixes three reproduced lifetime edge cases. A retiring startup group rejects new calls and shares one retirement promise; failed bounded retirement remains retryable. Transport identity uses a fixed field tuple rather than object-property order. Finishing a context rejects pending lookup waits without aborting an already-started backend or pretending the worker completed. All three new regressions failed before the fix; the updated foundation selection passes 20 tests, including actual IPC and failed-retirement retry. Server typechecking and added-line checks pass. These replace the earlier 17-case selection, rather than adding 20 unique cases to it. The original failures and passing reruns are retained. Apex follow-up `cb8860bab5ab9d6357a352e11f276be0d0cdb26d` keeps removed worker handles reachable while they own startup calls, accepted host handlers or retirement work. Drain/retirement includes active and retained handles; completion callbacks remove settled handles. Stop and replacement remove only the exact handle they observed. This adds no database/network operation, polling loop or generic shutdown wait. The SDK, Runner, context and transport bridge are unchanged. Seven additional IPC regressions failed on the prior manager and pass after the fix, covering removal/replacement, Stop-all, host-handler settlement, bounded retirement failure/retry, pruning, and replacement ownership. The updated foundation selection passes 27 tests, with server typechecking passing; these supersede the earlier 20-case selection. A separate real-PostgreSQL/child-IPC characterization uses the integration's actual SDK host, storage registry and run-control path with the exact corrected manager bytes. Before the fix, cleanup released run ownership while an accepted host write was blocked, then the late write restored obsolete metadata. After the fix, ownership and cleanup remain pending until the write completes; cleanup then removes the stale metadata before releasing ownership. The original failure, corrected result and source snapshots are preserved. Final head `cb8860bab5ab9d6357a352e11f276be0d0cdb26d` passes GitHub CI (53 successful checks, 2 expected skips, Snyk success), standard Greptile 5/5 and fresh Apex 5/5. All review threads are resolved and the PR is conflict-free. ## Risks The shared Runner helper now observes synchronous operation failures through its asynchronous cleanup path. The affected runtime file's 188 tests pass, but full regression coverage still depends on CI. The new lifetime APIs require honest settlement signals. A future caller must not treat a deadline, worker crash or Stop intent as verified retirement. Production activation remains dependent on the complete integration and its cleanup tests in #18405. This foundation introduces no shorter production startup budget or new cleanup policy by itself. The context's transport projection is process-local. It must not be persisted or used as a general authorization cache. No production caller supplies it in this PR. There is no database migration, new dependency, or deployment requirement for merging this foundation. ## Model Used OpenAI Codex, based on GPT-6, assisted with repository inspection, the split, tests and command execution. The exact serving model ID and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal issue identifier - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes — this description documents the inactive contracts; operational activation documentation remains with #18405 - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on `cb8860bab5` - [x] Current-head standard Greptile and Apex are 5/5 with no unresolved findings - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-11-02 08:22:42 -08:00
Nicky LeachandPaperclip	aab87ce707	Prove workspace delivery before inspecting cleanup candidates (#18493 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Completed tasks leave workspaces that need safe cleanup. > - Providers without ancestry inspection currently need a recorded merged PR before cleanup can start. > - A squash merge changes commit identity, so direct ancestry cannot prove that delivery. > - This change proves delivery from the finalized HEAD and captured destination before storage inspection. > - Fresh storage checks and a second delivery check still protect removal. > - This reuses the existing worker and receipts. It adds no table or timer. ## Linked Issues or Issue Description Refs #18410. Related: #13400 addresses workspace identity during inspection; this change adds destination-aware delivery discovery. ## What Changed - Resolve direct ancestry or discover an associated merged GitHub PR without a recorded work product. Require exact HEAD, destination repository and branch, and a reachable integration commit. - Use the latest settled finalization receipt as a candidate. Reject missing, failed, pending, conflicting, or changed evidence. - Inspect storage only after positive remote proof for providers that disable ancestry inspection. Recheck HEAD, branch, source identity and delivery before removal intent. - Add an optional provider capability for a specific PR recovery hint and an optional prepared destination receipt. Older providers keep their existing protocol. - Wake cleanup when finalization writes settle or gain a workspace binding. Keep remote-only retries for concrete unresolved delivery. Park missing or unsupported candidate identity. - Reuse the source project's explicit GitHub consent for delivery checks through the existing authorization service. Revalidate the bound repository and consent before removal; missing or revoked consent retains storage without switching identities. Public sources remain anonymous. - Update the contracts, documentation, access map, and targeted regressions. ## Verification - Initial-head GitHub CI passed at `5acd4c818be6f1764f8319f42891f9956c357c26`, including full build and typecheck. Standard Greptile is 5/5 with no open threads. The first Apex review scored 3/5 and found startup latency from remote proof lookups and removal retry identity changes across provider capability changes. Both corrections are pushed in `9d1a38df3d`, with the qualified destination-ref length fix. Red regressions reproduced each issue. Latest targeted checks passed: cleanup/worker 26, provider integration 41, SDK contract 8. Head `9d1a38df3d` passed standard and Apex reviews at 5/5 with no open threads, plus all CI including full build/typecheck. Canary Dry Run initially failed on an npm 404 fetching `rollup-4.67.3.tgz`; the exact tarball subsequently returned HTTP 200, and one targeted retry passed without source changes. Aggregate verification also passed. After #18403 merged, master introduced conflicts in shared workspace cleanup and metadata handling. Resolved them in `7700832536`, preserving remote snapshot cleanup and Git delivery-proof cleanup. Targeted merge verification: 93 tests passed in the first run (59 opt-in skips), then all 16 remote lifecycle tests passed with explicit database opt-in; map checks passed. The isolated database is stopped. The merge head received standard review 5/5. Its CI found two failures in unchanged test cleanup paths: native runtime temporary-directory ENOTEMPTY and agent-chat cancellation lock timeout. Those failures remain recorded and require successful final-head verification. The subsequent project-consent change has targeted coverage: authorization36, resolver33 and cleanup19 tests passed, including revoked consent, anonymous public access and source-binding races. The positive consent test failed before the fix. Current head `c772679e05` has green GitHub CI, including full build/typecheck and both previously failing test jobs, plus standard Greptile 5/5 with no open threads. Neither cleanup failure required changes to its test path. Apex also completed at 5/5 on this exact head (check `124651548532`, summary updated 2026-10-31 20:45 UTC), with no open review threads. The PR remains draft. Earlier failures remain recorded above. - Final focused run on the rebased branch: 182 tests passed across six files, covering the resolver, delivery cleanup, provider bridge, finalization notifications, existing workspace service and SDK contracts. The disposable test database was stopped afterward. - Targeted resolver tests: 33 passed. These cover ancestry, discovered squash PRs, wrong HEAD/base/repository, removed integration commits, bounded responses, cancellation and credential isolation. - Targeted cleanup integration tests: 17 passed. Provider bridge integration: 36 passed. SDK contract tests: 7 passed. Finalization notification tests: 9 passed. - A negative control that removed the final delivery check failed the race regression as expected. The check was restored. - Read-only live GitHub probe: the new resolver discovered PR #18410 from HEAD `3f5ade4cd42b707040e9a82fdaed343e3d736a09`, without PR hints. It proved integration commit `6484f96c28bfb11a793ce4be6cc5ac15643a5a02` reachable from observed master `5bc029bf6c1db6647e96acea323f3723fd1d3eba`. GitHub also advertised the exact original HEAD at `refs/pull/18410/head`. - `node scripts/check-maps.mjs` passed. Structural checks do not prove deployment behavior. - Local tests are targeted by user request. Full build, typecheck and test suite run in GitHub CI. A standalone server typecheck could not complete because local Runner build declarations were absent; no changed-file diagnostics were reported. - Initial local execution hit a stalled pnpm wrapper, sandbox socket denial and stale generated SDK declarations. Direct installed binaries, the authorized isolated test database and rebuilt SDK resolved those setup failures. No deployment was performed. ## Risks - Destructive cleanup still requires fresh storage safety and lifecycle checks. Remote proof alone never authorizes removal. A persisted removal intent preserves existing recovery semantics. - Legacy workspaces without an unambiguous destination or completion HEAD remain retained. This does not guess the original default branch. - Private repositories use the explicit standing GitHub consent on the bound source project, with current membership, grant, repository access and trust checks. No separate company token is required when that consent is valid. Missing or revoked consent and unavailable or rate-limited access retain storage; the worker does not borrow a task or agent identity. - A known undelivered commit keeps the existing five-minute retry because external GitHub changes have no guaranteed ingress to this lane. This can keep cleanup scheduled. Missing evidence parks instead. - Providers need the new recovery capability to clean a squash merge whose original HEAD is reachable only through its PR ref. Providers without it remain conservative. - The resolver currently supports GitHub.com. Existing local and other-provider ancestry paths stay in place. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools and parallel code agents. The exact runtime model ID and context-window size were not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-11-02 07:59:17 -08:00
Dotta devinfoley Paperclip	06bc8db75c	fix(runner): install verified API artifacts in execution workspaces (#18494 ) ## Thinking Path > - Paperclip manages AI agents and their work. > - Native agents use authorized API tools to read saved results. > - Copying a large response as model-generated text can lose bytes. > - The execution workspace needs an exact copy of the saved artifact. > - This change copies the authorized bytes without shell API credentials. > - The runtime verifies size and SHA-256 before it reports success. > - A live evaluation checks capture origin, installation and delivered bytes. ## Linked Issues or Issue Description What happened? A native agent lost one byte while copying a 24576-byte API text page. Existing capture and paging controls passed. The runtime had no verified saved-artifact-to-workspace copy tool. A separate historical screening refusal does not establish capture corruption. Expected behavior An authorized native run can install the complete saved artifact. The stored and installed bytes must match in size and SHA-256. A conflicting file must remain unchanged. A lost publication receipt must not cause an unsafe repeated write. Steps to reproduce Run `server/src/services/native-runtime/artifact-transfer.integration.test.ts` on baseline `c46b8f35add9255fc71be294d27f5a7865bcb47f`. Its capability assertion fails because `download_artifact` is absent. Apply this PR and run the same regression with the filesystem and API controls. It verifies 388131 synthetic bytes through the authority and installation boundary. Refs #18429. Related #17393, #17086 and #17285 cover paging, transfer foundations and remote output reads. This PR adds the inbound installation contract. ## What Changed - Add `download_artifact` for supported standard-mode native runs, with a 10 MiB limit. - Reuse current company, task, run, asset-content and responsible-user access checks. Check authority again before publication. - Bind installation to the current execution lease. Stage through confined Linux descriptors and publish with an atomic create-only link. Keep API credentials out of execution commands. - Verify bytes at storage and execution boundaries. Reject unsafe paths, symlinks, different existing files, truncation and corruption. Reconcile an unknown publication through verification only; an identical explicit retry remains safe. - Add API documentation, access-map entries and content-free audit records. - Preserve only the pinned Daytona SDK numeric HTTP status when failed creation cleanup is unconfirmed. Keep provider exception messages and headers private. - Keep only the finished runner binary and portable provider pack in their builder layers. Remove Rust targets/downloads and the Node install workspace/store before export. Report tree sizes. - Add one explicit Codex/Daytona Product E2E cell. Its independent oracle checks the server capture audit, redacted tool receipts, stored source, delivered attachment and browser link after reload. Uploaded substitutes and one-byte loss fail. ## Verification - Merge update at `6766b46553cab5b9787ade87a002eccaf50e68fc`: merge current master and preserve both explicit suites in the catalog. Focused catalog, image and oracle checks: 122 pass. Filesystem and authority integration: 20 pass. Maps, module boundaries and whitespace checks pass. New-head CI and Greptile are running. Full local build and typecheck remain unavailable because cargo is absent. - Behavioral red/green: the absent-tool regression failed before implementation and passed afterward. The initial missing-package attempt is recorded as setup failure. - Linux filesystem and real PostgreSQL/authority integration: 19 filesystem tests and 1 authority test pass. Existing API authority, capture and paging controls: 70 pass, 2 skipped. Provider transport in these integration controls is simulated. - Qualification calibration: 118 focused tests pass. The real persisted-event redaction case and a byte-identical uploaded substitute each failed before their oracle repair. The unchanged controls then passed. Harness typecheck passes. - Full recursive typecheck and build pass locally on the integrated production source at `086f7a2901`. Later commits change only qualification tests and documentation. - Structural map/module checks and whitespace checks pass. - Full local `pnpm test:run` was attempted by the implementation worker and stopped under its 8 GiB memory limit, with no full-suite verdict. Current-head CI must supply the full test result. - The complete local Product E2E support suite has 13 failures. The unchanged pre-qualification baseline reproduces all 13: host PostgreSQL semaphore capacity, stale timeout/profile/workflow inventory expectations, and expired fixed-date allowance fixtures. This is not a full local suite pass. - Earlier CI at `086f7a2901` failed a browser test during cleanup cancellation with a database lock timeout. It remains recorded. All browser shards passed on both later fully checked heads. - Two qualification campaigns were canceled during setup before provider execution while correcting the oracle. They provide no live product verdict. - Previous-head CI at `28fa001457f59df1c300c016dc569049e72d0061`: 53 checks pass, 3 intentional selection skips; no failed or pending checks. - Real image/export regression: campaign 41272351780 compiled the Linux runner and passed root/unprivileged CLI and transport checks, then failed BuildKit export with `no space left on device` (exit 102, filesystem 99.99%). No provider cell ran. Removing the Rust build trees alone was insufficient: campaign 41273604815 also failed during provider-pack export with exit 102. Commit `5bbb549c54` also consolidates dependency installation and pack construction so the disposable workspace/store can be removed in the same layer. All 122 image/oracle/catalog checks pass. Image and remote-pack publication then passed in campaign 41275196741 after cleanup changes plus a scoped 60-to-120 GiB expansion of only its delete-on-termination builder disk. The VM terminated and the expanded volume was deleted. This does not establish that cleanup alone makes the default 60 GiB builder sufficient. - The Daytona admission probe then failed before agent creation, with unconfirmed provider cleanup. The queued continuity case was canceled because it shares this prerequisite. No artifact-transfer or continuity success is claimed. - Status-diagnostic regression: the original generic wrapper failed the HTTP-status assertion. After the repair, the full plugin suite passes 501 tests, with 7 skips; its standalone TypeScript check passes. Controls reject invalid status values, avoid accessors, preserve the original create status and exclude credential canaries. - Previous-head Greptile: 5/5; its summary names `28fa001457f59df1c300c016dc569049e72d0061`. All review threads are resolved. - Live artifact transfer and three-turn warm continuity: NOT VERIFIED. The diagnostic case in https://github.com/paperclipai/paperclip/actions/runs/41277598289 confirmed Daytona HTTP 401 before agent creation. The configured `DAYTONA_API_KEY` in the existing `runner-e2e-paid` environment needs verification or refresh. The queued continuity control was not run after its shared admission prerequisite failed. No further provider attempt is planned until the credential changes. Unconfirmed provider cleanup remains recorded. ## Risks - Source-provenance changes can repeat the provider dependency install. The combined build layer trades install-cache granularity for lower retained disk use. - Linux descriptor semantics and Node are required on the execution host. Unsupported providers and restricted runtime profiles fail closed. - Storage reads buffer at most 10 MiB. A provider outage can leave a reserved staging file. Partial bytes never become the destination. - A lost publication receipt can leave a complete file with an unknown result. Recovery verifies the existing file without installation. Conflicts are never overwritten. - Artifact contents remain untrusted. The copy does not bypass screening or authorize downstream execution. - Live qualification uses the governed Codex API profile. It does not establish every model, subscription profile, platform or historical failure cause. ## Model Used OpenAI GPT-6 in Codex assisted with diagnosis, code, tests and qualification. The desktop runtime does not expose a more specific model deployment ID or context-window size. The model used reasoning, source inspection, tools and code execution. The native implementation agent is configured with OpenAI `gpt-6.1-sol`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: devinfoley <[email protected]> Co-authored-by: Paperclip <[email protected]>	2026-11-02 08:27:09 -06:00
DottaandPaperclip	2b2b5a43b7	fix(daytona): restore the recorded directory for file recovery (#18527 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A stopped sandbox can hold workspace files that still need a durable copy. > - File recovery must reopen the exact allocation and directory recorded by the run. > - Workspace preparation can place that directory below the allocation root. > - Daytona resume returned the allocation root even when recovery needed the recorded child directory. > - This pull request lets the file-recovery owner request that exact directory. > - Ordinary resume keeps its allocation-root behavior. The recovery source checks stay strict. ## Linked Issues or Issue Description What happened? A stopped native run had a prepared remote workspace. The recovery service stopped and retained its allocation. The next file-recovery step failed with `source_binding_invalid` and `reopened_cwd_mismatch`. Daytona resumed the correct allocation and matched its sentinel, but returned the allocation root instead of the workspace directory recorded on the lease. Expected behavior The file-recovery owner must reopen its exact recorded directory without creating an allocation, an empty directory, or an agent turn. Ordinary execution resume must keep returning the allocation root for normal workspace preparation. Steps to reproduce 1. Prepare a task workspace below a Daytona allocation root. 2. Stop and retain the allocation with unsaved workspace files. 3. Request source-bound file recovery. 4. Before this change, the provider returns the allocation root and the strict cwd check rejects it. Paperclip version or commit Reproduced from `050365e09d88411960fc3bacc1c08c46c0d9b6fa`. Deployment mode Docker with the bundled Daytona provider. Local tests use synthetic provider boundaries and disposable PostgreSQL. Refs: #18517, #15749. Related retained-export work: #17916. That work does not change the Daytona resume handler. ## What Changed - Add an optional driver-only `requestedCwd` resume field to the SDK contract and existing host forwarders. - Set the field only after the owned file-recovery path verifies the recorded source. - Require Daytona to match the requested directory to the recorded lease directory. Require a normalized absolute path without NUL bytes. Keep the allocation and sentinel checks. - Keep ordinary resume unchanged when the field is absent. - Add provider, host, and real workspace-preparation regressions. Document the separate directory contracts. ## Verification - The original provider regression failed for reusable and ephemeral leases. The scoped regression also failed to read a real retained file from the wrong root. Ordinary resume controls passed before the fix. - Current provider handler: 345/345 passed. This includes exact-directory, malformed, missing, foreign, sentinel, reusable, ephemeral, ordinary-root, and real-file cases. - Removing only the owned producer's cwd request makes its real PostgreSQL regression fail with the original `reopened_cwd_mismatch`; the source file was restored exactly afterward. - Current host selection: 10/10 passed against disposable PostgreSQL 17. It covers the inspected owner, sandbox forwarding, generic plugin forwarding, normal resume without the field, same and later task workspace preparation, and ownership rejection. The same-workspace control retains saved bytes and does not append another workspace path. - Current SDK and Daytona builds, server build, module boundaries, maps, and diff checks passed. The server used its normal unchanged-Runner cache. Its canonical input digest matched the previously verified build, and the canonical assessor accepted the copied original stamp and outputs. The changed SDK and server compiled from current source. - Local package-wide validation exposed two unchanged 30-second timeouts in the 145,000-member tar-listing cases. A single-file control reproduced both timeouts. That test mocks this provider module. The failures remain recorded; no timeout changed. The earlier package run had 497 passes, two failures, and seven skips. It is historical evidence from the abandoned broader resume experiment, not final-source package acceptance. - Initial host attempts timed out during database setup before any assertion. The final host selection used the existing supported pre-migrated disposable template. No test timeout changed. - Greptile reviewed exact head `8d0e577d1234ad394887255f13707c010dee6d24`: 5/5, no actionable issues, zero review threads. [Current review](https://github.com/paperclipai/paperclip/pull/18527#issuecomment-6555253315). - The official [Ponytail review](https://github.com/paperclipai/paperclip/pull/18527#issuecomment-6555237850) also assessed exact head `8d0e577d1234ad394887255f13707c010dee6d24`: 5/5, no findings. Its check is terminal neutral; its completed source assessment is not a test-execution claim. - [Primary CI](https://github.com/paperclipai/paperclip/actions/runs/41286802674) initially lost its spot runner in the UI/CLI workspace lane. The log records an explicit shutdown signal and no test assertion result. UI, CLI, lane scripts and dependencies are unchanged; the adjacent identical-source baseline lane passed. The single failed-lane retry passed at the same source. The initial failure is retained. All exact-head CI gates are now green. - No live repair or image deployment is claimed by this PR. Commands: ```sh node packages/plugins/sandbox-providers/daytona/node_modules/vitest/vitest.mjs run --config packages/plugins/sandbox-providers/daytona/vitest.config.ts packages/plugins/sandbox-providers/daytona/src/plugin.test.ts node node_modules/vitest/vitest.mjs run --project @paperclipai/server server/src/services/workspace-recovery.test.ts server/src/__tests__/environment-runtime.test.ts -t 'recovers a protected retained source\|owned export directory\|prepares the (same\|later) task workspace\|does not reopen a retained source\|resumes only the exact stopped sandbox lifecycle\|explicit export directory' pnpm -C packages/plugins/sandbox-providers/daytona build pnpm --filter @paperclipai/server build node scripts/check-module-boundaries.mjs node scripts/check-maps.mjs git diff --check ``` ## Risks - A provider that does not support the optional field can still return a different directory. Recovery keeps failing closed on that mismatch. - Ordinary resumes omit the field. The tests cover the same workspace and a later task so that normal preparation does not gain an extra path prefix. - The field grants no source ownership, stop proof, file-persistence proof, or execution authority. Existing allocation, company, task, run, claim, and cwd checks remain in force. - No schema migration, retry-policy change, model replay, or historical-row repair is included. Existing failed attempts remain historical facts. Deployed file recovery still needs its normal admission and independent persistence and physical-stop receipts. ## Model Used - OpenAI GPT-6 through Codex. Used reasoning, source inspection, tool use, and code execution. The exact serving model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <[email protected]>	2026-11-01 02:47:30 -06:00
DottaandPaperclip	45a3a92e6d	fix(environments): persist sandbox ownership before creation (#18357 ) ## Thinking Path > - Paperclip manages companies and the agents that do their work. > - Remote execution creates provider resources that need a durable owner. > - The server could create a sandbox before it saved its lease. > - A failed lease write could leave cleanup data only on the old web server disk. > - This change saves a company-scoped operation before creation and transfers ownership atomically. > - Recovery finds the original Daytona resource or records an explicit manual reconciliation outcome. ## Linked Issues or Issue Description What happened? A sandbox could exist without a lease after a database failure or company rebind. Failed cleanup recording could then create a local fallback file. Replacing that web server could lose the cleanup reference. Expected behavior A failed admission write makes no provider call. An interrupted creation retains a committed operation. Recovery must not create a second sandbox or destroy an attached lease. Steps to reproduce 1. Run the disposable PostgreSQL cases in `server/src/__tests__/sandbox-creation-operations.test.ts`. 2. Reject the lease insert after the controlled provider creates its resource. 3. Start a new receiver process with a different empty web disk. It must find the same resource and create one cleanup owner. Related ownership report: Refs #14672. This change addresses new allocations before lease persistence. It does not replace existing lease recovery. ## What Changed - Add an idempotent migration and a small creation-operation table. Retain original company, environment and run identities across deletion. - Commit creation intent before run, ad-hoc lease, and plugin task replacement calls. Fence duplicate requests and ambiguous lease commits. - Retain a known returned task resource ID in the existing operation before attachment. Preserve manual reconciliation and never delete an ambiguously attached resource. - Keep company and run authority checks and atomic lease transfer with the lease owner. - Reuse Daytona names, labels, read-only inspection and claimed cleanup. Keep provider payload handling in the provider. - Retain nonsecret credential identity hashes. Refuse recovery lookup under a changed credential value. - Record manual reconciliation for missing/deleted or explicitly inactive saved credentials, versions, and provider vaults. Keep temporary resolution failures retryable. Never reactivate a disabled credential. - Use transaction signals and concrete deadlines. Failed records back off independently. Manual reconciliation stays durable and quiet. - Remove new orphan-file fallback writes from the covered paths. Keep legacy file replay and its tests for cutover. - Replace obsolete fallback-producer assertions with normal-entrypoint durability tests. Keep cleanup, diagnostics and warm handoff coverage. ## Verification - [Independent current-head verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6534744320) passed 190/190 combined tests, 40/40 incoming TestOps tests, full workspace typecheck, and build. PR head `958af986130f25e136065ad30d1fdd843c46255c` has tree `e6c3e5b02182d5941a70ccf5bf55638df3c2e28d`, identical to tested combined commit `b9d7a9c2e9b30f0153685fb24a5f70b56132201f`. Real PostgreSQL, compiled Rust Runner and controlled object providers were used; external-S3 integration remains outside this local proof. - Current head `958af986130f25e136065ad30d1fdd843c46255c` normally merges master `8595efc20bda3ce20a9fbbf34a5d156a6fcf6d57`. The sole conflict was the maintenance test: a parameterized case now preserves both legacy-local `local_storage` and valid shared-S3 `provider_mismatch` checks, including the parent local receipt, exact version identity, missing-snapshot/no-seed refusal, and unchanged persisted row. The three master TestOps files remain intact. No production, schema, migration, or package changes were made after the fully qualified prior head `0b4d05505e`. Independent review approved the resolution; the full managed-skill-content file and Cloud guard controls pass 58/58 (105.11 seconds). [Required PR CI passed on this exact merge head, first attempt](https://github.com/paperclipai/paperclip/actions/runs/41136987412): 54 successful checks and three intended skips. The additional Paperclip review is neutral at 4/5; two external Superagent checks (Contributor trust and Security Scan) remain in progress with zero annotations and no assessment, so they are not passing scan evidence. Greptile reviewed exact head `958af986130f25e136065ad30d1fdd843c46255c` at 5/5; the final thread audit has zero unresolved threads. All completed evidence below retains its recorded revision scope. The additional Paperclip Review on `958af98613` is neutral at 4/5 with a Ship verdict and no must-fix or should-fix item. Its optional observation is that a generic create rejection remains pending/manual even when an operator believes allocation did not occur. This change preserves uncertainty because the generic create-error contract does not prove provider absence; no new error classification or provider-specific compensation was added. - [Independent final-source verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6534328247) passed all 189 real-PostgreSQL/Runner and controlled-fixture cases, full workspace typecheck, and build. The integration commit `c8411a8528386f4824b340ec5fdaaed1efb9ea69` has the same source tree as final PR head `0b4d05505e` (tree `bda65efc14b4c904d0f589154db77bb160641c8c`). Controlled object-provider coverage passed; the separately guarded external-S3 suite was not run because the bounded local MinIO pull was unavailable. This includes the two final compatibility fixes. - The first final-head CI attempt exposed one E2E timing failure in `disconnect-write-recovery.spec.ts:181`: the title stayed unchanged through the 15-second session-recovery poll. The same original case passed in 13.8 seconds on immediately prior head `7ec06f77b7`, and all UI/session/E2E source is identical across those heads. The failure log and CI artifacts are retained; no cause is claimed. One failed-job-only retry is authorized on this evidence, without changing a timeout, assertion, or production behavior. [The single failed-job retry completed successfully](https://github.com/paperclipai/paperclip/actions/runs/41133619710/attempts/2): all 35 E2E shard cases passed (6.6 minutes), including the original session-recovery case in 14.0 seconds. The initial failure remains disclosed and its cause remains unproven. - Final head `0b4d05505ee0aa72f2981cdff74761406960ec73` adds two independently reviewed integration fixes after the complete 7ec CI matrix exposed two deterministic failures. Server 9/12 failed legacy local-storage maintenance because the CLI reached the new Cloud server startup guard; server 5/12 found the Daytona package version did not match its bumped manifest. The failed run is retained at https://github.com/paperclipai/paperclip/actions/runs/41131204203. It is not passing evidence and was not blindly retried. - Maintenance now constructs the existing explicit storage dependency after loading the required persisted Cloud snapshot, and passes it to both maintenance owners. It can verify legacy local storage before provider cutover while normal Cloud server startup remains guarded. The original case reproduced red locally, then the complete managed-skill-content and managed-cloud-prerequisites files passed 57/57 (32.05 seconds), including missing-snapshot refusal, unchanged stored configuration, and existing server guard controls. Daytona package version now matches manifest 0.1.10; the complete bundled package identity file passed 14/14 (3.98 seconds), and explicit manifest/lifecycle controls passed 2/2 (328 intentionally filtered; 3.42 seconds). An initial workspace-filter command matched no project and is not counted as pass evidence. The final diff changes no sandbox runtime or schema behavior. [Full CI is green on this exact final head](https://github.com/paperclipai/paperclip/actions/runs/41133619710/attempts/2): 56 successful checks, three intended skips, one additional incomplete/neutral Paperclip Review, and no pending or failed checks. The neutral review is not a passing assessment. [Greptile reviewed the exact final head at 5/5](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6529396969), with no new actionable issues and all review threads resolved. - Final combined head `7ec06f77b7c14a745c569d18401f16df3d26b626` includes master managed-skills migration 0417 and shared-storage prerequisites from `198218bf55` through normal merges. Sandbox creation moves to generated `0418_curved_lady_mastermind.sql`; its idempotent SQL is byte-for-byte unchanged. The regenerated 0418 snapshot contains both sandbox creation operations and the master skill mutation receipts, and points to the master 0417 snapshot. Master SQL and journal history remain intact. Exactly five snapshots remain, 0414 through 0418. The only sandbox-specific changes since the approved credential policy are migration metadata/number and the matching test/document references; the sandbox runtime files are unchanged. - [Independent final combined verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6533960026) passed on integration revision `e6f05b4cf0cd`, containing final sandbox head `7ec06f77b7` and actual master `198218bf55`: migration/declaration checks, all 158 real-PostgreSQL/Runner and controlled-fixture cases (251 seconds), full workspace typecheck (81 seconds), and build (35 seconds). - Final combined migration proof: fresh real PostgreSQL has both tables. A separate original-0417 sandbox table with a retained pending row survives two applications of the renamed 0418 SQL unchanged, with the expected primary/index and foreign-key structure. Migration numbering/safety checks pass. The normal runtime migration reapply case passes 1/1 (20 intentionally filtered cases; 46.69 seconds). The first temporary qualification harness stopped before any database assertion because its external `.ts` file selected CommonJS; the `.mts` harness corrected that setup and passed. Independent review approved the exact metadata and reference diff. Earlier full suite and live-provider results below retain their stated revision scope. - Credential-policy head `c606d473b03709f27577a55a1dfc0b0d3b4359fd` handles the remaining inactive-credential review warning. The existing boundary accepts only concrete secret-owner 422 codes and exact no-details disabled/coming-soon vault messages. Unknown 422, authorization errors, provider wrappers, and temporary database/provider failures still propagate. Independent source review approved this three-file correction. No generic resolver, schema, provider protocol, retry cap, or activation behavior changed. - Creation/cleanup qualification at `c606d473b0`: all 39 cases pass against real PostgreSQL (81.27 seconds). The new cases use actual disabled/archived saved secrets, a valid disabled version, and provider vaults created ready before normal API updates disable them or mark them coming soon. Each remains manual through restart/repeated scans, with zero provider lookup and no retry deadline. The independent plugin idle blocker remains intact. The temporary 503 control retains retry ownership and later recovers one cleanup owner. Before this correction, all five inactive-state cases remained pending; that failed evidence is retained. The prior 18-case known-ID task proof below belongs to `237144fcf6`, whose helper and task path are unchanged by this correction. - Known-ID head `237144fcf69a249c0e0310563428c0f55ed6d55c` also addresses the independent task-replacement reliability warning. The existing creation operation now saves only the known nonsecret returned resource ID before expiry validation and attachment. The write is scoped to operation/company/original environment and allows only a missing or matching ID. It changes no ownership state and copies no resolved configuration, credentials, or provider payload. Independent source review approved this four-file fix. - The complete real-PostgreSQL task suite passes 18/18 (9.52 seconds). Rejected attachments retain the ID through manual reconciliation and repeated scans. A real committed attachment followed by an injected lost response retains the same active lease and attached operation, with no release or second create. Same-ID retries work; conflicting IDs and foreign companies cannot change it. The initial three regression cases failed because the ID was absent, before the fix. - The shipped Daytona manifest does not declare task support, so it cannot enter this generic replacement path. The dispatcher requires both declared task support and an environmentTask worker method. Generic drivers without creation lookup still use manual reconciliation, with a known returned ID retained when the database write can commit. No automatic lookup capability or unconditional compensation was added. - Previous boundary-fix head `8ba77def151ea85eeae2db7693254db7f87673b0` fixes the fresh review finding about missing credentials. Only known secret-owner 404 results at the recovery secret-resolution boundary select the existing manual outcome. Provider inspection and transient database/provider errors remain retryable. Independent source review approved this change. - Secret-boundary focused acceptance at `8ba77def15`: all 34 cases pass against isolated real PostgreSQL. The complete creation suite passes 16/16 (100.75 seconds); the complete cleanup coordinator suite passes 18/18 (99.83 seconds). The new cases use normal saved-secret creation and deletion, confirm zero provider lookup, restart/repeated scans, no automatic deadline, preserved independent plugin idle blocking, and no additional idle blocker after the controlled worker supplies its quiet proof. A temporary 503 resolution failure preserves pending ownership and later recovers one cleanup owner. No additional live provider resource was created. - Failed attempts remain part of the evidence. The initial red run reproduced the pending-operation bug and also exposed a numeric-deadline test harness mistake, which was corrected. A later combined run reported an existing replacement-receiver failure and a new idle-fixture assertion failure; it was interrupted while the next test file remained in imports. An isolated diagnostic confirmed that the manual state was correct but the installed fixture plugin correctly blocked sleep without its independent quiet proof. The corrected test retains that blocker assertion before supplying the controlled worker witness. The unchanged replacement-receiver case passes in the final complete 16-case run; its earlier failure is not counted as a pass or assigned a proven cause. No timeout or production idle guard was relaxed. - Greptile reviewed previous combined head `7ec06f77b7c14a745c569d18401f16df3d26b626` at 5/5, with no unresolved review threads. Its CI produced the two concrete integration failures addressed above. The earlier CI and live evidence below remain historical evidence, not a claim that the current full gates have completed. - Master integration update: merged `dbff3747b8` and then latest master `76fde891cc` without rewriting branch history. Only `doc/DATABASE.md` conflicted; the sandbox ownership, object-backed run evidence, and cloud configuration sections all remain. The master merges changed no sandbox implementation, migration, or provider code. Migration numbering/safety checks pass and exactly five schema snapshots remain. At merge head `f5d23d4a59828835d01c150f83fa2645fb2d3bec`, the two focused creation/coordinator suites pass all 32 cases against isolated real PostgreSQL (64.57 seconds). Those merge-head checks precede the missing-credential fix below. The earlier live Daytona proof still covers the unchanged normal creation/lookup/teardown path; the new missing-credential branch is qualified with real PostgreSQL and controlled provider hooks. The optional Superagent Security Scan was inconclusive on prior head `7ec06f77b7`: its pinned PR merge-base checkout failed or timed out, with a neutral conclusion and zero annotations. An earlier scan could not start within 120 seconds; a bounded rerun request returned HTTP 404. These scanner failures are not passing security-scan evidence. On final head `0b4d05505e`, the additional Paperclip Review check is neutral: its assigned review execution did not complete and produced no validated assessment. Fresh Greptile 5/5 and independent source review are separate evidence. Previous-head required CI, independent review, and the live Daytona qualification below passed. - [Complete previous-head CI](https://github.com/paperclipai/paperclip/actions/runs/41105726052) passed at `0b5cbb763ab8faa25aa0ff1d8a405cdc0d3db409`: 53 successful checks and three intended skips, including typecheck, full tests, build and E2E. Greptile is 5/5 on this head and both original findings are resolved. - Initial head: 200 PostgreSQL runtime, creation and coordinator tests passed. The corrected head also passes all 32 creation and coordinator tests, including deletion before/during lookup, a failed lookup after deletion, restart, settled polling and actual idle-safety checks. Cases cover failed admission, failed lease writes, concurrent recovery, lost provider replies, ambiguous transfer retries, company rebind, environment deletion, stopped runs, lookup backoff, restart, drain and settled SQL quiet. - Separate receiver processes: passed. A controlled HTTP provider retains the resource while three independent receiver processes use two empty web disks. Recovery creates one cleanup owner and never repeats creation. - Fresh PostgreSQL migration and two additional applications: passed. The pending creation row survives unchanged. - Server TypeScript and SDK/shared build checks pass after the corrections. Migration numbering/safety checks passed. Exactly five schema snapshots remain. - Daytona adapter: all 330 tests passed, including stable-name lookup, ownership checks, deletion and the updated manifest version. An earlier run exposed the version expectation and it was corrected. Previous-head CI passed. - Live Daytona acceptance passed on `0b5cbb763a`: isolated real PostgreSQL, normal runtime acquisition through independent receiver processes, actual Daytona plugin/SDK 0.203.0, one disposable Ubuntu 24.04 sandbox (1 CPU / 1 GiB RAM / 1 GiB disk). A forced lease insert failure retained the committed operation; a fresh receiver with empty disk found the same sandbox, repeated recovery retained one cleanup owner, and the actual claimed cleanup sweep expired the lease and resolved the operation. Independent SDK lookup returned typed not-found; ordinary recorded-ID inspection returned destroyed. This successful attempt made one create, one recovery lookup, and one teardown. Two earlier failed harness/provider-confirmation attempts are disclosed in the [live acceptance record](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6530313054). The coordinator independently queried all three exact resource IDs through the SDK and confirmed typed not-found for each. - Complete CI-affected heartbeat/plugin warm-task suites passed 21/21 after test corrections. An unchanged workspace exposure case passed in isolation after the prior CI shard reported an unhandled readiness rejection. Complete previous-head CI and refreshed review passed. ## Risks - Deleted companies retain an explicit manual reconciliation outcome. Recovery does not retry missing credentials or insert a lease for a deleted company. - Only Daytona declares automatic creation lookup. Other uncertain providers retain `needs_reconciliation`. A missing creation name alone cannot prove that a late allocation will never appear. - Pending operations use the existing process-local signal and startup recovery model. This does not add cross-replica wake delivery or prove deployed sleep. - Legacy direct provider probes and custom-image setup use separate lifecycles. They are outside this replacement of the run-lease orphan-file fallback. The normal Test path that acquires a runtime lease is covered. - Existing orphan files still require the separate cutover inventory and settlement. This change does not delete them. - The schema change is additive. The new SQL can be reapplied. Historical SQL and journal entries are preserved; only the oldest generated snapshot is pruned. ## Model Used OpenAI `gpt-6.1-sol`, with tool use, code execution and repository review. The harness did not report the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge The required branch status checks (`ci / verify`, `ci / e2e`) and all test/build jobs passed on this head. The non-required Superagent Contributor trust and Security Scan checks are still running without findings; they are not claimed as passes. No required check or branch rule is overridden. --------- Co-authored-by: Paperclip <[email protected]>	2026-10-30 17:38:23 -05:00
Nicky LeachandPaperclip	b056026407	fix(ui): keep task plugin views available for manual access (#18363 ) ## Thinking Path > - Paperclip helps people manage agents and their tasks. > - The task side panel can display a plugin's agent view. > - Users need that view even while a task is idle. > - Activity currently controls both manual discovery and automatic opening. > - This pull request keeps manual access available and uses activity to request automatic opening. > - The same authorized agent viewer serves both entry points. ## Linked Issues or Issue Description What happened? A task view disappeared when its run ended. Users could not select it until the current run reported matching activity and its readiness check passed. This also excluded declared manual views without an activity mapping. Expected behavior The task launcher offers declared views for its running or assigned agent. Selecting one uses the same viewer as the agent detail page. Matching activity and readiness open the pane and select the view by default, once per run. Steps to reproduce Open an idle task assigned to an agent with an installed task-panel contribution. Open the panel launcher. The view is currently missing. Related work: #18131 and #18239. ## What Changed - Return manual panel declarations independently of run activity and readiness. - Mark panels that should open automatically after matching activity and readiness. Keep existing run attribution, retry bounds, and dismissal behavior. - Keep an open viewer mounted when a run ends. Focus an already open manual view when activity starts. - Preserve saved manual tabs while plugin and initial run discovery load; remove contributions after discovery confirms they are absent. - Share the run-scoped dismissal key between the task page and panel controller so dismissed views do not reveal an empty pane. - Use the current running agent for declared manual views that have no activity mapping. - Update the plugin authoring contract and feature recipe. ## Verification - Two new regressions failed before the fix: idle discovery returned no view, and a manual contribution opened automatically. - 105 focused hook and panel tests pass. They cover idle/manual access, declarations without mappings, completion, saved-tab restoration, automatic focus, activity attribution, retry bounds, dismissal, and missing/disabled plugins. - Thirteen task-page panel integration tests pass. New regressions reproduce and fix a dismissed manual view revealing the outer pane. They cover desktop/mobile and both workspace modes, and verify that a later run can still open the view automatically. - `node scripts/check-token-gates.mjs`, `node scripts/check-maps.mjs`, and `git diff --check` pass. - Local UI typecheck failed with the reused dependency cache: workspace modules/exports and generated Storybook catalog files are missing or stale. It reported no errors in the changed files. The final head passed the clean GitHub typecheck. No broad local build or repository test run was attempted. - The preceding head passed [GitHub CI](https://github.com/paperclipai/paperclip/actions/runs/41116028688), with 48 jobs succeeding and one skipped. Its final review scored 5/5 and identified a narrow saved-view restoration race. This revision adds the missing initial-run loading guard. Two delayed-response regressions failed before the correction and now pass; they cover a different assignee and an unassigned task. - The final head `bbde3a28f7ac0390da8f3fe81eb9abd2f3020695` passed [GitHub CI](https://github.com/paperclipai/paperclip/actions/runs/41119772438) and the additional [full CI run](https://github.com/paperclipai/paperclip/actions/runs/41121827023), each with 48 successful jobs and one skipped. Standard Greptile scored 5/5 on this head; both review threads are resolved. The small final loading correction used the permitted exception to another final-review cycle. - The additional full CI run initially failed the canary bundle install because npm advertised a newly published `[email protected]` archive before its tarball was available (HTTP 404). After the archive became available, one retry of the failed canary and dependent aggregate passed. No dependency or lockfile change was needed. - Hosted acceptance of this follow-up requires merge and deployment; the earlier hosted activation checks cover the previously merged behavior. ## Risks Manual views can now open while idle or before readiness. Each viewer retains its normal authorization and status handling. Declarations without an activity mapping now appear in the launcher. Automatic opening still requires authenticated activity and the authorized readiness result. Saved tabs wait for plugin discovery before removal. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, terminal tools, browser verification, and code execution. The exact serving variant and context window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-10-30 13:44:41 -07:00
DottaandPaperclip	8f822bfcf7	fix(cloud): persist configuration and load release-image packages (#18335 ) Land the approved Cloud configuration and release-image package stack. Includes the separately reviewed adapter settings (#18335), external adapters (#18333), bundled plugins (#18336), startup configuration (#18339), and migration/package checks (#18341). Includes the final image-owned loader and linked-dependency inventory corrections. Verified stack head: `462716170d`. Co-Authored-By: Paperclip <[email protected]>	2026-10-30 19:21:33 +00:00
Nicky Leach	fba0ab8162	fix(runner): preserve GitHub helpers and Codex probe policy (#18212 )	2026-10-30 07:42:10 -07:00
Nicky LeachandPaperclip	1d571de5f9	Add shared native turn attribution boundaries (#18213 ) ## Thinking Path > - Paperclip manages work performed by AI agents. > - Native execution crosses the plugin worker, Runner, sidecar, and durable event boundaries. > - Existing outer spans leave repeated preparation and admission work unattributed. > - Optimizations need a common baseline with accurate ownership and event meanings. > - This change extends the existing opt-in tracing contract with bounded timing receipts. > - It preserves execution policy and keeps provider internals opaque where the API provides no timing proof. ## Linked Issues or Issue Description What existing behavior does this improve? Attribution of native task preparation, retained/cold Runner handoff, and completion-to-text work. Current behavior Worker spans use a flat provider contract. Workspace calls lack a consistent dispatch envelope. Submitted-to-started timing is named as a provider queue even though it includes other work. The transport does not expose a shared breakdown of its waits. Proposed improvement Add invocation-scoped nesting, bounded numeric remote receipts, current-run transport timing, durable receipt intervals, and distinct completion/text milestones. Reuse the existing exporter. No latency optimization is included. Related public work: #17842 consolidates existing startup tracing tests; it does not add this runtime coverage. ## What Changed - Extend the optional SDK tracer with nested async scopes and completed timing receipts. Keep the host responsible for parent identity, capability checks, attributes, clocks, depth, and count limits. - Add dispatch and bounded host-read timings for environment and workspace providers. - Resolve transport parents on each call so retained sessions do not retain an earlier run's trace. - Split cold/resume/attach/prepare/registration/command/started-event waits. Add opt-in numeric ACPX Runner/sidecar status and prompt-start receipts. - Distinguish first completion request, validated tool receipt, final result commit, first nonempty text, and event persistence. Rename the submitted-to-started interval to match its meaning. - Link completed earlier remote lifecycles to their host-owned observer rather than placing them under a later polling request. Preserve sampling, timestamp limits and budgets. - Document the versioned coverage map, causal clock limits, and baseline acceptance requirements. ## Verification - Current head: GitHub CI is fully green. Standard Greptile and Apex are both 5/5, with no unresolved review threads. Full-suite validation was performed in CI; follow-up local validation was limited to affected packages and tests. - Review follow-up: early rejection, missing provider capability and startup-parent regressions were reproduced and fixed. The affected worker/SDK/timing set passes 128 tests; 28 timing/context tests, SDK typecheck and direct server TypeScript checking pass. The server typecheck wrapper reached an unrelated Runner rebuild and stopped because Cargo was absent from PATH; full Runner/image validation remains in CI. - CI failure accounting: the earlier Turbo cache/force conflict and Linux OpenCode shell assertion were reproduced and diagnosed. A direct proc probe ruled out permissions; a portable shell comparison identified Bash skipping BASH_ENV with stripped SHLVL and socket stdin. Upstream #18230 now supplies the CI and ignored-stdin fixture fixes; this PR adopts them without duplicate CI changes. The strict Linux OOM assertion passed before this source-equivalent merge. - Retry follow-up: the three insertion/replay/failure timing cases and server typecheck pass. The existing database regression now asserts a single marker across identical result replay; the current-head CI integration gate passes. - Upstream CI alignment: all three real-Turbo 2.11.10 integration cases pass. The repository policy gate rejected committing a generated lockfile; the final PR contains no lockfile edit and uses the maintained inline stale-lockfile recovery. - Linked-lifecycle follow-up: 44 focused host/export tests pass (three optional SDK-dependent cases skip); SDK and server typechecks pass. The API contract test verifies an explicit root context, host-owned observer link and unsampled suppression. - SDK typecheck and focused worker/invocation tests: 30 passed, including clock changes, bounded legacy IPC, and manual-span creation-scope parentage. - Focused reviewed host/native/transaction-boundary checks: 44 passed. The 86-test worker-manager suite passes after updating its exact assertions for the host-owned timing budget found by CI. Earlier preparation/first-text checks also passed. - Affected server and Runner TypeScript checks pass after building local workspace declarations. - Merge follow-up: the profile/candidate startup subset passes 94 tests, the four Grok source-identity checks pass, and Runner TypeScript checking passes. A subsequent execution-factory merge passes 21 focused timing/trace tests and server TypeScript checking. - Pinned Rust build and the ACPX Rust subset pass. The hostile receipt projection test passes. Source profile compatibility checks: 127 passed, one existing skip. Grok identity checks: four passed. - Broad transport/ACPX suite: 410 passed after building all fake provider binaries. Earlier attempts failed on missing Runner/fake-provider outputs; building the required fixtures resolved those failures. - The first CI run correctly rejected stale source attestations. After integrating current master, Grok advances 13→14 and Copilot 84→85; master’s historical fixtures and Pi 35 stay byte-identical. Provider dependencies, models and permission defaults are unchanged. Identity generation/checks now pass. - The added real-Postgres result lock regression could not initialize its local embedded cluster. The insertion/replay/failure unit regressions pass. The current-head CI integration gate passes. - Comprehensive repository checks and image builds are delegated to CI, as requested for this instrumentation pass. No deployment, model run, collector-ingestion proof, or latency improvement is claimed. ## Risks The new SDK methods are optional. Legacy span calls and endpoint-off behavior remain supported. The host accepts only closed timing names and bounded attributes. Remote durations are anchored at local receipt; transport/scheduling uncertainty remains explicit. Trace trees have count/depth limits. The first rollout must inspect actual collector exports before treating the baseline as accepted. Fresh live qualification for the new source identities remains a rollout requirement. Revert these commits to remove the new measurements; no schema migration is required. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and local code/tool execution. The exact runtime model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the enhancement template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Standard Greptile and Apex are both 5/5, with no unresolved findings - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-10-29 17:45:56 -07:00
Nicky LeachandPaperclip	35ab90ad0f	Map runtime activity to authorized task plugin panels (#18131 ) ## Thinking Path > - Paperclip manages agents and tasks. > - Plugins contribute views to the task sidebar. > - A ready view can belong to an earlier task. > - Runner records activity under its authenticated run. > - This PR maps that activity to installed task-panel declarations. > - The sidebar checks readiness after current-run use and restores use after reconnect. ## Linked Issues or Issue Description Builds on merged contract #18129 and Runner publisher #18050. This PR targets master and is ready for review. Earlier combined verification is retained in [the verification archive](https://github.com/paperclipai/paperclip/pull/18050#issuecomment-6489369676). Subsystem affected Run-event reads, plugin task contributions and the task sidebar. Problem or motivation Readiness polling can show a view left ready by another task. Reload and reconnect must recover current-run use without losing concurrent live activity. Proposed solution Map recorded activity to installed task-panel declarations. Merge authorized history and live activity, then perform bounded readiness checks for the running agent. Preserve the existing read-only view. ## What Changed - Add optional `{ namespace, activityKey }` activation mappings to task-panel declarations. Legacy declarations remain valid but contribute no task panel. - Add `view=plugin-panels` to the existing run-events route. Preserve run-read and telemetry-read authorization, secret redaction and private-run denial. Map activity to installed contributions and return at most 128 panel identities. - Scope activation caches to current panel versions and activity declarations. Preserve unchanged panels during unrelated plugin updates and history reads. - Merge history and live hints without losing concurrent activations. Filter by company, task, run and agent. Restore history on mount and reconnect. - Replace periodic availability checks with up to three readiness attempts after activity. Fresh activity restarts settled transient failures. Pending checks, successful readiness and HTTP 4xx failures are left alone. - Reuse live run updates with the panel cache key. Recover only panel history after a pending subscription read. Mark expected not-ready query results so the global error reporter skips them; real failures remain reportable. - Preserve current-run dismissal and the read-only plugin view. Update manifest comments and producer/consumer documentation. ## Verification - Current head: `5e04d757c`, based on master `09c05c550`. Range comparison confirmed all six feature commits were unchanged by the rebase. The final follow-up keys readiness by the plugin installation ID and availability path. - `cd ui && node ../node_modules/vitest/vitest.mjs run src/hooks/useTaskPluginPanels.test.tsx` — all 45 cases passed. `cd ui && node ../node_modules/typescript/bin/tsc -b` passed. - The shared query-client and live-update recovery selection passed 96 cases on `49c544cd2`. Token gates passed before the final hook-only changes. - Earlier focused verification passed 57 manifest cases, 69 database/history cases, event-registry cases and route authorization cases, including private-run denial. - Regression coverage includes subscription/history races, task changes during recovery, unfocused run completion, bounded readiness retries, denied reads, expected not-ready responses, changed activity mappings, unchanged panel preservation, old cache scopes, and same-version route or installation changes. - [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/40942001707) passed: 54 successful checks, two skips, no failures or pending checks. - [Standard and final review](https://github.com/paperclipai/paperclip/pull/18131#issuecomment-6489842168) both scored 5/5 on `5e04d757c`. The final review used the Apex tier, added no comments, and left no unresolved findings. The PR is conflict-free and ready for review. - Local checks stay targeted. GitHub CI provides the broad build, test, typecheck, browser and Runner coverage. No deployment or live-provider acceptance is claimed. <details> <summary>Retained failures and recovery evidence</summary> - `ac9329548` twice failed a browser reload assertion while conversation history was concealed. Master fixed its readiness wait in #17923. The approve journey then passed locally on `fea63a418` with installed Chrome (1 passed, 39 seconds). All eight browser shards passed in later CI. The browser test is outside this diff. - A canary checkout failed with Git HTTP 408; its unchanged-head retry passed. - `de07d09fa` failed the mobile Cases card click after a ten-second stability wait. The same read-only journey passed locally on that exact head (1 passed, 26 seconds). Its unchanged-head CI retry passed 22 cases, with one skip. The original artifact is retained; the cause of that click timeout is not established. - [CI on `49c544cd2`](https://github.com/paperclipai/paperclip/actions/runs/40903679075) passed 54 checks with two skips. CI also passed on `de07d09fa` after the recorded retry. - The `1e59ab441` merge checkout passed all browser and server suites. Runner shard 2 passed 2,252 cases and timed out waiting for a controller-recovery command. The test had landed on master in #18161 after the former branch base. Rebasing aligned the local checkout with that code; Runner shard 2 then passed on `fcf7380d3`. The original timeout cause remains unproven; no unrelated timeout was changed. - [CI on `fcf7380d3`](https://github.com/paperclipai/paperclip/actions/runs/40940235193) recorded runner shutdown signals in typecheck and a workspace shard. A chat recovery test observed a null wake deadline, and one Runner transport fixture failed during close. Those separate assertion failures remain unexplained. All of those jobs passed on the final readiness-key head. Those later passes establish recovery; they do not prove the cause of the earlier assertion failures. - Local browser setup first stopped at pnpm version-download verification, then a sandbox IPC restriction, then a missing bundled browser. Installed pnpm 9.15.4 and Chrome completed the journeys. No dependency files were changed. - Earlier optional-property TypeScript errors were corrected. Embedded PostgreSQL needed its install hook run from its package directory. Older CI also recorded fixture-cleanup lock timeouts, attention-fixture timeouts, and canceled server shards after runner shutdown. The parent fixed its FIFO snapshot-copy failure before merge. The linked verification archive retains earlier combined evidence. </details> ## Risks Activity is a hint and grants no access. Missing plugins, denied reads and unavailable views contribute no panel. Older runs without activity records show no task plugin tab. A used view stays available for the running task; reconnect reconciles readiness. Matching producer and consumer builds are required. No migration, dependency, image publication or deployment is included. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools and code execution. The exact serving snapshot is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-10-29 13:56:48 -07:00
Nicky Leach	e2cfdc7e0c	feat(plugins): expose GitHub credential broker to task providers (#18178 )	2026-10-29 10:52:44 -07:00
DottaandPaperclip	6a932bb979	fix(daytona): preserve workspace exports under disk pressure (#18172 ) ## Thinking Path > - Paperclip manages AI agents and preserves their work when execution stops. > - Remote workspace recovery must save the retained files before another executor can start. > - Daytona directory export first writes a complete tar on the source sandbox. > - When that disk is full, each preservation retry fails while writing the same temporary archive. > - This change streams an archive after a storage-capacity failure, without another full copy on the sandbox. > - Existing source ownership, archive validation, and task admission gates remain in force. ## Linked Issues or Issue Description Related: #14402 established safe workspace export and retained-source recovery. This PR handles the remaining temporary-archive storage requirement. What happened? A sandbox ran out of disk space. The provider could no longer save its state, and workspace preservation then failed with tar `No space left on device` or `Wrote only ... bytes`. Repeating the export could not resolve that failure because it required free space on the same disk. Expected behavior Export retained files without needing a second full copy on the source disk. Reject incomplete transfers and unsafe archives. Preserve the source and newer host files if export fails. Steps to reproduce Run the capacity regression with real tar and a file-size limit. The original export fails during archive creation. The fixed export transfers the same bytes through a real PTY. The credential-gated live test uses a real Daytona sandbox and `/dev/full` as the archive sink to reproduce ENOSPC without filling a provider disk. Paperclip version or commit Reproduced on `27be859044`; fix rebased onto `6d59f5ef7e`. Deployment mode Source checkout on macOS arm64; live provider verification on a disposable Ubuntu 24.04 Daytona sandbox with SDK 0.203.0. ## What Changed - Fall back only after archive storage-capacity errors. Keep the ordinary bulk download path. - Transfer 40 KiB archive chunks over a raw PTY. Wait for each host write acknowledgement and a final receipt acknowledgement. Send long scripts only after raw-mode readiness. - Require successful tar exit and complete framing before the existing host archive validator can import files. Keep the bounded safe-link retry and exclude rules. - Bound total archive bytes by the provider-reported disk allocation, preserve a 1 GiB host free-space reserve, and bound terminal cleanup to five seconds while closing the host file independently. - Add real-tar/PTY regressions, stream failure tests, a credential-gated provider test, and recovery documentation. ## Verification - Red before production changes: the capacity regression failed on the real tar write with exit 153. Green afterward: unchanged bytes, file mode, confined links, and source files survive. A tar producer that fails after sending bytes cannot overwrite newer host files. - 43 focused tests pass. Missing receipts, nonzero exits, transport loss, invalid base64, oversized frames, stalled transfers, and late connection cleanup fail closed. - Live Daytona test passes: stop/start, actual tar ENOSPC, matching file hashes and modes, unsafe-link filtering, intact source, then another stop/start and export with a later saved file. The disposable sandbox was deleted. Initial default-sandbox creation hit account quota; the successful test used a 1 GiB image. Large output bursts exposed a real socket closure, corrected by bounded chunks and verified in the successful run. - Repository `pnpm -r typecheck` and `pnpm build` pass. `node scripts/check-maps.mjs` passes. Standalone plugin typecheck passes. - Standalone plugin build and final typecheck pass. The final broad plugin run passed 482 tests, skipped seven credential-gated cases, and hit one existing 30-second archive-listing timeout. Earlier runs hit both large-listing cases. Both timeouts reproduce with unchanged `file-sync.ts` from master `6d59f5ef7e`, with 22 other archive-listing cases passing on both base and fixed source. Parallel, serial, and GNU-tar local runs retain those timeouts. The serial local `pnpm test:run` was stopped after 36 minutes once the complete current-head CI test suite passed; it is not counted as a full local pass. GitHub Actions on production-code head `9ffbd9e56e283a01c22f005c022b8f6838cf0143` passed typecheck, all general and serialized test shards, Runner verification, build, E2E, and canary dry run. - Review fixes have passed the live Daytona test again. Sparse-file budget enforcement, rejection before host writes, low host reserve, and stalled provider cleanup are covered by the 43 focused tests. Greptile is 5/5 and security scanning passes on `9ffbd9e56e283a01c22f005c022b8f6838cf0143`; both review threads are resolved. - Test portability: `d2280108f5` verifies GNU tool versions and Python before running the real-PTY regression. All 43 focused tests pass with prerequisites installed; restricting PATH to the macOS system tools cleanly skips the three integration cases. Standalone plugin typecheck passes. Current-head CI passes typecheck, build, all test shards, Runner verification, E2E, and canary dry run: https://github.com/paperclipai/paperclip/actions/runs/40910736373. Greptile and Paperclip Review both assess the final head 5/5, security checks pass, and no review threads remain unresolved. - Paperclip Review assessed the final head 5/5 with no actionable defects after the test-portability fix. Remaining accepted tradeoffs are recorded under Risks. - This is live provider-boundary acceptance. It does not prove a full task Retry journey, repair lost provider session state, or recover the original deployed task. ## Risks - Streaming requires Bash and GNU tar, dd, and base64. Missing tools or transfer failure reject export and retain the recovery obligation. - Acknowledged terminal transfer is slower than bulk download, especially across high-latency links. Very large exports can still reach the configured deadline. The ordinary bulk path is unchanged. - The fallback checks host free space before every 40 KiB write; this deliberately favors a strict reserve check over reducing local syscalls. It carries archive-download error context but does not yet emit a dedicated transfer span. Provider cleanup is bounded; a kernel-stalled local filesystem operation can still delay completion and file close. - This change does not increase sandbox capacity or reconstruct provider state that failed to persist. The plugin must reach the target installation before recovery can use it. No merge, deployment, or original-task replay is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, code editing, shell execution, and automated testing. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-10-29 11:41:01 -05:00
yzxcj797andAndrew Aymeloglu	81dcc4f6bf	docs(kubernetes): fix agent-sandbox install manifest URL (#11410 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Kubernetes sandbox provider supports the Sandbox CR backend. > - Its migration guide tells users to install the agent-sandbox controller. > - The command names install.yaml, which is not a release asset. > - This pull request changes the asset name to sandbox.yaml. > - Users can then download the base controller and CRD manifest. ## Linked Issues or Issue Description Issue type Incorrect information. Where is the issue? packages/plugins/sandbox-providers/kubernetes/README.md, under “Migrating from job to sandbox-cr”. What's wrong? The installation command uses releases/latest/download/install.yaml. The upstream agent-sandbox release does not publish that asset. Suggested fix Use releases/latest/download/sandbox.yaml, the base installation manifest. ## What Changed - Replace install.yaml with sandbox.yaml in the controller installation command. ## Verification - Checked upstream release v1.0.8. Its assets are sandbox.yaml, extensions.yaml, and sandbox-with-extensions.yaml. - Reviewed the one-line documentation diff. - No Kubernetes installation or local tests were run. ## Risks - Low risk. This changes one documentation URL. - The URL follows the latest upstream release, so future upstream asset changes can affect it. ## Model Used Original author model use was not reported. Adoption and description update assisted by OpenAI GPT-6 (Codex), with repository and GitHub tool access. A more specific runtime model ID is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [ ] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Andrew Aymeloglu <[email protected]>	2026-10-29 11:12:05 -05:00
Nicky LeachandPaperclip	27be859044	Record runtime activity under Runner authority (#18050 ) ## Thinking Path > - Paperclip manages agents and tasks. > - Runner owns each authenticated run and its provider process. > - Runtime clients need to report successful activity in that run. > - A retained process can outlive a run, so an operation needs current authority. > - This PR captures that authority before the operation and records activity afterward. > - Core can consume the event through the existing run stream. ## Linked Issues or Issue Description Builds on merged #18129 and targets master. The dependent consumer is #18131. Test stability fixes were separated into merged #18128. Earlier combined-head verification remains in [the archive](https://github.com/paperclipai/paperclip/pull/18050#issuecomment-6489369676). Subsystem affected Runner activity recording and provider command environments. Problem or motivation A retained provider process can complete an operation after its original run ends. Runtime clients need a host-attributed event and a helper that works inside the provider sandbox. Proposed solution Capture opaque Runner context before use. Publish a structured notification after success. Runner checks current authority and records `runtime.activity.recorded`. ## What Changed - Add `notification-context` and `record-activity CONTEXT NAMESPACE ACTIVITY_KEY`. Rotate context on warm attachment and process recovery. Accept no caller run ID. - Use one private named pipe. Reuse Runner's bounded JSON-line reader. Serialize concurrent publishers with a read-only lock. A blocking reader queues at most 128 records; the existing control loop handles at most 16 per pass. - Grant Codex access to the exact helper, descriptor, lock and pipe. Preserve other filesystem and network limits. Preserve host discovery values through provider launch boundaries and reject configured-environment overrides. - Update source identities to Pi 34, Copilot 81 and Grok 12. Keep historical identity fixtures and recovery behavior. ## Verification - Targeted release notification and sandbox-policy tests: seven passed. They cover run rotation, recovery, Unicode limits, concurrent writers, malformed and partial records, bounded queues, failed persistence and fixed-path cleanup. - Real CLI integration and pinned Codex 0.160.0 sandbox checks: passed. The sandbox check uses no model, credential or network call. It verifies activity recording while unrelated state reads and parent-directory writes remain denied. - ACPX profile generator check, formatting and diff checks: passed. - Rebased onto master `31b927e30` at head `64d0248a8`. Master used the same profile versions, so the resolution preserves its Pi 33, Copilot 79 and Grok 11 fixtures byte-for-byte and creates new identities. All identity checks pass. Five targeted provider/environment/recovery files pass 168 tests with one existing skip; the Rust Pi admission regression passes. The notification module and Codex sandbox implementation are byte-identical to the tested pipe head `671373fe0`. - Removed the mailbox traversal reported by security review. Its regression replaces the pipe path with a 100-level tree and an unreadable leaf. Shutdown preserves the leaf, performs no traversal and completes promptly, including with a partial record. - An initial targeted test failed because its run-rotation fixture updated config without updating durable state. The corrected fixture passes. Runner also rejects inconsistent run bindings. - Previous head `26abbf62b` passed Runner, build and browser CI. Server shard 11/12 failed in unchanged `attention-service.test.ts`: cleanup/test timeouts were followed by a duplicate `ATQ` fixture key. The final head needs fresh CI and review. - CI on `64d0248a8` completed with 52 successful checks and four intentional skips. Runner Vitest shard 1/2 failed when retained-state copies encountered the live FIFO; the aggregate also failed. Standard review scored 4/5 and found relative pipe paths plus late context publication. Follow-up `5c3f14d32` places IPC in a private canonical temporary directory and publishes context before welcome commands. Its relative-state CLI test, two authenticated reconnect cases, seven notification/policy cases and real Codex sandbox pass. One affected retained-settlement case passes with the matching release binary; the other 18 cases remain for CI. The sandbox denies sibling-directory and lock writes. All build, test and security checks pass on `5c3f14d32`; standard review is 5/5 with no unresolved threads. The requested Apex review is 5/5 on this exact head, with no findings or unresolved threads. - Earlier broad test results in the archive apply to their recorded source revisions. Per the requested workflow, final local checks stay targeted; GitHub CI supplies broad coverage. - After Apex completed, master advanced again and conflicted on Copilot identity. Head `e94a9359d` is rebased on master `8ac5a128e`; it preserves the published Copilot 80 fixture and uses Copilot 81. The notification module, Codex sandbox and durable control-loop code are byte-identical to Apex-reviewed `5c3f14d32`. All 131 targeted installation/compatibility/recovery cases and identity generator checks pass. All 55 checks pass with four intentional skips on the rebased head. Standard review is 5/5 with no unresolved threads. Apex is 5/5 on `e94a9359d`, with zero findings and no unresolved threads. GitHub confirms this head is conflict-free and ready for review. ## Risks Activity remains an optional hint and grants no access. A successful helper exit means publication to the pipe; Runner persists accepted events through its existing durable outbox. Full queues, unavailable helpers or failed writes can drop a hint without failing the operation. Context reads are local and do not wait for the control loop. Publishers have a 500 ms lock deadline and a 2 KiB record limit. IPC lives outside durable state. Idle transport blocks; cleanup removes its fixed files and empty private directory. The host must provide `/usr/bin/mkfifo`. Rebuild matching Runner/provider artifacts for the updated source identities. No dependency, migration, live provider call or deployment is included. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools and code execution. The exact serving snapshot is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-10-29 07:40:50 -07:00
MichielMAnalyticsandClaude Opus 5.5	644937c931	feat(sandbox-providers): add boxd sandbox provider (#17284 ) <p align="center"> <img src="https://raw.githubusercontent.com/MichielMAnalytics/paperclip/08f1cdda04443ed10fc43e7097ae0b2d14b5cc19/boxd-x-paperclip.png" alt="boxd × Paperclip" width="700"> </p> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent work runs in sandboxes that provider plugins supply > - An operator selects a provider for each environment, and a reused lease keeps one sandbox for an issue across heartbeats > - The time to resume a reused sandbox delays every heartbeat on that issue (see #14684) > - boxd runs each sandbox as an isolated microVM that suspends and resumes in milliseconds, with disk, memory and processes kept > - This pull request adds a boxd sandbox provider plugin > - The benefit is that operators get one more provider, with fast resume for reused leases ## Linked Issues or Issue Description Refs #14684. That pull request keys the remote session identity on the provider lease ID. This provider keeps the provider lease ID equal to the boxd machine ID, so the ID stays the same across suspend and resume. No issue exists for boxd. The request is described below with the fields of `adapter_request.yml`. Agent or provider boxd (https://boxd.sh). boxd runs each sandbox as a KVM microVM. The plugin uses the public TypeScript SDK, `@boxd-sh/sdk` 0.2.20 or later. Why this adapter is useful boxd suspends a machine in about 17 ms and resumes it in about 0.2 s, with the disk, the memory and the running processes kept. This fits `reuseLease`: the next heartbeat on an issue continues on the same machine. Each machine is created with `isolated: true`. An isolated machine has no metadata endpoint, no in-VM `boxd` CLI and no boxd integrations, so sandbox code cannot reach the provider management interface. How the agent is invoked Build and install the plugin by local path, as the package README describes. Open Instance Settings, then Environments. Select the `boxd` provider and supply a boxd API key. Agents that use the environment run their adapter commands in a boxd machine. Are you willing to implement it? Yes. This pull request is the implementation. ## What Changed - Adds the `boxd` sandbox provider under `packages/plugins/sandbox-providers/boxd`. Its structure follows the E2B and exe.dev plugins. The plugin has no code comments, to follow `CONTRIBUTING.md`. - Implements the validate, probe, acquire, resume, release, stop, destroy, inspect, realize-workspace and execute hooks. The manifest declares `sandboxCapabilities: { reusableLeases: true }`. - With `reuseLease`, release suspends the machine and resume wakes it. A suspended, hibernated or stopped machine is woken by the matching boxd call. A release that cancels active work stops the machine and keeps its disk. - Release and destroy return a `destroyed` receipt only after boxd no longer lists the machine. Stop returns a `stopped` receipt only after boxd reports the `stopped` state. - When the host sets `requestedExpiresAt`, the plugin sets a boxd delete timer at or before that time and returns `expiresAt` on the lease. The timer is set after the create call returns, so a failure while setting it cannot lose the machine ID. - Commands run through `sh` with quoted arguments and per-command environment. Stdin is staged with boxd's file API in a temporary file that only the machine user can read, and removed afterwards; the workspace directory is created the same way. A machine whose boxd agent predates that API, such as one restored from an older snapshot, falls back to the older upload and `mkdir` calls. A command timeout ends the remote process group and returns `timedOut: true`. A command on a stopped machine starts the machine and runs one more time. - A failed acquisition deletes its machine. If boxd cannot confirm the deletion, the plugin throws `PluginEnvironmentCreationCleanupError` with the machine ID, so the host records it and retries the teardown; the destroy hook checks the account and the machine labels before it deletes a failed creation. A probe whose machine cannot be deleted reports a failure, and probe machines carry a 15-minute boxd delete timer. - The plugin keeps one SDK client for each API key. This prevents a session token exchange for each command. - Machines carry the labels `paperclip-provider`, `paperclip-environment` and `paperclip-run`, so an operator can find them with `boxd machine list --label paperclip-provider=boxd`. - Registers the optional bundled catalog key `boxd` and the trusted `BOXD_API_KEY` host fallback, with tests that mirror the CreateOS cases. - Adds the package to the release manifest with `publishFromCi: false` until a maintainer bootstraps the first npm publish. - Adds unit tests with a mocked SDK (40 tests) and opt-in live tests (`BOXD_LIVE_TEST=1`, 9 tests) that run every hook against boxd. - Adds a section to `doc/plugins/PLUGIN_AUTHORING_GUIDE.md` and adds boxd to the README provider list. ## Verification Provider package (`packages/plugins/sandbox-providers/boxd`): - `pnpm install --ignore-workspace --no-lockfile && pnpm typecheck`: passes. - `pnpm test`: 40 passed, 9 live tests skipped. - `BOXD_LIVE_TEST=1 BOXD_API_KEY=... pnpm test`: 49 passed against production boxd on 2026-10-28 with `@boxd-sh/sdk` 0.2.20. The live tests cover: - validation, and a probe at 2 vCPU that deletes its machine (a bad key returns a failed probe); - the labels, the isolation and the org of an acquired machine; - stdin, env, argument quoting, the exit code and stderr, and the removal of the staged stdin file; - an empty stdin, and a staged stdin file that only the machine user can read (mode 600); - a 5 MiB round trip through the base64-over-exec path (the SHA-256 matches); - a command timeout, which returns `timedOut` and ends the remote processes; - suspend and resume (a file and a running process stay), stop and resume (the file stays), and inspect in each state; - a command on a stopped machine, which starts the machine and runs; - resume and stop of a hibernated machine, stop of a suspended machine, and release with `cancelActiveWork` and with `stop_and_retain`; - release without reuse, which deletes the machine; - a `requestedExpiresAt` 45 s away, which boxd enforces by deleting the machine; - a run restored from a snapshot, and a fresh machine at 2 vCPU / 8 GiB; - resume after destroy, which returns an expired lease, and a final check that no test machines remain. - Measured times: probe 1.5 s, acquire 1.4 s, release with suspend 17 ms, resume from suspend 0.13 s, resume from hibernate 0.18 s, stop 14 ms, resume from stop 3.2 s, acquire from snapshot 1.6 s, confirmed destroy 30 ms. - On a machine started before boxd's file API (the test host), the plugin's hooks fell back to the older calls: the workspace was created, stdin was staged (mode 644, the upload path), an empty stdin worked, and no staged file was left. Host: - `npx vitest run src/__tests__/bundled-plugins.test.ts src/__tests__/plugin-database.test.ts` in `server`: 73 passed, 6 of them new (re-run on 2026-10-28). - `npx vitest run src/__tests__/cli-invocation-safety.test.ts` in `server`: 40 passed. The README install commands now use the direct-exec CLI form. - `npx tsc --noEmit` in `server`: passes. - `node --test scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs`: 18 passed. Full stack, on a Linux host with `pnpm dev:once` in `local_trusted` mode: - I installed the plugin with `POST /api/plugins/install` and a local path, then created boxd environments with a secret reference for the API key. - Ephemeral environment: the environment probe passed. A `claude_local` agent (`claude-sonnet-5-5`, legacy runner) ran a task in a boxd machine. It reported Ubuntu 24.04.5 and the workspace `/home/boxd/paperclip-workspace`, and it marked the task done. The lease was released and the machine was deleted. - Reused-lease environment: the first run wrote a file and started a background loop. After release, the machine was in the `suspended` state. A comment with `reopen: true` started a second run. That run used the same machine, so the provider lease ID was the same on both lease rows. The file was there and the background loop was still running. - Validation: a config with `vcpu: 3`, and a config with both `vcpu` and `snapshot`, return 422 with the plugin error. The unsaved-config probe (`/environments/probe-config`) passed in 2.0 s and deleted its machine. A saved probe with `vcpu: 4` created a 4 vCPU / 16 GiB isolated machine. - Cancel: cancelling the first run on a reused lease marked the lease `expired`, and the host destroyed the machine through the plugin about 12 s later. - Delete: deleting an environment that holds a reused lease returns 409. With `?destroyReusableSandboxLeases=true` it returns 200, and the host destroys the suspended machine through the plugin. - I re-ran this flow on 2026-10-26 and again on 2026-10-28 with `@boxd-sh/sdk` 0.2.20, from a script, with each step checked. On 2026-10-28 the ephemeral run posted its proof and both of its machines were deleted. The reused-lease follow-up ran on the same machine (both lease rows have the same provider lease ID), and found the file and a background loop that had kept running. The validation errors returned 422, the unsaved-config probe passed in 1.8 s, and deleting the environments left no machines. Every check passed except the host gap below. - After the runs, `boxd machine list` showed no machines with the test prefix. Not run: `pnpm -r typecheck`, `pnpm test:run` and `pnpm build` at the root. The test host has no Rust toolchain for the runner binary. On that host, `scripts/check-forbidden-tokens.mjs` adds the local username (`boxd`) as a token. It then flags every `boxd` substring, and it also fails on `master` there (`InboxDismissal`). A CI runner uses a different username. ## Risks - Host gap, not in this plugin: when a run that resumed a reused lease is cancelled, the host marks the run `cancelled` but does not stop the agent in the sandbox. The adapter keeps polling the file bridge, and the host calls no release, stop or destroy hook. The lease stays `active` until the environment is deleted. A plugin-side trace showed only bridge-polling `execute` calls after the cancel. boxd is the only provider that declares `reusableLeases`, so this path may not have been exercised before. A cancel on the first run of a lease works as expected. - Low risk to existing behavior. The plugin is opt-in and is not installed by default. The host changes add one catalog entry, one credential passthrough entry and their tests. - boxd machine creation has no idempotency key. If a create response is lost, a machine can stay behind. The labels above help an operator find it. - A suspended machine keeps its memory image until boxd hibernates it after a period of idle time. An operator who keeps many reused leases pays for that storage. - boxd stops a non-PTY command that has no output for 30 minutes. Agent CLIs write output during a run, so this does not affect normal runs. - boxd has one region (EU), amd64 machines only, and no GPUs. Machine sizes are 1/4 GB, 2/8 GB, 4/16 GB and 8/32 GB. - The new `package.json` needs code-owner approval. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use and shell execution on a boxd machine. The session does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused checks; the limits are listed under Verification) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01HNWtBYxLByQmmNKjuaou1P --------- Co-authored-by: Claude Opus 5.5 <[email protected]>	2026-10-28 12:34:21 -07:00
DottaandPaperclip	d54f776ca0	fix(daytona): bound ephemeral eval lifetime at creation (#17980 ) ## Thinking Path > - Paperclip manages agent work through Runner and sandbox providers. > - Product E2E runs paid work in disposable Daytona sandboxes. > - A controller deadline cannot stop remote compute after the controller crashes. > - Inactivity cleanup does not establish an independent wall-clock lifetime. > - Daytona supports a lifetime in the creation request. > - This change exposes that option in the Daytona driver and verifies the provider expiry. > - Ephemeral evals can retain a finite lifetime even when their controller is unavailable. ## Linked Issues or Issue Description Refs #17947 and #17978. No duplicate creation-lifetime PR was found. What happened? The ephemeral Product E2E fixture configures inactivity stop, archive, and delete intervals. Ordinary task leases do not supply a requested expiry. A crashed controller therefore has no confirmed wall-clock sandbox lifetime for its cost bound. Expected behavior Each ephemeral eval sandbox must carry a provider-enforced lifetime from creation. The driver must reject invalid bounds and reject a sandbox whose provider expiry is missing, expired, or later than the configured remaining lifetime. Steps to reproduce Acquire a Daytona lease from the ephemeral fixture. Inspect the creation request and returned expiry. Before this patch, the request has no lifetime and the lease has no expiry unless its caller supplies one. Paperclip version or commit Current source based on master at `4c314117d2`. Deployment mode Source checkout and the bundled Daytona provider. Verification makes no live provider or model calls. ## What Changed - Add optional positive whole-minute `ttlMinutes` to the Daytona environment configuration. Bump the bundled manifest version so installed manifests refresh. - Send the lifetime in image and snapshot creation requests. Read the provider expiry within the configured liveness timeout before returning the sandbox. Existing ownership-checked cleanup handles a failed expiry check. A cleared optional field is treated as unset. - Return the confirmed expiry on a new lease. Preserve an earlier creation expiry when the caller requests a later deadline. Shorten it for an earlier caller deadline. - Set a 60-minute lifetime on the ephemeral Product E2E fixture. An invariant checks that every selected attempt plus its acquisition budget fits inside that lifetime. - Keep omitted lifetime behavior and the separate warm continuity fixture unchanged. ## Verification - Before implementation, all 12 focused creation-lifetime regressions failed. - Full Daytona plugin suite: 470 passed, six live tests skipped. It covers image/snapshot creation, invalid bounds before allocation, missing/expired/excessive provider proof, cleanup, and both deadline orders. - Product catalog and operations contracts: 95 passed. - Four review regressions failed before the fixes. They cover a cleared field and stalled expiry reads in acquire, probe, and interactive setup. - Standalone Daytona typecheck and build passed against its pinned SDK 0.203.0 and the freshly built plugin SDK. - The pinned SDK forwards `ttlMinutes` in the create API. [Daytona documents its wall-clock semantics](https://www.daytona.io/docs/en/typescript-sdk/daytona/). - Root workspace verification remains the fresh CI gate. Local installed Runner exports are stale in this checkout. - No live model request, sandbox allocation, credential creation, spending-limit change, or schedule activation occurred. ## Risks - A configured lifetime destroys the sandbox and its files at expiry, including while active. Existing environment configurations omit it and retain their current behavior. - Ephemeral evals get 60 minutes so their longest declared 30-minute attempt and five-minute acquisition budget fit. Future longer cases fail the catalog invariant until their lifetime is reviewed. - Provider expiry confirmation adds one read during configured creation. No polling or background worker is added. - Expiry confirmation uses the controller clock and fails closed without a skew tolerance. Live qualification must verify the reported expiry. A tolerance is deferred until there is measured provider evidence; it must not silently extend the declared bound. - These are contract tests. Actual Daytona creation, expiry proof, model usage, and cleanup still need the remaining live qualification. ## Model Used OpenAI Codex, GPT-6 based, with tool execution and extended reasoning. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-10-27 23:52:52 -05:00
Nicky Leach	baec3abd67	Authorize background project Git with explicit Connector consent (#17798 )	2026-10-26 17:22:49 -07:00
Nicky LeachandPaperclip	c7c12dbc79	Support plugin panels in task sidebars (#17796 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task side panels keep live work beside the conversation. > - Plugins can already provide agent detail tabs. > - Those tabs have no generic way to appear beside the task that is using the agent. > - This pull request adds an optional task-side-panel declaration to agent detail tabs. > - The plugin owns the label, component and availability. Core supplies the task's running agent. ## Linked Issues or Issue Description What existing behavior does this improve? Access to plugin-contributed live views from task chat. Subsystem affected Plugin manifest validation, slot discovery, and task side-panel UI/state. Current behavior Agent detail-tab contributions appear on agent pages. A task cannot offer the same view through a plugin-owned availability decision. Proposed behavior An agent detail tab can declare `taskSidePanel: { availabilityPath }`. Core substitutes the running agent into the plugin-local path. A declared board GET route scoped to that agent returns `{ available: boolean }`. Available contributions appear in desktop and mobile task side panels. Reason and benefit Keep the task and its live plugin views together. Preserve plugin ownership of provider protocols and product behavior. Core contains no concrete plugin identifiers, routes, labels or readiness rules. Breaking changes None. The field is optional. Existing contributions remain unchanged. Missing, disabled or unavailable plugins add no panel. Additional context Related: #17111 established the agent detail-tab and scoped API extension surfaces. Roadmap and related PR searches found no duplicate of this extension. ## What Changed - Extend the manifest with optional task-panel metadata. Validate safe plugin-local paths and require a declared board GET route with agent-scoped authorization. - Discover and load only opted-in agent contributions. Poll their generic availability responses and mount the existing components with agent context. - Add generic panel tabs, launcher entries, dismissal and inactive-view removal in desktop/mobile task chat. - Reuse the existing live-run polling guard so idle tasks with a healthy live connection stop polling. - Add provider-neutral fixtures, schema/authorization tests, task tests and SDK/feature documentation. ## Verification - PASS: 301 focused tests including the full task-detail file, generic panel discovery, persistence, slot loading and manifest validation. Regressions cover POST-before-GET route order, mobile drawer dismissal, recovery without focus and the absence of the hidden desktop task panel on mobile. - PASS: UI and shared-package typechecks; production UI build; token gates, feature-map and module-boundary checks; diff and secrets/PII review. - PASS on final head: all 795 UI files / 9,186 tests, and all 95 shared-package files / 1,038 tests. Commands: `node node_modules/vitest/vitest.mjs run --project @paperclipai/ui --maxWorkers 3` and the corresponding `--project @paperclipai/shared` command with local socket access. - On the final head, server shards 7/12 and 10/12 and workspace shard a-1/2 were interrupted by runner shutdown/communication loss. GitHub reported infrastructure failures; no assertion failure was surfaced in the available diagnostics. The queued summary was cancelled, then `gh run rerun --failed` successfully requested retries for only those three jobs. They are currently queued for CI runners; the 50 successful checks remain passing. Interrupted attempts are not counted as passing. [Final-head CI and retry](https://github.com/paperclipai/paperclip/actions/runs/40396870920). - Earlier verification exposed an empty-panel rerender loop, which was fixed. The sandboxed combined run also failed five shared port-registry tests because it could not bind loopback listeners; the full shared rerun with socket access passed. These earlier failures are retained here rather than reported as passing runs. - Full workspace typecheck/build were attempted with pinned pnpm 9 after resolving the initial local package-manager setup. Both stop at the unchanged Runner Rust step because `cargo` is not installed. Full local workspace checks are not reported as passing; final CI must provide those broader checks. - No live provider or plugin-specific transport acceptance is claimed by Core's synthetic fixtures. ## Risks - Availability is polled every three seconds only while an authorized running agent exists. Plugins own the side effects and meaning of that read. - A view uses the running agent's context. It is not a separate per-task resource. - Optional metadata adds no backend endpoint or permission bypass. The host requires existing board and agent authorization. - Reverting the optional contract and UI integration removes task panels without a data migration. ## Model Used OpenAI Codex (GPT-6), with reasoning, repository tools and code execution. The exact backend snapshot ID and context window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>	2026-10-26 14:47:15 -07:00
Nicky LeachandPaperclip	4ff493fb40	feat(plugins): include managed skills for selected environments (#17774 ) ## Thinking Path > - Paperclip manages agents and their execution environments. > - Plugins can vendor company skills and provide environments. > - Environment tools need their instructions before the provider starts. > - Persistent computers also need those skills before any run begins. > - This change adds environment requirements to normal skill selection and lifecycle synchronization. > - Plugin names prevent ambiguity, and required skills take delivery precedence without deleting company data. ## Linked Issues or Issue Description Problem or motivation An environment can provide desktop tools while its agent receives no instructions for those tools. A first-run-only import also leaves idle computers without required skills. Same-name company skills can make provider discovery ambiguous. Proposed solution Let plugin environment drivers declare their managed `skillKeys`. Install missing requirements during lifecycle reconciliation or new-run preparation. Deliver updates through the durable runtime event journal. Give managed imports stable plugin names and report shadowed assignments. Alternatives considered ROM links do not make a skill a normal Paperclip selection. Polling duplicates the existing lifecycle delivery queue. Replacing a same-name company record would destroy independent data and assignments. Roadmap alignment This extends the shipped Plugin system and Skills Manager. The existing related-PR search found no duplicate contract. ## What Changed - Validate `environmentDrivers[].skillKeys` as unique references to the same plugin's managed skills. Require a ready, enabled plugin with `skills.managed` authority. - Resolve company-scoped skills before version capture. Preserve explicit pins, company edits, and saved recovery selections. Serialize concurrent first installs. - Allow lifecycle callers to request missing-skill installation through `agents.getRuntime`. Ordinary task and recovery reads do not install new declarations. Pass task-selected environment IDs through the SDK. - Add SQL migration `0378_environment_skill_runtime_hooks`. Required skill edits, effective default changes, environment changes, and plugin requirement changes enqueue durable runtime events. Reuse lifecycle delivery and retries; add no polling. - Import managed defaults as `plugin-<short-plugin-id>-<skill-slug>`, including the SKILL.md name. Preserve canonical keys. Keep same-name company records separate without counter suffixes. - Required environment skills win delivery over same-name company assignments. Preserve saved assignments and library content. Record shadowed and winning keys in run warnings, run snapshots, and plugin runtime state. Removing the environment requirement restores the ordinary assignment. - Document the contract for `driver: "plugin"`. Other drivers and plugins without requirements keep existing behavior. Existing managed copies retain edits and names until reset. ## Verification - Latest targeted local checks: environment skills (5), managed skills (9), heartbeat execution configuration (27), and SDK worker RPC host (23): 65 tests passed (including the targeted schema-drift check). Disposable PostgreSQL exercised the migration, first install before any run, concurrent lifecycle reads, implicit edit events, inherited defaults, rollback, plugin requirement changes, shadowing, preserved records, version capture, and recovery. - Review follow-up also verifies case-insensitive/runtime-name aliases, recovery of a task-mentioned shadowed skill, preserved quoted/multiline YAML, and zero runtime events for repeated local-folder settings writes. - Migration numbering and safety checks passed. Whitespace and secret/PII scans passed. - Earlier targeted manifest validation passed (43 tests). No validator behavior changed in this follow-up. - Broad tests, typecheck, build and end-to-end verification passed in [GitHub CI](https://github.com/paperclipai/paperclip/actions/runs/40387647219) for current head `ce57f2a4d`. Greptile is 5/5 with no unresolved threads. The [commitperclip review check](https://github.com/paperclipai/paperclip/actions/runs/40387646208) also passed after restoring its required duplicate-search checkbox. - Earlier attempts found a SQL alias collision, an unparenthesized CASE expression, and fixture assumptions about bundled catalog rows and YAML quoting. These were corrected before the passing checks. The first broad CI run also exposed a schema-drift check that assumed every journal entry had a schema snapshot; it now reads the latest actual snapshot, so SQL-only migrations need no invented snapshot. A test callback replacement also affected an unrelated fixture; it was restored before the final passing execution-configuration suite. Sandbox runs could not start PostgreSQL; the reported database tests used local socket access. ## Risks - Apply migration 0378 before using the updated lifecycle contract. It adds trigger functions and event producers; it does not rewrite agent assignments or skill content. - Required environment skills take precedence over same-name assignments, including task-selected skills. Both library records remain. Run warnings identify omissions. - Company edits are preserved. Plugin upgrades do not reset installed content or rename existing copies. Explicit reset adopts current defaults. - Namespaced library slugs can be shared by distinct canonical records. Callers must use IDs or canonical keys when a slug is ambiguous. - No live provider request or deployment was performed. The downstream plugin must be built against this Core revision. Persistent-home synchronization retains its existing edit-conflict and retry behavior. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, and code execution. The exact model identifier and context-window size are not exposed in this task context. ## Checklist - [x] Thinking path and issue description included - [x] Model and available capabilities recorded - [x] Roadmap and related PRs checked - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] Public references only; no internal issue IDs or instance URLs - [x] Targeted local tests pass; broad verification delegated to GitHub CI - [x] New behavior covered and documentation updated - [x] Current-head CI green and Greptile 5/5 with no unresolved comments --------- Co-authored-by: Paperclip <[email protected]>	2026-10-26 14:11:35 -07:00

1 2 3 4 5 ...